Make recovery source partially non-blocking #37291

dnhatn · 2019-01-10T09:13:26Z

Today a peer-recovery may run into a deadlock if the value of
node_concurrent_recoveries is too high. This happens because the
peer-recovery is executed in a blocking fashion. This commit attempts to
make the recovery source partially non-blocking. I will make three
follow-ups to make it fully non-blocking: (1) send translog operations,
(2) primary relocation, (3) send commit files.

Relates #36195

Today a peer-recovery may run into a deadlock if the value of `node_concurrent_recoveries` is too high. This happens because the peer-recovery is executed in a blocking fashion. This commit attempts to make the recovery source partially non-blocking. I will make three follow-ups to make it fully non-blocking: (1) send translog operations, (2) primary relocation, (3) send commit files.

elasticmachine · 2019-01-10T09:13:28Z

Pinging @elastic/es-distributed

s1monw

did a first pass. I would love to minimize the steps we make async in this PR even further.

server/src/main/java/org/elasticsearch/indices/recovery/PeerRecoverySourceService.java

server/src/main/java/org/elasticsearch/indices/recovery/RecoveryTarget.java

server/src/main/java/org/elasticsearch/indices/recovery/RemoteRecoveryTargetHandler.java

server/src/main/java/org/elasticsearch/indices/recovery/RecoverySourceHandler.java

dnhatn · 2019-01-10T23:25:57Z

@s1monw Thanks for looking. I've minimized changes in this PR - just try to provide the infra for the next steps. Would you please take another look?

dnhatn · 2019-01-11T00:15:40Z

@elasticmachine run gradle build tests 1

Relates #37291

s1monw

took another round

server/src/main/java/org/elasticsearch/indices/recovery/PeerRecoverySourceService.java

server/src/main/java/org/elasticsearch/indices/recovery/RecoverySourceHandler.java

dnhatn · 2019-01-11T17:59:09Z

@s1monw I pushed changes. Can you have another look?

This commit introduces StepListener which provides a simple way to write a flow consisting of multiple asynchronous steps without having nested callbacks. Relates #37291

Relates #37291

This commit introduces StepListener which provides a simple way to write a flow consisting of multiple asynchronous steps without having nested callbacks. Relates #37291

s1monw

LGTM left 2 comments

server/src/main/java/org/elasticsearch/common/util/CancellableThreads.java

server/src/main/java/org/elasticsearch/indices/recovery/RecoverySourceHandler.java

dnhatn · 2019-01-12T17:48:47Z

Thanks @s1monw.

Relates elastic#37291

Today a peer-recovery may run into a deadlock if the value of node_concurrent_recoveries is too high. This happens because the peer-recovery is executed in a blocking fashion. This commit attempts to make the recovery source partially non-blocking. I will make three follow-ups to make it fully non-blocking: (1) send translog operations, (2) primary relocation, (3) send commit files. Relates #36195

Relates #37291

This commit prepares the required infra to make send a translog snapshot of the recovery source non-blocking. I'll make a follow-up to make the send snapshot method non-blocking. Relates #37291

Backport of elastic/elasticsearch#37291

dnhatn added >enhancement :Distributed Indexing/Recovery Anything around constructing a new shard, either from a local or a remote source. v7.0.0 v6.7.0 labels Jan 10, 2019

dnhatn requested review from s1monw, ywelsch and DaveCTurner January 10, 2019 09:13

dnhatn changed the title ~~Partially make recovery source non-blocking~~ Jan 10, 2019

s1monw suggested changes Jan 10, 2019

View reviewed changes

This was referenced Jan 10, 2019

Introduce StepListener #37327

Merged

Add runAfter and notifyOnce wrapper to ActionListener #37331

Merged

dnhatn added 4 commits January 10, 2019 16:44

Add runAfter and notifyOnce wrapper to ActionListener

5d9e8f0

wrap listeners

ef9a752

minimize changes

a2f7b3f

Merge branch 'master' into non-blocking-recovery

5363f14

dnhatn requested a review from s1monw January 10, 2019 23:26

stylecheck

8737b4c

dnhatn added a commit that referenced this pull request Jan 11, 2019

Add runAfter and notifyOnce wrapper to ActionListener (#37331)

360c430

Relates #37291

Merge branch 'master' into non-blocking-recovery

f5f6d13

s1monw suggested changes Jan 11, 2019

View reviewed changes

dnhatn added 2 commits January 11, 2019 12:57

simonw’s feedback

8c2a632

Merge branch 'master' into non-blocking-recovery

d0f0957

dnhatn requested a review from s1monw January 11, 2019 17:59

dnhatn added a commit that referenced this pull request Jan 11, 2019

Introduce StepListener (#37327)

70cee18

This commit introduces StepListener which provides a simple way to write a flow consisting of multiple asynchronous steps without having nested callbacks. Relates #37291

style

0e0a4d7

dnhatn added 2 commits January 11, 2019 15:23

Merge branch 'master' into non-blocking-recovery

18dad84

remove debug log

bcbc637

dnhatn added a commit that referenced this pull request Jan 12, 2019

Add runAfter and notifyOnce wrapper to ActionListener (#37331)

773d4c2

Relates #37291

dnhatn added a commit that referenced this pull request Jan 12, 2019

Introduce StepListener (#37327)

3e419ba

This commit introduces StepListener which provides a simple way to write a flow consisting of multiple asynchronous steps without having nested callbacks. Relates #37291

s1monw approved these changes Jan 12, 2019

View reviewed changes

server/src/main/java/org/elasticsearch/common/util/CancellableThreads.java Show resolved Hide resolved

server/src/main/java/org/elasticsearch/indices/recovery/RecoverySourceHandler.java Outdated Show resolved Hide resolved

dnhatn added 2 commits January 12, 2019 06:13

setOnce

7c84c09

release resources

ce14139

dnhatn merged commit 44a1071 into elastic:master Jan 12, 2019

dnhatn deleted the non-blocking-recovery branch January 12, 2019 17:49

dnhatn added the backport pending label Jan 12, 2019

dnhatn added a commit to dnhatn/elasticsearch that referenced this pull request Jan 12, 2019

Make finalize step of recovery source non-blocking

97e9196

Relates elastic#37291

dnhatn mentioned this pull request Jan 12, 2019

Make finalize step of recovery source non-blocking #37388

Merged

dnhatn removed the backport pending label Jan 13, 2019

dnhatn added a commit that referenced this pull request Jan 14, 2019

Make finalize step of recovery source non-blocking (#37388)

397f315

Relates #37291

dnhatn mentioned this pull request Jan 15, 2019

Prepare to make send translog of recovery non-blocking #37458

Merged

dnhatn added a commit that referenced this pull request Jan 15, 2019

Make finalize step of recovery source non-blocking (#37388)

821bb24

Relates #37291

colings86 added v7.0.0-beta1 and removed v7.0.0 labels Feb 7, 2019

kovrus added a commit to crate/crate that referenced this pull request Sep 11, 2019

Make recovery source partially non-blocking.

340a4e8

Backport of elastic/elasticsearch#37291

kovrus added a commit to crate/crate that referenced this pull request Sep 11, 2019

Make recovery source partially non-blocking.

41d2f85

Backport of elastic/elasticsearch#37291

kovrus added a commit to crate/crate that referenced this pull request Sep 12, 2019

Make recovery source partially non-blocking.

db318d0

Backport of elastic/elasticsearch#37291

kovrus added a commit to crate/crate that referenced this pull request Sep 12, 2019

Make recovery source partially non-blocking.

8e2a0d5

Backport of elastic/elasticsearch#37291

kovrus added a commit to crate/crate that referenced this pull request Sep 12, 2019

Make recovery source partially non-blocking.

23a2b07

Backport of elastic/elasticsearch#37291

mergify bot pushed a commit to crate/crate that referenced this pull request Sep 17, 2019

Make recovery source partially non-blocking.

4e2185e

Backport of elastic/elasticsearch#37291

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Make recovery source partially non-blocking #37291

Make recovery source partially non-blocking #37291

Uh oh!

dnhatn commented Jan 10, 2019

elasticmachine commented Jan 10, 2019

s1monw left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dnhatn commented Jan 10, 2019

dnhatn commented Jan 11, 2019

s1monw left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dnhatn commented Jan 11, 2019

s1monw left a comment

Uh oh!

Uh oh!

dnhatn commented Jan 12, 2019

Make recovery source partially non-blocking #37291

Make recovery source partially non-blocking #37291

Uh oh!

Conversation

dnhatn commented Jan 10, 2019

elasticmachine commented Jan 10, 2019

s1monw left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dnhatn commented Jan 10, 2019

dnhatn commented Jan 11, 2019

s1monw left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dnhatn commented Jan 11, 2019

s1monw left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dnhatn commented Jan 12, 2019