Skip to content

Threadpool merge scheduler #120869

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 236 commits into from
Mar 18, 2025
Merged

Threadpool merge scheduler #120869

Changes from 1 commit
Commits
Show all changes
236 commits
Select commit Hold shift + click to select a range
bf557d2
ExecutorMergeScheduler
albertzaharovits Jan 16, 2025
a3f87df
Merge branch 'main' into threadpool-merge-scheduler
albertzaharovits Jan 16, 2025
f5a1a8d
[CI] Auto commit changes from spotless
Jan 16, 2025
f0b72fe
wrap for merge in the executor merge scheduler
albertzaharovits Jan 16, 2025
9b03950
spotless
albertzaharovits Jan 16, 2025
26e4043
Merge branch 'main' into threadpool-merge-scheduler
albertzaharovits Jan 17, 2025
aba69d0
Fix InternalEngineTests
albertzaharovits Jan 17, 2025
52796b5
Merge branch 'main' into threadpool-merge-scheduler
albertzaharovits Jan 17, 2025
c0667bf
implemented Throttling
albertzaharovits Jan 18, 2025
2da753f
Merge branch 'main' into threadpool-merge-scheduler
albertzaharovits Jan 18, 2025
2c8dc7f
[CI] Auto commit changes from spotless
Jan 18, 2025
81cc0f1
Checkstyle
albertzaharovits Jan 18, 2025
f58120f
Fix threadpool size for SnapshotResiliencyTests
albertzaharovits Jan 19, 2025
5ca992d
Spotless
albertzaharovits Jan 19, 2025
3c203cb
Nit
albertzaharovits Jan 19, 2025
6c21654
Implemented max thread setting
albertzaharovits Jan 19, 2025
68079d9
Throttling ?
albertzaharovits Jan 19, 2025
7b68ba9
Checkstyle
albertzaharovits Jan 19, 2025
9e467a1
Indexing throttling !
albertzaharovits Jan 19, 2025
a8f5297
Better throttling logging
albertzaharovits Jan 19, 2025
928fd32
Merge branch 'main' into threadpool-merge-scheduler
albertzaharovits Jan 20, 2025
3f5b4a8
Don't wrap errors during merging
albertzaharovits Jan 20, 2025
0e714a1
Merge branch 'main' into threadpool-merge-scheduler
albertzaharovits Jan 20, 2025
0297cce
Merge branch 'main' into threadpool-merge-scheduler
albertzaharovits Jan 21, 2025
2b79809
Refresh config
albertzaharovits Jan 21, 2025
57c2a5c
Nit
albertzaharovits Jan 21, 2025
60a71b8
WIP
albertzaharovits Jan 23, 2025
68db209
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges
albertzaharovits Jan 23, 2025
4099ac5
IO throttling
albertzaharovits Jan 24, 2025
5554bc2
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges
albertzaharovits Jan 24, 2025
17b682f
[CI] Auto commit changes from spotless
Jan 24, 2025
f3506da
this-escape
albertzaharovits Jan 24, 2025
6a3911a
Unregister
albertzaharovits Jan 24, 2025
00070cf
Sort merge schedulers
albertzaharovits Jan 24, 2025
0cf8b0c
Rename to v1
albertzaharovits Jan 24, 2025
f6ed56d
Hmmm, looks OK
albertzaharovits Jan 26, 2025
d1fb28d
Rename WIP
albertzaharovits Jan 26, 2025
d8ca2a9
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges-t…
albertzaharovits Jan 26, 2025
ef0d757
rename in progress
albertzaharovits Jan 26, 2025
35a5ef6
Rename in progress 2
albertzaharovits Jan 26, 2025
33bd59c
Done?
albertzaharovits Jan 26, 2025
715c7e1
FollowingEngineTests.java
albertzaharovits Jan 26, 2025
2b2c790
[CI] Auto commit changes from spotless
Jan 26, 2025
25d9eef
Ooops set IO rate only if merge task supports it
albertzaharovits Jan 26, 2025
27bffb2
remove mergeDone from onFailure
albertzaharovits Jan 27, 2025
573084d
Execute not submit!
albertzaharovits Jan 27, 2025
c87931e
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges-t…
albertzaharovits Jan 27, 2025
3857959
synchronizedSet in ThreadPoolMergeQueue
albertzaharovits Jan 27, 2025
5dcf91b
ConcurrentCollections.newConcurrentSet()
albertzaharovits Jan 27, 2025
c5b05e5
submitMergeTask
albertzaharovits Jan 27, 2025
932f3ce
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges-t…
albertzaharovits Jan 27, 2025
7b52d8e
Ooops
albertzaharovits Jan 27, 2025
a315198
remove explicit unthrottled IO rate
albertzaharovits Jan 27, 2025
cb95685
assert estimated merge bytes is set
albertzaharovits Jan 27, 2025
6a96a02
Remove ThreadLocal<MergeRateLimiter> onGoingMergeRateLimiter
albertzaharovits Jan 27, 2025
60b7b3a
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges-t…
albertzaharovits Jan 27, 2025
d170de8
Ooops
albertzaharovits Jan 28, 2025
4c974af
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges-t…
albertzaharovits Jan 28, 2025
d98f350
Node level setting
albertzaharovits Jan 28, 2025
f3c6d5f
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges-t…
albertzaharovits Jan 28, 2025
7a84fdf
[CI] Auto commit changes from spotless
Jan 28, 2025
cf11da1
Register setting
albertzaharovits Jan 28, 2025
69fc3fc
NOOP
albertzaharovits Jan 28, 2025
67c9a42
revert typo
albertzaharovits Jan 28, 2025
236a6db
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges-t…
albertzaharovits Jan 28, 2025
aac7490
InternalTestCluster random merge scheduler
albertzaharovits Jan 28, 2025
bafc31b
IT
albertzaharovits Jan 28, 2025
9482d24
Do not build the thread pool executor if the merge scheduler is not used
albertzaharovits Jan 28, 2025
b8cdc13
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges-t…
albertzaharovits Jan 28, 2025
da0a9e3
[CI] Auto commit changes from spotless
Jan 28, 2025
04f82ba
Remove estimated merge bytes assertion
albertzaharovits Jan 28, 2025
37b9f81
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges-t…
albertzaharovits Jan 28, 2025
42ee5a9
Remove unused random setting in test cluster
albertzaharovits Jan 29, 2025
5dc5d77
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges-t…
albertzaharovits Jan 29, 2025
22df845
[CI] Auto commit changes from spotless
Jan 29, 2025
08e272a
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges-t…
albertzaharovits Jan 29, 2025
26037ba
Update docs/changelog/120869.yaml
albertzaharovits Jan 29, 2025
59ce648
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges-t…
albertzaharovits Jan 30, 2025
d70ba6c
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges-t…
albertzaharovits Jan 30, 2025
838cea7
Address review comments 1
albertzaharovits Jan 30, 2025
2b2c60d
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges-t…
albertzaharovits Jan 30, 2025
b16307a
EngineThreadPoolMergeScheduler
albertzaharovits Jan 30, 2025
89eb0d7
enable/disable indexing throttling
albertzaharovits Jan 30, 2025
a9f4823
Update comment
albertzaharovits Jan 30, 2025
5e1b508
AtomicLong for merge start time
albertzaharovits Jan 30, 2025
c74d9e8
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges-t…
albertzaharovits Jan 30, 2025
8ad122f
[CI] Auto commit changes from spotless
Jan 30, 2025
6ae7425
no-op onFailure
albertzaharovits Jan 30, 2025
680868e
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges-t…
albertzaharovits Jan 30, 2025
779eace
Checkstyle
albertzaharovits Jan 30, 2025
6304852
make MergeTask a Runnable rather than an AbstractRunnable
albertzaharovits Jan 30, 2025
8d2c382
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges-t…
albertzaharovits Feb 14, 2025
189b1b7
InternalEngineMergeIT
albertzaharovits Feb 14, 2025
436ef52
Assert empty merge queue after internal IT
albertzaharovits Feb 14, 2025
57ad622
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges-t…
albertzaharovits Feb 15, 2025
c3d4001
Random merge queue in RefreshListenersTests
albertzaharovits Feb 15, 2025
b943275
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges-t…
albertzaharovits Feb 15, 2025
4fdd97f
Fix activeIOThrottledMergeTasksCount
albertzaharovits Feb 15, 2025
9f103e1
comment on maybeUpdateTargetMBPerSec
albertzaharovits Feb 16, 2025
32dfaf8
targetIORateBytesPerSec
albertzaharovits Feb 16, 2025
3e4f726
maybeUpdateIORateBytesPerSec
albertzaharovits Feb 16, 2025
091dcba
submittedIOThrottlingMergeTasks
albertzaharovits Feb 16, 2025
a0874ee
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges-t…
albertzaharovits Feb 16, 2025
79f8462
Nits
albertzaharovits Feb 17, 2025
f2b1472
ESIntegTestCase#getShardSegments
albertzaharovits Feb 17, 2025
9cb4f48
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges-t…
albertzaharovits Feb 17, 2025
5f9a293
[CI] Auto commit changes from spotless
Feb 17, 2025
9d2d29d
Closing not quite there....
albertzaharovits Feb 18, 2025
1e78f21
closing WIP2
albertzaharovits Feb 20, 2025
137b8f5
Closeable finally sorted out
albertzaharovits Feb 20, 2025
2d42ded
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges-t…
albertzaharovits Feb 20, 2025
a2a6362
Maybe
albertzaharovits Feb 20, 2025
1348863
Avoid wait/notify
albertzaharovits Feb 20, 2025
cc24fce
[CI] Auto commit changes from spotless
Feb 20, 2025
8bd1d7d
Fix close
albertzaharovits Feb 20, 2025
0431299
Nits
albertzaharovits Feb 20, 2025
2b14346
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges-t…
albertzaharovits Feb 20, 2025
6401d86
[CI] Auto commit changes from spotless
Feb 20, 2025
b09863a
Rename ThreadPoolMergeQueue -> ThreadPoolMergeExecutorService
albertzaharovits Feb 21, 2025
2614b5d
Nit
albertzaharovits Feb 21, 2025
5122b7f
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges-t…
albertzaharovits Feb 21, 2025
88e8754
[CI] Auto commit changes from spotless
Feb 21, 2025
5ae44c5
Fixed close after removing wait/notify
albertzaharovits Feb 21, 2025
b2e65a2
Handle aborted merge tasks at any point
albertzaharovits Feb 23, 2025
feed4ca
synchronized close
albertzaharovits Feb 23, 2025
ee0e3e2
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges-t…
albertzaharovits Feb 23, 2025
b0e2da0
Nits
albertzaharovits Feb 23, 2025
257a13f
[CI] Auto commit changes from spotless
Feb 23, 2025
12944a5
Sometimes merges are disabled
albertzaharovits Feb 23, 2025
795c9b0
[CI] Auto commit changes from spotless
Feb 23, 2025
8bb890c
Rename
albertzaharovits Feb 23, 2025
9b09dba
Run merges when aborted too
albertzaharovits Feb 25, 2025
28d96f2
Nits
albertzaharovits Feb 25, 2025
dcd2af1
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges-t…
albertzaharovits Feb 25, 2025
519c48e
Simple tests
albertzaharovits Feb 25, 2025
c15785f
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges-t…
albertzaharovits Feb 26, 2025
79ccc74
[CI] Auto commit changes from spotless
Feb 26, 2025
822501b
testBackloggedMergeTasksAreAllExecutedExactlyOnce
albertzaharovits Feb 26, 2025
6931a95
[CI] Auto commit changes from spotless
Feb 26, 2025
a76744a
Nit
albertzaharovits Feb 26, 2025
ba4c7ee
WIP
albertzaharovits Feb 27, 2025
f163f36
testBackloggedMergeTasksExecuteInSizeOrder
albertzaharovits Feb 28, 2025
66fa090
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges-t…
albertzaharovits Feb 28, 2025
4eb5707
[CI] Auto commit changes from spotless
Feb 28, 2025
23ae2b4
synchronized (shouldThrottleIncomingMerges)
albertzaharovits Feb 28, 2025
93ea664
test nits
albertzaharovits Feb 28, 2025
e215fae
Nits
albertzaharovits Feb 28, 2025
9729e9c
testIORateAdjustedForNewlySubmittedTasks
albertzaharovits Feb 28, 2025
a4da44a
[CI] Auto commit changes from spotless
Feb 28, 2025
48dce91
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges-t…
albertzaharovits Mar 2, 2025
68c9bcf
testIORateAdjustedForSubmittedTasksWhenExecutionRateIs*
albertzaharovits Mar 2, 2025
58fa5b3
[CI] Auto commit changes from spotless
Mar 2, 2025
9f9fc86
Nit
albertzaharovits Mar 2, 2025
dce963f
testThreadPoolStatsWithBackloggedMergeTasks
albertzaharovits Mar 2, 2025
6da15aa
[CI] Auto commit changes from spotless
Mar 2, 2025
2002d0a
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges-t…
albertzaharovits Mar 3, 2025
30e4860
Fix RetrySearchIntegTests
albertzaharovits Mar 3, 2025
cd39b81
testMergeTasksRunConcurrently
albertzaharovits Mar 3, 2025
9e1215a
[CI] Auto commit changes from spotless
Mar 3, 2025
769e43b
Fix testThreadPoolStatsWithBackloggedMergeTasks
albertzaharovits Mar 3, 2025
aeca6b8
Test nits
albertzaharovits Mar 3, 2025
86e6117
Test method name nit
albertzaharovits Mar 3, 2025
261902e
testIORateAdjustedForRunningTasks
albertzaharovits Mar 4, 2025
7f6ce86
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges-t…
albertzaharovits Mar 4, 2025
155c110
Ooops remove +1 from floor/ceiling
albertzaharovits Mar 4, 2025
d0a4a3f
Test comments
albertzaharovits Mar 4, 2025
31b0867
testTargetIORateChangesWhenSubmittingMergeTasks
albertzaharovits Mar 4, 2025
0003f10
testMergesExecuteInSizeOrder
albertzaharovits Mar 5, 2025
321a332
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges-t…
albertzaharovits Mar 5, 2025
553b08a
[CI] Auto commit changes from spotless
Mar 5, 2025
dc80bef
Test nit
albertzaharovits Mar 5, 2025
34ca0fa
testMergeSourceWithFollowUpMergesRunSequentially
albertzaharovits Mar 5, 2025
fda3f86
[CI] Auto commit changes from spotless
Mar 5, 2025
3586ee1
test nit comments
albertzaharovits Mar 6, 2025
1fa7ba8
testMergesRunConcurrently
albertzaharovits Mar 6, 2025
0b55e35
[CI] Auto commit changes from spotless
Mar 6, 2025
054fda0
Move catch (InterruptedException e) in ThreadPoolMergeExecutorService
albertzaharovits Mar 6, 2025
0959714
Comment about interrupt
albertzaharovits Mar 6, 2025
e75c94c
comment on 0 for estimatedMergeBytes
albertzaharovits Mar 6, 2025
e57f06b
testSchedulerCloseWaitsForRunningMerge
albertzaharovits Mar 6, 2025
910289f
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges-t…
albertzaharovits Mar 6, 2025
669a87e
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges-t…
albertzaharovits Mar 7, 2025
34ab7f6
abortOnGoingMerge -> abort
albertzaharovits Mar 7, 2025
669b349
Nit
albertzaharovits Mar 7, 2025
c2b0cce
nits
albertzaharovits Mar 8, 2025
d03d808
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges-t…
albertzaharovits Mar 8, 2025
1d7a94e
currentlyRunningMergeTasks -> runningMergeTasks
albertzaharovits Mar 9, 2025
d975a1c
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges-t…
albertzaharovits Mar 9, 2025
c35518a
nit
albertzaharovits Mar 9, 2025
a217d12
ThreadPoolMergeScheduler.Schedule schedule = smallestMergeTask.schedu…
albertzaharovits Mar 9, 2025
1fba8a1
[CI] Auto commit changes from spotless
Mar 9, 2025
7ac3328
currentlySubmittedIOThrottledMergeTasksCount -> ioThrottledMergeTasks…
albertzaharovits Mar 9, 2025
fe7e9eb
Fix ThreadPoolMergeExecutorServiceTests
albertzaharovits Mar 9, 2025
171fc22
[CI] Auto commit changes from spotless
Mar 9, 2025
79e6abf
Enhance ThreadPoolMergeExecutorServiceTests post MergeTask#abort
albertzaharovits Mar 9, 2025
a25940d
Checkstyle
albertzaharovits Mar 9, 2025
507896e
AtomicIORate
albertzaharovits Mar 9, 2025
eb6279e
[CI] Auto commit changes from spotless
Mar 9, 2025
79030ed
AtomicIORate
albertzaharovits Mar 10, 2025
0d65aeb
Ooops
albertzaharovits Mar 10, 2025
b6f3f0a
testEnqueuedAndBackloggedMergesAreStillExecutedWhenThreadPoolIsShutdown
albertzaharovits Mar 10, 2025
3ca893f
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges-t…
albertzaharovits Mar 10, 2025
c82a4d9
[CI] Auto commit changes from spotless
Mar 10, 2025
7c9729c
Checkstyle main
albertzaharovits Mar 10, 2025
79a4652
Remove @Before from ThreadPoolMergeSchedulerTests
albertzaharovits Mar 10, 2025
b7fd765
simple tests for backlogging and reenqueueing
albertzaharovits Mar 10, 2025
cc71070
[CI] Auto commit changes from spotless
Mar 10, 2025
660a2a6
Update server/src/test/java/org/elasticsearch/index/engine/ThreadPool…
albertzaharovits Mar 13, 2025
cb99c07
Assert executors are terminated
albertzaharovits Mar 13, 2025
4f19df6
force merge with expunge deletes is not merging down to one segment only
albertzaharovits Mar 13, 2025
0019f58
Nit
albertzaharovits Mar 13, 2025
76dab39
Update server/src/test/java/org/elasticsearch/index/engine/ThreadPool…
albertzaharovits Mar 13, 2025
8005779
10
albertzaharovits Mar 13, 2025
77ce8b4
Better way to assert that IO rate is set for running merge tasks
albertzaharovits Mar 13, 2025
1ffbfbe
Ooops
albertzaharovits Mar 13, 2025
46e4471
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges-t…
albertzaharovits Mar 13, 2025
4d4fd7f
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges-t…
albertzaharovits Mar 13, 2025
2ad13b3
Randomly set USE_THREAD_POOL_MERGE_SCHEDULER_SETTING
albertzaharovits Mar 13, 2025
55c142d
[CI] Auto commit changes from spotless
Mar 13, 2025
169e06b
Fix randomization of thread pool based merge scheduler
albertzaharovits Mar 13, 2025
0918dcc
[CI] Auto commit changes from spotless
Mar 13, 2025
15b3e4c
ThreadPoolMergeSchedulerStressTestIT
albertzaharovits Mar 16, 2025
5ce463f
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges-t…
albertzaharovits Mar 16, 2025
dc85b15
[CI] Auto commit changes from spotless
Mar 16, 2025
f3e7ffb
Done ThreadPoolMergeSchedulerStressTestIT
albertzaharovits Mar 17, 2025
40631b3
Also run a force merge at the end
albertzaharovits Mar 17, 2025
a728b6a
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges-t…
albertzaharovits Mar 17, 2025
0e300a7
Ensure merge threads are blocked
albertzaharovits Mar 17, 2025
e4216d2
ThreadPoolMergeSchedulerStressTestIT no static
albertzaharovits Mar 17, 2025
4d0cab2
[CI] Auto commit changes from spotless
Mar 17, 2025
a3f73b7
Meh ThreadPoolMergeSchedulerStressTestIT
albertzaharovits Mar 17, 2025
b187e1c
force merge without max num segments
albertzaharovits Mar 17, 2025
cec66c5
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges-t…
albertzaharovits Mar 18, 2025
1a7db64
Remove segments count before merging caught up
albertzaharovits Mar 18, 2025
ad6d7b2
[CI] Auto commit changes from spotless
Mar 18, 2025
4d23a54
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges-t…
albertzaharovits Mar 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Merge branch 'main' into threadpool-merge-scheduler-sort-all-merges
  • Loading branch information
@albertzaharovits
albertzaharovits committed Jan 24, 2025
commit 5554bc2a8dfdbcd47355e8eec1a1e854ef83b506

This merge commit was added into this branch cleanly.

There are no new changes to show, but you can still view the diff.