[CI] CrossClusterAsyncQueryIT testSuccessfulPathways failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic #6697 / concurrent-search-tests
• elasticsearch-periodic-platform-support #6648 / sles-15_platform-support-unix

Reproduction Line:

./gradlew ":x-pack:plugin:esql:internalClusterTest" --tests "org.elasticsearch.xpack.esql.action.CrossClusterAsyncQueryIT.testAsyncQueriesWithLimit0" -Dtests.seed=400DB4231EF1C473 -Dtests.jvm.argline="-Des.concurrent_search=true" -Dtests.locale=su-Latn-ID -Dtests.timezone=EAT -Druntime.java=23

Applicable branches:
9.0

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: timeout waiting for query response

Issue Reasons:

• [9.0] 2 failures in test testSuccessfulPathways (1.2% fail rate in 164 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[dnhatn]

@smalyshev I took a quick look at the test failure, and it seems the first test to fail is testStopQueryLocal. Would you mind taking a look? Let me know if you need any help with it. Thanks!

2> REPRODUCE WITH: ./gradlew ":x-pack:plugin:esql:internalClusterTest" --tests "org.elasticsearch.xpack.esql.action.CrossClusterAsyncQueryIT.testStopQueryLocal" -Dtests.seed=14C1D323891169E4 -Dtests.locale=en-GY -Dtests.timezone=Africa/Conakry -Druntime.java=23
2> java.lang.AssertionError: 
  Expected: not <running>
       but: was <running>
      at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
      at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
      at org.elasticsearch.test.ESTestCase.assertThat(ESTestCase.java:2625)
      at org.elasticsearch.xpack.esql.action.EsqlAsyncTestUtils.lambda$waitForCluster$0(EsqlAsyncTestUtils.java:99)
      at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:1485)
      at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:1458)
      at org.elasticsearch.xpack.esql.action.EsqlAsyncTestUtils.waitForCluster(EsqlAsyncTestUtils.java:89)
      at org.elasticsearch.xpack.esql.action.CrossClusterAsyncQueryIT.testStopQueryLocal(CrossClusterAsyncQueryIT.java:367)

[smalyshev]

@dnhatn Yes, testStopQueryLocal has the stability issue now. See also #121672. Not sure yet what's going on there, somehow it seems to not run smoothly even though other tests that are doing the same things are fine. I'll ping you later if I can't find what's up there.

[smalyshev]

Ah, this is also 9.0 so likely related to #123334. I've backported the lock fix but if it keeps causing issues we may also want to backport moving stop tests to their own class like they are in main.

[smalyshev]

Looks like one-time issue caused by testStopQueryLocal and not repeating anymore, so closing this for now.