[CI] TracesApmIT testApmIntegration failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic #6830 / openjdk21_checkpart1_java-fips-matrix
• elasticsearch-pull-request #59805 / part-1
• elasticsearch-pull-request #59780 / part-1
• elasticsearch-pull-request #59780 / part-1

Reproduction Line:

./gradlew ":test:external-modules:test-apm-integration:javaRestTest" --tests "org.elasticsearch.test.apmintegration.TracesApmIT.testApmIntegration" -Dtests.seed=70562D7B3B3B3228 -Dtests.locale=pa-Guru -Dtests.timezone=Antarctica/Rothera -Druntime.java=21 -Dtests.fips.enabled=true

Applicable branches:
9.0

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: State is not RESPONSE (REQUEST)

Issue Reasons:

• [9.0] 4 failures in test testApmIntegration (0.7% fail rate in 567 executions)
• [9.0] 3 failures in step part-1 (4.2% fail rate in 71 executions)
• [9.0] 2 failures in pipeline elasticsearch-pull-request (2.8% fail rate in 71 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[jdconrad]

This is probably related to #109335 where the apm server isn't aware that the cluster is shutting down. There is probably a way to make this test more robust prior to converting to the OTEL SDK. This is low-risk for these reasons.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 2 failures in test testApmIntegration (1.2% fail rate in 170 executions)

Build Scans:

• elasticsearch-periodic-platform-support #6499 / almalinux-8-aarch64_checkpart1_platform-support-arm
• elasticsearch-pull-request #58255 / part-1-entitlements

[ldematte]

The PR failure (2nd here) is bogus, as it was related to adding Files instrumentation and the APM agent does some strange stuff. Fixed now. I wonder if we should unmute the test, or like @jdconrad says if we should make it more lenient before.

[alexey-ivanov-es]

There is another muted APM test (muted after two failures, one of which occurred in the same PR): 123022. Since the failure in the PR was caused by an issue in the PR itself and there is only one failure outside the PR, I would just unmute them for now

[elasticsearchmachine]

This has been muted on branch 9.0

Mute Reasons:

• [9.0] 4 failures in test testApmIntegration (0.7% fail rate in 567 executions)
• [9.0] 3 failures in step part-1 (4.2% fail rate in 71 executions)
• [9.0] 2 failures in pipeline elasticsearch-pull-request (2.8% fail rate in 71 executions)

Build Scans:

• elasticsearch-periodic #6830 / openjdk21_checkpart1_java-fips-matrix
• elasticsearch-pull-request #59805 / part-1
• elasticsearch-pull-request #59780 / part-1
• elasticsearch-pull-request #59780 / part-1