[CI] MlWithSecurityIT test {yaml=ml/3rd_party_deployment/Test start and stop multiple deployments} failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic #7318 / openjdk17_checkpart4_java-fips-matrix
• elasticsearch-periodic #7283 / openjdk17_checkpart4_java-matrix

Reproduction Line:

./gradlew ":x-pack:plugin:ml:qa:ml-with-security:yamlRestTest" --tests "org.elasticsearch.smoketest.MlWithSecurityIT.test {yaml=ml/3rd_party_deployment/Test start and stop multiple deployments}" -Dtests.seed=36C1F9B1BBB9F987 -Dtests.locale=shi-Tfng-MA -Dtests.timezone=Europe/Bucharest -Druntime.java=17 -Dtests.fips.enabled=true

Applicable branches:
8.18

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: Failure at [ml/3rd_party_deployment:633]: expected [2xx] status code but api [ml.infer_trained_model] returned [408 Request Timeout] [{"error":{"root_cause":[{"type":"status_exception","reason":"timeout [10s] waiting for inference result","stack_trace":"org.elasticsearch.ElasticsearchStatusException: timeout [10s] waiting for inference result\n\tat org.elasticsearch.ml@8.18.0-SNAPSHOT/org.elasticsearch.xpack.ml.inference.deployment.AbstractPyTorchAction.onTimeout(AbstractPyTorchAction.java:68)\n\tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:977)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\n"}],"type":"status_exception","reason":"timeout [10s] waiting for infere
[truncated]

Issue Reasons:

• [8.18] 2 failures in test test {yaml=ml/3rd_party_deployment/Test start and stop multiple deployments} (1.5% fail rate in 137 executions)
• [8.18] 2 failures in pipeline elasticsearch-periodic (40.0% fail rate in 5 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 3 failures in test test {yaml=ml/3rd_party_deployment/Test start and stop multiple deployments} (0.4% fail rate in 813 executions)
• [main] 2 failures in step part-4 (0.6% fail rate in 350 executions)
• [main] 2 failures in pipeline elasticsearch-pull-request (0.6% fail rate in 353 executions)

Build Scans:

• elasticsearch-intake #19620 / part4
• elasticsearch-pull-request #60598 / part-4
• elasticsearch-pull-request #60254 / part-4

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)