[CI] MlWithSecurityIT test {yaml=ml/3rd_party_deployment/Test start deployment fails while model download in progress} failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic-platform-support #6780 / windows-2022_checkpart4_platform-support-windows
• elasticsearch-pull-request #59998 / part-4
• elasticsearch-pull-request #59534 / part-4-fips

Reproduction Line:

gradlew ":x-pack:plugin:ml:qa:ml-with-security:yamlRestTest" --tests "org.elasticsearch.smoketest.MlWithSecurityIT.test {yaml=ml/3rd_party_deployment/Test start deployment fails while model download in progress}" -Dtests.seed=DCCBFB74A3C4D289 -Dtests.locale=tk-TM -Dtests.timezone=America/Guadeloupe -Druntime.java=23

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: Failure at [ml/3rd_party_deployment:221]: expected [2xx] status code but api [ml.put_trained_model] returned [400 Bad Request] [{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Failed to load package metadata","stack_trace":"org.elasticsearch.ElasticsearchException$1: Failed to load package metadata\r\n\tat org.elasticsearch.server@9.1.0-SNAPSHOT/org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:725)\r\n\tat org.elasticsearch.server@9.1.0-SNAPSHOT/org.elasticsearch.ElasticsearchException.generateFailureXContent(ElasticsearchException.java:639)\r\n\tat org.elasticsearch.server@9.1.0-SNAPSHOT/org.elasticsearch.rest.RestResponse.build(RestResponse.java:202)\r\n\tat org.elasticsearch.server@9.1.0-SNAPSHOT/org.elasticsearch.rest.RestResponse.<init>(RestResponse.java:161)\r\n\tat org.elasticsearch.server@9.1.0-SNAPSHOT/org.elasticsearch.rest.RestResponse.<init>(RestResponse.java:122)\r\n\tat org.elasticsearch.
[truncated]

Issue Reasons:

• [main] 3 failures in test test {yaml=ml/3rd_party_deployment/Test start deployment fails while model download in progress} (0.3% fail rate in 999 executions)
• [main] 2 failures in pipeline elasticsearch-pull-request (0.5% fail rate in 410 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[davidkyle]

The test failed because the ml feature reset failed

[2025-01-24T16:08:59,350][ERROR][o.e.x.m.MachineLearning  ] [yamlRestTest-0] failed to reset machine learning
org.elasticsearch.action.FailedNodeException: Failed node [yA0vy4BNTgalVxF6VeOxlg]
	....

Caused by: org.elasticsearch.transport.RemoteTransportException: [yamlRestTest-0][127.0.0.1:36665][cluster:monitor/tasks/lists[n]]
Caused by: org.elasticsearch.ElasticsearchTimeoutException: timed out after [30s/30000ms]
	at org.elasticsearch.action.support.SubscribableListener.lambda$scheduleTimeout$5(SubscribableListener.java:556) ~[elasticsearch-8.18.0-SNAPSHOT.jar:?]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:977) ~[elasticsearch-8.18.0-SNAPSHOT.jar:?]
	at org.elasticsearch.threadpool.ThreadPool$1.run(ThreadPool.java:514) ~[elasticsearch-8.18.0-SNAPSHOT.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]

[elasticsearchmachine]

This has been muted on branch 8.18

Mute Reasons:

• [8.18] 2 consecutive failures in step openjdk17_checkpart4_java-fips-matrix
• [8.18] 3 failures in test test {yaml=ml/3rd_party_deployment/Test start deployment fails while model download in progress} (2.3% fail rate in 130 executions)
• [8.18] 2 failures in step openjdk17_checkpart4_java-fips-matrix (66.7% fail rate in 3 executions)
• [8.18] 2 failures in pipeline elasticsearch-periodic (50.0% fail rate in 4 executions)

Build Scans:

• elasticsearch-periodic #6660 / openjdk17_checkpart4_java-fips-matrix
• elasticsearch-pull-request #59430 / part-4-fips
• elasticsearch-periodic #6653 / openjdk17_checkpart4_java-fips-matrix

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 3 failures in test test {yaml=ml/3rd_party_deployment/Test start deployment fails while model download in progress} (0.3% fail rate in 999 executions)
• [main] 2 failures in pipeline elasticsearch-pull-request (0.5% fail rate in 410 executions)

Build Scans:

• elasticsearch-periodic-platform-support #6780 / windows-2022_checkpart4_platform-support-windows
• elasticsearch-pull-request #59998 / part-4
• elasticsearch-pull-request #59534 / part-4-fips