Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] InferenceGetServicesIT class failing #124185

Closed
elasticsearchmachine opened this issue Mar 6, 2025 · 4 comments
Closed

[CI] InferenceGetServicesIT class failing #124185

elasticsearchmachine opened this issue Mar 6, 2025 · 4 comments
Labels
low-risk An open issue or test failure that is a low risk to future releases :ml Machine learning Team:ML Meta label for the ML team >test-failure Triaged test failures from CI

Comments

@elasticsearchmachine
Copy link
Collaborator

elasticsearchmachine commented Mar 6, 2025 •
edited
Loading

Build Scans:

Reproduction Line:

undefined

Applicable branches:
8.18

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

undefined

Issue Reasons:

  • [8.18] 2 failures in class org.elasticsearch.xpack.inference.InferenceGetServicesIT (0.3% fail rate in 770 executions)
  • [8.18] 2 failures in pipeline elasticsearch-periodic-platform-support (10.0% fail rate in 20 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.
@elasticsearchmachine elasticsearchmachine added :ml Machine learning >test-failure Triaged test failures from CI Team:ML Meta label for the ML team needs:risk Requires assignment of a risk label (low, medium, blocker) labels Mar 6, 2025
@elasticsearchmachine
Copy link
Collaborator Author

Pinging @elastic/ml-core (Team:ML)

@jonathan-buttner
Copy link
Contributor

For the release tests the failure is:

        java.lang.IllegalArgumentException: Can't run executable: `/opt/buildkite-agent/builds/bk-agent-prod-gcp-1740864863961808188/elastic/elasticsearch-periodic/x-pack/plugin/inference/qa/inference-service-tests/build/testrun/javaRestTest/temp/test-cluster3174178169262148509/test-cluster-0/distro/bin/elasticsearch-plugin` does not exist.
            at org.elasticsearch.test.cluster.util.ProcessUtils.exec(ProcessUtils.java:54)
            at org.elasticsearch.test.cluster.local.AbstractLocalClusterFactory$Node.runToolScript(AbstractLocalClusterFactory.java:923)
            at org.elasticsearch.test.cluster.local.AbstractLocalClusterFactory$Node.installPlugins(AbstractLocalClusterFactory.java:752)
            at org.elasticsearch.test.cluster.local.AbstractLocalClusterFactory$Node.start(AbstractLocalClusterFactory.java:171)
            at org.elasticsearch.test.cluster.local.DefaultLocalClusterHandle.lambda$start$0(DefaultLocalClusterHandle.java:79)
            at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
            at java.base/java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:722)
            at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:570)
            at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291)
            at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:759)
            at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:507)
            at java.base/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:676)
            at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:160)
            at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:174)
            at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:264)
            at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:636)
            at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:810)
            at org.elasticsearch.test.cluster.local.DefaultLocalClusterHandle.lambda$start$1(DefaultLocalClusterHandle.java:79)
            at org.elasticsearch.test.cluster.local.DefaultLocalClusterHandle.lambda$execute$16(DefaultLocalClusterHandle.java:297)
            at java.base/java.util.concurrent.ForkJoinTask$AdaptedInterruptibleCallable.compute(ForkJoinTask.java:1689)
            at java.base/java.util.concurrent.ForkJoinTask$InterruptibleTask.exec(ForkJoinTask.java:1641)
            at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:507)
            at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1458)
            at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:2034)
            at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:189)

Maybe this is a build related issue 🤔

@jonathan-buttner
Copy link
Contributor

For elasticsearch-periodic-platform-support #6806 / almalinux-8_platform-support-unix

[2025-03-05T22:53:46,202][INFO ][process-output           ] [[elasticsearch-plugin-log-forwarder]] -> Downloading file:///opt/local-ssd/buildkite/builds/bk-agent-prod-gcp-1741239269952963741/elastic/elasticsearch-periodic-platform-support/x-pack/plugin/inference/qa/test-service-plugin/build/distributions/inference-service-test-8.18.0-SNAPSHOT.zip
[2025-03-05T22:53:46,206][ERROR][process-output           ] [[elasticsearch-plugin-log-forwarder]] java.nio.file.NoSuchFileException: /opt/local-ssd/buildkite/builds/bk-agent-prod-gcp-1741239269952963741/elastic/elasticsearch-periodic-platform-support/x-pack/plugin/inference/qa/inference-service-tests/build/testrun/javaRestTest/temp/test-cluster14121365567700162289/test-cluster-0/distro/plugins/.installing-5937250166822199906
[2025-03-05T22:53:46,207][ERROR][process-output           ] [[elasticsearch-plugin-log-forwarder]] 	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
[2025-03-05T22:53:46,213][ERROR][process-output           ] [[elasticsearch-plugin-log-forwarder]] 	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
[2025-03-05T22:53:46,213][ERROR][process-output           ] [[elasticsearch-plugin-log-forwarder]] 	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
[2025-03-05T22:53:46,213][ERROR][process-output           ] [[elasticsearch-plugin-log-forwarder]] 	at java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:414)
[2025-03-05T22:53:46,213][ERROR][process-output           ] [[elasticsearch-plugin-log-forwarder]] 	at java.base/java.nio.file.Files.createDirectory(Files.java:647)
[2025-03-05T22:53:46,213][ERROR][process-output           ] [[elasticsearch-plugin-log-forwarder]] 	at java.base/java.nio.file.TempFileHelper.create(TempFileHelper.java:130)
[2025-03-05T22:53:46,213][INFO ][process-output           ] [[elasticsearch-plugin-log-forwarder]] -> Failed installing file:///opt/local-ssd/buildkite/builds/bk-agent-prod-gcp-1741239269952963741/elastic/elasticsearch-periodic-platform-support/x-pack/plugin/inference/qa/test-service-plugin/build/distributions/inference-service-test-8.18.0-SNAPSHOT.zip
[2025-03-05T22:53:46,214][ERROR][process-output           ] [[elasticsearch-plugin-log-forwarder]] 	at java.base/java.nio.file.TempFileHelper.createTempDirectory(TempFileHelper.java:162)
[2025-03-05T22:53:46,214][INFO ][process-output           ] [[elasticsearch-plugin-log-forwarder]] -> Rolling back file:///opt/local-ssd/buildkite/builds/bk-agent-prod-gcp-1741239269952963741/elastic/elasticsearch-periodic-platform-support/x-pack/plugin/inference/qa/test-service-plugin/build/distributions/inference-service-test-8.18.0-SNAPSHOT.zip
[2025-03-05T22:53:46,214][ERROR][process-output           ] [[elasticsearch-plugin-log-forwarder]] 	at java.base/java.nio.file.Files.createTempDirectory(Files.java:890)
[2025-03-05T22:53:46,214][ERROR][process-output           ] [[elasticsearch-plugin-log-forwarder]] 	at org.elasticsearch.plugins.cli.InstallPluginAction.stagingDirectory(InstallPluginAction.java:831)
[2025-03-05T22:53:46,214][INFO ][process-output           ] [[elasticsearch-plugin-log-forwarder]] -> Rolled back file:///opt/local-ssd/buildkite/builds/bk-agent-prod-gcp-1741239269952963741/elastic/elasticsearch-periodic-platform-support/x-pack/plugin/inference/qa/test-service-plugin/build/distributions/inference-service-test-8.18.0-SNAPSHOT.zip
[2025-03-05T22:53:46,214][ERROR][process-output           ] [[elasticsearch-plugin-log-forwarder]] 	at org.elasticsearch.plugins.cli.InstallPluginAction.unzip(InstallPluginAction.java:778)
[2025-03-05T22:53:46,214][ERROR][process-output           ] [[elasticsearch-plugin-log-forwarder]] 	at org.elasticsearch.plugins.cli.InstallPluginAction.execute(InstallPluginAction.java:254)
[2025-03-05T22:53:46,214][ERROR][process-output           ] [[elasticsearch-plugin-log-forwarder]] 	at org.elasticsearch.plugins.cli.InstallPluginCommand.execute(InstallPluginCommand.java:90)
[2025-03-05T22:53:46,214][ERROR][process-output           ] [[elasticsearch-plugin-log-forwarder]] 	at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:55)
[2025-03-05T22:53:46,214][ERROR][process-output           ] [[elasticsearch-plugin-log-forwarder]] 	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:101)
[2025-03-05T22:53:46,214][ERROR][process-output           ] [[elasticsearch-plugin-log-forwarder]] 	at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:95)
[2025-03-05T22:53:46,216][ERROR][process-output           ] [[elasticsearch-plugin-log-forwarder]] 	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:101)
[2025-03-05T22:53:46,217][ERROR][process-output           ] [[elasticsearch-plugin-log-forwarder]] 	at org.elasticsearch.cli.Command.main(Command.java:54)
[2025-03-05T22:53:46,217][ERROR][process-output           ] [[elasticsearch-plugin-log-forwarder]] 	at org.elasticsearch.launcher.CliToolLauncher.main(CliToolLauncher.java:65)
[2025-03-05T22:53:46,354][INFO ][o.e.t.c.l.DefaultLocalClusterHandle] [suite] Stopping Elasticsearch test cluster 'test-cluster', forcibly: true

Seems like it failed to download some testing zip file?

@jonathan-buttner jonathan-buttner added low-risk An open issue or test failure that is a low risk to future releases and removed needs:risk Requires assignment of a risk label (low, medium, blocker) labels Mar 6, 2025
@jozala
Copy link
Contributor

jozala commented Mar 13, 2025

All the failures with Can't run executable: [...] does not exist, java.lang.RuntimeException: Execution of elasticsearch-plugin failed with exit code 74 or java.nio.file.NoSuchFileException: [...]/distro[..] have been caused by the test cluster setup bug fixed in: #124628.
I can see that all the failed tests mentioned in this issue failed with one of those.
I'll close this issue.

@jozala jozala closed this as completed Mar 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
low-risk An open issue or test failure that is a low risk to future releases :ml Machine learning Team:ML Meta label for the ML team >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jozala @jonathan-buttner @elasticsearchmachine