[CI] RepositoryS3RestReloadCredentialsIT class failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic-platform-support #6863 / sles-15_platform-support-unix
• elasticsearch-periodic-platform-support #6719 / rhel-8_platform-support-unix

Reproduction Line:

undefined

Applicable branches:
8.18

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

undefined

Issue Reasons:

• [8.18] 2 failures in class org.elasticsearch.repositories.s3.RepositoryS3RestReloadCredentialsIT (0.3% fail rate in 688 executions)
• [8.18] 2 failures in pipeline elasticsearch-periodic-platform-support (10.5% fail rate in 19 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

[DaveCTurner]

java.nio.file.NoSuchFileException: /opt/local-ssd/buildkite/builds/bk-agent-prod-gcp-1740893888138178008/elastic/elasticsearch-periodic-platform-support/modules/repository-s3/build/testrun/javaRestTest/temp/test-cluster15497271829683965258/test-cluster-0/distro/config looks like a build environment problem?

RepositoryS3RestReloadCredentialsIT > classMethod FAILED	
    java.lang.RuntimeException: An error occurred orchestrating test cluster.	
        at __randomizedtesting.SeedInfo.seed([B5E3CA7FDF564F53]:0)	
        at org.elasticsearch.test.cluster.local.DefaultLocalClusterHandle.execute(DefaultLocalClusterHandle.java:291)	
        at org.elasticsearch.test.cluster.local.DefaultLocalClusterHandle.execute(DefaultLocalClusterHandle.java:296)	
        at org.elasticsearch.test.cluster.local.DefaultLocalClusterHandle.start(DefaultLocalClusterHandle.java:79)	
        at org.elasticsearch.test.cluster.local.DefaultLocalElasticsearchCluster$1.evaluate(DefaultLocalElasticsearchCluster.java:47)	
        at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)	
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)	
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)	
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
        at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)	
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)	
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)	
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
        at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)	
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)	
        at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)	
        at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)	
        at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)	
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)	
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)	
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)	
        at java.base/java.lang.Thread.run(Thread.java:1575)	
        Caused by:	
        java.nio.file.NoSuchFileException: /opt/local-ssd/buildkite/builds/bk-agent-prod-gcp-1740893888138178008/elastic/elasticsearch-periodic-platform-support/modules/repository-s3/build/testrun/javaRestTest/temp/test-cluster15497271829683965258/test-cluster-0/distro/config	
            at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)	
            at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)	
            at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)	
            at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:56)	
            at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:172)	
            at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)	
            at java.base/java.nio.file.Files.readAttributes(Files.java:1865)	
            at java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:220)	
            at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:277)	
            at java.base/java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:323)	
            at java.base/java.nio.file.FileTreeIterator.<init>(FileTreeIterator.java:71)	
            at java.base/java.nio.file.Files.walk(Files.java:3923)	
            at java.base/java.nio.file.Files.walk(Files.java:3979)	
            at org.elasticsearch.test.cluster.local.AbstractLocalClusterFactory$Node.writeConfiguration(AbstractLocalClusterFactory.java:426)	
            at org.elasticsearch.test.cluster.local.AbstractLocalClusterFactory$Node.start(AbstractLocalClusterFactory.java:178)	
            at org.elasticsearch.test.cluster.local.DefaultLocalClusterHandle.lambda$start$0(DefaultLocalClusterHandle.java:79)	
            at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)	
            at java.base/java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:722)	
            at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:570)	
            at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291)	
            at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:759)	
            at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:507)	
            at java.base/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:676)	
            at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:160)	
            at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:174)	
            at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:264)	
            at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:636)	
            at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:810)	
            at org.elasticsearch.test.cluster.local.DefaultLocalClusterHandle.lambda$start$1(DefaultLocalClusterHandle.java:79)	
            at org.elasticsearch.test.cluster.local.DefaultLocalClusterHandle.lambda$execute$16(DefaultLocalClusterHandle.java:297)	
            at java.base/java.util.concurrent.ForkJoinTask$AdaptedInterruptibleCallable.compute(ForkJoinTask.java:1689)	
            at java.base/java.util.concurrent.ForkJoinTask$InterruptibleTask.exec(ForkJoinTask.java:1641)	
            at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:507)	
            at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1458)	
            at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:2034)	
            at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:189)	

[elasticsearchmachine]

Pinging @elastic/es-delivery (Team:Delivery)

[jozala]

All the failures with Can't run executable: [...] does not exist, java.lang.RuntimeException: Execution of elasticsearch-plugin failed with exit code 74 or java.nio.file.NoSuchFileException: [...]/distro[..] have been caused by the test cluster setup bug fixed in: #124628.
I can see that all the failed tests mentioned in this issue failed with one of those.
I'll close this issue.