[CI] LogsdbIndexingRollingUpgradeIT testIndexing {upgradedNodes=2} failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-intake #20048 / 9.0.0_bwc-snapshots
• elasticsearch-pull-request #62598 / 9.0.0_bwc-snapshots
• elasticsearch-pull-request #62595 / 9.0.0_bwc-snapshots
• elasticsearch-pull-request #62597 / 9.0.0_bwc-snapshots
• elasticsearch-pull-request #62592 / 9.0.0_bwc-snapshots
• elasticsearch-intake #20046 / 9.0.0_bwc-snapshots
• elasticsearch-pull-request #62590 / 9.0.0_bwc-snapshots
• elasticsearch-pull-request #62589 / 9.0.0_bwc-snapshots
• elasticsearch-intake #20044 / 9.0.0_bwc-snapshots
• elasticsearch-pull-request #62581 / 9.0.0_bwc-snapshots

Reproduction Line:

./gradlew ":qa:rolling-upgrade:v9.0.0#bwcTest" -Dtests.class="org.elasticsearch.upgrades.LogsdbIndexingRollingUpgradeIT" -Dtests.method="testIndexing {upgradedNodes=1}" -Dtests.seed=BC808F086923CB28 -Dtests.bwc=true -Dtests.locale=es-PA -Dtests.timezone=Pacific/Fiji -Druntime.java=23

Applicable branches:
main

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: timed out waiting for green state for index [logs-bwc-test] cluster state [{
  "cluster_name" : "test-cluster",
  "cluster_uuid" : "bDRMLq-FTiaBRwWDcOp73Q",
  "version" : 92,
  "state_uuid" : "wJZwpQztSJSWdertfdwIqA",
  "master_node" : "V20gptq2RlyMtqOAmf0Qgw",
  "blocks" : { },
  "nodes" : {
    "V20gptq2RlyMtqOAmf0Qgw" : {
      "name" : "test-cluster-1",
      "ephemeral_id" : "45sNebf6Rr6Vw-t2H7lMQw",
      "transport_address" : "127.0.0.1:39141",
      "external_id" : "test-cluster-1",
      "attributes" : {
        "testattr" : "test",
        "transform.config_version" : "10.0.0",
        "xpack.installed" : "true",
        "ml.allocated_processors" : "32",
        "ml.max_jvm_size" : "536870912",
        "ml.config_version" : "12.0.0",
        "ml.machine_memory" : "101237063680",
        "ml.allocated_processors_double" : "32.0"
      },
      "roles" : [
        "data",
        "data_cold",
        "data_content",
        "data_frozen",
        "data
[truncated]

Issue Reasons:

• [main] 7 consecutive failures in step 9.0.0_bwc-snapshots
• [main] 16 failures in test testIndexing {upgradedNodes=2} (1.7% fail rate in 947 executions)
• [main] 13 failures in step 9.0.0_bwc-snapshots (4.5% fail rate in 286 executions)
• [main] 2 failures in step 9.1.0_bwc-snapshots (0.7% fail rate in 288 executions)
• [main] 3 failures in pipeline elasticsearch-intake (4.5% fail rate in 67 executions)
• [main] 11 failures in pipeline elasticsearch-pull-request (4.7% fail rate in 236 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 2 consecutive failures in step 9.0.0_bwc-snapshots
• [main] 8 failures in test testIndexing {upgradedNodes=2} (0.9% fail rate in 904 executions)
• [main] 5 failures in step 9.0.0_bwc-snapshots (1.8% fail rate in 276 executions)
• [main] 2 failures in step 9.1.0_bwc-snapshots (0.7% fail rate in 277 executions)
• [main] 5 failures in pipeline elasticsearch-pull-request (2.2% fail rate in 228 executions)

Build Scans:

• elasticsearch-intake #20044 / 9.0.0_bwc-snapshots
• elasticsearch-pull-request #62581 / 9.0.0_bwc-snapshots
• elasticsearch-pull-request #62574 / 9.0.0_bwc-snapshots
• elasticsearch-pull-request #62504 / 9.0.0_bwc-snapshots
• elasticsearch-pull-request #62049 / 8.19.0_bwc-snapshots
• elasticsearch-pull-request #62049 / 9.0.0_bwc-snapshots
• elasticsearch-pull-request #62049 / 9.1.0_bwc-snapshots
• elasticsearch-pull-request #62005 / 9.1.0_bwc-snapshots

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)

[martijnvg]

This and #124831 and #124833, can be unmuted. These test failed briefly because #124803 was merged and #124787 wasn't.