bug: CI passes despite test failures!

[Kira-Pilot]

CI passes here but tests are actually failing. The failures are logged in Datadog. See below context - this is due to a retry mechanism. We should stop retrying as doing so compromises the feedback loop for engineering.

Maybe we're passing the wrong exit codes on Windows?

Posting @hugodutka's comment here:

We automatically retry failing tests now. If the test passes on the next run, we don't report the CI run as failed. The output isn't formatted great, but that's what happened here. The key line here is:

DONE 2 runs, 9735 tests, 229 skipped, 2 failures in 162.909s
2 runs means that the tests were run twice, and on the second run they passed.

[hugodutka]

We automatically retry failing tests now. If the test passes on the next run, we don't report the CI run as failed. The output isn't formatted great, but that's what happened here. The key line here is:

DONE 2 runs, 9735 tests, 229 skipped, 2 failures in 162.909s

2 runs means that the tests were run twice, and on the second run they passed.

[Kira-Pilot]

@hugodutka thanks for the context! We discussed this with Jon today and would like to revert the retries PR if you can help us out with that. We're worried we're sweeping flakes under the rug as engineers may not be combing Datadog as frequently as they are interacting with CI.