flake: Database cleaner subprocess timeout on macOS - TestScheduleShow, TestCli

[blink-so]

CI Failure Details

CI Run: https://github.com/coder/coder/actions/runs/18002107653
Failed Job: test-go-pg (macos-latest)
Commit: da467bad2af6922488faef40470f2e3150472694
Date: 2025-09-25

Failing Tests

1. TestScheduleShow (cli/schedule_test.go:83)
2. TestCli (cli/clitest/clitest_test.go:18)

Root Cause Analysis

Both tests are failing with the same error during database setup:

    github.com/coder/coder/v2/coderd/database/dbtestutil.(*Broker).init
        /Users/runner/work/coder/coder/coderd/database/dbtestutil/broker.go:155
  - context deadline exceeded

Analysis

• The failure occurs in the recently introduced database cleaner subprocess (PR #19844, commit e2f5401fb)
• The cleaner uses go run github.com/coder/coder/v2/coderd/database/dbtestutil/cleanercmd to start a subprocess
• This subprocess has a 20-second timeout to start and send an "OK" response
• This failure is specific to macOS runners - Ubuntu and Windows tests pass
• The timeout suggests either:
  1. Go module compilation is slow on macOS
  2. Subprocess startup is slow on macOS
  3. Network/dependency issues on macOS runners

Assignment Analysis

Using git blame on the failing test functions:

TestScheduleShow (lines 83-90):

$ git blame -L 83,90 cli/schedule_test.go
a4f1319108 (Cian Johnston 2023-11-10 13:51:49 +0000 83) func TestScheduleShow(t *testing.T) {

TestCli (lines 18-25):

$ git blame -L 18,25 cli/clitest/clitest_test.go
154b9bce57 (Kyle Carberry    2022-02-12 13:34:04 -0600 18) func TestCli(t *testing.T) {

However, this is not a test-specific issue. The failure is in the database infrastructure code added by @spikecurtis in commit e2f5401fb. The actual issue is in the database cleaner startup process, not the individual test logic.

Recommendations

1. Increase timeout for macOS specifically (if this is expected slower startup)
2. Pre-compile the cleaner binary instead of using go run
3. Add better logging to understand where the delay occurs
4. Consider macOS-specific handling if this is a platform limitation

Related Issues

This cleaner was added to fix #927 (database leaks), but the fix is causing reliability issues on macOS.

[blink-so]

New Occurrence - September 25, 2025

CI Run: https://github.com/coder/coder/actions/runs/18007200335
Failed Job: test-go-pg (macos-latest)
Commit: 289f0217c7d0ee0ecb8692b772ef9d3052eebe92
Date: 2025-09-25 12:56:47 GMT

Failing Tests

1. TestCli (cli/clitest/clitest_test.go:21)
2. TestScaleTestWorkspaceTraffic_UseHostLogin (cli/exptest/exptest_scaletest_test.go:24)

Same Error Pattern

Both tests failing with identical database cleaner timeout:

start test db cleaner:
    github.com/coder/coder/v2/coderd/database/dbtestutil.(*Broker).init
        /Users/runner/work/coder/coder/coderd/database/dbtestutil/broker.go:155
  - context deadline exceeded

Test Duration

• Both tests ran for ~20.4 seconds before timing out
• This confirms the 20-second timeout in the database cleaner initialization

Note: This is the same macOS-specific infrastructure issue - not a new flake type.

[blink-so]

New Occurrence

CI Run: https://github.com/coder/coder/actions/runs/18166156920
Failed Job: test-go-pg (macos-latest)
Commit: 257fb768828e991df3715da69f6a67d10256ee3b
Author: @ethanndickson
Date: 2025-10-01

Same failure pattern - database cleaner subprocess timing out during initialization with context deadline exceeded error.

Affected Tests

• TestScheduleShow (cli/schedule_test.go:92)
• TestCli (cli/clitest/clitest_test.go:21)

Error trace identical to previous occurrence:

start test db cleaner:
    github.com/coder/coder/v2/coderd/database/dbtestutil.(*Broker).init
        /Users/runner/work/coder/coder/coderd/database/dbtestutil/broker.go:155
  - context deadline exceeded

[blink-so]

New Occurrence

CI Run: https://github.com/coder/coder/actions/runs/18269707147
Failed Job: test-go-pg (macos-latest)
Commit: c1357d4e275559221007a6efaef7a647af5217d0
Date: 2025-10-06
Test: TestExecutorAutostartSkipsWhenNoProvisionersAvailable

Error Details

Same database cleaner subprocess timeout on macOS:

lifecycle_executor_test.go:1699: Setting provisioner daemon tags: map[test-tag:asdf]
lifecycle_executor_test.go:1701: 
    Error Trace:	/Users/runner/work/coder/coder/coderd/database/dbtestutil/db.go:120
                				/Users/runner/work/coder/coder/coderd/autobuild/lifecycle_executor_test.go:1701
    Error:      	Received unexpected error:
                	start test db cleaner:
                	    github.com/coder/coder/v2/coderd/database/dbtestutil.(*Broker).init
                	        /Users/runner/work/coder/coder/coderd/database/dbtestutil/broker.go:155
                	  - context deadline exceeded
    Test:       	TestExecutorAutostartSkipsWhenNoProvisionersAvailable

This confirms the pattern - the database cleaner subprocess continues to timeout on macOS runners across different tests. The issue is not specific to individual test code but is a broader infrastructure problem with the database cleaner startup time on macOS.

[spikecurtis]

@deansheather we discussed having the dbcleaner stderr connected up to the test process stderr. I've found that gotestsum only includes data written to stderr outside the context of a test case.

When a test is running, data written to stderr is silently discarded, even in the output for test cases that fail.

So, first thing I did was implement explicitly compiling the dbcleaner binary. If that turns out not to be the cause of the macOS flakes, then I will look into explicitly reading stderr from the cleaner and logging it using t.Logf. It's a bit complicated because we need to ensure that

a) no goroutines are leaked to be flagged by goleak
b) we stop writing to the t.Logf once the test ends

This can be done, but I don't want to add it unless we see some reason we need it.

[blink-so]

New Occurrence - October 9, 2025

CI Run: https://github.com/coder/coder/actions/runs/18365170055
Failed Job: test-go-pg (macos-latest)
Workflow: nightly-gauntlet
Commit: c84ab728a8c209fd1bf291594c8b5cec86c47100
Date: 2025-10-09 04:20:59 GMT

Multiple Test Failures

This occurrence is more severe - 65 total test failures in the nightly gauntlet run, with multiple tests hitting the same database cleaner timeout issue:

1. TestPGCoordinatorSingle_AgentInvalidIP (enterprise/tailnet/pgcoord_test.go:111)
2. Test_Runner/Derp+Simple (scaletest/agentconn/run_test.go:224)
3. Test_Runner/OK (scaletest/createworkspaces/run_test.go:55)
4. Test_Run (scaletest/dashboard/run_test.go:45)
5. TestRun (scaletest/dynamicparameters/run_test.go:20)

Additionally:

• TestMultiplePresetsPerTemplateVersion - different timeout in prebuilds reconciliation (context deadline exceeded getting prebuilds group)
• TestPGCoordinatorPropogatedPeerContext - RequireSend: context expired

Common Error Pattern

Error Trace:	/Users/runner/work/coder/coder/coderd/database/dbtestutil/db.go:120
Error:      	Received unexpected error:
            	start test db cleaner:
            	    github.com/coder/coder/v2/coderd/database/dbtestutil.(*Broker).init
            	        /Users/runner/work/coder/coder/coderd/database/dbtestutil/broker.go:155
            	  - context deadline exceeded

Cleaner Build Times

The logs show the cleaner binary is taking 20-30+ seconds to build on macOS:

• TestPGCoordinatorSingle_AgentInvalidIP: 20.018s build time
• Test_Runner/Derp+Simple: 30.195s build time
• Test_Runner/OK: 26.921s build time
• Test_Run: 25.994s build time
• TestRun: 22.337s build time

This confirms that the cleaner binary compilation itself is exceeding or nearly exceeding the timeout.

Severity

This is a widespread failure affecting the nightly gauntlet run with 65 test failures total. The macOS database cleaner timeout issue is blocking multiple test suites and appears to be getting worse. The pre-compilation fix mentioned by @spikecurtis in the last comment is urgently needed.

[blink-so]

New Occurrence - October 9, 2025

CI Run: https://github.com/coder/coder/actions/runs/18376967667
Failed Job: test-go-pg (macos-latest)
Commit: ed90ecf00e2a7eba5b2ca060cf5348831091a9d3
Author: @mtojek
Date: 2025-10-09 12:59:33 GMT
Run Attempt: 1

Failing Test

TestProvisionerDaemon_PrometheusEnabled (enterprise/cli/provisionerdaemonstart_test.go:459)

Error Details

Same database cleaner subprocess timeout on macOS:

coderdenttest.go:84: 
    Error Trace:	/Users/runner/work/coder/coder/coderd/database/dbtestutil/db.go:120
                				/Users/runner/work/coder/coder/coderd/coderdtest/coderdtest.go:270
                				/Users/runner/work/coder/coder/enterprise/coderd/coderdenttest/coderdenttest.go:105
                				/Users/runner/work/coder/coder/enterprise/coderd/coderdenttest/coderdenttest.go:84
                				/Users/runner/work/coder/coder/enterprise/cli/provisionerdaemonstart_test.go:465
    Error:      	Received unexpected error:
                	start test db cleaner:
                	    github.com/coder/coder/v2/coderd/database/dbtestutil.(*Broker).init
                	        /Users/runner/work/coder/coder/coderd/database/dbtestutil/broker.go:155
                	  - context deadline exceeded
    Test:       	TestProvisionerDaemon_PrometheusEnabled

Cleaner Binary Build Time

The logs show:

cleaner_posix.go:19: [2025-10-09 12:58:01.501] starting cleaner binary build
cleaner_posix.go:31: [2025-10-09 12:58:24.665] cleaner binary /var/folders/.../cleaner built in 23.16390425s

The cleaner binary took 23.16 seconds to build, which is very close to exceeding the timeout threshold. This confirms the root cause - go run compilation on macOS is too slow.

Test Ownership Analysis

Using git blame on the test function (line 459):

$ git blame -L 459,475 enterprise/cli/provisionerdaemonstart_test.go
966c888e9d (Marcin Tojek 2024-07-23 12:31:27 +0200) func TestProvisionerDaemon_PrometheusEnabled(t *testing.T) {

However, as noted in the original issue analysis, this is an infrastructure problem, not a test-specific issue. The failure is in the database cleaner startup process, not the test logic itself.

Status

This occurrence happened on the main branch after the explicit binary compilation fix was mentioned (previous comment from @spikecurtis on Oct 8). It appears this fix may not have been merged yet, as the logs still show the cleaner being built at runtime. The issue continues to affect various tests on macOS.