Create load generator for Task App Statuses

[ethanndickson]

From the plan:

Goal:

• To observe how Coder handles Task usage at scale.

User target	Tasks	Regions	coderd replicas	Provisioners	Task update frequency
1k	400	3	3	60	6/minute
2k	800	3	3	120	6/minute
3k	1200	3	4	180	6/minute
10k	4000	3	10	600	6/minute

Infrastructure:

• Kubernetes cluster per region, each cluster has:
  • 1/3 of total workspaces.
  • 1/3 of total provisioners.
  • 1 workspace proxy in secondary regions (those w/o the coder deployment).

Implementation:

• Tools:
  • scaletest create-workspaces
  • Custom tool to create fake AI agents in each workspace (i.e. scaletest fake-ai ).
    • These fake AI agents should make tool calls to the Coder MCP server running in the workspace, which then talk to coderd .
• Build concurrency equal to provisioners, to minimize build load.

Run Sequence:

1. Create users.
2. Create workspaces with a Coder MCP server, and a fake AI agent.
3. Have each user client connect to the /api/v2/workspaces/{workspace}/watch-ws endpoint, and listen for status updates.
4. Establish baseline (10 minutes)
5. Have fake AI agents report app status updates every 10 seconds.
6. Run for 10-15 minutes.

Key Results:

• Latency from app status update to update receipt by client.
• DB utilization/load during update fan out.
• If any status updates were missed altogether.

Other Results:

• Coder deployment + database resource utilization (incl. pprofs, Grafana Pyroscope).

[spikecurtis]

Note to self: https://github.com/coder/coder/pull/20426/files gives a description of how to set up terraform for AI tasks.