Create load generator for Dynamic Parameters

[ethanndickson]

From the plan:

Goal:

• To observe how Coder handles dynamic parameters being manipulated at scale.

User target	coderd replicas	Concurrent evaluations
1k	3	100
2k	3	200
3k	4	300
10k	10	1000

Implementation:

• Re-evaluate template parameters via the API, no workspace builds.
• Template with a 5 level dependency chain:
  • Provider → Region → Zone → Instance Type → Storage(?)
  • We only need a single template uploaded on coderd, as I don't believe there's any per-template caching.

Run sequence:

1. Create users.
2. Create a complex template.
3. Establish baseline (<5 minutes)
4. Have each user client manipulate the 'root' parameter via the API.
5. Await responses on all requests.

Key Results:

• API response times.
• API errors (%).

Other Results:

• Coder deployment + database resource utilization (incl. pprofs, Grafana Pyroscope).

[spikecurtis]

There is per-template-version caching of the files. So, I think we want to load the system with multiple versions being used simultaneously. These things tend to follow a power law in real life.

I suggest we partition the concurrent evaluations across templates such that the ratio of popularity drops by half as you go in order. I.e. if we have 600 evaluations, we'd get 10 templates with [300, 150, 75, 38, 19, 9, 5, 2, 1, 1] evaluations each.

[spikecurtis]

@Emyrk @ethanndickson we need to decide whether external modules should be included as part of this load generator.

IIUC, modules that are referenced in the template version terraform (but not included locally) are resolved/downloaded by terraform during the template import and then inserted as a separate File object into the database. This file is then acquired/cached when dynamic parameters are evaluated and overlaid on top of the original terraform.

To reproduce this arrangement, we need to take a dependency on some external module, e.g. hosted in Terraform's open registry or Coder's registry. I'm pretty hesitant to add that kind of dependency to our scale tests:

• it could cause undesirable coupling between the availability/performance of the external registry and the scale test results
• it makes this load generator type unusable in an Internet-denied setting

It also will just take more time to implement and test.

My suggestion is that we rule it out of scope for now and take it under consideration for a later improvement to this load generator, as an optional inclusion.

[ethanndickson]

My suggestion is that we rule it out of scope for now and take it under consideration for a later improvement to this load generator, as an optional inclusion.

That sounds good to me.

It could cause undesirable coupling between the availability/performance of the external registry and the scale test results

Should we consider standing up our own single-module registry as part of our scaletest infrastructure? That would at least give us a registry with no other traffic and consistent/controllable performance,

[spikecurtis]

Should we consider standing up our own single-module registry as part of our scaletest infrastructure? That would at least give us a registry with no other traffic and consistent/controllable performance,

Yeah, if and when we add this functionality to the load generator.

[Emyrk]

IIUC, modules that are referenced in the template version terraform (but not included locally) are resolved/downloaded by terraform during the template import and then inserted as a separate File object into the database. This file is then acquired/cached when dynamic parameters are evaluated and overlaid on top of the original terraform.

This is correct.

To reproduce this arrangement, we need to take a dependency on some external module, e.g. hosted in Terraform's open registry or Coder's registry.

The source has a few options such as Github and https: https://developer.hashicorp.com/terraform/language/block/module#source

I agree it would be unfortunate to rely on the terraform registry. There are lighter options than running our own internal registry if we decide to test this.