We want to ensure Coder v2 can scale to support large deployments and write docs to help users set up the right infrastructure!

Initial requirements

• We scale test Coder to ensure it can support workloads with 100, 1000, and 3000 users (e.g. a realistic number of concurrent builds, workspace connections, API requests, etc)
• Users can reproduce our results and get a sense of what infrastructure they need via "reference architecture" docs we provide
  • Reference environments are on Kubernetes (Coder's most common enterprise environment)
    • We share the configuration for cluster, coderd, and nodes (see discussions in docs: scaling Coder #5206)
• Provide coder loadtest command used and any flags used (template name, num workspaces, num connections, etc)
• Document how the coder loadtest works and how a user can use it against with their own environments, since their "mileage will vary" depending on various factors on their environment (e.g. workspace size, template, etc)

Vision

• We provide reference architectures for various infrastructure types (Docker, VMs, etc)
• We simulate actual developer workflows inside workspaces (e.g. intermittently running builds of a large Java project)
  • This helps identify provisioning rates, node sizing, etc.
• Reference architectures can be defined as code with Terraform. Users can use Terraform to deploy reference architecture
• We regularly run scale tests via CRON to identify regressions
• Scale test results against reference environments (tracing, build stats) are published on a publicly-accessible URL

References/inspiration

• GitLab: Performance & scalability
• GitLab: How we perform the tests
• Clickhouse benchmarks