update README

coder · johnstcn · Jun 8, 2023 · Jun 2, 2023 · Jun 2, 2023 · Jun 2, 2023
commit fd9886e1b50f95389a47c5e7601dbe1d882286cb
diff --git a/scaletest/README.md b/scaletest/README.md
@@ -1,43 +1,84 @@
-# Load Test Terraform
+# Scale Testing
 
-This folder contains Terraform code and scripts to aid in performing load tests of Coder.
-It does the following:
+This folder contains CLI commands, Terraform code, and scripts to aid in performing load tests of Coder.
+At a high level, it performs the following steps:
 
-- Creates a GCP VPC.
-- Creates a CloudSQL instance with a global peering rule so it's accessible inside the VPC.
-- Creates a GKE cluster inside the VPC with separate nodegroups for Coder and workspaces.
-- Installs Coder in a new namespace, using the CloudSQL instance.
+- Using the Terraform code in `./terraform`, stands up a preconfigured Google Cloud environment
+  consisting of a VPC, GKE Cluster, and CloudSQL instance.
+  > **Note: You must have an existing Google Cloud project available.**
+- Creates a dedicated namespace for Coder and installs Coder using the Helm chart in this namespace.
+- Configures the Coder deployment with random credentials and a predefined Kubernetes template.
+  > **Note:** These credentials are stored in `${PROJECT_ROOT}/scaletest/.coderv2/coder.env`.
+- Creates a number of workspaces and waits for them to all start successfully. These workspaces
+  are ephemeral and do not contain any persistent resources.
+- Waits for 10 minutes to allow things to settle and establish a baseline.
+- Generates web terminal traffic to all workspaces for 30 minutes.
+- Directly after traffic generation, captures goroutine and heap snapshots of the Coder deployment.
+- Tears down all resources (unless `--skip-cleanup` is specified).
 
-## Usage
 
-> You must have an existing Google Cloud project available.
+## Usage
 
-1. Create a file named `override.tfvars` with the following content, modifying as appropriate:
+The main entrypoint is the `scaletest.sh` script.
 
-```terraform
-name = "some_unique_identifier"
-project_id = "some_google_project_id"
+```console
+$ scaletest.sh --help
+Usage: scaletest.sh --name <name> --project <project> --num-workspaces <num-workspaces> --scenario <scenario> [--dry-run] [--skip-cleanup]
 ```
 
-1. Inspect `vars.tf` and override any other variables you deem necessary.
+### Required arguments:
+
+- `--name`: Name for the loadtest. This is added as a prefix to resources created by Terraform (e.g. `joe-big-loadtest`).
+- `--project`: Google Cloud project in which to create the resources (example: `my-loadtest-project`).
+- `--num-workspaces`: Number of workspaces to create (example: `10`).
+- `--scenario`: Deployment scenario to use (example: `small`). See `terraform/scenario-*.tfvars`.
+
+> **Note:** In order to capture Prometheus metrics, you must define the environment variables
+ `SCALETEST_PROMETHEUS_REMOTE_WRITE_USER` and `SCALETEST_PROMETHEUS_REMOTE_WRITE_PASSWORD`.
+
+### Optional arguments:
+
+- `--dry-run`: Do not perform any action and instead print what would be executed.
+- `--skip-cleanup`: Do not perform any cleanup. You will be responsible for deleting any resources this creates.
 
-1. Run `terraform init`.
+### Environment Variables
 
-1. Run `terraform plan -var-file=override.tfvars` and inspect the output.
-   If you are not satisfied, modify `override.tfvars` until you are.
+All of the above arguments may be specified as environment variables. Consult the script for details.
 
-1. Run `terraform apply -var-file=override.tfvars`. This will spin up a pre-configured environment
-   and emit the Coder URL as an output.
+### Prometheus Metrics
 
-1. Run `coder_init.sh <coder_url>` to setup an initial user and a pre-configured Kubernetes
-   template. It will also download the Coder CLI from the Coder instance locally.
+To capture Prometheus metrics from the loadtest, two environment
+
+## Scenarios
+
+A scenario defines a number of variables that override the default Terraform variables.
+A number of existing scenarios are provided in `scaletest/terraform/scenario-*.tfvars`.
+
+For example, `scenario-small.tfvars` includes the following variable definitions:
+
+```
+nodepool_machine_type_coder      = "t2d-standard-2"
+nodepool_machine_type_workspaces = "t2d-standard-2"
+coder_cpu                        = "1000m" # Leaving 1 CPU for system workloads
+coder_mem                        = "4Gi"   # Leaving 4GB for system workloads
+```
 
-1. Do whatever you need to do with the Coder instance:
+To create your own scenario, simply add a new file `terraform/scenario-$SCENARIO_NAME.tfvars`.
+In this file, override variables as required, consulting `vars.tf` as needed.
+You can then use this scenario by specifying `--scenario $SCENARIO_NAME`.
+For example, if your scenario file were named `scenario-big-whopper2x.tfvars`, you would specify
+`--scenario=big-whopper2x`.
 
-   > Note: To run Coder commands against the instance, you can use `coder_shim.sh <command>`.
-   > You don't need to run `coder login` yourself.
+## Utility scripts
 
-   - To create workspaces, run `./coder_shim.sh scaletest create-workspaces --template="kubernetes" --count=N`
-   - To generate workspace traffic, run `./coder_trafficgen.sh <name of loadtest from your Terraform vars>`. This will keep running until you delete the pod `coder-scaletest-workspace-traffic`.
+A number of utility scripts are provided in `lib`, and are used by `scaletest.sh`:
 
-1. When you are finished, you can run `terraform destroy -var-file=override.tfvars`.
+- `coder_shim.sh`: a convenience script to run the `coder` binary with a predefined config root.
+  This is intended to allow running Coder CLI commands against the loadtest cluster without
+  modifying a user's existing Coder CLI configuration.
+- `coder_init.sh`: Performs first-time user setup of an existing Coder instance, generating
+  a random password for the admin user. The admin user is named `admin@coder.com` by default.
+  Credentials are written to `scaletest/.coderv2/coder.env`.
+- `coder_workspacetraffic.sh`: Runs traffic generation against the loadtest cluster and creates
+  a monitoring manifest for the traffic generation pod. This pod will restart automatically
+  after the traffic generation has completed.