|
1 | | -# Load Test Terraform |
| 1 | +# Scale Testing |
2 | 2 |
|
3 | | -This folder contains Terraform code and scripts to aid in performing load tests of Coder. |
4 | | -It does the following: |
| 3 | +This folder contains CLI commands, Terraform code, and scripts to aid in performing load tests of Coder. |
| 4 | +At a high level, it performs the following steps: |
5 | 5 |
|
6 | | -- Creates a GCP VPC. |
7 | | -- Creates a CloudSQL instance with a global peering rule so it's accessible inside the VPC. |
8 | | -- Creates a GKE cluster inside the VPC with separate nodegroups for Coder and workspaces. |
9 | | -- Installs Coder in a new namespace, using the CloudSQL instance. |
| 6 | +- Using the Terraform code in `./terraform`, stands up a preconfigured Google Cloud environment |
| 7 | + consisting of a VPC, GKE Cluster, and CloudSQL instance. |
| 8 | + > **Note: You must have an existing Google Cloud project available.** |
| 9 | +- Creates a dedicated namespace for Coder and installs Coder using the Helm chart in this namespace. |
| 10 | +- Configures the Coder deployment with random credentials and a predefined Kubernetes template. |
| 11 | + > **Note:** These credentials are stored in `${PROJECT_ROOT}/scaletest/.coderv2/coder.env`. |
| 12 | +- Creates a number of workspaces and waits for them to all start successfully. These workspaces |
| 13 | + are ephemeral and do not contain any persistent resources. |
| 14 | +- Waits for 10 minutes to allow things to settle and establish a baseline. |
| 15 | +- Generates web terminal traffic to all workspaces for 30 minutes. |
| 16 | +- Directly after traffic generation, captures goroutine and heap snapshots of the Coder deployment. |
| 17 | +- Tears down all resources (unless `--skip-cleanup` is specified). |
10 | 18 |
|
11 | | -## Usage |
12 | 19 |
|
13 | | -> You must have an existing Google Cloud project available. |
| 20 | +## Usage |
14 | 21 |
|
15 | | -1. Create a file named `override.tfvars` with the following content, modifying as appropriate: |
| 22 | +The main entrypoint is the `scaletest.sh` script. |
16 | 23 |
|
17 | | -```terraform |
18 | | -name = "some_unique_identifier" |
19 | | -project_id = "some_google_project_id" |
| 24 | +```console |
| 25 | +$ scaletest.sh --help |
| 26 | +Usage: scaletest.sh --name <name> --project <project> --num-workspaces <num-workspaces> --scenario <scenario> [--dry-run] [--skip-cleanup] |
20 | 27 | ``` |
21 | 28 |
|
22 | | -1. Inspect `vars.tf` and override any other variables you deem necessary. |
| 29 | +### Required arguments: |
| 30 | + |
| 31 | +- `--name`: Name for the loadtest. This is added as a prefix to resources created by Terraform (e.g. `joe-big-loadtest`). |
| 32 | +- `--project`: Google Cloud project in which to create the resources (example: `my-loadtest-project`). |
| 33 | +- `--num-workspaces`: Number of workspaces to create (example: `10`). |
| 34 | +- `--scenario`: Deployment scenario to use (example: `small`). See `terraform/scenario-*.tfvars`. |
| 35 | + |
| 36 | +> **Note:** In order to capture Prometheus metrics, you must define the environment variables |
| 37 | + `SCALETEST_PROMETHEUS_REMOTE_WRITE_USER` and `SCALETEST_PROMETHEUS_REMOTE_WRITE_PASSWORD`. |
| 38 | + |
| 39 | +### Optional arguments: |
| 40 | + |
| 41 | +- `--dry-run`: Do not perform any action and instead print what would be executed. |
| 42 | +- `--skip-cleanup`: Do not perform any cleanup. You will be responsible for deleting any resources this creates. |
23 | 43 |
|
24 | | -1. Run `terraform init`. |
| 44 | +### Environment Variables |
25 | 45 |
|
26 | | -1. Run `terraform plan -var-file=override.tfvars` and inspect the output. |
27 | | - If you are not satisfied, modify `override.tfvars` until you are. |
| 46 | +All of the above arguments may be specified as environment variables. Consult the script for details. |
28 | 47 |
|
29 | | -1. Run `terraform apply -var-file=override.tfvars`. This will spin up a pre-configured environment |
30 | | - and emit the Coder URL as an output. |
| 48 | +### Prometheus Metrics |
31 | 49 |
|
32 | | -1. Run `coder_init.sh <coder_url>` to setup an initial user and a pre-configured Kubernetes |
33 | | - template. It will also download the Coder CLI from the Coder instance locally. |
| 50 | +To capture Prometheus metrics from the loadtest, two environment |
| 51 | + |
| 52 | +## Scenarios |
| 53 | + |
| 54 | +A scenario defines a number of variables that override the default Terraform variables. |
| 55 | +A number of existing scenarios are provided in `scaletest/terraform/scenario-*.tfvars`. |
| 56 | + |
| 57 | +For example, `scenario-small.tfvars` includes the following variable definitions: |
| 58 | + |
| 59 | +``` |
| 60 | +nodepool_machine_type_coder = "t2d-standard-2" |
| 61 | +nodepool_machine_type_workspaces = "t2d-standard-2" |
| 62 | +coder_cpu = "1000m" # Leaving 1 CPU for system workloads |
| 63 | +coder_mem = "4Gi" # Leaving 4GB for system workloads |
| 64 | +``` |
34 | 65 |
|
35 | | -1. Do whatever you need to do with the Coder instance: |
| 66 | +To create your own scenario, simply add a new file `terraform/scenario-$SCENARIO_NAME.tfvars`. |
| 67 | +In this file, override variables as required, consulting `vars.tf` as needed. |
| 68 | +You can then use this scenario by specifying `--scenario $SCENARIO_NAME`. |
| 69 | +For example, if your scenario file were named `scenario-big-whopper2x.tfvars`, you would specify |
| 70 | +`--scenario=big-whopper2x`. |
36 | 71 |
|
37 | | - > Note: To run Coder commands against the instance, you can use `coder_shim.sh <command>`. |
38 | | - > You don't need to run `coder login` yourself. |
| 72 | +## Utility scripts |
39 | 73 |
|
40 | | - - To create workspaces, run `./coder_shim.sh scaletest create-workspaces --template="kubernetes" --count=N` |
41 | | - - To generate workspace traffic, run `./coder_trafficgen.sh <name of loadtest from your Terraform vars>`. This will keep running until you delete the pod `coder-scaletest-workspace-traffic`. |
| 74 | +A number of utility scripts are provided in `lib`, and are used by `scaletest.sh`: |
42 | 75 |
|
43 | | -1. When you are finished, you can run `terraform destroy -var-file=override.tfvars`. |
| 76 | +- `coder_shim.sh`: a convenience script to run the `coder` binary with a predefined config root. |
| 77 | + This is intended to allow running Coder CLI commands against the loadtest cluster without |
| 78 | + modifying a user's existing Coder CLI configuration. |
| 79 | +- `coder_init.sh`: Performs first-time user setup of an existing Coder instance, generating |
| 80 | + a random password for the admin user. The admin user is named `admin@coder.com` by default. |
| 81 | + Credentials are written to `scaletest/.coderv2/coder.env`. |
| 82 | +- `coder_workspacetraffic.sh`: Runs traffic generation against the loadtest cluster and creates |
| 83 | + a monitoring manifest for the traffic generation pod. This pod will restart automatically |
| 84 | + after the traffic generation has completed. |
0 commit comments