feat(cli): add dashboard load test command #8723
Conversation
| ) | ||
|
|
||
| var ( | ||
| AllRBACResources = []RBACResource{ |
There was a problem hiding this comment.
nit: It looks like something devs may forget to update in the future.
There was a problem hiding this comment.
We have rbac gen:
coder/coderd/rbac/object_gen.go
Line 1 in 4463012
I wonder if we can repurpose it for this too?
In a more general note though, we manually copy over all enums we have in the database package to the sdk package. So this problem already exists for all enum types.
| var stdout, stderr bytes.Buffer | ||
| inv.Stdout = &stdout | ||
| inv.Stderr = &stderr | ||
| err := inv.WithContext(ctx).Run() |
There was a problem hiding this comment.
nit: you don't need to perform any actions/waiters/or output string-checks?
There was a problem hiding this comment.
This is me being lazy 😅
scaletest/dashboard/rolltable.go
Outdated
| // D&D nerds will feel right at home here :-) | ||
| // Note that the order of the table is important! | ||
| // Entries must be in ascending order. | ||
| var allActions rollTable = []rollTableEntry{ |
There was a problem hiding this comment.
I'm not sure that I buy the idea of rolling a dice. Isn't it better to perform a set of separate tests for each of these cases?
There was a problem hiding this comment.
Do you mean like, save all the calls when hitting the "Users" page and gather them into 1 "scenario" that gets run?
There was a problem hiding this comment.
Because right now the "scenarios" are single api calls, but you could feasibly make them emulate page hits.
There was a problem hiding this comment.
Correct - we could build out each action to include multiple fetches in parallel.
For now I'm focusing on keeping it simple so that we don't have to constantly play catchup with the frontend.
There was a problem hiding this comment.
The "table" construct is primarily an implementation detail to make it easier to weight the probability of each scenario. I could imagine fetching HTTP logs of an existing instance, and setting the priority of each endpoint accordingly.
scaletest/dashboard/run.go
Outdated
| case <-ctx.Done(): | ||
| return | ||
| case <-t.C: | ||
| rolls <- rand.Intn(allActions.max() + 1) // nolint:gosec |
There was a problem hiding this comment.
My main concern is that randomized tests may be hard to reproduce if every scenario is different. I'd rather model a few scenarios that cover all dashboard API calls. Feel free to challenge it though!
There was a problem hiding this comment.
My main concern is that randomized tests may be hard to reproduce if every scenario is different.
We're not looking for 100% reproducibility here, I don't think. I was mainly looking to generate load that corresponds to a baseline number of requests per second. When performed over a period of time, you're going to end up with data points for all of the scenarios.
There was a problem hiding this comment.
I never even considered reproducibility for the scaletest. Is there a way to maybe log or use prom metrics to indicate how many of each "scenario" is run?
Not needed in this PR, but would be cool so if 1 scaletest just craps the bed, and we notice "Scenerio 5" was run an excessive amount of times.
Idk how the metrics or logs are hooked up for this though
There was a problem hiding this comment.
100% adding prom metrics, we need those.
Regarding reproucibility, we could use a seeded PRNG so we can seed a single test with the current time and be able to reproduce it if needed.
cli/exp_scaletest.go
Outdated
| clibase.Option{ | ||
| Flag: "scaletest-prometheus-wait", | ||
| Env: "CODER_SCALETEST_PROMETHEUS_WAIT", | ||
| Default: "5s", |
There was a problem hiding this comment.
Is 5s long enough? Isn't a default scraper 15s?
| {0, fetchWorkspaces, "fetch workspaces"}, | ||
| {1, fetchUsers, "fetch users"}, | ||
| {2, fetchTemplates, "fetch templates"}, | ||
| {3, authCheckAsOwner, "authcheck owner"}, |
There was a problem hiding this comment.
Might be worth getting the list of auth checks the UI does all the time. Not needed in this PR, but the FE does like 10ish checks on each load.
This PR adds a scaletest subcommand to simulate dashboard load using codersdk.
A table is defined consisting of a list of actions to perform and an associated probability.
Each tick, an action is chosen from the table based on the result of a random number, and executed.