Skip to content

chore: add scaletest convenience script #7819

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 17 commits into from
Jun 8, 2023
Merged

chore: add scaletest convenience script #7819

merged 17 commits into from
Jun 8, 2023

Conversation

johnstcn
Copy link
Member

@johnstcn johnstcn commented Jun 2, 2023

Adds a convenience script for scaletesting. This will automatically stand up a scale testing cluster, create a number of workspaces, and run traffic generation against the workspaces.

A number of sample scenarios are included. I'm open to changing the naming scheme, but for now I'm settling on T-shirt sizes with multipliers for the number of workspace nodes.

Additional changes:

  • Enables pprof endpoint by default
  • Captures pprof traces before tearing down infra.
  • coder_init.sh is more idempotent than it previously was.
  • Removed the promtest.Float64 invocations in workspacetraffic runner, these metrics will be in prometheus.
  • Increases default traffic sent to workspaces to 40KB/s.

@johnstcn johnstcn self-assigned this Jun 2, 2023
attempt_counter=0
max_attempts=6 # 60 seconds
echo -n "Waiting for Coder deployment at ${SCALETEST_CODER_URL} to become ready"
until curl --output /dev/null --silent --fail "${SCALETEST_CODER_URL}/healthz"; do
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not a fan of bash for this sort of thing because it makes it basically impossible to be cross-platform due to shelling out to commands that we don't control the version of.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you be more in favour of a kubectl wait?

Copy link
Contributor

@spikecurtis spikecurtis Jun 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not curl, in particular, that's the problem...

Copy link
Member

@mtojek mtojek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍 I left a few comments, but these can be applied iteratively or discarded.

I'd like to see a public runbook for developers trying to run these scale tests. It would be awesome to store it in our Coder docs. Some customers may want to perform these to examine their infra.

echo "Taking pprof snapshots"
maybedryrun "$DRY_RUN" curl --silent --fail --output "${SCALETEST_NAME}-heap.pprof.gz" http://localhost:6061/debug/pprof/heap
maybedryrun "$DRY_RUN" curl --silent --fail --output "${SCALETEST_NAME}-goroutine.pprof.gz" http://localhost:6061/debug/pprof/goroutine
maybedryrun "$DRY_RUN" kill $pfpid
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: Could remove trap handler here to avoid error on exit due to missing pid:

Suggested change
maybedryrun "$DRY_RUN" kill $pfpid
maybedryrun "$DRY_RUN" kill $pfpid
maybedryrun "$DRY_RUN" trap - EXIT

@johnstcn johnstcn added chore scaletest Issues related to scale testing. labels Jun 7, 2023
@johnstcn johnstcn merged commit efbb558 into main Jun 8, 2023
@johnstcn johnstcn deleted the cj/scaletest-script branch June 8, 2023 08:30
@github-actions github-actions bot locked and limited conversation to collaborators Jun 8, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
scaletest Issues related to scale testing.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants