Skip to content

scaletest: How can I avoid API rate limits? #5531

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
bpmct opened this issue Jan 2, 2023 · 5 comments · Fixed by #5570
Closed

scaletest: How can I avoid API rate limits? #5531

bpmct opened this issue Jan 2, 2023 · 5 comments · Fixed by #5570

Comments

@bpmct
Copy link
Member

bpmct commented Jan 2, 2023

I'm attempting to run a scale test on my Kubernetes cluster with the following command:

coder scaletest create-workspaces \
    --count 1000 \
    --template "kubernetes" \
    --concurrency 0 \
    --cleanup-concurrency 0 \
    --parameter "home_disk_size=10" \
    --run-command "sleep 2 && echo hello"

However, I'm only able to create roughly ~60 concurrent workspaces as I run into a rate limit error:

== FAIL: workspacebuild/794

        Error: login as new user: POST http://coder.example.com/api/v2/users/login: unexpected status code 429: You've been rate limited for sending more than 60 requests in 1m0s.

I have also set CODER_API_RATE_LIMIT=-1 for my coder server, but saw that some routes are still rate limited, regardless of this value:

  --api-rate-limit int                           Maximum number of requests per minute
                                                 allowed to the API per user, or per IP
                                                 address for unauthenticated users.
                                                 Negative values mean no rate limit. Some
                                                 API endpoints are always rate limited
                                                 regardless of this value to prevent
                                                 denial-of-service attacks.
                                                 Consumes $CODER_API_RATE_LIMIT
@bpmct bpmct changed the title How can I avoid rate limits with scale tests? scaletest: How can I avoid rate limits? Jan 2, 2023
@bpmct bpmct changed the title scaletest: How can I avoid rate limits? scaletest: How can I avoid API rate limits? Jan 2, 2023
@Emyrk
Copy link
Member

Emyrk commented Jan 2, 2023

Two values are hard coded and not respecting the input value.

httpmw.RateLimit(12, time.Minute),

r.Use(httpmw.RateLimit(60, time.Minute))

@bpmct
Copy link
Member Author

bpmct commented Jan 2, 2023

I think that is the desired behavior, based on the help text:

Some API endpoints are always rate limited regardless of this value to prevent denial-of-service attacks.

I wish the scale test was able to bypass it somehow

@Emyrk
Copy link
Member

Emyrk commented Jan 2, 2023

Yea, I would argue all rate limits are to prevent DOS attacks though.

@ammario
Copy link
Member

ammario commented Jan 2, 2023

There's a BypassRatelimitHeader option that I added for this very purpose. Have you tried it?

@deansheather
Copy link
Member

@ammario the load tests use that header but only when making requests as site owner, this is the login endpoint to login to a new user so it has a ratelimit of it's own. I think we just need to make that ratelimit configurable also :/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants