Skip to content

fix: Prevent autobuild executor from slowing down API requests #3726

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Sep 2, 2022
Merged

fix: Prevent autobuild executor from slowing down API requests #3726

merged 4 commits into from
Sep 2, 2022

Conversation

mafredri
Copy link
Member

With just a few workspaces, the autobuild executor can slow down API
requests every time it runs. This is because we started a long running
transaction and checked all eligible (for autostart) workspaces inside
that transaction. PostgreSQL doesn't know if we're modifying rows and as
such is locking the tables for read operations.

This commit changes the behavior so each workspace is checked in its own
transaction reducing the time the table/rows needs to stay locked.

For now concurrency has been arbitrarily limited to 10 workspaces at a
time, this could be made configurable or adjusted as the need arises.

Before (worst observed case):

coder list > /dev/null  0.04s user 0.01s system 33% cpu 0.144 total
coder list > /dev/null  0.04s user 0.01s system 33% cpu 0.147 total
coder list > /dev/null  0.05s user 0.01s system 6% cpu 0.937 total
coder list > /dev/null  0.05s user 0.01s system 34% cpu 0.179 total
coder list > /dev/null  0.05s user 0.01s system 27% cpu 0.208 total
coder list > /dev/null  0.05s user 0.02s system 23% cpu 0.283 total
coder list > /dev/null  0.05s user 0.01s system 31% cpu 0.198 total
coder list > /dev/null  0.05s user 0.01s system 23% cpu 0.243 total
coder list > /dev/null  0.05s user 0.01s system 34% cpu 0.170 total
coder list > /dev/null  0.05s user 0.01s system 19% cpu 0.305 total
coder list > /dev/null  0.05s user 0.01s system 32% cpu 0.196 total
coder list > /dev/null  0.05s user 0.01s system 40% cpu 0.147 total
coder list > /dev/null  0.04s user 0.01s system 31% cpu 0.153 total
coder list > /dev/null  0.05s user 0.01s system 15% cpu 0.380 total
coder list > /dev/null  0.05s user 0.01s system 34% cpu 0.184 total
coder list > /dev/null  0.05s user 0.01s system 24% cpu 0.243 total
coder list > /dev/null  0.05s user 0.02s system 24% cpu 0.285 total
coder list > /dev/null  0.05s user 0.01s system 39% cpu 0.160 total
coder list > /dev/null  0.04s user 0.01s system 24% cpu 0.216 total
coder list > /dev/null  0.05s user 0.02s system 36% cpu 0.167 total
coder list > /dev/null  0.05s user 0.02s system 9% cpu 0.661 total
coder list > /dev/null  0.05s user 0.02s system 12% cpu 0.581 total
coder list > /dev/null  0.05s user 0.01s system 32% cpu 0.199 total
coder list > /dev/null  0.05s user 0.01s system 25% cpu 0.240 total
coder list > /dev/null  0.05s user 0.01s system 34% cpu 0.172 total
coder list > /dev/null  0.04s user 0.01s system 30% cpu 0.175 total
coder list > /dev/null  0.05s user 0.01s system 10% cpu 0.550 total
coder list > /dev/null  0.05s user 0.02s system 9% cpu 0.712 total
coder list > /dev/null  0.05s user 0.02s system 17% cpu 0.389 total
coder list > /dev/null  0.05s user 0.01s system 25% cpu 0.250 total
coder list > /dev/null  0.05s user 0.01s system 34% cpu 0.175 total
coder list > /dev/null  0.04s user 0.01s system 39% cpu 0.135 total
coder list > /dev/null  0.04s user 0.01s system 18% cpu 0.292 total
coder list > /dev/null  0.05s user 0.02s system 39% cpu 0.168 total
coder list > /dev/null  0.05s user 0.01s system 36% cpu 0.148 total
coder list > /dev/null  0.04s user 0.01s system 35% cpu 0.138 total
coder list > /dev/null  0.04s user 0.01s system 34% cpu 0.144 total
coder list > /dev/null  0.04s user 0.01s system 32% cpu 0.153 total
coder list > /dev/null  0.05s user 0.01s system 17% cpu 0.330 total
coder list > /dev/null  0.05s user 0.01s system 23% cpu 0.263 total
coder list > /dev/null  0.05s user 0.02s system 5% cpu 1.127 total
coder list > /dev/null  0.05s user 0.01s system 33% cpu 0.186 total
coder list > /dev/null  0.05s user 0.01s system 23% cpu 0.268 total
coder list > /dev/null  0.05s user 0.01s system 32% cpu 0.193 total
coder list > /dev/null  0.05s user 0.01s system 38% cpu 0.149 total

And after:

coder list > /dev/null  0.04s user 0.01s system 36% cpu 0.129 total
coder list > /dev/null  0.04s user 0.01s system 40% cpu 0.118 total
coder list > /dev/null  0.04s user 0.01s system 12% cpu 0.368 total
coder list > /dev/null  0.06s user 0.02s system 35% cpu 0.214 total
coder list > /dev/null  0.06s user 0.02s system 40% cpu 0.177 total
coder list > /dev/null  0.05s user 0.01s system 39% cpu 0.137 total

(The after is much shorter, but it's the worst observed case after the change.)

@mafredri mafredri self-assigned this Aug 29, 2022
@mafredri mafredri requested review from johnstcn and a team August 29, 2022 09:52
johnstcn
johnstcn approved these changes Aug 29, 2022
View reviewed changes
Copy link
Member

@johnstcn johnstcn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work! Just a couple of suggestions; I think we can probably move more of the logic into Go and out of the DB layer to keep things all self-contained.

@mafredri
fix: Prevent autobuild executor from slowing down API requests
11b7eb8 
With just a few workspaces, the autobuild executor can slow down API
requests every time it runs. This is because we started a long running
transaction and checked all eligible (for autostart) workspaces inside
that transaction. PostgreSQL doesn't know if we're modifying rows and as
such is locking the tables for read operations.

This commit changes the behavior so each workspace is checked in its own
transaction reducing the time the table/rows needs to stay locked.

For now concurrency has been arbitrarily limited to 10 workspaces at a
time, this could be made configurable or adjusted as the need arises.
@mafredri
Add early return
3a345aa
@mafredri
chore: Refactor autostart check, do it in Go
45e4141
@mafredri
fix: Ensure deleted workspaces are not eligible
b4a650b
@mafredri mafredri force-pushed the mafredri/improve-performance branch from e31f80b to b4a650b Compare September 2, 2022 08:24
@mafredri mafredri merged commit 4c18034 into main Sep 2, 2022
@mafredri mafredri deleted the mafredri/improve-performance branch September 2, 2022 10:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@johnstcn johnstcn johnstcn approved these changes

Assignees

@mafredri mafredri

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@mafredri @johnstcn