-
Notifications
You must be signed in to change notification settings - Fork 1k
Closed
Labels
scaletestIssues related to scale testing.Issues related to scale testing.
Milestone
Description
Problem
Each running workspace agent reports stats every 30 seconds by default via POST to /api/v2/workspaceagents/me/report-stats
. This handler synchronously persists this statistic to the database.
With large numbers of agents, this can potentially cause large numbers of transactions. With 10,000 workspace agents, this would equate to roughly 333.4 requests per second, each of which would kick off a database transaction.
Proposed Solution
We can reduce this overhead by batching updates to workspace agent stats.
- Alter InsertWorkspaceAgentStat to persist multiple rows
- Store a batch of workspace agent stats with a limited size (e.g. 10)
- Insert the current batch every batch interval (e.g. 1s) or once the current batch reaches capacity, whichever happens first.
- It may be prudent to add a limited retry on batch insertion in order to avoid dropping stats.
Metadata
Metadata
Assignees
Labels
scaletestIssues related to scale testing.Issues related to scale testing.