feat: make agent stats' cardinality configurable #12468
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dannykopping
commented
Mar 8, 2024
mafredri
reviewed
Mar 8, 2024
| "golang.org/x/xerrors" | ||
|
|
||
| "github.com/coder/coder/v2/coderd/agentmetrics" |
There was a problem hiding this comment.
Unfortunately our formatter doesn't handle merging import groups and leaves things in a messy state (depending on what program injected them). 😔
If you notice these, please feel free to fix, but the standard is we try our best but sometimes these slip through, so don't worry too much.
codersdk/deployment.go
Outdated
| return nil | ||
| } | ||
|
|
||
| acceptable := make(map[string]any, len(AcceptedMetricAggregationLabels)) |
There was a problem hiding this comment.
I think using map[string]bool would be preferable/clearer here, also simplifies the map lookup later. Typically I'd use either bool or if we're looking for space savings, I'd use the zero struct (map[string]struct{}).
Signed-off-by: Danny Kopping <danny@coder.com>
Signed-off-by: Danny Kopping <danny@coder.com>
…ebug works Signed-off-by: Danny Kopping <danny@coder.com>
Signed-off-by: Danny Kopping <danny@coder.com>
Signed-off-by: Danny Kopping <danny@coder.com>
Signed-off-by: Danny Kopping <danny@coder.com>
Signed-off-by: Danny Kopping <danny@coder.com>
5ba04c5 to
3e569ff
Compare
Signed-off-by: Danny Kopping <danny@coder.com>
Signed-off-by: Danny Kopping <danny@coder.com>
johnstcn
reviewed
Mar 8, 2024
mtojek
reviewed
Mar 8, 2024
johnstcn
reviewed
Mar 8, 2024
Signed-off-by: Danny Kopping <danny@coder.com>
…/coder into dk/configurable-cardinality
Signed-off-by: Danny Kopping <danny@coder.com>
Signed-off-by: Danny Kopping <danny@coder.com>
881cd6a to
9b16a3b
Compare
…k/configurable-cardinality
mafredri
approved these changes
Mar 11, 2024
mtojek
approved these changes
Mar 11, 2024
Signed-off-by: Danny Kopping <danny@coder.com>
Signed-off-by: Danny Kopping <danny@coder.com>
johnstcn
approved these changes
Mar 11, 2024
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implements #12221
When stats from agents are collected, they are aggregated by 4 dimensions:
agent_name,template_name,workspace_name, andusername. This can result in some very high cardinality metrics being scraped by Prometheus in large environments.This PR adds the ability to tune which labels are include in this aggregation, therefore reducing the cardinality.
For example:
With hundreds of active workspaces, each having a unique name, the cardinality may be unacceptable. In this case the operator may choose to configure
CODER_PROMETHEUS_AGGREGATE_AGENT_STATS_BY=usernameto rather aggregate by user, summing all the metrics' values and only producing a single metric series:Multiple labels can be provided, e.g.
CODER_PROMETHEUS_AGGREGATE_AGENT_STATS_BY=username,agent_nameThe current behaviour remains the default; if no value is passed to
CODER_PROMETHEUS_AGGREGATE_AGENT_STATS_BYthere will be no change.Note to reviewer: I made a sweeping refactor to define the label names in a single location instead of duplicating them, which may make the PR larger than it seems.