Skip to content

feat: make agent stats' cardinality configurable #12535

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 22 commits into from
Mar 13, 2024

Conversation

dannykopping
Copy link
Contributor

@dannykopping dannykopping commented Mar 11, 2024

Part 2 of #12468

We initially had to revert the above PR because of a panic from Prometheus.

This was caused by the 4 default labels being passed in to AgentStats(), however these metrics do not contain a value for the template_name label. This was masked in a test by passing in nil for the aggregation labels which defaulted them to the 3 correct ones, but I subsequently changed the default value from nil in the actual code to all 4 labels for various reasons.

I implemented the fix in 3c1d016. I can't think of a cleaner approach than this; if you come up with a better approach, please share!

Note to reviewers: the code is otherwise identical to #12468, so simply reviewing 3c1d016 should suffice.

dannykopping and others added 18 commits March 11, 2024 16:38
Signed-off-by: Danny Kopping <danny@coder.com>
Signed-off-by: Danny Kopping <danny@coder.com>
…ebug works

Signed-off-by: Danny Kopping <danny@coder.com>
Signed-off-by: Danny Kopping <danny@coder.com>
Signed-off-by: Danny Kopping <danny@coder.com>
Signed-off-by: Danny Kopping <danny@coder.com>
Signed-off-by: Danny Kopping <danny@coder.com>
Signed-off-by: Danny Kopping <danny@coder.com>
Signed-off-by: Danny Kopping <danny@coder.com>
Signed-off-by: Danny Kopping <danny@coder.com>
Signed-off-by: Danny Kopping <danny@coder.com>
Signed-off-by: Danny Kopping <danny@coder.com>
Signed-off-by: Danny Kopping <danny@coder.com>
Signed-off-by: Danny Kopping <danny@coder.com>
Signed-off-by: Danny Kopping <danny@coder.com>
Signed-off-by: Danny Kopping <danny@coder.com>
Signed-off-by: Danny Kopping <danny@coder.com>
Signed-off-by: Danny Kopping <danny@coder.com>
@dannykopping dannykopping marked this pull request as ready for review March 12, 2024 07:31
@dannykopping dannykopping requested a review from johnstcn March 12, 2024 07:31
Comment on lines 14 to 18
"github.com/coder/coder/v2/coderd/agentmetrics"

"github.com/coder/coder/v2/coderd/batchstats"
"github.com/coder/coder/v2/coderd/database/dbtestutil"
"github.com/coder/coder/v2/coderd/database/dbtime"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: sorting?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hhmm, I ran make fmt and it didn't correct it so I think we have some things to fix in our CI. Will manually sort though 👍

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

goimports only ensures that imports are grouped, but doesn't handle merging multiple import groups from the same source. I recall that https://github.com/indeedeng/go-groups was built in anger to address this, but it has not been updated in a while.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh nice. I just ran it and it... changes a lot.
I'll raise a separate PR.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is extremely opinionated

Comment on lines 18 to 22
"github.com/coder/coder/v2/coderd/agentmetrics"

"github.com/coreos/go-oidc/v3/oidc"

"github.com/coder/coder/v2/buildinfo"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: here too

func filterInvalidLabels(labels []string) []string {
var out []string

for _, label := range labels {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe optimize it this way?

validLabels := make([]string, 0, len(labels))

for _, label := range labels {
	if label != agentmetrics.LabelTemplateName {
		validLabels = append(validLabels, label)
	}
}

// filterInvalidLabels handles a slightly messy situation whereby `prometheus-aggregate-agent-stats-by` can control on
// which labels agent stats are aggregated, but for these specific metrics in this file there is no `template` label value,
// and therefore we have to exclude it from the list of acceptable labels.
func filterInvalidLabels(labels []string) []string {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add an internal_test.go to verify this function?

Copy link
Member

@johnstcn johnstcn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only a couple of nits from me. Tested locally with a small scaletest.

"golang.org/x/xerrors"

"github.com/coder/coder/v2/coderd/agentmetrics"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: sorting

Signed-off-by: Danny Kopping <danny@coder.com>
Copy link
Member

@mtojek mtojek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@dannykopping dannykopping merged commit 7a7105a into coder:main Mar 13, 2024
@github-actions github-actions bot locked and limited conversation to collaborators Mar 13, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants