Skip to content

feat: make agent stats' cardinality configurable #12535

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 22 commits into from
Mar 13, 2024
Merged

feat: make agent stats' cardinality configurable #12535

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
975caa5
Initial implementation
dannykopping Mar 7, 2024
fcaebb3
Refactoring
dannykopping Mar 8, 2024
70102f8
Drive-by change to add delve to nix flake so ./scripts/develop.sh --d…
dannykopping Mar 8, 2024
e3bfa0d
Update deployment config
dannykopping Mar 8, 2024
92e538a
Linting
dannykopping Mar 8, 2024
fd19896
Control cardinality of coderd metrics as well
dannykopping Mar 8, 2024
a682843
Remove default value
dannykopping Mar 8, 2024
48b68ad
make fmt && make gen
dannykopping Mar 8, 2024
661f5c2
Appeasing CI
dannykopping Mar 8, 2024
7d3ded4
Appeasing CI, again...
dannykopping Mar 8, 2024
defcf74
...
dannykopping Mar 8, 2024
97f2bd1
Review comments & CI appeasement
dannykopping Mar 11, 2024
8f2c505
More review comments & CI appeasement
dannykopping Mar 11, 2024
c0896a7
Fixing lints
dannykopping Mar 11, 2024
c11dc61
chore(dogfood): update keys
johnstcn Mar 11, 2024
cdd6734
Fix flaky label evaluation by sorting both sides
dannykopping Mar 11, 2024
06f2f87
Update node key
dannykopping Mar 11, 2024
3c1d016
Filter out unacceptable labels in AgentStats
dannykopping Mar 11, 2024
f8ebdf2
make fmt
dannykopping Mar 11, 2024
05e5cd0
Merge branch 'dk/configurable-cardinality' of github.com:dannykopping…
dannykopping Mar 12, 2024
b1ef0d6
Merge branch 'main' of github.com:/coder/coder into dk/configurable-c…
dannykopping Mar 12, 2024
a669411
Review comments
dannykopping Mar 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion coderd/prometheusmetrics/aggregator_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -262,7 +262,7 @@ func TestUpdateMetrics_MetricsExpire(t *testing.T) {

// given
registry := prometheus.NewRegistry()
metricsAggregator, err := prometheusmetrics.NewMetricsAggregator(slogtest.Make(t, &slogtest.Options{IgnoreErrors: true}), registry, time.Millisecond, nil)
metricsAggregator, err := prometheusmetrics.NewMetricsAggregator(slogtest.Make(t, &slogtest.Options{IgnoreErrors: true}), registry, time.Millisecond, agentmetrics.LabelAll)
require.NoError(t, err)

ctx, cancelFunc := context.WithCancel(context.Background())
Expand Down
19 changes: 19 additions & 0 deletions coderd/prometheusmetrics/prometheusmetrics.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -338,6 +338,8 @@ func AgentStats(ctx context.Context, logger slog.Logger, registerer prometheus.R
aggregateByLabels = agentmetrics.LabelAgentStats
}

aggregateByLabels = filterInvalidLabels(aggregateByLabels)

metricsCollectorAgentStats := prometheus.NewHistogram(prometheus.HistogramOpts{
Namespace: "coderd",
Subsystem: "prometheusmetrics",
Expand Down Expand Up @@ -514,3 +516,20 @@ func AgentStats(ctx context.Context, logger slog.Logger, registerer prometheus.R
<-done
}, nil
}

// filterInvalidLabels handles a slightly messy situation whereby `prometheus-aggregate-agent-stats-by` can control on
// which labels agent stats are aggregated, but for these specific metrics in this file there is no `template` label value,
// and therefore we have to exclude it from the list of acceptable labels.
func filterInvalidLabels(labels []string) []string {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add an internal_test.go to verify this function?

var out []string

for _, label := range labels {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe optimize it this way?

validLabels := make([]string, 0, len(labels))

for _, label := range labels {
	if label != agentmetrics.LabelTemplateName {
		validLabels = append(validLabels, label)
	}
}

if label == agentmetrics.LabelTemplateName {
continue
}

out = append(out, label)
}

return out
}
4 changes: 3 additions & 1 deletion coderd/prometheusmetrics/prometheusmetrics_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ import (
"testing"
"time"

"github.com/coder/coder/v2/coderd/agentmetrics"

"github.com/coder/coder/v2/coderd/batchstats"
"github.com/coder/coder/v2/coderd/database/dbtestutil"
"github.com/coder/coder/v2/coderd/database/dbtime"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: sorting?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hhmm, I ran make fmt and it didn't correct it so I think we have some things to fix in our CI. Will manually sort though 👍

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

goimports only ensures that imports are grouped, but doesn't handle merging multiple import groups from the same source. I recall that https://github.com/indeedeng/go-groups was built in anger to address this, but it has not been updated in a while.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh nice. I just ran it and it... changes a lot.
I'll raise a separate PR.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is extremely opinionated

Expand Down Expand Up @@ -451,7 +453,7 @@ func TestAgentStats(t *testing.T) {
// and it doesn't depend on the real time.
closeFunc, err := prometheusmetrics.AgentStats(ctx, slogtest.Make(t, &slogtest.Options{
IgnoreErrors: true,
}), registry, db, time.Now().Add(-time.Minute), time.Millisecond, nil)
}), registry, db, time.Now().Add(-time.Minute), time.Millisecond, agentmetrics.LabelAll)
require.NoError(t, err)
t.Cleanup(closeFunc)

Expand Down