Skip to content

feat: add metrics to workspace agent scripts #11132

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 19 commits into from
Dec 13, 2023
Merged

feat: add metrics to workspace agent scripts #11132

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
2f978c3
wip: work on agent script metrics
Emyrk Dec 8, 2023
c567eab
push startup script metrics to agent
Emyrk Dec 11, 2023
5d39495
Add metrics pushing to prometheus
Emyrk Dec 11, 2023
07fe74a
make metric a normal prom metric exported, rather than a first class
Emyrk Dec 11, 2023
753f1a6
linting/fmt/naming
Emyrk Dec 11, 2023
aafd29e
add template_name label to workspace agent stats
Emyrk Dec 11, 2023
6c5a560
Fix prom tests
Emyrk Dec 11, 2023
ad3f47f
Update prom docs
Emyrk Dec 11, 2023
19428fa
seed template id in test
Emyrk Dec 11, 2023
8bfa9ad
fixup! seed template id in test
Emyrk Dec 12, 2023
044b942
PR feedback, _s -> seconds, start -> startup
Emyrk Dec 13, 2023
2db41b6
change string params to a struct
Emyrk Dec 13, 2023
dd32401
Change the name of the labels argument struct
Emyrk Dec 13, 2023
b4a3607
Clarify comment
Emyrk Dec 13, 2023
c5da250
Remove extra words
Emyrk Dec 13, 2023
eb1d173
measure workspace startup closer to script finish in lines of code
Emyrk Dec 13, 2023
e803436
Fix unit test call args
Emyrk Dec 13, 2023
99483fe
Make gen
Emyrk Dec 13, 2023
2f89b62
Formatting
Emyrk Dec 13, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add metrics pushing to prometheus
  • Loading branch information
@Emyrk
Emyrk committed Dec 11, 2023
commit 5d39495eceb7a16c7166b540bb4860c801361ec1
3 changes: 3 additions & 0 deletions coderd/batchstats/batcher.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ import (

"cdr.dev/slog"
"cdr.dev/slog/sloggers/sloghuman"

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: many of these changes are sneaking in, should we adjust our linter?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just added this to my editor config, maybe that will fix it:

coder/.golangci.yaml

Line 126 in 8bfa9ad

local-prefixes: coder.com,cdr.dev,go.coder.com,github.com/cdr,github.com/coder

The issue is that imports are not deterministic for the fmt from my experience. Both this in the PR, and grouping them is valid. There are packages like https://github.com/daixiang0/gci that make import ordering deterministic. Would prevent this, but require us to setup an extra tool.

"github.com/coder/coder/v2/coderd/database"
"github.com/coder/coder/v2/coderd/database/dbauthz"
"github.com/coder/coder/v2/coderd/database/dbtime"
Expand Down Expand Up @@ -161,6 +162,8 @@ func (b *Batcher) Add(
b.buf.SessionCountReconnectingPTY = append(b.buf.SessionCountReconnectingPTY, st.SessionCountReconnectingPTY)
b.buf.SessionCountSSH = append(b.buf.SessionCountSSH, st.SessionCountSSH)
b.buf.ConnectionMedianLatencyMS = append(b.buf.ConnectionMedianLatencyMS, st.ConnectionMedianLatencyMS)
b.buf.StartupScriptNs = append(b.buf.StartupScriptNs, st.StartupScriptNs)
b.buf.StartupScriptSuccess = append(b.buf.StartupScriptSuccess, st.StartupScriptSuccess)

// If the buffer is over 80% full, signal the flusher to flush immediately.
// We want to trigger flushes early to reduce the likelihood of
Expand Down
8 changes: 7 additions & 1 deletion coderd/database/dump.sql
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions coderd/database/migrations/000176_agent_startup_script_stats.down.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
ALTER TABLE workspace_agent_stats DROP COLUMN startup_script_ns;
ALTER TABLE workspace_agent_stats DROP COLUMN startup_script_success;
5 changes: 5 additions & 0 deletions coderd/database/migrations/000176_agent_startup_script_stats.up.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
ALTER TABLE workspace_agent_stats ADD COLUMN startup_script_ns BIGINT NOT NULL DEFAULT 0;
ALTER TABLE workspace_agent_stats ADD COLUMN startup_script_success BOOL NOT NULL DEFAULT false;

COMMENT ON COLUMN workspace_agent_stats.startup_script_ns IS 'The time it took to run the startup script in nanoseconds. If set to 0, the startup script was not run.';
COMMENT ON COLUMN workspace_agent_stats.startup_script_success IS 'Whether the startup script was run successfully. Will be false if the duration is 0, but the script has not been run.';
4 changes: 4 additions & 0 deletions coderd/database/models.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

66 changes: 46 additions & 20 deletions coderd/database/queries.sql.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

18 changes: 13 additions & 5 deletions coderd/database/queries/workspaceagentstats.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,9 @@ INSERT INTO
session_count_jetbrains,
session_count_reconnecting_pty,
session_count_ssh,
connection_median_latency_ms
connection_median_latency_ms,
startup_script_ns,
startup_script_success
)
SELECT
unnest(@id :: uuid[]) AS id,
Expand All @@ -60,7 +62,10 @@ SELECT
unnest(@session_count_jetbrains :: bigint[]) AS session_count_jetbrains,
unnest(@session_count_reconnecting_pty :: bigint[]) AS session_count_reconnecting_pty,
unnest(@session_count_ssh :: bigint[]) AS session_count_ssh,
unnest(@connection_median_latency_ms :: double precision[]) AS connection_median_latency_ms;
unnest(@connection_median_latency_ms :: double precision[]) AS connection_median_latency_ms,
unnest(@startup_script_ns :: bigint[]) AS startup_script_ns,
unnest(@startup_script_success :: bool[]) AS startup_script_success
;

-- name: GetTemplateDAUs :many
SELECT
Expand Down Expand Up @@ -163,21 +168,24 @@ WITH agent_stats AS (
coalesce(SUM(session_count_jetbrains), 0)::bigint AS session_count_jetbrains,
coalesce(SUM(session_count_reconnecting_pty), 0)::bigint AS session_count_reconnecting_pty,
coalesce(SUM(connection_count), 0)::bigint AS connection_count,
coalesce(MAX(connection_median_latency_ms), 0)::float AS connection_median_latency_ms
coalesce(MAX(connection_median_latency_ms), 0)::float AS connection_median_latency_ms,
-- TODO: Figure this out
coalesce(MAX(startup_script_ns), 0)::float AS startup_script_ns,
coalesce(MAX(startup_script_success), false)::float AS startup_script_success
FROM (
SELECT *, ROW_NUMBER() OVER(PARTITION BY agent_id ORDER BY created_at DESC) AS rn
FROM workspace_agent_stats
-- The greater than 0 is to support legacy agents that don't report connection_median_latency_ms.
WHERE created_at > $1 AND connection_median_latency_ms > 0
) AS a
WHERE a.rn = 1
GROUP BY a.user_id, a.agent_id, a.workspace_id, a.template_id
GROUP BY a.user_id, a.agent_id, a.workspace_id
)
SELECT
users.username, workspace_agents.name AS agent_name, workspaces.name AS workspace_name,
workspaces.template_id AS template_id, rx_bytes, tx_bytes,
session_count_vscode, session_count_ssh, session_count_jetbrains, session_count_reconnecting_pty,
connection_count, connection_median_latency_ms
connection_count, connection_median_latency_ms, startup_script_ns, startup_script_success, templates.name AS template_name
FROM
agent_stats
JOIN
Expand Down
Loading