Skip to content

feat: Add docs about workspace metric cardinality #1177

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Nov 21, 2022
Merged

feat: Add docs about workspace metric cardinality #1177

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6b2d327
feat: Add docs about workspace metric cardinality
Emyrk Nov 17, 2022
9307225
Update admin/prometheus.md
Emyrk Nov 18, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 57 additions & 1 deletion admin/prometheus.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,63 @@ spec:
- port: prom-coderd
```

## Metrics
## Workspace Metrics

Each coder workspace has an agent that connects to a single `coderd` instance.
Each coderd instance will include all metrics from the workspaces it manages.
The workspace metrics will all look like this:

```prometheus
coderd_workspace_<workspace_metric_name>{user_id="<user_id>",workspace_id="<workspace_id>"}
```

Due to the nature of workspace ids, this produces a
[high cardinality of metric labels](https://prometheus.io/docs/practices/naming/#labels).
This could be problematic for some configurations. If specific workspace metrics
are not of interest, or are causing issues, you can configure your metric
scraping service to drop these metrics.

Note that if a workspace connects to a new `coderd` (rebuild, network issue,
coder update, etc), the metrics for that workspace will be moved to the new
`coderd` metrics endpoint. The labels on the new metrics will likely have the
new `coderd` pod name. So when tracking a singular workspace, you should track
**only** by `workspace_id` throughout the lifetime of the workspace until it is
deleted.

### Drop workspace metrics config

[Prometheus Documentation](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#metric_relabel_configs)
about relabelling metrics. In this case we will drop all metrics that contain the
`workspace_id` label.

```yaml
metric_relabel_configs:
- source_labels: ["workspace_id"]
action: drop
```

In Prometheus Operator we can pass this config addition to our `coderd`
PodMonitor spec.

```yaml
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
name: master-monitor
namespace: coder
spec:
selector:
matchLabels:
app.kubernetes.io/component: coderd
podMetricsEndpoints:
- port: prom-coderd
relabelings:
- action: drop
sourceLabels:
- workspace_id
```

## Coderd Metrics

Below is a list of the various metrics emitted by Coder's Prometheus endpoint:

Expand Down