From 6b2d327ba70ab90af5c9dfaea63b571e507426cb Mon Sep 17 00:00:00 2001
From: Steven Masley <stevenmasley@coder.com>
Date: Thu, 17 Nov 2022 14:51:51 -0600
Subject: [PATCH 1/2] feat: Add docs about workspace metric cardinality

Future work can be to add the workspace metrics like coderd
metrics
---
 admin/prometheus.md | 58 ++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 57 insertions(+), 1 deletion(-)

diff --git a/admin/prometheus.md b/admin/prometheus.md
index 48a4d89b4..7a2b4d33a 100644
--- a/admin/prometheus.md
+++ b/admin/prometheus.md
@@ -32,7 +32,63 @@ spec:
     - port: prom-coderd
 ```
 
-## Metrics
+## Workspace Metrics
+
+Each coder workspace has an agent that connects to a single `coderd` instance.
+Each coderd instance will include all metrics from the workspaces it manages.
+The workspace metrics will all look like this:
+
+```prometheus
+coderd_workspace_<workspace_metric_name>{user_id="<user_id>",workspace_id="<workspace_id>"}
+```
+
+Due to the nature of workspace ids, this produces a
+[high cardinality of metric labels](https://prometheus.io/docs/practices/naming/#labels).
+This could be problematic for some configurations. If specific workspace metrics
+are not of interest, or are causing issues, you can configure your metric
+scraping service to drop these metrics.
+
+Note that if a workspace connects to a new `coderd` (rebuild, network issue,
+coder update, etc), the metrics for that workspace will be moved to the new
+`coderd` metrics endpoint. The labels on the new metrics will likely have the
+new `coderd` pod name. So when tracking a singular workspace, you should track
+**only** by `workspace_id` throughout the lifetime of the workspace until it is
+deleted.
+
+### Drop workspace metrics config
+
+[Prometheus Documentation](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#metric_relabel_configs)
+about relabling metrics. In this case we will drop all metrics that contain the
+`workspace_id` label.
+
+```yaml
+metric_relabel_configs:
+  - source_labels: ["workspace_id"]
+    action: drop
+```
+
+In Prometheus Operator we can pass this config addition to our `coderd`
+PodMonitor spec.
+
+```yaml
+apiVersion: monitoring.coreos.com/v1
+kind: PodMonitor
+metadata:
+  name: master-monitor
+  namespace: coder
+spec:
+  selector:
+    matchLabels:
+      app.kubernetes.io/component: coderd
+  podMetricsEndpoints:
+    - port: prom-coderd
+      relabelings:
+        - action: drop
+          sourceLabels:
+            - workspace_id
+```
+
+## Coderd Metrics
 
 Below is a list of the various metrics emitted by Coder's Prometheus endpoint:
 

From 93072250e7610fafb44ce1832559b3f6d5dbc6ce Mon Sep 17 00:00:00 2001
From: Steven Masley <Emyrk@users.noreply.github.com>
Date: Fri, 18 Nov 2022 11:06:41 -0600
Subject: [PATCH 2/2] Update admin/prometheus.md

Co-authored-by: Cian Johnston <cian@coder.com>
---
 admin/prometheus.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/admin/prometheus.md b/admin/prometheus.md
index 7a2b4d33a..ef8761835 100644
--- a/admin/prometheus.md
+++ b/admin/prometheus.md
@@ -58,7 +58,7 @@ deleted.
 ### Drop workspace metrics config
 
 [Prometheus Documentation](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#metric_relabel_configs)
-about relabling metrics. In this case we will drop all metrics that contain the
+about relabelling metrics. In this case we will drop all metrics that contain the
 `workspace_id` label.
 
 ```yaml