Use idiomatic prometheus collector pattern for coderd metrics #12792
Labels
bug risk
Prone to bugs
need-backend
Issues that need backend work
observability
Issues related to observability (metrics, dashboards, alerts, opentelemetry)
Follow-up of #12762
We expose a few metrics which are effectively being "exported" from our database. We should use the idiomatic approach of writing exporters as described in https://prometheus.io/docs/instrumenting/writing_exporters/#collectors. In other words, run the queries in response to scrape requests; currently we run these queries in
coderd/prometheusmetrics/prometheusmetrics.go
every minute and update the metrics.Our current approach leads to stale metrics between scrapes (the underlying data may have changed, but subsequent scrapes do not pull the "correct" data) and we leave ourselves open to a possible data races by scraping in the middle of updating metrics.
See https://github.com/prometheus-community/postgres_exporter/blob/master/collector/pg_locks.go for an example of how we should be approaching this.
The text was updated successfully, but these errors were encountered: