Skip to content

Use idiomatic prometheus collector pattern for coderd metrics #12792

Open
@dannykopping

Description

@dannykopping

Follow-up of #12762

We expose a few metrics which are effectively being "exported" from our database. We should use the idiomatic approach of writing exporters as described in https://prometheus.io/docs/instrumenting/writing_exporters/#collectors. In other words, run the queries in response to scrape requests; currently we run these queries in coderd/prometheusmetrics/prometheusmetrics.go every minute and update the metrics.

Our current approach leads to stale metrics between scrapes (the underlying data may have changed, but subsequent scrapes do not pull the "correct" data) and we leave ourselves open to a possible data races by scraping in the middle of updating metrics.

See https://github.com/prometheus-community/postgres_exporter/blob/master/collector/pg_locks.go for an example of how we should be approaching this.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bug riskProne to bugsneed-backendIssues that need backend workobservabilityIssues related to observability (metrics, dashboards, alerts, opentelemetry)

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions