Skip to content

Use idiomatic prometheus collector pattern for coderd metrics #12792

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dannykopping opened this issue Mar 28, 2024 · 0 comments
Open

Use idiomatic prometheus collector pattern for coderd metrics #12792

dannykopping opened this issue Mar 28, 2024 · 0 comments
Labels
bug risk Prone to bugs need-backend Issues that need backend work observability Issues related to observability (metrics, dashboards, alerts, opentelemetry)

Comments

@dannykopping
Copy link
Contributor

Follow-up of #12762

We expose a few metrics which are effectively being "exported" from our database. We should use the idiomatic approach of writing exporters as described in https://prometheus.io/docs/instrumenting/writing_exporters/#collectors. In other words, run the queries in response to scrape requests; currently we run these queries in coderd/prometheusmetrics/prometheusmetrics.go every minute and update the metrics.

Our current approach leads to stale metrics between scrapes (the underlying data may have changed, but subsequent scrapes do not pull the "correct" data) and we leave ourselves open to a possible data races by scraping in the middle of updating metrics.

See https://github.com/prometheus-community/postgres_exporter/blob/master/collector/pg_locks.go for an example of how we should be approaching this.

@dannykopping dannykopping added bug risk Prone to bugs need-backend Issues that need backend work observability Issues related to observability (metrics, dashboards, alerts, opentelemetry) labels Mar 28, 2024
@coder-labeler coder-labeler bot added the chore label Mar 28, 2024
@matifali matifali removed the chore label Oct 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug risk Prone to bugs need-backend Issues that need backend work observability Issues related to observability (metrics, dashboards, alerts, opentelemetry)
Projects
None yet
Development

No branches or pull requests

2 participants