feat: fetch prebuilds metrics state in background #17792
Conversation
Signed-off-by: Danny Kopping <dannykopping@gmail.com>
5da546e to
e73dae6
Compare
Signed-off-by: Danny Kopping <dannykopping@gmail.com>
e73dae6 to
fcbfb7f
Compare
| @@ -55,20 +57,34 @@ var ( | |||
| labels, | |||
| nil, | |||
| ) | |||
| lastUpdateDesc = prometheus.NewDesc( | |||
| "coderd_prebuilt_workspaces_metrics_last_updated", | |||
| "The unix timestamp when the metrics related to prebuilt workspaces were last updated; these metrics are cached.", | |||
There was a problem hiding this comment.
Is unix timestamp easy to alert on? Like can you do something like unix_now() - metric_value > 1000 or something in grafana and co? If not, it might be better if this was a duration since the last successful fetch instead.
There was a problem hiding this comment.
+1 from me for duration since last successful fetch
There was a problem hiding this comment.
The idiomatic approach is to use unix timestamps, see prometheus_config_last_reload_success_timestamp_seconds.
There was a problem hiding this comment.
So I guess we have an existing metric for the coder server start timestamp?
There was a problem hiding this comment.
I don't think so (or at least not one we export), but I think as long as this metric is updated relative to itself and up is taken into consideration, it should be useful.
Signed-off-by: Danny Kopping <dannykopping@gmail.com>
Collect()is called whenever the/metricsendpoint is hit to retrieve metrics.The queries used in prebuilds metrics collection are quite heavy, and we want to avoid having them running concurrently / too often to keep db load down.
Here I'm moving towards a background retrieval of the state required to set the metrics, which gets invalidated every interval.
Also introduces
coderd_prebuilt_workspaces_metrics_last_updatedwhich operators can use to determine when these metrics go stale.See #17789 as well.