Context

We want to place a tight SLO around coder workspace build times, so we can detect regression. However, buffy GPU instances often take a much longer time to start/stop than general instance, which frequently triggered our SLO alerts, even though it's only because of a few (expected) slow GPU builds. This is caused by the metrics we are using coderd_provisionerd_job_timing_seconds not having a dimension for template name (as we have a separate template for GPU and another for general instances).

Looking closer at the code, this metrics is also not the correct one to use either, because a Job can actually be many different things, not just a workspace build.

Intent

This PR introduces a new prometheus metrics for workspace_build_timing_seconds, which specifically reports workspace build times. To reduce cardinality, this metrics excludes workspace_name and workspace_owner that are present on the workspace_builds_total metrics.

This'd allow us to have different (and tight) SLOs for each of our template (GPU vs non-GPU) by filtering on the template_name (optionally template_version tag) as well as the workspace transition (as we noticed stop is often slower than start, but users don't care a lot about stop transitions).