feat: add hard-limited presets metric #18008
Conversation
32f43de to
39748bd
Compare
39748bd to
6fb6800
Compare
f6ea98b to
2667684
Compare
8db12a2 to
3100e01
Compare
|
|
||
| key := hardLimitedPresetKey{orgName: orgName, templateName: templateName, presetName: presetName} | ||
|
|
||
| mc.isPresetHardLimited[key] = isHardLimited |
There was a problem hiding this comment.
nit: With this approach, we’re accumulating entries that are no longer relevant over time. This can cause the metrics to keep reporting 0 for presets that should disappear (e.g., deleted or outdated presets).
A small improvement would be to remove presets from the map when isHardLimited == false:
if isHardLimited {
mc.isPresetHardLimited[key] = true
} else {
delete(mc.isPresetHardLimited, key)
}
This keeps the metric data cleaner and avoids unnecessary entries in Prometheus.
But there is a catch 🤔 With this change, there’s a brief window where Prometheus may still show the previous value (1) for a preset after it’s been removed from the map. This is expected behavior in Prometheus, metrics are not deleted immediately, but expire after some time without being reported.
There was a problem hiding this comment.
Good catch, fixed here: 80c89fe.
I was under a false assumption that it will report stale value indefinitely.
There was a problem hiding this comment.
Overall looks good to me, just a note on the metrics setting: removing entries from the map when isHardLimited keeps metrics clean and reduce cardinality. The trade-off is that Prometheus will show stale data briefly after removal, since metrics aren’t deleted immediately but expire after some time without being reported. This behavior is expected and I think acceptable in this context.
Co-authored-by: Susana Ferreira <ssncferreira@gmail.com>
Closes #17988
Define
preset_hard_limitedmetric which for every preset indicates whether a given preset has reached the hard failure limit (1 for hard-limited, 0 otherwise).CLI example:
NOTE:
Only active template version is tracked. If admin creates new template version - old value of metric (for previous template version) will be overwritten with new value of metric (for active template version).
Because
template_versionis not part of metric:Implementation is similar to implementation of
MetricResourceReplacementsCountmetric