Add kubelet_pod_emptydir_volume_{used,size_limit}_bytes Prometheus metrics #121489

machine424 · 2023-10-24T20:32:36Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

Exposes emptyDir usage and size limit per Pod as Prometheus metrics

Which issue(s) this PR fixes:

Fixes #69507

Special notes for your reviewer:

Do we need to expose thecapacityBytes as well (I don't think it's really informative)? What about cases when emptyDir.sizeLimit is set? DONE => for the capacity, maybe in a follow-up PR if it turns out relevant.
~~Could unit test more cases.~~
Linked the new collector in pkg/kubelet/kubelet.go, not sure if it's sufficient, any doc about that is welcome.

Does this PR introduce a user-facing change?

Add kubelet_pod_emptydir_volume_{used,size_limit}_bytes Prometheus metrics to expose the used and limit bytes for volumes on the default medium, per Pod.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

k8s-ci-robot · 2023-10-24T20:32:46Z

Hi @machine424. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

machine424 · 2023-10-24T20:32:58Z

pkg/kubelet/metrics/collectors/emptydir_metrics.go

+		podNamespace := podStat.PodRef.Namespace
+
+		if podStat.VolumeStats == nil {
+			// TODO: another level?


pkg/kubelet/metrics/collectors/emptydir_metrics.go

machine424 · 2023-10-24T20:33:49Z

/hold

machine424 · 2023-10-24T20:33:59Z

/cc @dgrisonnet

kannon92 · 2023-10-24T21:42:58Z

/ok-to-test
/release-note-edit Add kubelet_pod_emptydir_volume_used_bytes Prometheus metric to expose emptyDir used bytes Per Pod

machine424 · 2023-10-25T20:19:40Z

I fixed the duplicate metrics collector registration attempted, it was an unfortunate copy/past.

For future readers:

printing the conflicting collectors at /vendor/github.com/prometheus/client_golang/prometheus/registry.go is helpful (I don't know why we don't expose them by default)

 func (err AlreadyRegisteredError) Error() string {
-       return "duplicate metrics collector registration attempted"
+       return fmt.Sprintf("duplicate metrics collector registration attempted %+v %+v", err.ExistingCollector, err.NewCollector)
 }

pkg/kubelet/metrics/collectors/emptydir_metrics.go

machine424 · 2023-10-26T11:51:20Z

/retest-required

kannon92 · 2024-08-28T21:07:29Z

/release-note-edit

Add kubelet_pod_emptydir_volume_{used,size_limit_bytes}_bytes Prometheus metrics to expose emptyDir used and limit bytes per Pod

k8s-ci-robot · 2024-08-28T21:07:31Z

@kannon92: /release-note-edit must be used with a release note block.

In response to this:

/release-note-edit

Add kubelet_pod_emptydir_volume_{used,size_limit_bytes}_bytes Prometheus metrics to expose emptyDir used and limit bytes per Pod

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

kannon92

Do we want to distinguish between memory backed volumes and normal empty dir?

kannon92 · 2024-08-28T21:12:12Z

/release-note-edit

Add kubelet_pod_emptydir_volume_{used,size_limit_bytes}_bytes Prometheus metrics to expose emptyDir used and limit bytes per Pod

kannon92 · 2024-08-28T21:13:43Z

btw @machine424 you can always edit your PR in the release-notes section to add a release-note. I had to remember the syntax to do it from the PR..

…e limit per pod/volume. Only volumes on the default medium are considered. Signed-off-by: machine424 <ayoubmrini424@gmail.com>

machine424 · 2024-09-02T17:13:21Z

Do we want to distinguish between memory backed volumes and normal empty dir?

This could be useful, but I’ll stick to volumes on the default medium for now (I just added that explicitly to the metrics help text). Given that some maintainers already reviewed/started reviewing, I prefer not to make any major changes.

Also, given how we’re very cautious about cardinality, I wouldn’t add another dimension now (especially since StorageMediumHugePagesPrefix can be unbound). I’d leave that for a follow-up PR if needed.

mrueg · 2024-10-08T19:40:07Z

Any chance to get this into the next kubernetes release? I think those metrics would be very useful.

dgrisonnet · 2024-10-08T20:18:39Z

I'll bring it up to the next sig meeting

dgrisonnet · 2024-10-10T17:12:49Z

Coming back from the SIG meeting, the group was in favor of this change and approved the cardinality of these metrics. I am assigning @dashpole for instrumentation and kubelet review.

/unassign @logicalhan
/assign @dashpole

dashpole · 2024-11-22T20:16:53Z

pkg/kubelet/metrics/collectors/emptydir_metrics.go

+
+		for _, volumeStat := range podStat.VolumeStats {
+			if volume, found := podVolumes[volumeStat.Name]; found {
+				// Only consider volumes on the default medium.


I see there is already a VolumeStatsUsedBytesKey for Persistent volumes. Are emptyDir volumes the only volumes that are missing metrics today?

Not the only ones, many other volumes don't have metrics.
I suggested in #121489 (comment) to deprecate the the existing per source metrics and have a generic metrics (with the sources as metrics), that will be easier to discover and to extend, but that effort will need more effort from maintainers.

dashpole · 2024-11-22T20:18:28Z

pkg/kubelet/metrics/collectors/emptydir_metrics.go

+		for _, volumeStat := range podStat.VolumeStats {
+			if volume, found := podVolumes[volumeStat.Name]; found {
+				// Only consider volumes on the default medium.
+				if volume.EmptyDir != nil && volume.EmptyDir.Medium == v1.StorageMediumDefault {


Would it be useful for medium to be a label? It seems like memory-backed empty-dir usage and limits would also be useful, since we already have computed that.

#121489 (comment)

machine424 · 2025-01-03T21:50:39Z

Unfortunately, this has taken much longer than expected (not blaming anyone, I get that it’s not a top priority for everyone). It's been harder for me to dive back into the PR, and it feels like we’re spinning our wheels.

I won't be able to keep working on this without compromising on the quality.

If you think this isn't ready yet, I’m okay with closing the PR and letting someone else take over, maybe deprecate the existing metrics and switch to a generic metric, as mentioned in #121489 (comment) is the way to go.

rexagod · 2025-02-06T18:02:48Z

/assign @dgrisonnet

k8s-triage-robot · 2025-05-07T18:41:18Z

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Mark this PR as fresh with /remove-lifecycle stale
Close this PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

frittentheke · 2025-05-08T09:25:11Z

/remove-lifecycle stale

k8s-triage-robot · 2025-08-06T10:23:39Z

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Mark this PR as fresh with /remove-lifecycle stale
Close this PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

frittentheke · 2025-08-06T11:26:22Z

/remove-lifecycle stale

k8s-ci-robot added the needs-priority Indicates a PR lacks a `priority/foo` label and requires one. label Oct 24, 2023

machine424 commented Oct 24, 2023

View reviewed changes

k8s-ci-robot added area/kubelet sig/node Categorizes an issue or PR as relevant to SIG Node. sig/storage Categorizes an issue or PR as relevant to SIG Storage. labels Oct 24, 2023

machine424 commented Oct 24, 2023

View reviewed changes

pkg/kubelet/metrics/collectors/emptydir_metrics.go Outdated Show resolved Hide resolved

k8s-ci-robot removed the do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Oct 24, 2023

k8s-ci-robot requested review from mattcary and wzshiming October 24, 2023 20:33

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 24, 2023

k8s-ci-robot requested a review from dgrisonnet October 24, 2023 20:34

machine424 force-pushed the empty-metrics branch from 4f03012 to d1f6dce Compare October 24, 2023 20:38

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Oct 24, 2023

machine424 force-pushed the empty-metrics branch from d1f6dce to 7ca63be Compare October 25, 2023 20:14

machine424 commented Oct 25, 2023

View reviewed changes

pkg/kubelet/metrics/collectors/emptydir_metrics.go Show resolved Hide resolved

machine424 force-pushed the empty-metrics branch from 7ca63be to 455b460 Compare October 26, 2023 08:37

machine424 force-pushed the empty-metrics branch from 9b588b2 to 98b6a0b Compare August 28, 2024 17:22

machine424 changed the title ~~Add kubelet_pod_emptydir_volume_{used,size_limit_bytes}_bytes Prometheus metrics~~ Add kubelet_pod_emptydir_volume_{used,size_limit}_bytes Prometheus metrics Aug 28, 2024

kannon92 reviewed Aug 28, 2024

View reviewed changes

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Aug 28, 2024

Add Prometheus metrics to expose emptyDir ivolumes used bytes and siz…

7ecc832

…e limit per pod/volume. Only volumes on the default medium are considered. Signed-off-by: machine424 <ayoubmrini424@gmail.com>

machine424 force-pushed the empty-metrics branch from 98b6a0b to 7ecc832 Compare September 2, 2024 08:33

k8s-ci-robot assigned dashpole and unassigned logicalhan Oct 10, 2024

zeritti mentioned this pull request Oct 23, 2024

[prometheus] metrics for ephemeral-storage usage prometheus-community/helm-charts#4929

Closed

dashpole reviewed Nov 22, 2024

View reviewed changes

k8s-ci-robot assigned dgrisonnet Feb 6, 2025

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label May 7, 2025

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label May 8, 2025

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 6, 2025

k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Aug 6, 2025

Add kubelet_pod_emptydir_volume_{used,size_limit}_bytes Prometheus metrics #121489

Are you sure you want to change the base?

Add kubelet_pod_emptydir_volume_{used,size_limit}_bytes Prometheus metrics #121489

Conversation

machine424 commented Oct 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

Uh oh!

k8s-ci-robot commented Oct 24, 2023

Uh oh!

machine424 Oct 24, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

machine424 commented Oct 24, 2023

Uh oh!

machine424 commented Oct 24, 2023

Uh oh!

kannon92 commented Oct 24, 2023

Uh oh!

machine424 commented Oct 25, 2023

Uh oh!

Uh oh!

machine424 commented Oct 26, 2023

Uh oh!

kannon92 commented Aug 28, 2024

Uh oh!

k8s-ci-robot commented Aug 28, 2024

Uh oh!

kannon92 left a comment

Choose a reason for hiding this comment

Uh oh!

kannon92 commented Aug 28, 2024

Uh oh!

kannon92 commented Aug 28, 2024

Uh oh!

machine424 commented Sep 2, 2024

Uh oh!

mrueg commented Oct 8, 2024

Uh oh!

dgrisonnet commented Oct 8, 2024

Uh oh!

dgrisonnet commented Oct 10, 2024

Uh oh!

dashpole Nov 22, 2024

Choose a reason for hiding this comment

Uh oh!

machine424 Jan 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dashpole Nov 22, 2024

Choose a reason for hiding this comment

Uh oh!

kannon92 Nov 22, 2024

Choose a reason for hiding this comment

Uh oh!

machine424 commented Jan 3, 2025

Uh oh!

rexagod commented Feb 6, 2025

Uh oh!

k8s-triage-robot commented May 7, 2025

Uh oh!

frittentheke commented May 8, 2025

Uh oh!

k8s-triage-robot commented Aug 6, 2025

Uh oh!

frittentheke commented Aug 6, 2025

Uh oh!

Uh oh!

machine424 commented Oct 24, 2023 •

edited

Loading

machine424 Jan 3, 2025 •

edited

Loading