PodStartSLIDuration excludes init container runtime and excludes stateful pods #131950

alimaazamat · 2025-05-23T23:27:50Z

What type of PR is this?

/kind bug

What this PR does / why we need it:

PodStartSLIDuration should not include init container run time and only consider stateless pods according to https://github.com/kubernetes/community/blob/master/sig-scalability/slos/pod_startup_latency.md

How this was tested:

kind create cluster --name $CLUSTER_NAME
# rebuilt kubelet for Linux because kind runs linux containers
GOOS=linux GOARCH=amd64 go build -o _output/kubelet ./cmd/kubelet
# deploy custom kubelet to node
docker cp _output/kubelet ${CLUSTER_NAME}-control-plane:/usr/bin/kubelet
docker exec ${CLUSTER_NAME}-control-plane systemctl restart kubelet
kubectl apply -f stateless-stateful-test.yaml
# get metrics
kubectl get --raw="/api/v1/nodes/${CLUSTER_NAME}-control-plane/proxy/metrics" | grep "kubelet_pod_start_sli_duration_seconds"

If you want to test another deployment and reset metrics:

kubectl delete deployment stateless-test stateful-test
#reset metrics by restarting kubelet
docker exec ${CLUSTER_NAME}-control-plane systemctl restart kubelet
kubectl apply -f init-container-test.yaml
# get metrics
kubectl get --raw="/api/v1/nodes/${CLUSTER_NAME}-control-plane/proxy/metrics" | grep "kubelet_pod_start_sli_duration_seconds"

Tested this with 5 stateless and 10 stateful pods with the stateless-stateful-test.yaml deployment like:

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: stateless-test
  labels:
    app: stateless-test
spec:
  replicas: 5
  selector:
    matchLabels:
      app: stateless-test
  template:
    metadata:
      labels:
        app: stateless-test
    spec:
      initContainers:
      - name: init-container
        image: busybox:1.36
        imagePullPolicy: Always
        command: ["sh", "-c", " sleep 10"]
      containers:
      - name: main-container
        image: nginx:1.25-alpine 
        imagePullPolicy: Always
        ports:
        - containerPort: 80
        # No persistent volumes - this is stateless
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: stateful-test
  labels:
    app: stateful-test
spec:
  replicas: 10
  selector:
    matchLabels:
      app: stateful-test
  template:
    metadata:
      labels:
        app: stateful-test
    spec:
      initContainers:
      - name: init-container
        image: alpine:3.19
        imagePullPolicy: Always
        command: ["sh", "-c", "sleep 10"]
      containers:
      - name: main-container
        image: nginx:1.25-alpine
        imagePullPolicy: Always
        ports:
        - containerPort: 80
        volumeMounts:
        - name: persistent-storage
          mountPath: /data
      volumes:
      - name: persistent-storage
        persistentVolumeClaim:
          claimName: test-pvc

and got the correct behavior of kubelet_pod_start_sli_duration_seconds_count 5 So it correctly counts only 5 stateless pods.

Tested this with init container sleep time of 10 seconds with a init-container-test.yaml deployment like:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: init-container-test
spec:
  replicas: 50
  selector:
    matchLabels:
      app: test
  template:
    metadata:
      labels:
        app: test
    spec:
      nodeName: $YOUR_NODE_NAME
      initContainers:
      - name: init
        image: busybox
        command: ["sh", "-c", "sleep 10"]
      containers:
      - name: test
        image: k8s.gcr.io/pause:3.9

and got the correct behavior of

kubelet_pod_start_sli_duration_seconds_bucket{le="0.5"} 0
kubelet_pod_start_sli_duration_seconds_bucket{le="1"} 5
kubelet_pod_start_sli_duration_seconds_bucket{le="2"} 36
kubelet_pod_start_sli_duration_seconds_bucket{le="3"} 47
kubelet_pod_start_sli_duration_seconds_bucket{le="4"} 50
kubelet_pod_start_sli_duration_seconds_bucket{le="5"} 50

Where we are correctly incrementing in buckets less than le="10".

Which issue(s) this PR fixes:

Fixes #131733

Special notes for your reviewer:

Does this PR introduce a user-facing change?

kubelet_pod_start_sli_duration_seconds_bucket metric now matches pod startup latency SLI/SLO documentation.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

k8s-ci-robot · 2025-05-23T23:27:58Z

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot · 2025-05-23T23:27:58Z

Welcome @alimaazamat!

It looks like this is your first PR to kubernetes/kubernetes 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/kubernetes has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2025-05-23T23:27:59Z

Hi @alimaazamat. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot · 2025-05-23T23:28:20Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: alimaazamat
Once this PR has been reviewed and has the lgtm label, please assign random-liu for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

pkg/kubelet/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

pkg/kubelet/util/pod_startup_latency_tracker.go

alimaazamat · 2025-05-27T18:59:23Z

/sig node

alimaazamat · 2025-05-28T22:44:42Z

Waiting for feedback from Sig Node before proceeding...
We expect additional changes to be added (excluding init container runtime, excluding some stateful pods, and excluding schedulable pods)
SLO definition doesn't match the formula being used, but perhaps the definition needs to be updated.

alimaazamat · 2025-07-10T20:34:30Z

/retest

alimaazamat · 2025-07-11T19:26:47Z

/retest

alimaazamat · 2025-07-11T20:21:43Z

/test pull-kubernetes-e2e-kind-ipv6

alimaazamat · 2025-07-11T21:32:53Z

/retest

…teless

alimaazamat · 2025-07-25T17:36:44Z

/retest

jackfrancis · 2025-07-29T17:07:26Z

cc @haircommander, this is the metrics fix/change mentioned in SIG Node today

cc @dgrisonnet, is this something you'd have thoughts on?

dgrisonnet · 2025-07-31T15:29:42Z

PodStartSLIDuration should not include init container run time and only consider stateless pods

I think that this statement makes sense. Excluding all the external factors that could impact pod startup will make the PodStartSLIDuration metric more accurate to define SLOs for kubelet.

pkg/kubelet/util/pod_startup_latency_tracker.go

… true after all possible metrics recorded

alimaazamat · 2025-08-05T17:37:46Z

@dgrisonnet Thank you for your comment, fixed accordingly (also put state.metricRecorded = true at the end after all possible metrics could be recorded)! PTAL!

dgrisonnet · 2025-08-06T14:33:14Z

Perfect, thanks.

This looks good to me from a sig-instrumentation standpoint.

/lgtm

k8s-ci-robot · 2025-08-06T14:33:24Z

LGTM label has been added.

Git tree hash: c51b8f2afb23294dfc6aefdf39399ac90b642dd2

jackfrancis · 2025-08-06T22:55:48Z

/ok-to-test

k8s-ci-robot · 2025-08-07T01:12:46Z

@alimaazamat: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-kubernetes-cmd	`89336bb`	link	true	`/test pull-kubernetes-cmd`
pull-kubernetes-e2e-kind	`89336bb`	link	true	`/test pull-kubernetes-e2e-kind`
pull-kubernetes-conformance-kind-ga-only-parallel	`89336bb`	link	true	`/test pull-kubernetes-conformance-kind-ga-only-parallel`
pull-kubernetes-e2e-gce	`89336bb`	link	true	`/test pull-kubernetes-e2e-gce`
pull-kubernetes-integration	`89336bb`	link	true	`/test pull-kubernetes-integration`
pull-kubernetes-verify	`89336bb`	link	true	`/test pull-kubernetes-verify`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label May 23, 2025

k8s-ci-robot added the needs-priority Indicates a PR lacks a `priority/foo` label and requires one. label May 23, 2025

k8s-ci-robot requested review from feiskyer and kannon92 May 23, 2025 23:28

k8s-ci-robot added area/kubelet sig/node Categorizes an issue or PR as relevant to SIG Node. labels May 23, 2025

github-project-automation bot added this to SIG Node: code and documentation PRs May 23, 2025

k8s-ci-robot removed the do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label May 23, 2025

github-project-automation bot moved this to Triage in SIG Node: code and documentation PRs May 23, 2025

alimaazamat changed the title ~~[WIP]: initContainerDuration SLI created and subtracted from podStartSLOdura…~~ [WIP]: PodStartSLIDuration excludes init container runtime May 23, 2025

alimaazamat force-pushed the subtract-init-container-runtime-from-slo branch 2 times, most recently from b606046 to 84042ad Compare May 23, 2025 23:47

jim-minter reviewed May 26, 2025

View reviewed changes

pkg/kubelet/util/pod_startup_latency_tracker.go Outdated Show resolved Hide resolved

jim-minter reviewed May 26, 2025

View reviewed changes

pkg/kubelet/util/pod_startup_latency_tracker.go Outdated Show resolved Hide resolved

hashim21223445 approved these changes May 29, 2025

View reviewed changes

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels May 30, 2025

alimaazamat force-pushed the subtract-init-container-runtime-from-slo branch 2 times, most recently from e3021d0 to 2cd2f30 Compare July 10, 2025 23:59

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 22, 2025

alimaazamat added 3 commits July 24, 2025 17:11

initContainerDuration calculated and subtracted from podStartSLOduration

e756e58

matched initContainerDuration logic to match imagepulling logic

19cddf7

fix: exclude init container + image pulling time and check pod is sta…

d7accc6

…teless

alimaazamat force-pushed the subtract-init-container-runtime-from-slo branch from 4e25a34 to d7accc6 Compare July 25, 2025 00:20

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 25, 2025

alimaazamat changed the title ~~[WIP]: PodStartSLIDuration excludes init container runtime~~ PodStartSLIDuration excludes init container runtime and excludes stateful pods Jul 25, 2025

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 25, 2025

dgrisonnet reviewed Jul 31, 2025

View reviewed changes

pkg/kubelet/util/pod_startup_latency_tracker.go Outdated Show resolved Hide resolved

fix: PodStartTotalDuration records all pods and metricRecorded set to…

89336bb

… true after all possible metrics recorded

alimaazamat requested review from dgrisonnet and jim-minter August 4, 2025 21:03

k8s-ci-robot assigned dgrisonnet Aug 6, 2025

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 6, 2025

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Aug 6, 2025

PodStartSLIDuration excludes init container runtime and excludes stateful pods #131950

Are you sure you want to change the base?

PodStartSLIDuration excludes init container runtime and excludes stateful pods #131950

Conversation

alimaazamat commented May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

What this PR does / why we need it:

How this was tested:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

Uh oh!

k8s-ci-robot commented May 23, 2025

Uh oh!

k8s-ci-robot commented May 23, 2025

Uh oh!

k8s-ci-robot commented May 23, 2025

Uh oh!

k8s-ci-robot commented May 23, 2025

Uh oh!

Uh oh!

Uh oh!

alimaazamat commented May 27, 2025

Uh oh!

alimaazamat commented May 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alimaazamat commented Jul 10, 2025

Uh oh!

alimaazamat commented Jul 11, 2025

Uh oh!

alimaazamat commented Jul 11, 2025

Uh oh!

alimaazamat commented Jul 11, 2025

Uh oh!

alimaazamat commented Jul 25, 2025

Uh oh!

jackfrancis commented Jul 29, 2025

Uh oh!

dgrisonnet commented Jul 31, 2025

Uh oh!

Uh oh!

alimaazamat commented Aug 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dgrisonnet commented Aug 6, 2025

Uh oh!

k8s-ci-robot commented Aug 6, 2025

Uh oh!

jackfrancis commented Aug 6, 2025

Uh oh!

k8s-ci-robot commented Aug 7, 2025

Uh oh!

Uh oh!

alimaazamat commented May 23, 2025 •

edited

Loading

alimaazamat commented May 28, 2025 •

edited

Loading

alimaazamat commented Aug 5, 2025 •

edited

Loading