Skip to content

Fix: Add garbage collection to handle Approved-Unissued CSRs #133116

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

264nm
Copy link

@264nm 264nm commented Jul 22, 2025

What type of PR is this?

/kind bug

What this PR does / why we need it:

Currently there is no mechanism to account for garbage collection of Approved-But-Unissued kubelet client CSRs which may happen in scenarios like HashiVault managed certificate signing, which over time will lead to excessive etcd space consumption. This implements the suggestion in issue #132947 where a condition is added to the cleaner which will treat certificates in this state, in the same way it treats Pending CSRs so they can get garbage collected every 24 hours.

Which issue(s) this PR is related to:

#132947

Special notes for your reviewer:

  • Easy to reproduce as demonstrated in the investigation notes on the issue.
  • Change is perceived to be minimal and carries low risk of regression.
  • If implemented a seperate documentation update might be required that explains that it treats GC of Approved CSRs which get issued certificates differently, or alternatively this PR could be adjusted to delete them hourly unless there is a specific reason why this would want to be avoided.

Does this PR introduce a user-facing change?

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

- [Docs: Kubelet TLS Bootstrapping] https://kubernetes.io/docs/reference/access-authn-authz/kubelet-tls-bootstrapping/

@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. kind/bug Categorizes issue or PR as related to a bug. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jul 22, 2025
Copy link

linux-foundation-easycla bot commented Jul 22, 2025

CLA Signed

The committers listed above are authorized under a signed CLA.

  • ✅ login: 264nm / name: Christopher J. Ward (036f4bc)

@k8s-ci-robot k8s-ci-robot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jul 22, 2025
@k8s-ci-robot
Copy link
Contributor

Welcome @264nm!

It looks like this is your first PR to kubernetes/kubernetes 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/kubernetes has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

@k8s-ci-robot k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Jul 22, 2025
@k8s-ci-robot
Copy link
Contributor

Hi @264nm. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added sig/apps Categorizes an issue or PR as relevant to SIG Apps. sig/auth Categorizes an issue or PR as relevant to SIG Auth. labels Jul 22, 2025
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. label Jul 22, 2025
@github-project-automation github-project-automation bot moved this to Needs Triage in SIG Apps Jul 22, 2025
@k8s-ci-robot k8s-ci-robot requested review from dims and liggitt July 22, 2025 01:28
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: 264nm
Once this PR has been reviewed and has the lgtm label, please assign liggitt for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@264nm 264nm force-pushed the fix-approved-unissued-csrs branch from 2fd7180 to 036f4bc Compare July 22, 2025 04:54
@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Jul 22, 2025
@264nm
Copy link
Author

264nm commented Jul 22, 2025

Force pushed an amend commit so the CLA emails lined up properly.. Let me know what you guys think.

@enj enj moved this to Needs Triage in SIG Auth Jul 22, 2025
@liggitt
Copy link
Member

liggitt commented Jul 23, 2025

thanks for the PR, it's in the review queue but won't make it before 1.34 code freeze tomorrow, will take a look some time after 1.34 wraps up

@micahhausler
Copy link
Member

/triage accepted

@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jul 28, 2025
@micahhausler micahhausler moved this from Needs Triage to Backlog in SIG Auth Jul 28, 2025
// isApprovedUnissuedPastDeadline checks if the certificate has an Approved status but
// no certificate has been issued, and the approval time has passed the deadline
// that pending requests are maintained for.
func isApprovedUnissuedPastDeadline(logger klog.Logger, csr *capi.CertificateSigningRequest) bool {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a pre-existing issue with some of the other check functions, but let's improve this to hoist the checks that don't change out of the loop, e.g.:

func isApprovedUnissuedPastDeadline(logger klog.Logger, csr *capi.CertificateSigningRequest) bool {
  if isIssued(csr) {
    return false
  }
	for _, c := range csr.Status.Conditions {
		if c.Type == capi.CertificateApproved && isOlderThan(c.LastUpdateTime, pendingExpiration) {
      ...

can make the same change to isIssuedPastDeadline

func isIssuedPastDeadline(logger klog.Logger, csr *capi.CertificateSigningRequest) bool {
	if !isIssued(csr) {
		return false
	}
	for _, c := range csr.Status.Conditions {
		if c.Type == capi.CertificateApproved && isOlderThan(c.LastUpdateTime, approvedExpiration) {

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/bug Categorizes issue or PR as related to a bug. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. release-note-none Denotes a PR that doesn't merit a release note. sig/apps Categorizes an issue or PR as relevant to SIG Apps. sig/auth Categorizes an issue or PR as relevant to SIG Auth. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
Status: Needs Triage
Status: Backlog
Development

Successfully merging this pull request may close these issues.

4 participants