-
Notifications
You must be signed in to change notification settings - Fork 41.1k
Optimize RS Controller Performance: Reduce Work Duration Time & Minimize Cache Locking #130961
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
/assign @soltysh |
cc: @dims |
/assign @wojtek-t |
@hakuna-matatah the linter CI job is a easy fix, do you want to do that then we can ping folks for review? |
/triage accepted |
Sorry, this slipped through the cracks. @hakuna-matatah can you rebase the PR, fix that linter error (adding |
…ize Cache Locking
@soltysh Sorry for the delayed response, I have been on vacation. Please check now if you get a chance. |
/retest |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/lgtm
/approve
LGTM label has been added. Git tree hash: a9606cc0a2acc8b2239c5fd9255724556c746b4e
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: hakuna-matatah, soltysh The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
What type of PR is this?
/kind cleanup
What this PR does / why we need it:
This PR introduces a Daemonset indexer for the PodInformer to efficiently query Pods belonging to Daemonsets or Orphans from the InformerCache, avoiding a full namespace scan.
This improves performance and correctness at scale by minimizing the time a read lock is held on the cache, reducing blockage of CacheController delta processing. It also helps lower workqueue processing time per object/key.
I did some benchmarking against Indexing (vs) purely listing the pods from Store/Cache
For 10k pods in store, using Index it took ~30 µs (vs) ~320 µs
For 500k pods in store, using Index it took ~35 µs (vs) 18.080762ms
We can tell that, listing from store using Index is 500X times better at higher scale.
Following are the results :
Optimizing this list call will provide the following benefits:
Made similar improvement for DS controller and Statefulset controller
Which issue(s) this PR fixes:
Fixes # #130767
Special notes for your reviewer:
Does this PR introduce a user-facing change?
Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.: