Events+weighted borrowing #128974

MikeSpreitzer · 2024-11-26T06:25:47Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

This PR updates the default API Priority and Fairness configuration to have a FlowSchema and a corresponding priority level for events. This is a draft implementation of kubernetes/enhancements#4978.

This is built on #128087.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?

A new flow schema and priority level have been added to the default configuration for API Priority and Fairness. These give requests to create/update/delete Event objects rather low priority.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

[KEP]: https://github.com/kubernetes/enhancements/pull/4978

Signed-off-by: Mike Spreitzer <mspreitz@us.ibm.com>

k8s-ci-robot · 2024-11-26T06:26:08Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: MikeSpreitzer
Once this PR has been reviewed and has the lgtm label, please assign deads2k for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

api/OWNERS
pkg/apis/OWNERS
pkg/generated/openapi/OWNERS
pkg/registry/OWNERS
staging/src/k8s.io/api/OWNERS
staging/src/k8s.io/apiserver/pkg/apis/OWNERS
~~staging/src/k8s.io/apiserver/pkg/util/flowcontrol/OWNERS~~ [MikeSpreitzer]
staging/src/k8s.io/client-go/applyconfigurations/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

MikeSpreitzer · 2024-11-26T06:26:55Z

/cc @linxiulei
/cc @tkashem
/cc @wojtek-t

k8s-triage-robot · 2024-11-26T07:05:11Z

This PR may require API review.

If so, when the changes are ready, complete the pre-review checklist and request an API review.

Status of requested reviews is tracked in the API Review project.

Trying out kubernetes/enhancements#4978 Signed-off-by: Mike Spreitzer <mspreitz@us.ibm.com>

MikeSpreitzer · 2024-11-26T07:45:20Z

/test pull-kubernetes-e2e-gce-100-performance

MikeSpreitzer · 2024-11-27T08:46:00Z

See https://docs.google.com/document/d/1C0sCkHqnWCHWlDuXvtM_2A72kZnIsFWKdRcXf4wT9js for a study of the Prometheus metrics from the pull-kubernetes-e2e-gce-100-performance run https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/128974/pull-kubernetes-e2e-gce-100-performance/1861315292341211136

linxiulei · 2024-11-27T19:27:44Z

staging/src/k8s.io/apiserver/pkg/apis/flowcontrol/bootstrap/default.go

+			Limited: &flowcontrol.LimitedPriorityLevelConfiguration{
+				NominalConcurrencyShares: ptr.To(int32(5)),
+				LendablePercent:          ptr.To(int32(0)),
+				BorrowingLimitPercent:    ptr.To(int32(100)),


Do we still need borrowing limit with weighted borrowing feature?

Yes, I think so. For example, this makes it possible to "jail" some requests to a priority level with a fixed (over time, for a given sever) upper limit.

linxiulei · 2024-12-01T21:22:48Z

staging/src/k8s.io/apiserver/pkg/apis/flowcontrol/bootstrap/default.go

+			Subjects: groups(user.AllAuthenticated), // the nodes group
+			ResourceRules: []flowcontrol.ResourcePolicyRule{
+				resourceRule(
+					[]string{"create", "update", "delete"},


Event client does use Patch requests for EventSeries https://github.com/kubernetes/client-go/blob/master/tools/events/event_broadcaster.go#L240

Good catch. Fixed here and in #129008

fedebongio · 2024-12-03T21:21:40Z

/triage accepted

Signed-off-by: Mike Spreitzer <mspreitz@us.ibm.com>

MikeSpreitzer · 2024-12-20T17:08:31Z

See https://docs.google.com/document/d/1Px2fEiYWG9YPlLpt13ko0L-fNmwmO9AFVnN4YoZ3PVQ for graphs for a trial that @linxiulei did on Dec 1.

MikeSpreitzer · 2025-02-10T19:36:59Z

PRs to compare:

Add weighting to APF concurrency borrowing #128087: weighted borrowing
Events+weighted borrowing #128974: weighted borrowing + segregated events (graphs also included more shares for leader-election)
Events + weighted borrowing + faster seat demand smoothing #129337: weighted borrowing + segregated events + faster seat demand decay (graphs also included more shares for leader-election)
Increase leader-election nominal concurrency shares from 10 to 40 #129646: more shares for leader-election

Also see:

https://docs.google.com/document/d/1kuQ2STE2mB_XFhWNxqkrx5wCh8o3SHZ6BUzWTuBUsps , which compares kube 1.31 normal and re-jiggered nominal concurrency shares
https://docs.google.com/document/d/1ouKtwLIEJSL0qgpUnOouhKH6oMdlB7JpLkEzsQIhdL4 , which looks at re-jiggered nominal concurrency shares plus segregating events

k8s-triage-robot · 2025-05-11T21:16:23Z

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Mark this PR as fresh with /remove-lifecycle stale
Close this PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot · 2025-06-10T22:01:15Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Mark this PR as fresh with /remove-lifecycle rotten
Close this PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot · 2025-07-10T22:43:47Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Reopen this PR with /reopen
Mark this PR as fresh with /remove-lifecycle rotten
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

k8s-ci-robot · 2025-07-10T22:43:55Z

@k8s-triage-robot: Closed this PR.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied

After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied

After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Reopen this PR with /reopen

Mark this PR as fresh with /remove-lifecycle rotten

Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

MikeSpreitzer added 3 commits November 26, 2024 00:47

Add borrowing weight to APF API

5c95187

Signed-off-by: Mike Spreitzer <mspreitz@us.ibm.com>

make update

18a798e

Signed-off-by: Mike Spreitzer <mspreitz@us.ibm.com>

Implement weighted borrowing

6688ab2

Signed-off-by: Mike Spreitzer <mspreitz@us.ibm.com>

k8s-ci-robot requested review from aojea and dims November 26, 2024 06:26

k8s-ci-robot requested review from linxiulei, tkashem and wojtek-t November 26, 2024 06:26

Add flow schema and priority level for events

863c90b

Trying out kubernetes/enhancements#4978 Signed-off-by: Mike Spreitzer <mspreitz@us.ibm.com>

MikeSpreitzer force-pushed the events+weighted-borrowing branch from 371e5ad to 863c90b Compare November 26, 2024 07:44

MikeSpreitzer mentioned this pull request Nov 26, 2024

KEP-4977: Dedicated Priority Level For Event Requests kubernetes/enhancements#4978

Closed

This was referenced Nov 27, 2024

Added missing 'time' for a field manager that server-side-applied same configuration #127939

Open

Events reject+weighted borrowing #129008

Open

linxiulei reviewed Nov 27, 2024

View reviewed changes

linxiulei reviewed Dec 1, 2024

View reviewed changes

k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Dec 3, 2024

Make events FlowSchema match verb==patch too

690c0f1

Signed-off-by: Mike Spreitzer <mspreitz@us.ibm.com>

MikeSpreitzer mentioned this pull request Dec 20, 2024

Events + weighted borrowing + faster seat demand smoothing #129337

Closed

This was referenced Feb 10, 2025

Add weighting to APF concurrency borrowing #128087

Closed

Increase leader-election nominal concurrency shares from 10 to 40 #129646

Closed

MikeSpreitzer mentioned this pull request Mar 12, 2025

Rejigger API Priority and Fairness config #130459

Open

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label May 11, 2025

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jun 10, 2025

k8s-ci-robot closed this Jul 10, 2025

Events+weighted borrowing #128974

Events+weighted borrowing #128974

Uh oh!

Conversation

MikeSpreitzer commented Nov 26, 2024

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

Uh oh!

k8s-ci-robot commented Nov 26, 2024

Uh oh!

MikeSpreitzer commented Nov 26, 2024

Uh oh!

k8s-triage-robot commented Nov 26, 2024

Uh oh!

MikeSpreitzer commented Nov 26, 2024

Uh oh!

MikeSpreitzer commented Nov 27, 2024

Uh oh!

linxiulei Nov 27, 2024

Choose a reason for hiding this comment

Uh oh!

MikeSpreitzer Dec 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

linxiulei Dec 1, 2024

Choose a reason for hiding this comment

Uh oh!

MikeSpreitzer Dec 4, 2024

Choose a reason for hiding this comment

Uh oh!

fedebongio commented Dec 3, 2024

Uh oh!

MikeSpreitzer commented Dec 20, 2024

Uh oh!

MikeSpreitzer commented Feb 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-triage-robot commented May 11, 2025

Uh oh!

k8s-triage-robot commented Jun 10, 2025

Uh oh!

k8s-triage-robot commented Jul 10, 2025

Uh oh!

k8s-ci-robot commented Jul 10, 2025

Uh oh!

Uh oh!

MikeSpreitzer Dec 4, 2024 •

edited

Loading

MikeSpreitzer commented Feb 10, 2025 •

edited

Loading