[WIP] reuse etcd client to monitor storage metric #125771

Yiqing-Liu · 2024-06-27T20:55:13Z

What type of PR is this?

/kind bug

What this PR does / why we need it:

motivation

We have recently noticed an increase in gRPC errors (specifically, use of closed network connection) in apiserver logs.

I0619 00:22:[31.927056 11](tel:3192705611) http2_client.go:959] "[transport] [client-transport 0xc004144900] Closing: connection error: desc = \"error reading from server: read tcp 10.0.33.210:43716→10.0.32.16:2379: use of closed network connection\"\n"
would like to rca for the recurring error message.

suspect

The following is tcpdump screenshot,

noticed there is etcdserverpb.Maintenance/Status call around the grpc error occurrence timeline.

We suspect the status call was introduced by PR: https://github.com/kubernetes/kubernetes/pull/118812/commits
where etcd client is frequently initialized and shutDown.

validation

the error message seems clear with this PR change (reuse the existing etcd cilent)

Which issue(s) this PR fixes:

Fixes #125770

Special notes for your reviewer:

We are looking for more evidence to build the causality between the grpc erros and the suspected PR https://github.com/kubernetes/kubernetes/pull/118812/commits.

Any insight is appreciated.

Does this PR introduce a user-facing change?

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

NONE

k8s-ci-robot · 2024-06-27T20:55:15Z

Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

linux-foundation-easycla · 2024-06-27T20:55:16Z

The committers listed above are authorized under a signed CLA.

✅ login: Yiqing-Liu / name: Yiqing (f6732cd, 9755bc5)

k8s-ci-robot · 2024-06-27T20:55:21Z

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot · 2024-06-27T20:55:21Z

Welcome @Yiqing-Liu!

It looks like this is your first PR to kubernetes/kubernetes 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/kubernetes has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2024-06-27T20:55:22Z

Hi @Yiqing-Liu. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Yiqing-Liu · 2024-06-27T20:58:37Z

:/cc @dims @chaochn47

dims · 2024-06-27T21:05:32Z

/ok-to-test

dims · 2024-06-27T21:05:51Z

cc @serathius

shyamjvs · 2024-07-11T18:46:41Z

staging/src/k8s.io/apiserver/pkg/storage/etcd3/metrics/metrics.go

+	// get storage metric monitor is protected by mutex
+	monitorMutex sync.Mutex


I see a mutex above monitorGetter already whose purpose seems similar. Can we reuse that?

yea, also this testing PR changed a bit of etcd client behavior. (testing is telling us we need figure out how to shut down the "singleton" etcd client)
More code refactoring may be need if we confirmed this is the right fix.

currently we are looking for more evidence to build the causality between the gRPC error and the original PR https://github.com/kubernetes/kubernetes/pull/118812/commits.

k8s-triage-robot · 2024-10-23T18:57:47Z

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Mark this PR as fresh with /remove-lifecycle stale
Close this PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot · 2024-11-22T19:08:07Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Mark this PR as fresh with /remove-lifecycle rotten
Close this PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot · 2024-12-22T19:43:53Z

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Reopen this PR with /reopen
Mark this PR as fresh with /remove-lifecycle rotten
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

k8s-ci-robot · 2024-12-22T19:43:58Z

@k8s-triage-robot: Closed this PR.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied

After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied

After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

Reopen this PR with /reopen

Mark this PR as fresh with /remove-lifecycle rotten

Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

chaochn47 · 2025-08-11T23:27:41Z

/reopen

k8s-ci-robot · 2025-08-11T23:27:47Z

@chaochn47: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Yiqing-Liu · 2025-08-11T23:29:48Z

/reopen

k8s-ci-robot · 2025-08-11T23:29:54Z

@Yiqing-Liu: Reopened this PR.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot · 2025-08-11T23:29:55Z

Please note that we're already in Test Freeze for the release-1.34 branch. This means every merged PR will be automatically fast-forwarded via the periodic ci-fast-forward job to the release branch of the upcoming v1.34.0 release.

Fast forwards are scheduled to happen every 6 hours, whereas the most recent run was: Mon Aug 11 22:23:43 UTC 2025.

k8s-ci-robot · 2025-08-11T23:30:12Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Yiqing-Liu
Once this PR has been reviewed and has the lgtm label, please assign logicalhan for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

staging/src/k8s.io/apiserver/pkg/storage/etcd3/metrics/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2025-08-12T01:50:05Z

@Yiqing-Liu: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-kubernetes-unit	`9755bc5`	link	true	`/test pull-kubernetes-unit`
pull-kubernetes-node-e2e-containerd	`9755bc5`	link	true	`/test pull-kubernetes-node-e2e-containerd`
pull-kubernetes-integration	`9755bc5`	link	true	`/test pull-kubernetes-integration`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

reuse etcd client to monitor storage metric

f6732cd

k8s-ci-robot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jun 27, 2024

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Jun 27, 2024

k8s-ci-robot requested review from serathius and wojtek-t June 27, 2024 20:55

k8s-ci-robot added area/apiserver sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. sig/etcd Categorizes an issue or PR as relevant to SIG Etcd. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Jun 27, 2024

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Jun 27, 2024

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Jun 27, 2024

remove unused func: getGetter()

9755bc5

shyamjvs reviewed Jul 11, 2024

View reviewed changes

drewgonzales360 mentioned this pull request Sep 20, 2024

Fix grpc warnings in API server DataDog/kubernetes#76

Closed

drewgonzales360 mentioned this pull request Oct 2, 2024

drew/1.28.11/grpc warnings DataDog/kubernetes#81

Closed

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 23, 2024

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Nov 22, 2024

k8s-ci-robot closed this Dec 22, 2024

k8s-ci-robot reopened this Aug 11, 2025

		// get storage metric monitor is protected by mutex
		monitorMutex sync.Mutex

[WIP] reuse etcd client to monitor storage metric #125771

Are you sure you want to change the base?

[WIP] reuse etcd client to monitor storage metric #125771

Conversation

Yiqing-Liu commented Jun 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

What this PR does / why we need it:

motivation

suspect

validation

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

Uh oh!

k8s-ci-robot commented Jun 27, 2024

Uh oh!

linux-foundation-easycla bot commented Jun 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Jun 27, 2024

Uh oh!

k8s-ci-robot commented Jun 27, 2024

Uh oh!

k8s-ci-robot commented Jun 27, 2024

Uh oh!

Yiqing-Liu commented Jun 27, 2024

Uh oh!

dims commented Jun 27, 2024

Uh oh!

dims commented Jun 27, 2024

Uh oh!

shyamjvs Jul 11, 2024

Choose a reason for hiding this comment

Uh oh!

Yiqing-Liu Jul 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

k8s-triage-robot commented Oct 23, 2024

Uh oh!

k8s-triage-robot commented Nov 22, 2024

Uh oh!

k8s-triage-robot commented Dec 22, 2024

Uh oh!

k8s-ci-robot commented Dec 22, 2024

Uh oh!

chaochn47 commented Aug 11, 2025

Uh oh!

k8s-ci-robot commented Aug 11, 2025

Uh oh!

Yiqing-Liu commented Aug 11, 2025

Uh oh!

k8s-ci-robot commented Aug 11, 2025

Uh oh!

k8s-ci-robot commented Aug 11, 2025

Uh oh!

k8s-ci-robot commented Aug 11, 2025

Uh oh!

k8s-ci-robot commented Aug 12, 2025

Uh oh!

Uh oh!

Yiqing-Liu commented Jun 27, 2024 •

edited

Loading

linux-foundation-easycla bot commented Jun 27, 2024 •

edited

Loading

Yiqing-Liu Jul 25, 2024 •

edited

Loading