Skip to content

use pull if not present for eviction tests #133359

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

use pull if not present for eviction tests #133359

wants to merge 1 commit into from

Conversation

kannon92
Copy link
Contributor

@kannon92 kannon92 commented Aug 1, 2025

What type of PR is this?

/kind cleanup

What this PR does / why we need it:

Follow up from @SergeyKanzhelev below.

ref: #133349 (comment)

Which issue(s) this PR is related to:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

@k8s-ci-robot k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Aug 1, 2025
@k8s-ci-robot
Copy link
Contributor

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@k8s-ci-robot k8s-ci-robot added needs-priority Indicates a PR lacks a `priority/foo` label and requires one. area/test sig/node Categorizes an issue or PR as relevant to SIG Node. sig/testing Categorizes an issue or PR as relevant to SIG Testing. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Aug 1, 2025
@kannon92 kannon92 force-pushed the eviction-always-main branch from fbaab50 to ef0fcd3 Compare August 1, 2025 20:57
@k8s-ci-robot k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 1, 2025
@kannon92 kannon92 force-pushed the eviction-always-main branch from ef0fcd3 to 48f514b Compare August 1, 2025 20:58
@kannon92
Copy link
Contributor Author

kannon92 commented Aug 1, 2025

/milestone v1.35

/hold

to confirm tests.

cc @SergeyKanzhelev

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 1, 2025
@k8s-ci-robot k8s-ci-robot added this to the v1.35 milestone Aug 1, 2025
@kannon92 kannon92 mentioned this pull request Aug 1, 2025
Merged
SergeyKanzhelev
SergeyKanzhelev approved these changes Aug 1, 2025
View reviewed changes
Copy link
Member

@SergeyKanzhelev SergeyKanzhelev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/approve

thank you for the follow up, appreciate the improvement. I am tagging with approve, I do not expect issues with tests

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kannon92, SergeyKanzhelev

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 1, 2025
HirazawaUi
HirazawaUi reviewed Aug 2, 2025
View reviewed changes
Copy link
Contributor

@HirazawaUi HirazawaUi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this isn't a valid change, when the ImagePullPolicy value isn't PullAlways, it always gets reset to PullNever by the e2e framework.
ref:

kubernetes/test/e2e/framework/pod/pod_client.go

Lines 272 to 287 in 8e6d788

for i := range pod.Spec.Containers {
c := &pod.Spec.Containers[i]
if c.ImagePullPolicy == v1.PullAlways {
// If the image pull policy is PullAlways, the image doesn't need to be in
// the allow list or pre-pulled, because the image is expected to be pulled
// in the test anyway.
continue
}
// If the image policy is not PullAlways, the image must be in the pre-pull list and
// pre-pulled.
gomega.Expect(ImagePrePullList.Has(c.Image)).To(gomega.BeTrueBecause("Image %q is not in the pre-pull list, consider adding it to PrePulledImages in test/e2e/common/util.go or NodePrePullImageList in test/e2e_node/image_list.go", c.Image))
// Do not pull images during the tests because the images in pre-pull list should have
// been prepulled.
c.ImagePullPolicy = v1.PullNever
}
}

@HirazawaUi
Copy link
Contributor

On the contrary, we should modify the ImagePullPolicy to PullAlways for all podSpecs in this file. but @ajaysundark has already opened a PR to implement this:
#131273

@BenTheElder
Copy link
Member

I think this isn't a valid change, when the ImagePullPolicy value isn't PullAlways, it always gets reset to PullNever by the e2e framework.
ref:

Huhhhhh, I think that's a misguided feature.

We should attempt to pre-pull, but we should not clear the image to Never. Images can of course be evicted.

IMHO we should stop clearing this setting.

@BenTheElder BenTheElder mentioned this pull request Aug 4, 2025
Open
@BenTheElder
Copy link
Member

cc @SergeyKanzhelev @tallclair (would've cced @kannon92...); @pohly @aojea

clobbering the image pull policy like this seems like a bug IMHO

@SergeyKanzhelev
Copy link
Member

cc @SergeyKanzhelev @tallclair (would've cced @kannon92...); @pohly @aojea

clobbering the image pull policy like this seems like a bug IMHO

I agree. Is there a bug tracking improving this? @ajaysundark @HirazawaUi do you know?

@kannon92
Copy link
Contributor Author

kannon92 commented Aug 5, 2025

Thank you for calling this out @HirazawaUi!

@ajaysundark
Copy link
Contributor

ajaysundark commented Aug 5, 2025 •
edited
Loading

we should modify the ImagePullPolicy to PullAlways for all podSpecs in this file

This didn't help all the test cases, as we also have a test ImageGCNoEviction that relies on 'prepulling of images' to induce the disk-pressure.
I tried to hack it to selectively prepull but there were other problems and didn't get time to look into it.

I think we could ImagePull Always for all Eviction tests, and rewrite this test to manage disk-pressure explicitly without assumptions.

clobbering the image pull policy like this seems like a bug IMHO

@BenTheElder I agree with this, but I'd also want to be cautious to not handle this improvement as part of release efforts.

cc @SergeyKanzhelev @tallclair (would've cced @kannon92...); @pohly @aojea
clobbering the image pull policy like this seems like a bug IMHO

I agree. Is there a bug tracking improving this? @ajaysundark @HirazawaUi do you know?

#131142 was for tracking the containerd eviction failures.

@BenTheElder
Copy link
Member

@BenTheElder I agree with this, but I'd also want to be cautious to not handle this improvement as part of release efforts.

I'm skeptical of this breaking anything that isn't already flaky / unreliable, it's fundamentally broken to assume that the image once pulled will remain available.

@natasha41575 natasha41575 moved this from Triage to PRs Waiting on Author in SIG Node CI/Test Board Aug 6, 2025
@bart0sh bart0sh moved this from Triage to Needs Reviewer in SIG Node: code and documentation PRs Aug 8, 2025
@chenggu88 chenggu88 mentioned this pull request Aug 11, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@HirazawaUi HirazawaUi HirazawaUi left review comments

@SergeyKanzhelev SergeyKanzhelev SergeyKanzhelev approved these changes

@bobbypage bobbypage Awaiting requested review from bobbypage

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/test cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. release-note-none Denotes a PR that doesn't merit a release note. sig/node Categorizes an issue or PR as relevant to SIG Node. sig/testing Categorizes an issue or PR as relevant to SIG Testing. size/S Denotes a PR that changes 10-29 lines, ignoring generated files.
Projects
SIG Node CI/Test Board
Status: PRs Waiting on Author
SIG Node: code and documentation PRs
Status: Needs Reviewer
Milestone
v1.35
Development

Successfully merging this pull request may close these issues.
6 participants
@kannon92 @k8s-ci-robot @HirazawaUi @BenTheElder @SergeyKanzhelev @ajaysundark