Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kube-up can't create Windows nodes because NODE_BINARY_TAR_URL is empty #130203

Closed
mauriciopoppe opened this issue Feb 16, 2025 · 5 comments
Closed

kube-up can't create Windows nodes because NODE_BINARY_TAR_URL is empty #130203

mauriciopoppe opened this issue Feb 16, 2025 · 5 comments
Labels
kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. sig/node Categorizes an issue or PR as relevant to SIG Node. sig/storage Categorizes an issue or PR as relevant to SIG Storage.

Comments

@mauriciopoppe
Copy link
Member

mauriciopoppe commented Feb 16, 2025 •
edited
Loading

Which jobs are failing?

The out of tree SIG Storage project CSI Proxy relies on kube-up and some env vars to create a GCE Windows node on GCP. The job is triggered on every pull request on CSI Proxy like in kubernetes-csi/csi-proxy#369, unfortunately the job is not triggered periodically to find out the commit in k/k that could explain why.

Which tests are failing?

https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-csi_csi-proxy/369/pull-kubernetes-csi-csi-proxy-integration/1889120572684636160

Since when has it been failing?

From the history of recent runs https://prow.k8s.io/job-history/gs/kubernetes-ci-logs/pr-logs/directory/pull-kubernetes-csi-csi-proxy-integration, approximately after January 7 2025,

Testgrid link

N/A (not a CI job)

Reason for failure (if possible)

The serial logs of the Windows node show:

2025/02/11 01:26:18 GCEMetadataScripts: windows-startup-script-ps1: Exception caught in script:
2025/02/11 01:26:18 GCEMetadataScripts: windows-startup-script-ps1: At C:\k8s-node-setup.psm1:383 char:32
2025/02/11 01:26:18 GCEMetadataScripts: windows-startup-script-ps1: +   $filename = Split-Path -leaf $urls[0]
2025/02/11 01:26:18 GCEMetadataScripts: windows-startup-script-ps1: +                                ~~~~~~~~
2025/02/11 01:26:18 GCEMetadataScripts: windows-startup-script-ps1: Kubernetes Windows node setup failed: Cannot bind argument to parameter 'Path' because it is an empty string.

Which points to

kubernetes/cluster/gce/windows/k8s-node-setup.psm1

Lines 382 to 383 in edc1fd2

$urls = ${kube_env}['NODE_BINARY_TAR_URL'].Split(",")
$filename = Split-Path -leaf $urls[0]

The env var NODE_BINARY_TAR_URL is empty

2025/02/11 01:25:37 GCEMetadataScripts: windows-startup-script-ps1: NODE_BINARY_TAR_URL:

In the last successful run, the variable was not empty:

2025/01/08 04:03:44 GCEMetadataScripts: windows-startup-script-ps1: NODE_BINARY_TAR_URL: https://storage.googleapis.com/kubernetes-staging-1057edcb61-us-central1/e2e-8d17c314d5-e3991-devel/kubernetes-node-windows-amd64.tar.gz

Anything else we need to know?

No response

Relevant SIG(s)

/sig storage
/sig node
@mauriciopoppe mauriciopoppe added the kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. label Feb 16, 2025
@k8s-ci-robot k8s-ci-robot added sig/storage Categorizes an issue or PR as relevant to SIG Storage. sig/node Categorizes an issue or PR as relevant to SIG Node. labels Feb 16, 2025
@k8s-ci-robot k8s-ci-robot added the needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. label Feb 16, 2025
@k8s-ci-robot
Copy link
Contributor

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@mauriciopoppe mauriciopoppe mentioned this issue Feb 16, 2025
Open
@mauriciopoppe
Copy link
Member Author

mauriciopoppe commented Feb 16, 2025 •
edited
Loading

Taking a deeper look into the kubetest up logs:

Good run

2025/01/08 03:55:18 process.go:153: Running: /workspace/get-kube.sh
Downloading kubernetes release v1.33.0-alpha.0.356+14dafc30c99299
  from https://storage.googleapis.com/k8s-release-dev/ci/v1.33.0-alpha.0.356+14dafc30c99299/kubernetes.tar.gz
  to /home/prow/go/src/github.com/kubernetes-csi/csi-proxy/kubernetes.tar.gz
Copying gs://k8s-release-dev/ci/v1.33.0-alpha.0.356+14dafc30c99299/kubernetes.tar.gz...
/ [0 files][    0.0 B/512.0 KiB]                                                
/ [1 files][512.0 KiB/512.0 KiB]                                                
Operation completed over 1 objects/512.0 KiB.                                    
Unpacking kubernetes release v1.33.0-alpha.0.356+14dafc30c99299
Kubernetes release: v1.33.0-alpha.0.356+14dafc30c99299
Server: linux/amd64  (to override, set KUBERNETES_SERVER_ARCH)
Client: linux/amd64  (autodetected)  (to override, set KUBERNETES_CLIENT_OS and/or KUBERNETES_CLIENT_ARCH)

Will download kubernetes-server-linux-amd64.tar.gz from https://storage.googleapis.com/k8s-release-dev/ci/v1.33.0-alpha.0.356+14dafc30c99299
Will download and extract kubernetes-client-linux-amd64.tar.gz from https://storage.googleapis.com/k8s-release-dev/ci/v1.33.0-alpha.0.356+14dafc30c99299
Will download and extract kubernetes-node-windows-amd64.tar.gz from https://storage.googleapis.com/k8s-release-dev/ci/v1.33.0-alpha.0.356+14dafc30c99299
Will download and extract kubernetes-test tarball(s) from https://storage.googleapis.com/k8s-release-dev/ci/v1.33.0-alpha.0.356+14dafc30c99299

Bad run

2025/02/11 01:13:49 process.go:153: Running: /workspace/get-kube.sh
Downloading kubernetes release v1.33.0-alpha.1.100+80bf507facbd11
  from https://storage.googleapis.com/k8s-release-dev/ci/v1.33.0-alpha.1.100+80bf507facbd11/kubernetes.tar.gz
  to /home/prow/go/src/github.com/kubernetes-csi/csi-proxy/kubernetes.tar.gz
Copying gs://k8s-release-dev/ci/v1.33.0-alpha.1.100+80bf507facbd11/kubernetes.tar.gz...
/ [0 files][    0.0 B/510.0 KiB]                                                
/ [1 files][510.0 KiB/510.0 KiB]                                                
Operation completed over 1 objects/510.0 KiB.                                    
Unpacking kubernetes release v1.33.0-alpha.1.100+80bf507facbd11
Kubernetes release: v1.33.0-alpha.1.100+80bf507facbd11
Server: linux/amd64  (to override, set KUBERNETES_SERVER_ARCH)
Client: linux/amd64  (autodetected)  (to override, set KUBERNETES_CLIENT_OS and/or KUBERNETES_CLIENT_ARCH)

Will download kubernetes-server-linux-amd64.tar.gz from https://storage.googleapis.com/k8s-release-dev/ci/v1.33.0-alpha.1.100+80bf507facbd11
Will download and extract kubernetes-client-linux-amd64.tar.gz from https://storage.googleapis.com/k8s-release-dev/ci/v1.33.0-alpha.1.100+80bf507facbd11
Will download and extract kubernetes-test tarball(s) from https://storage.googleapis.com/k8s-release-dev/ci/v1.33.0-alpha.1.100+80bf507facbd11

I don't see this line printed anymore:

Will download and extract kubernetes-node-windows-amd64.tar.gz from https://storage.googleapis.com/k8s-release-dev/ci/v1.33.0-alpha.0.356+14dafc30c99299

Maybe the binary build process changed?

@mauriciopoppe
Copy link
Member Author

In one of the latest kubernetes CI builds at 2025-02-16 14:49:41 +0000 UTC I see the windows/amd64 target built.

@mauriciopoppe
Copy link
Member Author

I found some env vars in the good run that aren't in the bad run. KUBERNETES_NODE_PLATFORM=windows isn't in the bad run.

This env var was set through the Prow preset preset-common-gce-windows that was removed in kubernetes/test-infra#34103.

Next: partially undo the PR keeping the presets alive but not setting any new CI jobs.

@mauriciopoppe mauriciopoppe mentioned this issue Feb 16, 2025
Merged
@mauriciopoppe
Copy link
Member Author

Fixed with kubernetes/test-infra#34338 and kubernetes/test-infra#34370.

@github-project-automation github-project-automation bot moved this from Triage to Done in SIG Node CI/Test Board Feb 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. sig/node Categorizes an issue or PR as relevant to SIG Node. sig/storage Categorizes an issue or PR as relevant to SIG Storage.
Projects
SIG Node CI/Test Board
Status: Done
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mauriciopoppe @k8s-ci-robot