Skip to content

CI Update codecov uploader to 0.7.1 #28361

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Feb 5, 2024
Merged

CI Update codecov uploader to 0.7.1 #28361

merged 4 commits into from
Feb 5, 2024

Conversation

lesteve
Copy link
Member

@lesteve lesteve commented Feb 5, 2024

This may get rid of some codecov upload issues that we see from time to time.

For example retry is done in more cases here codecov/uploader#1203, which is released in 0.6.3 and we use 0.4.1.

The kind of errors that were seen in #28348:

Codecov report uploader 0.4.1
[2024-02-05T08:23:29.118Z] ['info'] => Project root located at: /home/vsts/work/1/s
[2024-02-05T08:23:29.122Z] ['info'] ->  Token found by arguments
[2024-02-05T08:23:29.136Z] ['info'] Searching for coverage files...
[2024-02-05T08:23:29.305Z] ['info'] => Found 1 possible coverage files:
  coverage.xml
[2024-02-05T08:23:29.306Z] ['info'] Processing /home/vsts/work/1/s/coverage.xml...
[2024-02-05T08:23:29.386Z] ['info'] Detected Azure Pipelines as the CI provider.
[2024-02-05T08:23:29.396Z] ['info']     Fixing merge commit SHA a1e9563299b3d02eaccd4d1231c826beeb1b064d -> 92f7a58fb1fc8d5c889d61c550f9b783a0c71c9a
[2024-02-05T08:23:29.399Z] ['info'] Pinging Codecov: https://codecov.io/upload/v4?package=uploader-0.4.1&token=*******&branch=refs%2Fpull%2F28348%2Fmerge&build=20240205.7&build_url=https%3A%2F%2Fdev.azure.com%2Fscikit-learn%2Fscikit-learn%2F_build%2Fresults%3FbuildId%3D63528&commit=92f7a58fb1fc8d5c889d61c550f9b783a0c71c9a&job=63528&pr=28348&project=scikit-learn&server_uri=https%3A%2F%2Fdev.azure.com%2Fscikit-learn%2F&service=azure_pipelines&slug=scikit-learn%2Fscikit-learn&name=&tag=&flags=&parent=
[2024-02-05T08:23:59.581Z] ['error'] There was an error running the uploader: Error uploading to https://codecov.io/: Error: There was an error fetching the storage URL during POST: 502 - 
<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>502 Server Error</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Server Error</h1>
<h2>The server encountered a temporary error and could not complete your request.<p>Please try again in 30 seconds.</h2>
<h2></h2>
</body></html>

Codecov report uploader 0.4.1
[2024-02-05T08:30:09.558Z] ['info'] => Project root located at: /home/vsts/work/1/s
[2024-02-05T08:30:09.561Z] ['info'] ->  Token found by arguments
[2024-02-05T08:30:09.574Z] ['info'] Searching for coverage files...
[2024-02-05T08:30:09.686Z] ['info'] => Found 1 possible coverage files:
  coverage.xml
[2024-02-05T08:30:09.687Z] ['info'] Processing /home/vsts/work/1/s/coverage.xml...
[2024-02-05T08:30:09.761Z] ['info'] Detected Azure Pipelines as the CI provider.
[2024-02-05T08:30:09.770Z] ['info']     Fixing merge commit SHA a1e9563299b3d02eaccd4d1231c826beeb1b064d -> 92f7a58fb1fc8d5c889d61c550f9b783a0c71c9a
[2024-02-05T08:30:09.772Z] ['info'] Pinging Codecov: https://codecov.io/upload/v4?package=uploader-0.4.1&token=*******&branch=refs%2Fpull%2F28348%2Fmerge&build=20240205.7&build_url=https%3A%2F%2Fdev.azure.com%2Fscikit-learn%2Fscikit-learn%2F_build%2Fresults%3FbuildId%3D63528&commit=92f7a58fb1fc8d5c889d61c550f9b783a0c71c9a&job=63528&pr=28348&project=scikit-learn&server_uri=https%3A%2F%2Fdev.azure.com%2Fscikit-learn%2F&service=azure_pipelines&slug=scikit-learn%2Fscikit-learn&name=&tag=&flags=&parent=
[2024-02-05T08:30:12.934Z] ['error'] There was an error running the uploader: Error uploading to https://codecov.io/: Error: There was an error fetching the storage URL during POST: 503 - upstream connect error or disconnect/reset before headers. reset reason: connection failure

@lesteve lesteve changed the title CI Update codecov uploader to latest version CI Update codecov uploader to 0.7.1 Feb 5, 2024
@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 5, 2024 •
edited
Loading

✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

Generated for commit: 1f69510. Link to the linter CI: here

@ogrisel
Copy link
Member

ogrisel commented Feb 5, 2024

Unfortunately not all codecov step pass, even with this update.

For future reference, here is the error message (here for the pylatest_conda_forge_mkl):

Codecov report uploader 0.7.1
[2024-02-05T09:26:14.239Z] ['info'] => Project root located at: /home/vsts/work/1/s
[2024-02-05T09:26:14.244Z] ['info'] ->  Token found by arguments
[2024-02-05T09:26:14.267Z] ['info'] Searching for coverage files...
[2024-02-05T09:26:14.417Z] ['info'] => Found 1 possible coverage files:
  coverage.xml
[2024-02-05T09:26:14.417Z] ['info'] Processing /home/vsts/work/1/s/coverage.xml...
[2024-02-05T09:26:14.510Z] ['info'] Detected Azure Pipelines as the CI provider.
[2024-02-05T09:26:14.519Z] ['info']     Fixing merge commit SHA 135b5e81d263e9511507dd920f1a90e0c39d67eb -> 780e769f06e4a346a6b3a1b0a94ff97372f8dcc6
[2024-02-05T09:26:14.522Z] ['info'] Pinging Codecov: https://codecov.io/upload/v4?package=uploader-0.7.1&token=*******&branch=refs%2Fpull%2F28361%2Fmerge&build=20240205.13&build_url=https%3A%2F%2Fdev.azure.com%2Fscikit-learn%2Fscikit-learn%2F_build%2Fresults%3FbuildId%3D63534&commit=780e769f06e4a346a6b3a1b0a94ff97372f8dcc6&job=63534&pr=28361&project=scikit-learn&server_uri=https%3A%2F%2Fdev.azure.com%2Fscikit-learn%2F&service=azure_pipelines&slug=scikit-learn%2Fscikit-learn&name=&tag=&flags=&parent=
[2024-02-05T09:26:14.750Z] ['error'] There was an error running the uploader: Error uploading to https://codecov.io/: Error: There was an error fetching the storage URL during POST: 503 - upstream connect error or disconnect/reset before headers. reset reason: connection failure

##[error]Bash exited with code '255'.
##[warning]RetryHelper encountered task failure, will retry (attempt #: 1 out of 5) after 1000 ms

So we get a 503 error which hints at a permission problem. But on the same build, we do also get successful codecov uploads such as pymin_conda_defaults_openblas for instance:

Codecov report uploader 0.7.1
[2024-02-05T09:36:30.039Z] ['info'] => Project root located at: /home/vsts/work/1/s
[2024-02-05T09:36:30.044Z] ['info'] ->  Token found by arguments
[2024-02-05T09:36:30.062Z] ['info'] Searching for coverage files...
[2024-02-05T09:36:30.150Z] ['info'] => Found 1 possible coverage files:
  coverage.xml
[2024-02-05T09:36:30.150Z] ['info'] Processing /home/vsts/work/1/s/coverage.xml...
[2024-02-05T09:36:30.215Z] ['info'] Detected Azure Pipelines as the CI provider.
[2024-02-05T09:36:30.224Z] ['info']     Fixing merge commit SHA 135b5e81d263e9511507dd920f1a90e0c39d67eb -> 780e769f06e4a346a6b3a1b0a94ff97372f8dcc6
[2024-02-05T09:36:30.226Z] ['info'] Pinging Codecov: https://codecov.io/upload/v4?package=uploader-0.7.1&token=*******&branch=refs%2Fpull%2F28361%2Fmerge&build=20240205.13&build_url=https%3A%2F%2Fdev.azure.com%2Fscikit-learn%2Fscikit-learn%2F_build%2Fresults%3FbuildId%3D63534&commit=780e769f06e4a346a6b3a1b0a94ff97372f8dcc6&job=63534&pr=28361&project=scikit-learn&server_uri=https%3A%2F%2Fdev.azure.com%2Fscikit-learn%2F&service=azure_pipelines&slug=scikit-learn%2Fscikit-learn&name=&tag=&flags=&parent=
[2024-02-05T09:36:30.757Z] ['info'] https://app.codecov.io/github/scikit-learn/scikit-learn/commit/780e769f06e4a346a6b3a1b0a94ff97372f8dcc6
https://storage.googleapis.com/codecov/v4/raw/2024-02-05/067C463327404A63BFA478ACBE94C8FB/780e769f06e4a346a6b3a1b0a94ff97372f8dcc6/96420237-ece9-4b70-9fdf-5d9ceb57b119.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=GOOG1EJOGFN2JQ4OCTGA2MU5AEIT7OT5Z7HTFOAN2SPG4NWSN2UJYOY5U6LZQ%2F20240205%2FUS%2Fs3%2Faws4_request&X-Amz-Date=20240205T093630Z&X-Amz-Expires=10&X-Amz-SignedHeaders=host&X-Amz-Signature=3468f428c154d3ce824fd1eb217250647887cf9fa52f9e35aaa7e5b02db5e55e
[2024-02-05T09:36:30.758Z] ['info'] Uploading...
[2024-02-05T09:36:31.043Z] ['info'] {"status":"processing","resultURL":"[https://app.codecov.io/github/scikit-learn/scikit-learn/commit/780e769f06e4a346a6b3a1b0a94ff97372f8dcc6"}](https://app.codecov.io/github/scikit-learn/scikit-learn/commit/780e769f06e4a346a6b3a1b0a94ff97372f8dcc6%22%7D)

So this kind of weird because I don't think we have per-job permission settings that could explain this discrepancy.

@lesteve
Copy link
Member Author

lesteve commented Feb 5, 2024

So this kind of weird because I don't think we have per-job permission settings that could explain this discrepancy.

I am guessing this is some random codecov server issues. They seem to happen more often recently i.e. seen at least in this PR and #28348.

@ogrisel
Copy link
Member

ogrisel commented Feb 5, 2024

The status page of codecov is all green:

and I could not find any issue with the 503 error code in the codecov uploader issue tracker:

@adrinjalali
Copy link
Member

This gave me a lot of headache on skops, e.g. here.

We ended up fixing the issue by including the upload token as plain text in the repo / CI config, and then never saw the issue again.

Since this token is only used to upload coverage, and w/o the token one can upload coverage anyway, I don't see any security risks in it.

@lesteve
Copy link
Member Author

lesteve commented Feb 5, 2024

As far as I can see we are passing the token. Also codecov is used from Azure Pipelines but and the discussion you linked to seems to be specifically about codecov being used inside Github Actions.

Not sure what is going on, let's wait and see ...

@ogrisel ogrisel mentioned this pull request Feb 5, 2024
Closed
@adrinjalali
Copy link
Member

The issue AFAIK is that the token used by default reaches limits and upload fails. That's why having a hard coded token works.

@ogrisel
Copy link
Member

ogrisel commented Feb 5, 2024

Interesting. Let me try to see if I can generate a new version of the token an update the matching Azure Pipelines secret env variable for CODECOV_TOKEN.

@ogrisel
Copy link
Member

ogrisel commented Feb 5, 2024

Ok I did it. I got an error message from codecov the first time I tried to press the "regenerate token" button but it worked after waiting a bit and doing a page refresh. I updated the Azure Pipelines CODECOV_TOKEN variable for scikit-learn.

glemaitre
glemaitre approved these changes Feb 5, 2024
View reviewed changes
Copy link
Member

@glemaitre glemaitre left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm happy with updating the uploader in any case. Let see if @ogrisel push is successful.

@lesteve
Copy link
Member Author

lesteve commented Feb 5, 2024

🤔 looks like there is an issue with the new token?

See build log:

Codecov report uploader 0.7.1
[2024-02-05T14:44:01.604Z] ['info'] => Project root located at: /home/vsts/work/1/s
[2024-02-05T14:44:01.608Z] ['info'] ->  Token found by arguments
[2024-02-05T14:44:01.609Z] ['error'] There was an error running the uploader: Token found by arguments with length 50 did not pass validation
[2024-02-05T14:44:01.610Z] ['verbose'] The error stack is: Error: Token found by arguments with length 50 did not pass validation
    at getToken (/snapshot/repo/dist/src/helpers/token.js)
    at main (/snapshot/repo/dist/src/index.js)
    at Object.<anonymous> (/snapshot/repo/dist/bin/codecov.js)
    at Module._compile (pkg/prelude/bootstrap.js:1926:22)
    at Module._extensions..js (node:internal/modules/cjs/loader:1166:10)
    at Module.load (node:internal/modules/cjs/loader:988:32)
    at Module._load (node:internal/modules/cjs/loader:834:12)
    at Function.runMain (pkg/prelude/bootstrap.js:1979:12)
    at node:internal/main/run_main_module:17:47
[2024-02-05T14:44:01.610Z] ['verbose'] End of uploader: 8 milliseconds

@ogrisel
Copy link
Member

ogrisel commented Feb 5, 2024

pylatest_conda_forge_mkl failed with:

[2024-02-05T14:44:01.609Z] ['error'] There was an error running the uploader: Token found by arguments with length 50 did not pass validation
[2024-02-05T14:44:01.610Z] ['verbose'] The error stack is: Error: Token found by arguments with length 50 did not pass validation
    at getToken (/snapshot/repo/dist/src/helpers/token.js)
    at main (/snapshot/repo/dist/src/index.js)
    at Object.<anonymous> (/snapshot/repo/dist/bin/codecov.js)
    at Module._compile (pkg/prelude/bootstrap.js:1926:22)
    at Module._extensions..js (node:internal/modules/cjs/loader:1166:10)
    at Module.load (node:internal/modules/cjs/loader:988:32)
    at Module._load (node:internal/modules/cjs/loader:834:12)
    at Function.runMain (pkg/prelude/bootstrap.js:1979:12)
    at node:internal/main/run_main_module:17:47
[2024-02-05T14:44:01.610Z] ['verbose'] End of uploader: 8 milliseconds

Maybe I made a mistake when setting up the CODECOV_TOKEN variable the first time. I deleted it and set it again after checking that the content of my clipboard has the expected value (36 characters long).

ogrisel
ogrisel approved these changes Feb 5, 2024
View reviewed changes
Copy link
Member

@ogrisel ogrisel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Finally all green!

@ogrisel ogrisel merged commit f17bc7e into scikit-learn:main Feb 5, 2024
@ogrisel
Copy link
Member

ogrisel commented Feb 5, 2024

The issue AFAIK is that the token used by default reaches limits and upload fails. That's why having a hard coded token works.

Not sure about why hard-coding a token in the config file vs passing it as an env variable would avoid this buggy rate limited-related errors.

Anyways, it works for now. If it means we need to rotate the token twice a year, I think it's bearable (once you know that those errors can be fixed by rotating the token).

@adrinjalali
Copy link
Member

For me the issue was that I thought the secret was available, but it was empty in PRs, and making them available in GH actions is a pain.

@ogrisel
Copy link
Member

ogrisel commented Feb 6, 2024

For me the issue was that I thought the secret was available, but it was empty in PRs, and making them available in GH actions is a pain.

Alright so this was probably a different issue because here the logs never suggested that the env variable was empty or undefined.

glemaitre pushed a commit to glemaitre/scikit-learn that referenced this pull request Feb 10, 2024
@lesteve @ogrisel @glemaitre
CI Update codecov uploader to 0.7.1 (scikit-learn#28361)
4cc9145 
Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
glemaitre pushed a commit to glemaitre/scikit-learn that referenced this pull request Feb 13, 2024
@lesteve @ogrisel @glemaitre
CI Update codecov uploader to 0.7.1 (scikit-learn#28361)
591a64d 
Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
glemaitre pushed a commit that referenced this pull request Feb 13, 2024
@lesteve @ogrisel @glemaitre
CI Update codecov uploader to 0.7.1 (#28361)
314ec14 
Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ogrisel ogrisel ogrisel approved these changes

@glemaitre glemaitre glemaitre approved these changes

Assignees
No one assigned
Labels
Build / CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@lesteve @ogrisel @adrinjalali @glemaitre