Skip to content

Create coverage report for GNU tests #3045

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Feb 20, 2022
Merged

Create coverage report for GNU tests #3045

merged 4 commits into from
Feb 20, 2022

Conversation

xxyzz
Copy link
Contributor

@xxyzz xxyzz commented Feb 3, 2022

  • It seems codecov-action can only upload reports to the same project.
  • The code runs slower with coverage report enabled, so I extend the timeout limit to four hours.
  • codecov-action@v1 is deprecated, I didn't update the action in CICD.yml in case it breaks something.
  • The origin windows coverage report action failed on my forked repo sometimes, but I don't think it's caused by my changes.
  • Install valgrind and libexpect-perl for tests that require it.

Closes #2946.

@sylvestre
Copy link
Contributor

excellent stuff, thanks :)
I was more thinking at a new CI job then extend "Run GNU tests"

What about "Run GNU tests" and "Run GNU tests with coverage" ?

@rivy
Copy link
Member

rivy commented Feb 5, 2022

I very much appreciate this effort, but I'm concerned about adding this workflow to the usual CI runs.
The workflow triples the CI completion time from (an already too long) 1 hour to 3 hours.

And, in looking at the coverage diff, I don't see much of a coverage change either... only a few lines here with lots of them being changes to coverage of help message lines.

Maybe only enable it for merges to 'main'?

@xxyzz
Copy link
Contributor Author

xxyzz commented Feb 5, 2022

I think this link shows the correct coverage change of this pull request.

@sylvestre
Copy link
Contributor

Hmm, if I understand it correctly, it is adding on top of the current one ?
it isn't a separate result?

@rivy
Copy link
Member

rivy commented Feb 6, 2022

I think this link shows the correct coverage change of this pull request.

@xxyzz , That's definitely a much better gain (on the order of ~500 net line coverage gain)!

Thanks for showing me that comparison mode.

So, any ideas about how can we add this in a way that doesn't severely slow the CI for regular PRs?
An hour seems manageable, if only barely, for the current commit cadence, but three seems too far out.

@xxyzz
Copy link
Contributor Author

xxyzz commented Feb 6, 2022

I suspect the test runs slower because optimizations are disabled. I already let the new job only runs on the main branch in the last commit as you suggested.

@sylvestre
Copy link
Contributor

I approved earlier a patch from @rivy before
it caused conflicts here :( sorry
could you please fix them?
thanks

@rivy
Copy link
Member

rivy commented Feb 6, 2022

@xxyzz , I'm happy to fix the merge conflicts (especially since I caused them ☺️).

I want to merge some maintenance work (PR #3090) on the configuration and 'util' scripts first, then I'll rebase your additions on top and push them back here. Ok?

@rivy rivy self-assigned this Feb 6, 2022
@xxyzz
Copy link
Contributor Author

xxyzz commented Feb 7, 2022

Alright, thanks for solving the conflicts.

@rivy
Copy link
Member

rivy commented Feb 12, 2022

@xxyzz , I've rebased this on top of #3090, preserving your intent as I could see it.
When #3090 is merged, I'll update this commit set for you to review and revise.

@sylvestre sylvestre merged commit 9adaf5c into uutils:main Feb 20, 2022
@sylvestre
Copy link
Contributor

let's see :)
many thanks

@xxyzz
Copy link
Contributor Author

xxyzz commented Feb 20, 2022

I should run sudo at the end of the shell file... There are many permission errors in the log: https://github.com/xxyzz/coreutils/runs/5180735222?check_suite_focus=true#step:8:1688

@sylvestre
Copy link
Contributor

@xxyzz quick question, how can I use codecov to compare the gnu vs rust testsuite ? :)

@sylvestre
Copy link
Contributor

I noticed that you have

        flags: gnutests
        name: gnutests

but I don't see them on https://app.codecov.io/gh/uutils/coreutils

@xxyzz
Copy link
Contributor Author

xxyzz commented Feb 22, 2022

The GNU tests report only created when commits pushed to the main branch, so only the merge commits have it. Codecov says "Flags have been temporarily removed" in the compare page overview tab, but the app subdomain still has it. And as the notice said, flags can be used on files.

@sylvestre
Copy link
Contributor

thanks again :)

One more question @xxyzz
it seems that the coverage diff analysis in PR is using GNU's as comparison
does it ring a bell ?

 codecov/project Successful in 1s — 69.31% (-4.34%) compared to f11ac4b 

@xxyzz
Copy link
Contributor Author

xxyzz commented Mar 4, 2022

f11ac4b is a merge commit, so it has one more GNU coverage report then a PR. Now the GNU coverage job takes around one hour to finish, maybe it's time to enable it in pull requests so it can be compared?

@sylvestre
Copy link
Contributor

Sounds good :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add a new job in the CI to compute the code coverage of this project using the GNU testsuite
3 participants