feat(CI): add Codecov Test Analytics for flaky and failed tests #29881
Conversation
There was a problem hiding this comment.
Thank you for opening your first PR into Matplotlib!
If you have not heard from us in a week or so, please leave a new comment below and that should bring it to our attention. Most of our reviewers are volunteers and sometimes things fall through the cracks.
You can also join us on gitter for real-time discussion.
For details on testing, writing docs, and our review process, please see the developer guide
We strive to be a welcoming and open project. Please follow our Code of Conduct.
ps.: the action that uploads the test results doesn't have a great time differentiating 'linux' from 'linux-arm64', so I had to add the matrix values for the `os:` option
fa88a74 to
3c13d7f
Compare
There was a problem hiding this comment.
I guess the results are here: https://app.codecov.io/gh/matplotlib/matplotlib/tests/giovanni-guidini%3Agio%2Fcodecov-test-analytics
But it doesn't say too much since it's just the one commit, and there are no trends to speak of.
41ad7d5 to
887c1e9
Compare
There was a problem hiding this comment.
This looks like a nice new feature, and I'd definitely be 👍 to trying it out.
The other comment I have, is would you recommend us adding any tags? The codecov docs seem to recommend the Python version, so perhaps we should add that as a tag? We have definitely had tests fail on certain Python versions before, so I think adding that tag could be helpful.
.github/workflows/tests.yml
Outdated
| uses: codecov/test-results-action@v1 | ||
| with: | ||
| token: ${{ secrets.CODECOV_TOKEN }} | ||
| os: ${{ matrix.codecov-test-results-action-os }} |
There was a problem hiding this comment.
Is there a reason we need to manually specify the os - the docs say "Override the assumed OS.", so I would have assumed that the os can be autodetected somehow? I was thinking of saving having to hard code the os as a variable for every element of the test matrix.
There was a problem hiding this comment.
I completely agree with you there, it would be better to rely on the autodetect. It works mostly fine, but it's not perfect.
Originally I was relying on that, but this run shows that specifically the ubuntu-22.04-arm is being detected as "linux" not "linux-arm64". Hopefully this will be addressed in the Action's side, of course, but in the mean time using the os var can side step this limitation and make sure all uploads are being done correctly.
An alternative approach that I didn't think would work is to conditionally pass the os var only in that one case that would need it. This would be a much smaller footprint in the yaml. I can try that approach.
We appreciate the interest in this feature! About the flags, I think the python version is indeed a good candidate for flags. The |
|
Can you explain where the Python tag can be used on codecov to filter results (or for other functionality)? I tried looking at https://app.codecov.io/gh/matplotlib/matplotlib/commit/bc70afb98830826cfd18ee24dc185bcc1af16a66 and https://app.codecov.io/gh/matplotlib/matplotlib/tests/giovanni-guidini%3Agio%2Fcodecov-test-analytics, but I can't obviously see where the tags show up. |
Good observation. I've confirmed with the team that at this time you can only filter the test results by flag in the The team did say they are considering expanding filtering to any branch, but I won't make promises, nor do I have a timeframe for such feature. Sorry :E |
PR summary
Hi, Gio from Codecov here. I saw that you used Codecov already, and I thought you could benefit from catching any flaky or failed tests. We released a Test Analytics product that might be helpful and would love your feedback.
Changes:
PR checklist