Skip to content

Random image failures with test_curvelinear4 #10602

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
efiring opened this issue Feb 25, 2018 · 4 comments
Closed

Random image failures with test_curvelinear4 #10602

efiring opened this issue Feb 25, 2018 · 4 comments
Labels
topic: testing
Milestone
v3.0.0

Comments

@efiring
Copy link
Member

efiring commented Feb 25, 2018

We are getting frequent but inconsistent failures on Travis:

FAIL lib/mpl_toolkits/tests/test_axisartist_floating_axes.py::test_curvelinear4[png]

e.g., https://travis-ci.org/matplotlib/matplotlib/jobs/346056502
@tacaswell tacaswell added this to the v3.0 milestone Feb 25, 2018
@QuLogic
Copy link
Member

QuLogic commented Feb 26, 2018

Also seems to affect AppVeyor, e.g., #10426 failed but then worked on rebuild.

@efiring
Copy link
Member Author

efiring commented Feb 26, 2018

The Travis output indicates that we should be able to download the result images as a tarball, but I haven't been able to access the given URL. Is there some trick to it?

@efiring
Copy link
Member Author

efiring commented Feb 26, 2018

In 2 examples, one from Travis and the other from Appveyor, the mismatch is 0.011, but the tolerance is 0.01. My guess is that there has always been something slightly non-deterministic in this test, and whatever it is, it has simply gotten slightly worse. In the spirit of triage, I think we should just bump the tolerance to 0.02 and move on. Tracking down the root cause is probably not the best use of anyone's time right now.

@QuLogic
Copy link
Member

QuLogic commented Feb 26, 2018

Yes, when I transferred those examples from the source files into real tests, they were always a bit flaky, and I was always hopeful that things like #10403 will help with it (e.g., I've just noticed that set_axes_locator in the toolkit version does not set Axes.stale.)

efiring added a commit to efiring/matplotlib that referenced this issue Feb 26, 2018
@efiring
Bump a tolerance in test_axisartist_floating_axes.
ef62911 
This is to stop random test failures with test_curvelinear4.png.
It's original tolerance was 0.01; the mismatch is sometimes 0.011.

Closes matplotlib#10602.
@efiring efiring mentioned this issue Feb 26, 2018
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
topic: testing
Projects
None yet
Milestone
v3.0.0
Development

No branches or pull requests
4 participants
@efiring @tacaswell @QuLogic @anntzer