Skip to content

Increase a few test tolerances on some arches #24786

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jan 4, 2023
Merged

Increase a few test tolerances on some arches #24786

merged 1 commit into from
Jan 4, 2023

Conversation

QuLogic
Copy link
Member

@QuLogic QuLogic commented Dec 20, 2022

PR Summary

I previously had this downstream in Fedora, but as we may soon get real ARM CI (#24597), it makes sense to put it in here now.

PR Checklist

Documentation and Tests

  • Has pytest style unit tests (and pytest passes)
  • [n/a] Documentation is sphinx and numpydoc compliant (the docs should build without error).
  • [n/a] New plotting related features are documented with examples.

Release Notes

  • [n/a] New features are marked with a .. versionadded:: directive in the docstring and documented in doc/users/next_whats_new/
  • [n/a] API changes are marked with a .. versionchanged:: directive in the docstring and documented in doc/api/next_api_changes/
  • [n/a] Release notes conform with instructions in next_whats_new/README.rst or next_api_changes/README.rst

@oscargus
Copy link
Member

oscargus commented Dec 20, 2022 •
edited
Loading

Isn't it beter to swap so that the troublesome platforms get higher tolerance and the rest 0/lower?

Turns out that arm64 (Mac m1) does not need higher tolerance, but aarch64 does.

Edit: also, amd64 seems to be a viable option as well, further stressing that we should special case the ones that require higher tolerance rather than setting lower tolerance for 'x86_64`.

@tacaswell tacaswell added this to the v3.7.0 milestone Dec 21, 2022
@QuLogic
Copy link
Member Author

QuLogic commented Dec 30, 2022 •
edited
Loading

I ran a scratch build and here are the exact faiures:

aarch64

test_contour_addlines[png] (RMS 0.146)
test_colorbar/test_contour_colorbar[png] (RMS 0.007)
test_axes/test_contour_colorbar[png] (RMS 0.012)
test_axes/test_contour_colorbar[svg] (RMS 0.013)
test_imshow_clip[pdf] (RMS 1.238)
test_imshow_clip[png] (RMS 0.009)
test_imshow_clip[svg] (RMS 0.274)
test_rgba_antialias[png] (RMS 0.006)
test_set_line_coll_dash_image[pdf] (RMS 0.402)
test_set_line_coll_dash_image[svg] (RMS 0.614)

ppc64le

test_contour_addlines[png] (RMS 0.146)
test_colorbar/test_contour_colorbar[png] (RMS 0.007)
test_axes/test_contour_colorbar[png] (RMS 0.012)
test_axes/test_contour_colorbar[svg] (RMS 0.013)
test_imshow_clip[pdf] (RMS 1.238)
test_imshow_clip[png] (RMS 0.009)
test_imshow_clip[svg] (RMS 0.274)
test_mathfont_rendering[png-mathfont-cm-34] (RMS 0.007)
test_mathfont_rendering[png-mathfont-cm-35] (RMS 0.007)
test_mathfont_rendering[png-mathfont-cm-45] (RMS 0.007)
test_mathfont_rendering[png-mathfont-cm-49] (RMS 0.007)
test_mathfont_rendering[png-mathfont-dejavusans-9] (RMS 0.010)
test_mathfont_rendering[png-mathfont-dejavusans-34] (RMS 0.007)
test_mathfont_rendering[png-mathfont-dejavusans-45] (RMS 0.007)
test_mathfont_rendering[png-mathfont-dejavusans-49] (RMS 0.007)
test_mathfont_rendering[png-mathfont-dejavuserif-34] (RMS 0.007)
test_mathfont_rendering[png-mathfont-dejavuserif-35] (RMS 0.007)
test_mathfont_rendering[png-mathfont-dejavuserif-45] (RMS 0.007)
test_mathfont_rendering[png-mathfont-dejavuserif-49] (RMS 0.007)
test_mathfont_rendering[png-mathfont-stix-9] (RMS 0.009)
test_mathfont_rendering[png-mathfont-stix-11] (RMS 0.005)
test_mathfont_rendering[png-mathfont-stix-21] (RMS 0.010)
test_mathfont_rendering[png-mathfont-stix-34] (RMS 0.007)
test_mathfont_rendering[png-mathfont-stix-35] (RMS 0.007)
test_mathfont_rendering[png-mathfont-stix-45] (RMS 0.007)
test_mathfont_rendering[png-mathfont-stix-49] (RMS 0.007)
test_mathfont_rendering[png-mathfont-stix-52] (RMS 0.007)
test_mathfont_rendering[png-mathfont-stix-55] (RMS 0.005)
test_mathfont_rendering[png-mathfont-stix-57] (RMS 0.007)
test_mathfont_rendering[png-mathfont-stixsans-1] (RMS 0.007)
test_mathfont_rendering[png-mathfont-stixsans-6] (RMS 0.005)
test_mathfont_rendering[png-mathfont-stixsans-10] (RMS 0.007)
test_mathfont_rendering[png-mathfont-stixsans-16] (RMS 0.007)
test_mathfont_rendering[png-mathfont-stixsans-21] (RMS 0.010)
test_mathfont_rendering[png-mathfont-stixsans-34] (RMS 0.007)
test_mathfont_rendering[png-mathfont-stixsans-45] (RMS 0.007)
test_mathfont_rendering[png-mathfont-stixsans-49] (RMS 0.007)
test_mathfont_rendering[png-mathfont-stixsans-52] (RMS 0.007)
test_mathfont_rendering[png-mathfont-stixsans-55] (RMS 0.005)
test_mathfont_rendering[png-mathfont-stixsans-57] (RMS 0.007)
test_mathtext_rendering[png-mathtext-stix-12] (RMS 0.007)
test_mathtext_rendering[png-mathtext-stix-13] (RMS 0.007)
test_mathtext_rendering[png-mathtext-stix-18] (RMS 0.010)
test_mathtext_rendering[png-mathtext-stix-20] (RMS 0.007)
test_mathtext_rendering[png-mathtext-stix-21] (RMS 0.007)
test_mathtext_rendering[png-mathtext-stix-27] (RMS 0.010)
test_mathtext_rendering[png-mathtext-stix-31] (RMS 0.007)
test_mathtext_rendering[png-mathtext-stix-54] (RMS 0.007)
test_mathtext_rendering[png-mathtext-stix-63] (RMS 0.010)
test_mathtext_rendering[png-mathtext-stix-64] (RMS 0.007)
test_mathtext_rendering[png-mathtext-stix-81] (RMS 0.007)
test_mathtext_rendering[png-mathtext-stixsans-11] (RMS 0.007)
test_mathtext_rendering[png-mathtext-stixsans-12] (RMS 0.007)
test_mathtext_rendering[png-mathtext-stixsans-13] (RMS 0.007)
test_mathtext_rendering[png-mathtext-stixsans-18] (RMS 0.010)
test_mathtext_rendering[png-mathtext-stixsans-20] (RMS 0.007)
test_mathtext_rendering[png-mathtext-stixsans-21] (RMS 0.007)
test_mathtext_rendering[png-mathtext-stixsans-27] (RMS 0.010)
test_mathtext_rendering[png-mathtext-stixsans-31] (RMS 0.007)
test_mathtext_rendering[png-mathtext-stixsans-54] (RMS 0.007)
test_mathtext_rendering[png-mathtext-stixsans-63] (RMS 0.010)
test_mathtext_rendering[png-mathtext-stixsans-64] (RMS 0.007)
test_mathtext_rendering[png-mathtext-stixsans-81] (RMS 0.007)
test_rgba_antialias[png] (RMS 0.006)
test_set_line_coll_dash_image[pdf] (RMS 0.402)
test_set_line_coll_dash_image[svg] (RMS 0.614)

s390x

test_contour_addlines[png] (RMS 0.146)
test_colorbar/test_contour_colorbar[png] (RMS 0.007)
test_axes/test_contour_colorbar[png] (RMS 0.012)
test_axes/test_contour_colorbar[svg] (RMS 0.013)
test_imshow_clip[pdf] (RMS 1.238)
test_imshow_clip[png] (RMS 0.009)
test_imshow_clip[svg] (RMS 0.274)
test_mathfont_rendering[png-mathfont-cm-34] (RMS 0.007)
test_mathfont_rendering[png-mathfont-cm-35] (RMS 0.007)
test_mathfont_rendering[png-mathfont-cm-45] (RMS 0.007)
test_mathfont_rendering[png-mathfont-cm-49] (RMS 0.007)
test_mathfont_rendering[png-mathfont-dejavusans-9] (RMS 0.010)
test_mathfont_rendering[png-mathfont-dejavusans-34] (RMS 0.007)
test_mathfont_rendering[png-mathfont-dejavusans-45] (RMS 0.007)
test_mathfont_rendering[png-mathfont-dejavusans-49] (RMS 0.007)
test_mathfont_rendering[png-mathfont-dejavuserif-34] (RMS 0.007)
test_mathfont_rendering[png-mathfont-dejavuserif-35] (RMS 0.007)
test_mathfont_rendering[png-mathfont-dejavuserif-45] (RMS 0.007)
test_mathfont_rendering[png-mathfont-dejavuserif-49] (RMS 0.007)
test_mathfont_rendering[png-mathfont-stix-9] (RMS 0.009)
test_mathfont_rendering[png-mathfont-stix-11] (RMS 0.005)
test_mathfont_rendering[png-mathfont-stix-21] (RMS 0.010)
test_mathfont_rendering[png-mathfont-stix-34] (RMS 0.007)
test_mathfont_rendering[png-mathfont-stix-35] (RMS 0.007)
test_mathfont_rendering[png-mathfont-stix-45] (RMS 0.007)
test_mathfont_rendering[png-mathfont-stix-49] (RMS 0.007)
test_mathfont_rendering[png-mathfont-stix-52] (RMS 0.007)
test_mathfont_rendering[png-mathfont-stix-55] (RMS 0.005)
test_mathfont_rendering[png-mathfont-stix-57] (RMS 0.007)
test_mathfont_rendering[png-mathfont-stixsans-1] (RMS 0.007)
test_mathfont_rendering[png-mathfont-stixsans-6] (RMS 0.005)
test_mathfont_rendering[png-mathfont-stixsans-10] (RMS 0.007)
test_mathfont_rendering[png-mathfont-stixsans-16] (RMS 0.007)
test_mathfont_rendering[png-mathfont-stixsans-21] (RMS 0.010)
test_mathfont_rendering[png-mathfont-stixsans-34] (RMS 0.007)
test_mathfont_rendering[png-mathfont-stixsans-45] (RMS 0.007)
test_mathfont_rendering[png-mathfont-stixsans-49] (RMS 0.007)
test_mathfont_rendering[png-mathfont-stixsans-52] (RMS 0.007)
test_mathfont_rendering[png-mathfont-stixsans-55] (RMS 0.005)
test_mathfont_rendering[png-mathfont-stixsans-57] (RMS 0.007)
test_mathtext_rendering[png-mathtext-stix-12] (RMS 0.007)
test_mathtext_rendering[png-mathtext-stix-13] (RMS 0.007)
test_mathtext_rendering[png-mathtext-stix-18] (RMS 0.010)
test_mathtext_rendering[png-mathtext-stix-20] (RMS 0.007)
test_mathtext_rendering[png-mathtext-stix-21] (RMS 0.007)
test_mathtext_rendering[png-mathtext-stix-27] (RMS 0.010)
test_mathtext_rendering[png-mathtext-stix-31] (RMS 0.007)
test_mathtext_rendering[png-mathtext-stix-54] (RMS 0.007)
test_mathtext_rendering[png-mathtext-stix-63] (RMS 0.010)
test_mathtext_rendering[png-mathtext-stix-64] (RMS 0.007)
test_mathtext_rendering[png-mathtext-stix-81] (RMS 0.007)
test_mathtext_rendering[png-mathtext-stixsans-11] (RMS 0.007)
test_mathtext_rendering[png-mathtext-stixsans-12] (RMS 0.007)
test_mathtext_rendering[png-mathtext-stixsans-13] (RMS 0.007)
test_mathtext_rendering[png-mathtext-stixsans-18] (RMS 0.010)
test_mathtext_rendering[png-mathtext-stixsans-20] (RMS 0.007)
test_mathtext_rendering[png-mathtext-stixsans-21] (RMS 0.007)
test_mathtext_rendering[png-mathtext-stixsans-27] (RMS 0.010)
test_mathtext_rendering[png-mathtext-stixsans-31] (RMS 0.007)
test_mathtext_rendering[png-mathtext-stixsans-54] (RMS 0.007)
test_mathtext_rendering[png-mathtext-stixsans-63] (RMS 0.010)
test_mathtext_rendering[png-mathtext-stixsans-64] (RMS 0.007)
test_mathtext_rendering[png-mathtext-stixsans-81] (RMS 0.007)
test_rgba_antialias[png] (RMS 0.006)
test_set_line_coll_dash_image[pdf] (RMS 0.402)
test_set_line_coll_dash_image[svg] (RMS 0.614)

@QuLogic
Copy link
Member Author

QuLogic commented Dec 30, 2022

A scratch build with this updated PR. For some reason, something in test_constrainedlayout has started failing now, but not sure why.

ksunden
ksunden reviewed Jan 3, 2023
View reviewed changes
lib/matplotlib/tests/test_backend_pgf.py Outdated
@@ -169,7 +169,7 @@ def test_pathclip():
# test mixed mode rendering
@needs_pgf_xelatex
@pytest.mark.backend('pgf')
@image_comparison(['pgf_mixedmode.pdf'], style='default')
@image_comparison(['pgf_mixedmode.pdf'], style='default', tol=0.9)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason this tolerance bump is not conditional like the rest of them?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because it probably shouldn't have been there. As noted above, this test doesn't appear in the list. I think this may have been for different Ghostscript versions, but is unnecessary now.

@ksunden
Copy link
Member

ksunden commented Jan 3, 2023

@QuLogic do you wish to resolve the new test failure before merge?

@tacaswell tacaswell merged commit a0442f4 into matplotlib:main Jan 4, 2023
@QuLogic QuLogic deleted the arm-tolerance branch January 4, 2023 21:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ksunden ksunden ksunden approved these changes

Assignees
No one assigned
Labels
topic: testing
Projects
None yet
Milestone
v3.7.0
Development

Successfully merging this pull request may close these issues.
4 participants
@QuLogic @oscargus @ksunden @tacaswell