TST: Add future dependency tests as a weekly CI job #21634

greglucas · 2021-11-14T20:41:53Z

PR Summary

Test future numpy versions with Matplotlib to see if anything needs to be done in the future to address deprecations
or other pending changes.

Currently will run once per week at 03:47 UTC on Saturdays.
Only runs on Linux and for 3.9/3.10 currently as this is mostly just a smoke test for future issues.
I think it will upload an issue on failure to notify the repo, but that is untested. Perhaps there is a better way here that someone knows about.

Link to a run on my branch (it should only run on matplotlib/matplotlib now)
https://github.com/greglucas/matplotlib/runs/4204821100?check_suite_focus=true

3.9 + numpy-1.22.dev failures

FAILED lib/matplotlib/tests/test_units.py::test_plot_masked_units[png] - matp...
= 1 failed, 8352 passed, 148 skipped, 13 xfailed, 4 xpassed in 537.93s (0:08:57) =

3.10 + numpy-1.22.dev failures

FAILED lib/matplotlib/tests/test_axes.py::test_errorbar[png] - matplotlib.tes...
FAILED lib/matplotlib/tests/test_axes.py::test_errorbar[svg] - matplotlib.tes...
FAILED lib/matplotlib/tests/test_axes.py::test_errorbar[pdf] - matplotlib.tes...
FAILED lib/matplotlib/tests/test_backends_interactive.py::test_webagg - Asser...
FAILED lib/matplotlib/tests/test_units.py::test_plot_masked_units[png] - matp...
FAILED lib/matplotlib/tests/test_streamplot.py::test_direction[png] - matplot...
FAILED lib/mpl_toolkits/tests/test_axisartist_grid_helper_curvelinear.py::test_axis_direction[png]
FAILED lib/mpl_toolkits/tests/test_mplot3d.py::test_trisurf3d[png] - matplotl...
FAILED lib/mpl_toolkits/tests/test_mplot3d.py::test_stem3d[png] - matplotlib....
= 9 failed, 8344 passed, 148 skipped, 13 xfailed, 4 xpassed, 4 warnings in 414.94s (0:06:54) =

Additional things to consider

Perhaps we should also consider adding a nightly cibuildwheel to that repository for others to test out Matplotlib and report back to us easier?

PR Checklist

Tests and Styling

Has pytest style unit tests (and pytest passes).
Is Flake 8 compliant (install flake8-docstrings and run flake8 --docstring-convention=all).

Documentation

New features are documented, with examples if plot related.
New features have an entry in doc/users/next_whats_new/ (follow instructions in README.rst there).
API changes documented in doc/api/next_api_changes/ (follow instructions in README.rst there).
Documentation is sphinx and numpydoc compliant (the docs should build without error).

greglucas · 2021-11-14T20:48:28Z

.github/workflows/tests-future-deps.yml

+        run: |
+          xvfb-run -a python -mpytest -raR -n auto \
+            --maxfail=50 --timeout=300 --durations=25 --log-level=DEBUG \
+            -W error::UserWarning


We actually would like to see the DeprecationWarning's as well (-W error without the specific UserWarning), but on Python 3.10 Numpy's testing import causes deprecation warnings due to the distutils future removal in Python 3.12. So, we would have to wait for Numpy's removal of distutils before we did that. Unless I'm missing a way to catch a specific string of deprecation warnings so we could ignore only that specific warning.

Sure, you can be more specific with warnings filters: https://docs.python.org/3/library/warnings.html#describing-warning-filters

Thanks for pointing that out. I tried a few things out and it was more complicated that I hoped.
See here for other discussion: https://bugs.python.org/issue43862

While the Python API warnings.filterwarnings(action, message="", ...) uses the message as a regular expression, -W and PYTHONWARNINGS require to match exactly the whole message.

I got one match, and then another import failed, so not sure how many options/messages we want to match on if we do. I couldn't figure out how to use the "module" piece either, I thought that could refer to errors raise from distutils directly, or even by putting numpy.testing in there for where the warnings were arising from, but neither of those worked for me...

In summary, I'd leave it with the UserWarning -> Error for now and deal with DeprecationWarnings after Numpy fixes their distutils imports.

Alternatively, you could change pytest.ini to ignore what you want, as it seems to support regex.

cat >> pytest.ini <<EOF filterwarnings = error:message:DeprecationWarning EOF

tacaswell · 2021-11-14T21:05:05Z

I think some of the 3.10 + numpy-1.22.dev failures are fixed on numpy's default branch (at one point I was seeing issuse with distutils warnings, but they are gone for me locally).

I am in principle 👍🏻 👍🏻 on this idea!

greglucas · 2021-11-22T19:30:42Z

Looks like the test suite is passing with the nightly pandas and numpy wheels now.

https://github.com/greglucas/matplotlib/runs/4290939395?check_suite_focus=true

.github/workflows/tests-future-deps.yml

QuLogic · 2021-11-23T01:56:15Z

.github/workflows/tests-future-deps.yml

+        run: |
+          xvfb-run -a python -mpytest -raR -n auto \
+            --maxfail=50 --timeout=300 --durations=25 --log-level=DEBUG \
+            -W error::UserWarning


Alternatively, you could change pytest.ini to ignore what you want, as it seems to support regex.

cat >> pytest.ini <<EOF filterwarnings = error:message:DeprecationWarning EOF

greglucas · 2021-11-24T00:45:41Z

Some of the tests seem to be a little flaky when they are run: https://github.com/greglucas/matplotlib/actions/runs/1495511858
The new changes seem to be good, but test_errorbar is failing, even though it passed on a previous run with the same dependency versions installed...

greglucas · 2021-12-23T21:56:36Z

I pushed a local copy of this up to my fork: https://github.com/greglucas/matplotlib/runs/4621007693?check_suite_focus=true
and both 3.9 and 3.10 with latest numpy is failing. However, when I try to run this all locally (installing numpy from nightly wheels) the tests all pass for me. There must be something about the runner environment that makes a few of the tests flaky, but I haven't been able to figure it out yet looking through the logs.

QuLogic · 2021-12-23T22:01:37Z

The result images should be obtainable on the Summary page. It doesn't look like the failures are anywhere other than test images.

greglucas · 2021-12-23T22:08:08Z

Yep, I did download them and couldn't find any obvious reason for the failures (they are all quite small). The largest one test_errorbar has an expanded y-limit, causing the number of ticks to be increased, but the data in the plot appears to be correct. My initial guess is that the failures are due to floating point discrepancies, causing a ylim to be expanded to the next decade, but I'm not sure where that would be coming from since I can't reproduce locally.

greglucas · 2021-12-24T22:55:37Z

Putting some more notes down here:
This test suite does pass CI if I don't upgrade numpy to the nightly wheel: https://github.com/greglucas/matplotlib/actions/runs/1617303489 (I was worried I'd updated some other config to make a standard run fail). I also tried pip installing nightly numpy before everything else and after everything else (including matplotlib), to make sure it wasn't affecting the compilation of any dependencies and it failed in every case there as well.

I can't reproduce these failures on a local macosx or ubuntu machine either, so my debugging capability here is limited unfortunately.

greglucas · 2022-01-03T16:17:23Z

Perhaps this is related to AVX512 instructions now being available in the wheel...
https://numpy.org/devdocs/release/1.22.0-notes.html#vectorize-umath-module-using-avx-512

On the 3.8-3.10 runners:

Supported SIMD extensions in this NumPy install:
    baseline = SSE,SSE2,SSE3
    found = SSSE3,SSE41,POPCNT,SSE42,AVX,F16C,FMA3,AVX2,AVX512F,AVX512CD,AVX512_SKX
    not found = AVX512_KNL,AVX512_KNM,AVX512_CLX,AVX512_CNL,AVX512_ICL

On the 3.7 runner (numpy 1.21):

Supported SIMD extensions in this NumPy install:
    baseline = SSE,SSE2,SSE3
    found = SSSE3,SSE41,POPCNT,SSE42,AVX,F16C,FMA3,AVX2
    not found = AVX512F,AVX512CD,AVX512_KNL,AVX512_KNM,AVX512_SKX,AVX512_CLX,AVX512_CNL,AVX512_ICL

My local Ubuntu is on a chip without AVX512 instructions, which could explain why I'm not able to reproduce this locally. If someone does have a system with a chip that supports AVX512 and could test locally that could verify this guess: python -c "import numpy; numpy.show_config()"
or if someone knows of a way to disable that instruction set from use within numpy at runtime rather than build time, I could push that up to the runner as well.

dstansby · 2022-01-03T16:25:45Z

Perhaps this is related to AVX512 instructions now being available in the wheel... https://numpy.org/devdocs/release/1.22.0-notes.html#vectorize-umath-module-using-avx-512

I suspect this is correct - I'm seeing similar small floating point differences on another project that seems to be related to both numpy version and the specific machine CI is being run on.

greglucas · 2022-01-24T04:50:47Z

I decided to rework this and incorporate it into the current tests.yml file, which seems better than trying to maintain two copies of the CI file. There are new floating-point issues that cropped up again in Normalize with longdouble. It must be a new numpy optimization in .min(), as specific item-access works, so I've updated the tests to account for these changes.

.github/workflows/tests.yml

Test future numpy and pandas versions with Matplotlib to see if anything needs to be done in the future to address deprecations or other pending changes. Turns all warnings into errors, but filters out the distutils deprecation and find_spec warnings.

QuLogic · 2022-08-22T19:47:35Z

lib/matplotlib/tests/test_colors.py

    # This returns exactly 0.5 when longdouble is extended precision (80-bit),
    # but only a value close to it when it is quadruple precision (128-bit).
-    assert 0 < norm(1 + 50 * eps) < 1
+    np.testing.assert_array_almost_equal_nulp(norm(1 + 50 * eps), 0.5)


This seems to be a bit too strict now, as it is failing on aarch64, ppc64le, and s390x, where the result is 0.50096339.

Also, on my 64-bit AMD system, which is apparently using np.float128 for np.longdouble (though I don't know if that means 80-bit internally), it seems to return 0.5 exactly, which seems the opposite of the comment.

greglucas commented Nov 14, 2021

View reviewed changes

ianhi mentioned this pull request Nov 15, 2021

Also upload a subset of nightly wheels #21637

Closed

greglucas force-pushed the future-deps branch from 3042f7a to e79d051 Compare November 22, 2021 19:28

QuLogic reviewed Nov 23, 2021

View reviewed changes

greglucas force-pushed the future-deps branch from a97d5a9 to 9a4a1a6 Compare November 24, 2021 00:46

tacaswell added this to the v3.6.0 milestone Nov 25, 2021

greglucas mentioned this pull request Jan 2, 2022

See if pinning numpy to < 1.22 fixes tests #22076

Closed

6 tasks

greglucas force-pushed the future-deps branch from 9a4a1a6 to 7830c85 Compare January 24, 2022 04:46

jklymak approved these changes Feb 1, 2022

View reviewed changes

jklymak added status: needs review topic: testing labels Feb 1, 2022

QuLogic reviewed Feb 2, 2022

View reviewed changes

.github/workflows/tests.yml Show resolved Hide resolved

greglucas force-pushed the future-deps branch 2 times, most recently from d0e5605 to b4d1b14 Compare February 2, 2022 15:33

greglucas force-pushed the future-deps branch from b4d1b14 to 4cf54f4 Compare February 12, 2022 03:20

QuLogic approved these changes Feb 12, 2022

View reviewed changes

QuLogic merged commit 0407b9f into matplotlib:main Feb 12, 2022

QuLogic removed the status: needs review label Feb 12, 2022

greglucas deleted the future-deps branch February 12, 2022 15:15

QuLogic reviewed Aug 22, 2022

View reviewed changes

QuLogic mentioned this pull request Aug 22, 2022

test_Normalize fails on aarch64/ppc64le/s390x #23707

Closed

QuLogic mentioned this pull request Jun 27, 2023

CI: Add pre-release installs to upcoming tests #26197

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TST: Add future dependency tests as a weekly CI job #21634

TST: Add future dependency tests as a weekly CI job #21634

greglucas commented Nov 14, 2021

greglucas Nov 14, 2021

QuLogic Nov 15, 2021

greglucas Nov 16, 2021

QuLogic Nov 23, 2021

tacaswell commented Nov 14, 2021

greglucas commented Nov 22, 2021

QuLogic Nov 23, 2021

greglucas commented Nov 24, 2021

greglucas commented Dec 23, 2021

QuLogic commented Dec 23, 2021

greglucas commented Dec 23, 2021

greglucas commented Dec 24, 2021

greglucas commented Jan 3, 2022

dstansby commented Jan 3, 2022

greglucas commented Jan 24, 2022

QuLogic Aug 22, 2022

TST: Add future dependency tests as a weekly CI job #21634

TST: Add future dependency tests as a weekly CI job #21634

Conversation

greglucas commented Nov 14, 2021

PR Summary

3.9 + numpy-1.22.dev failures

3.10 + numpy-1.22.dev failures

Additional things to consider

PR Checklist

greglucas Nov 14, 2021

Choose a reason for hiding this comment

QuLogic Nov 15, 2021

Choose a reason for hiding this comment

greglucas Nov 16, 2021

Choose a reason for hiding this comment

QuLogic Nov 23, 2021

Choose a reason for hiding this comment

tacaswell commented Nov 14, 2021

greglucas commented Nov 22, 2021

QuLogic Nov 23, 2021

Choose a reason for hiding this comment

greglucas commented Nov 24, 2021

greglucas commented Dec 23, 2021

QuLogic commented Dec 23, 2021

greglucas commented Dec 23, 2021

greglucas commented Dec 24, 2021

greglucas commented Jan 3, 2022

dstansby commented Jan 3, 2022

greglucas commented Jan 24, 2022

QuLogic Aug 22, 2022

Choose a reason for hiding this comment