Skip to content

[MNT]: Python 3.14.0a7 test failures #29959

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
befeleme opened this issue Apr 22, 2025 · 4 comments
Open

[MNT]: Python 3.14.0a7 test failures #29959

befeleme opened this issue Apr 22, 2025 · 4 comments
Labels
Maintenance

Comments

@befeleme
Copy link

befeleme commented Apr 22, 2025 •
edited
Loading

Summary

I ran tox on the main branch of matplotlib with the fix from #29393 applied on top. I ran the same set of tests which we run during the Fedora matplotlib build, so invocation looked like this:

tox -e py314 -- -k 'not test_invisible_Line_rendering and not test_form_widget_get_with_datetime_and_date_fields' --pyargs matplotlib mpl_toolkits.axes_grid1 mpl_toolkits.axisartist mpl_toolkits.mplot3d

And the results:

________________________________________________ test_hexbin_log[png] _________________________________________________

args = (), kwds = {'extension': 'png', 'request': <FixtureRequest for <Function test_hexbin_log[png]>>}

    @wraps(func)
    def inner(*args, **kwds):
        with self._recreate_cm():
>           return func(*args, **kwds)
E           matplotlib.testing.exceptions.ImageComparisonFailure: images not close (RMS 35.934):
E           	result_images/test_axes/hexbin_log.png
E           	result_images/test_axes/hexbin_log-expected.png
E           	result_images/test_axes/hexbin_log-failed-diff.png

/usr/lib64/python3.14/contextlib.py:85: ImageComparisonFailure
_______________________________________________ test_hexbin_linear[png] _______________________________________________

args = (), kwds = {'extension': 'png', 'request': <FixtureRequest for <Function test_hexbin_linear[png]>>}

    @wraps(func)
    def inner(*args, **kwds):
        with self._recreate_cm():
>           return func(*args, **kwds)
E           matplotlib.testing.exceptions.ImageComparisonFailure: images not close (RMS 20.427):
E           	result_images/test_axes/hexbin_linear.png
E           	result_images/test_axes/hexbin_linear-expected.png
E           	result_images/test_axes/hexbin_linear-failed-diff.png

/usr/lib64/python3.14/contextlib.py:85: ImageComparisonFailure
__________________________________________________ test_symlog2[pdf] __________________________________________________

args = (), kwds = {'extension': 'pdf', 'request': <FixtureRequest for <Function test_symlog2[pdf]>>}

    @wraps(func)
    def inner(*args, **kwds):
        with self._recreate_cm():
>           return func(*args, **kwds)
E           matplotlib.testing.exceptions.ImageComparisonFailure: images not close (RMS 12.445):
E           	result_images/test_axes/symlog2_pdf.png
E           	result_images/test_axes/symlog2-expected_pdf.png
E           	result_images/test_axes/symlog2_pdf-failed-diff.png

/usr/lib64/python3.14/contextlib.py:85: ImageComparisonFailure
_______________________________________ test_mollweide_forward_inverse_closure ________________________________________

    def test_mollweide_forward_inverse_closure():
        # test that the round-trip Mollweide forward->inverse transformation is an
        # approximate identity
        fig = plt.figure()
        ax = fig.add_subplot(projection='mollweide')
    
        # set up 1-degree grid in longitude, latitude
        lon = np.linspace(-np.pi, np.pi, 360)
        # The poles are degenerate and thus sensitive to floating point precision errors
        lat = np.linspace(-np.pi / 2.0, np.pi / 2.0, 180)[1:-1]
        lon, lat = np.meshgrid(lon, lat)
        ll = np.vstack((lon.flatten(), lat.flatten())).T
    
        # perform forward transform
>       xy = ax.transProjection.transform(ll)

/home/ksurma/dev/matplotlib/lib/matplotlib/tests/test_axes.py:5021: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/home/ksurma/dev/matplotlib/lib/matplotlib/transforms.py:1523: in transform
    res = self.transform_affine(self.transform_non_affine(values))
/home/ksurma/dev/matplotlib/lib/matplotlib/projections/geo.py:372: in transform_non_affine
    delta, large_delta = d(theta)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

theta = array([ 3.22129331e+27,  3.22129331e+27,  3.22129331e+27, ...,
       -3.22129331e+27, -3.22129331e+27, -3.22129331e+27], shape=(61200,))

    def d(theta):
>       delta = (-(theta + np.sin(theta) - pi_sin_l)
                 / (1 + np.cos(theta)))
E       RuntimeWarning: overflow encountered in divide

/home/ksurma/dev/matplotlib/lib/matplotlib/projections/geo.py:355: RuntimeWarning
_______________________________________ test_mollweide_inverse_forward_closure ________________________________________

    def test_mollweide_inverse_forward_closure():
        # test that the round-trip Mollweide inverse->forward transformation is an
        # approximate identity
        fig = plt.figure()
        ax = fig.add_subplot(projection='mollweide')
    
        # set up grid in x, y
        x = np.linspace(0, 1, 500)
        x, y = np.meshgrid(x, x)
        xy = np.vstack((x.flatten(), y.flatten())).T
    
        # perform inverse transform
        ll = ax.transProjection.inverted().transform(xy)
    
        # perform forward transform
>       xy2 = ax.transProjection.transform(ll)

/home/ksurma/dev/matplotlib/lib/matplotlib/tests/test_axes.py:5045: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/home/ksurma/dev/matplotlib/lib/matplotlib/transforms.py:1523: in transform
    res = self.transform_affine(self.transform_non_affine(values))
/home/ksurma/dev/matplotlib/lib/matplotlib/projections/geo.py:372: in transform_non_affine
    delta, large_delta = d(theta)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

theta = array([ 0.        ,  0.        ,  0.        , ..., -3.14323857,
       -3.14323857, -3.14323857], shape=(250000,))

    def d(theta):
>       delta = (-(theta + np.sin(theta) - pi_sin_l)
                 / (1 + np.cos(theta)))
E       RuntimeWarning: divide by zero encountered in divide

/home/ksurma/dev/matplotlib/lib/matplotlib/projections/geo.py:355: RuntimeWarning
______________________________________________ test_specgram_angle[png] _______________________________________________

args = (), kwds = {'extension': 'png', 'request': <FixtureRequest for <Function test_specgram_angle[png]>>}

    @wraps(func)
    def inner(*args, **kwds):
        with self._recreate_cm():
>           return func(*args, **kwds)
E           matplotlib.testing.exceptions.ImageComparisonFailure: images not close (RMS 23.460):
E           	result_images/test_axes/specgram_phase_freqs.png
E           	result_images/test_axes/specgram_phase_freqs-expected.png
E           	result_images/test_axes/specgram_phase_freqs-failed-diff.png

/usr/lib64/python3.14/contextlib.py:85: ImageComparisonFailure
__________________________________________ test_proportional_colorbars[png] ___________________________________________

args = (), kwds = {'extension': 'png', 'request': <FixtureRequest for <Function test_proportional_colorbars[png]>>}

    @wraps(func)
    def inner(*args, **kwds):
        with self._recreate_cm():
>           return func(*args, **kwds)
E           matplotlib.testing.exceptions.ImageComparisonFailure: images not close (RMS 47.976):
E           	result_images/test_colorbar/proportional_colorbars.png
E           	result_images/test_colorbar/proportional_colorbars-expected.png
E           	result_images/test_colorbar/proportional_colorbars-failed-diff.png

/usr/lib64/python3.14/contextlib.py:85: ImageComparisonFailure
_______________________________________________ test_rgb_hsv_round_trip _______________________________________________

x = array([0.5488135 , 0.71518937, 0.60276338, ..., 0.31486942, 0.95593047,
       0.76646949], shape=(750000,))
y = array([], dtype=float64)

    def compare(x, y):
        try:
            if npany(isinf(x)) or npany(isinf(y)):
                xinfid = isinf(x)
                yinfid = isinf(y)
                if not (xinfid == yinfid).all():
                    return False
                # if one item, x and y is +- inf
                if x.size == y.size == 1:
                    return x == y
                x = x[~xinfid]
                y = y[~yinfid]
        except (TypeError, NotImplementedError):
            pass
    
        # make sure y is an inexact type to avoid abs(MIN_INT); will cause
        # casting of x later.
        dtype = result_type(y, 1.)
        y = np.asanyarray(y, dtype)
>       z = abs(x - y)
E       ValueError: operands could not be broadcast together with shapes (750000,) (0,)

/home/ksurma/dev/matplotlib/.tox/py314/lib64/python3.14/site-packages/numpy/testing/_private/utils.py:1165: ValueError

During handling of the above exception, another exception occurred:

    def test_rgb_hsv_round_trip():
        for a_shape in [(500, 500, 3), (500, 3), (1, 3), (3,)]:
            np.random.seed(0)
            tt = np.random.random(a_shape)
>           assert_array_almost_equal(
                tt, mcolors.hsv_to_rgb(mcolors.rgb_to_hsv(tt)))
E           ValueError: 
E           error during assertion:
E           
E           Traceback (most recent call last):
E             File "/home/ksurma/dev/matplotlib/.tox/py314/lib64/python3.14/site-packages/numpy/testing/_private/utils.py", line 851, in assert_array_compare
E               val = comparison(x, y)
E             File "/home/ksurma/dev/matplotlib/.tox/py314/lib64/python3.14/site-packages/numpy/testing/_private/utils.py", line 1165, in compare
E               z = abs(x - y)
E                       ~~^~~
E           ValueError: operands could not be broadcast together with shapes (750000,) (0,) 
E           
E           
E           Arrays are not almost equal to 6 decimals
E            ACTUAL: array([0.548814, 0.715189, 0.602763, ..., 0.314869, 0.95593 , 0.766469],
E                 shape=(750000,))
E            DESIRED: array([], dtype=float64)

/home/ksurma/dev/matplotlib/lib/matplotlib/tests/test_colors.py:947: ValueError
_________________________________________ test_light_source_topo_surface[png] _________________________________________

args = (), kwds = {'extension': 'png', 'request': <FixtureRequest for <Function test_light_source_topo_surface[png]>>}

    @wraps(func)
    def inner(*args, **kwds):
        with self._recreate_cm():
>           return func(*args, **kwds)
E           matplotlib.testing.exceptions.ImageComparisonFailure: images not close (RMS 31.976):
E           	result_images/test_colors/light_source_shading_topo.png
E           	result_images/test_colors/light_source_shading_topo-expected.png
E           	result_images/test_colors/light_source_shading_topo-failed-diff.png

/usr/lib64/python3.14/contextlib.py:85: ImageComparisonFailure
__________________________________________________ test_labels[png] ___________________________________________________

args = (), kwds = {'extension': 'png', 'request': <FixtureRequest for <Function test_labels[png]>>}

    @wraps(func)
    def inner(*args, **kwds):
        with self._recreate_cm():
>           return func(*args, **kwds)
E           matplotlib.testing.exceptions.ImageComparisonFailure: images not close (RMS 30.608):
E           	result_images/test_contour/contour_test_label_transforms.png
E           	result_images/test_contour/contour_test_label_transforms-expected.png
E           	result_images/test_contour/contour_test_label_transforms-failed-diff.png

/usr/lib64/python3.14/contextlib.py:85: ImageComparisonFailure
___________________________________________ test_mask_image_over_under[png] ___________________________________________

args = (), kwds = {'extension': 'png', 'request': <FixtureRequest for <Function test_mask_image_over_under[png]>>}

    @wraps(func)
    def inner(*args, **kwds):
        with self._recreate_cm():
>           return func(*args, **kwds)
E           matplotlib.testing.exceptions.ImageComparisonFailure: images not close (RMS 41.585):
E           	result_images/test_image/mask_image_over_under.png
E           	result_images/test_image/mask_image_over_under-expected.png
E           	result_images/test_image/mask_image_over_under-failed-diff.png

/usr/lib64/python3.14/contextlib.py:85: ImageComparisonFailure
______________________________________________ test_rgba_antialias[png] _______________________________________________

args = (), kwds = {'extension': 'png', 'request': <FixtureRequest for <Function test_rgba_antialias[png]>>}

    @wraps(func)
    def inner(*args, **kwds):
        with self._recreate_cm():
>           return func(*args, **kwds)
E           matplotlib.testing.exceptions.ImageComparisonFailure: images not close (RMS 59.938):
E           	result_images/test_image/rgba_antialias.png
E           	result_images/test_image/rgba_antialias-expected.png
E           	result_images/test_image/rgba_antialias-failed-diff.png

/usr/lib64/python3.14/contextlib.py:85: ImageComparisonFailure
_______________________________________________ test_downsampling[png] ________________________________________________

args = (), kwds = {'extension': 'png', 'request': <FixtureRequest for <Function test_downsampling[png]>>}

    @wraps(func)
    def inner(*args, **kwds):
        with self._recreate_cm():
>           return func(*args, **kwds)
E           matplotlib.testing.exceptions.ImageComparisonFailure: images not close (RMS 71.138):
E           	result_images/test_image/downsampling.png
E           	result_images/test_image/downsampling-expected.png
E           	result_images/test_image/downsampling-failed-diff.png

/usr/lib64/python3.14/contextlib.py:85: ImageComparisonFailure
_______________________________________________ test_quiver_memory_leak _______________________________________________

    @pytest.mark.skipif(platform.python_implementation() != 'CPython',
                        reason='Requires CPython')
    def test_quiver_memory_leak():
        fig, ax = plt.subplots()
    
        Q = draw_quiver(ax)
        ttX = Q.X
        Q.remove()
    
        del Q
    
>       assert sys.getrefcount(ttX) == 2
E       AssertionError

/home/ksurma/dev/matplotlib/lib/matplotlib/tests/test_quiver.py:33: AssertionError
_____________________________________________ test_quiver_key_memory_leak _____________________________________________

    @pytest.mark.skipif(platform.python_implementation() != 'CPython',
                        reason='Requires CPython')
    def test_quiver_key_memory_leak():
        fig, ax = plt.subplots()
    
        Q = draw_quiver(ax)
    
        qk = ax.quiverkey(Q, 0.5, 0.92, 2, r'$2 \frac{m}{s}$',
                          labelpos='W',
                          fontproperties={'weight': 'bold'})
>       assert sys.getrefcount(qk) == 3
E       AssertionError

/home/ksurma/dev/matplotlib/lib/matplotlib/tests/test_quiver.py:46: AssertionError
_________________________________________________ test_fft_peaks[png] _________________________________________________

    @image_comparison(['fft_peaks'], remove_text=True)
    def test_fft_peaks():
        fig, ax = plt.subplots()
        t = np.arange(65536)
        p1 = ax.plot(abs(np.fft.fft(np.sin(2*np.pi*.01*t)*np.blackman(len(t)))))
    
        # Ensure that the path's transform takes the new axes limits into account.
        fig.canvas.draw()
        path = p1[0].get_path()
        transform = p1[0].get_transform()
        path = transform.transform_path(path)
        simplified = path.cleaned(simplify=True)
    
>       assert simplified.vertices.size == 36
E       AssertionError

/home/ksurma/dev/matplotlib/lib/matplotlib/tests/test_simplification.py:421: AssertionError
_________________________________________________ test_fft_peaks[pdf] _________________________________________________

    @image_comparison(['fft_peaks'], remove_text=True)
    def test_fft_peaks():
        fig, ax = plt.subplots()
        t = np.arange(65536)
        p1 = ax.plot(abs(np.fft.fft(np.sin(2*np.pi*.01*t)*np.blackman(len(t)))))
    
        # Ensure that the path's transform takes the new axes limits into account.
        fig.canvas.draw()
        path = p1[0].get_path()
        transform = p1[0].get_transform()
        path = transform.transform_path(path)
        simplified = path.cleaned(simplify=True)
    
>       assert simplified.vertices.size == 36
E       AssertionError

/home/ksurma/dev/matplotlib/lib/matplotlib/tests/test_simplification.py:421: AssertionError
_________________________________________________ test_fft_peaks[svg] _________________________________________________

    @image_comparison(['fft_peaks'], remove_text=True)
    def test_fft_peaks():
        fig, ax = plt.subplots()
        t = np.arange(65536)
        p1 = ax.plot(abs(np.fft.fft(np.sin(2*np.pi*.01*t)*np.blackman(len(t)))))
    
        # Ensure that the path's transform takes the new axes limits into account.
        fig.canvas.draw()
        path = p1[0].get_path()
        transform = p1[0].get_transform()
        path = transform.transform_path(path)
        simplified = path.cleaned(simplify=True)
    
>       assert simplified.vertices.size == 36
E       AssertionError

/home/ksurma/dev/matplotlib/lib/matplotlib/tests/test_simplification.py:421: AssertionError
___________________________________________ test_tri_smooth_contouring[png] ___________________________________________

args = (), kwds = {'extension': 'png', 'request': <FixtureRequest for <Function test_tri_smooth_contouring[png]>>}

    @wraps(func)
    def inner(*args, **kwds):
        with self._recreate_cm():
>           return func(*args, **kwds)
E           matplotlib.testing.exceptions.ImageComparisonFailure: images not close (RMS 123.603):
E           	result_images/test_triangulation/tri_smooth_contouring.png
E           	result_images/test_triangulation/tri_smooth_contouring-expected.png
E           	result_images/test_triangulation/tri_smooth_contouring-failed-diff.png

/usr/lib64/python3.14/contextlib.py:85: ImageComparisonFailure

FAILED tests/test_axes.py::test_hexbin_log[png] - matplotlib.testing.exceptions.ImageComparisonFailure: images not close (RMS 35.934):
FAILED tests/test_axes.py::test_hexbin_linear[png] - matplotlib.testing.exceptions.ImageComparisonFailure: images not close (RMS 20.427):
FAILED tests/test_axes.py::test_symlog2[pdf] - matplotlib.testing.exceptions.ImageComparisonFailure: images not close (RMS 12.445):
FAILED tests/test_axes.py::test_mollweide_forward_inverse_closure - RuntimeWarning: overflow encountered in divide
FAILED tests/test_axes.py::test_mollweide_inverse_forward_closure - RuntimeWarning: divide by zero encountered in divide
FAILED tests/test_axes.py::test_specgram_angle[png] - matplotlib.testing.exceptions.ImageComparisonFailure: images not close (RMS 23.460):
FAILED tests/test_colorbar.py::test_proportional_colorbars[png] - matplotlib.testing.exceptions.ImageComparisonFailure: images not close (RMS 47.976):
FAILED tests/test_colors.py::test_rgb_hsv_round_trip - ValueError: 
FAILED tests/test_colors.py::test_light_source_topo_surface[png] - matplotlib.testing.exceptions.ImageComparisonFailure: images not close (RMS 31.976):
FAILED tests/test_contour.py::test_labels[png] - matplotlib.testing.exceptions.ImageComparisonFailure: images not close (RMS 30.608):
FAILED tests/test_image.py::test_mask_image_over_under[png] - matplotlib.testing.exceptions.ImageComparisonFailure: images not close (RMS 41.585):
FAILED tests/test_image.py::test_rgba_antialias[png] - matplotlib.testing.exceptions.ImageComparisonFailure: images not close (RMS 59.938):
FAILED tests/test_image.py::test_downsampling[png] - matplotlib.testing.exceptions.ImageComparisonFailure: images not close (RMS 71.138):
FAILED tests/test_quiver.py::test_quiver_memory_leak - AssertionError
FAILED tests/test_quiver.py::test_quiver_key_memory_leak - AssertionError
FAILED tests/test_simplification.py::test_fft_peaks[png] - AssertionError
FAILED tests/test_simplification.py::test_fft_peaks[pdf] - AssertionError
FAILED tests/test_simplification.py::test_fft_peaks[svg] - AssertionError
FAILED tests/test_triangulation.py::test_tri_smooth_contouring[png] - matplotlib.testing.exceptions.ImageComparisonFailure: images not close (RMS 123.603):
================= 19 failed, 9423 passed, 507 skipped, 2 deselected, 30 xfailed in 330.41s (0:05:30) ==================

Proposed fix

No response
@ngoldbaum
Copy link
Contributor

FYI, a NumPy optimization that elides temporaries is broken on 3.14. Probably best to wait until numpy/numpy#28748 (comment) is merged to work on downstream libraries, otherwise you'll be dealing with deeply confusing test failures.

@befeleme
Copy link
Author

FYI, I jumped to test matplotlib on numpy 2.2.5 with that PR patched on top - it fixed most of the problems, remaining ones seem to be genuine for matplotlib:

___________________________ test_quiver_memory_leak ____________________________
[gw0] linux -- Python 3.14.0 /usr/bin/python3

    @pytest.mark.skipif(platform.python_implementation() != 'CPython',
                        reason='Requires CPython')
    def test_quiver_memory_leak():
        fig, ax = plt.subplots()
    
        Q = draw_quiver(ax)
        ttX = Q.X
        Q.remove()
    
        del Q
    
>       assert sys.getrefcount(ttX) == 2
E       assert 1 == 2
E        +  where 1 = <built-in function getrefcount>(array([0, 1, 2, 3, 4, 5, 6, 0, 1, 2, 3, 4, 5, 6, 0, 1, 2, 3, 4, 5, 6, 0,\n       1, 2, 3, 4, 5, 6, 0, 1, 2, 3, 4, 5, 6, 0, 1, 2, 3, 4, 5, 6, 0, 1,\n       2, 3, 4, 5, 6]))
E        +    where <built-in function getrefcount> = sys.getrefcount

../BUILDROOT/usr/lib64/python3.14/site-packages/matplotlib/tests/test_quiver.py:33: AssertionError
_________________________ test_quiver_key_memory_leak __________________________
[gw0] linux -- Python 3.14.0 /usr/bin/python3

    @pytest.mark.skipif(platform.python_implementation() != 'CPython',
                        reason='Requires CPython')
    def test_quiver_key_memory_leak():
        fig, ax = plt.subplots()
    
        Q = draw_quiver(ax)
    
        qk = ax.quiverkey(Q, 0.5, 0.92, 2, r'$2 \frac{m}{s}$',
                          labelpos='W',
                          fontproperties={'weight': 'bold'})
>       assert sys.getrefcount(qk) == 3
E       assert 2 == 3
E        +  where 2 = <built-in function getrefcount>(<matplotlib.quiver.QuiverKey object at 0x7f7fc4262120>)
E        +    where <built-in function getrefcount> = sys.getrefcount

@tacaswell
Copy link
Member

It looks like the refcount is coming in one better than expected (I suspect this is due to an optimization in refcounting where it changes using "borrow" bytecodes when it decides it is safe so the refcount inside of getrefcount() is not added), we probably need to make the expected number python dependent.

@ngoldbaum
Copy link
Contributor

In NumPy I handled updating tests that depend on refcounts to check the change in the refcount rather than a constant refcount value. That should be the same in all Python versions. There was only one other obscure case in NumPy where that didn’t work because we happened to trigger another new refcount optimization on top of the stackref optimization.

@timhoffm timhoffm mentioned this issue Apr 26, 2025
Merged
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Maintenance
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@tacaswell @ngoldbaum @befeleme