Matplotlib version: 3.1.3
Other libraries: pytest 5.4.3

You need to update Matplotlib, or downgrade pytest.

@QuLogic
Thanks, with mpl 3.3.0 works fine, with 3.2 does not work.

I still have an issue with testing. Tests work fine when I run them one by one, but when I run pytest on a whole folder, each following test remembers about previously opened figures, and tries to compare all of them with the saved baseline image.

Should I implicitly close all the opened figures in the beginning of the test with plt.close('all')? This works...
Input:

from matplotlib.testing.decorators import image_comparison
from matplotlib import pyplot as plt

@image_comparison(baseline_images=['line_dashes1'], remove_text=True, extensions=['png'])
def test_line_dashes1():
    fig, ax = plt.subplots()
    ax.plot(range(10), linestyle=(0, (3, 3)), lw=5)

@image_comparison(baseline_images=['line_dashes2'], remove_text=True, extensions=['png'])
def test_line_dashes2():
    fig, ax = plt.subplots()
    ax.plot(range(10), linestyle=(0, (3, 3)), lw=5)

Output of pytest on this file:

============================= test session starts =============================
platform win32 -- Python 3.7.7, pytest-5.4.3, py-1.9.0, pluggy-0.13.1 -- C:\Users\smirn\.conda\envs\gofish\python.exe
cachedir: .pytest_cache
rootdir: D:\astrowork\prodige\imager
collecting ... collected 2 items

test_mpl_sanity.py::test_line_dashes1[png] 
test_mpl_sanity.py::test_line_dashes2[png] PASSED                        [ 50%]FAILED                        [100%]
test\test_mpl_sanity.py:8 (test_line_dashes2[png])
extension = 'png'
request = <FixtureRequest for <Function test_line_dashes2[png]>>, args = ()
kwargs = {}, __tracebackhide__ = True
img = <matplotlib.testing.decorators._ImageComparisonBase object at 0x000001BD9B0F0BC8>
needs_lock = False, our_baseline_images = ['line_dashes2']

    @functools.wraps(func)
    @pytest.mark.parametrize('extension', extensions)
    @pytest.mark.style(style)
    @_checked_on_freetype_version(freetype_version)
    @functools.wraps(func)
    def wrapper(*args, extension, request, **kwargs):
        __tracebackhide__ = True
        if 'extension' in old_sig.parameters:
            kwargs['extension'] = extension
        if 'request' in old_sig.parameters:
            kwargs['request'] = request
    
        img = _ImageComparisonBase(func, tol=tol, remove_text=remove_text,
                                   savefig_kwargs=savefig_kwargs)
        matplotlib.testing.set_font_settings_for_testing()
        func(*args, **kwargs)
    
        # If the test is parametrized in any way other than applied via
        # this decorator, then we need to use a lock to prevent two
        # processes from touching the same output file.
        needs_lock = any(
            marker.args[0] != 'extension'
            for marker in request.node.iter_markers('parametrize'))
    
        if baseline_images is not None:
            our_baseline_images = baseline_images
        else:
            # Allow baseline image list to be produced on the fly based on
            # current parametrization.
            our_baseline_images = request.getfixturevalue(
                'baseline_images')
    
        assert len(plt.get_fignums()) == len(our_baseline_images), (
            "Test generated {} images but there are {} baseline images"
>           .format(len(plt.get_fignums()), len(our_baseline_images)))
E       AssertionError: Test generated 2 images but there are 1 baseline images

C:\Users\smirn\.conda\envs\gofish\lib\site-packages\matplotlib\testing\decorators.py:280: AssertionError

Assertion failed

Assertion failed

Assertion failed

Assertion failed


================================== FAILURES ===================================
___________________________ test_line_dashes2[png] ____________________________

extension = 'png'
request = <FixtureRequest for <Function test_line_dashes2[png]>>, args = ()
kwargs = {}, __tracebackhide__ = True
img = <matplotlib.testing.decorators._ImageComparisonBase object at 0x000001BD9B0F0BC8>
needs_lock = False, our_baseline_images = ['line_dashes2']

    @functools.wraps(func)
    @pytest.mark.parametrize('extension', extensions)
    @pytest.mark.style(style)
    @_checked_on_freetype_version(freetype_version)
    @functools.wraps(func)
    def wrapper(*args, extension, request, **kwargs):
        __tracebackhide__ = True
        if 'extension' in old_sig.parameters:
            kwargs['extension'] = extension
        if 'request' in old_sig.parameters:
            kwargs['request'] = request
    
        img = _ImageComparisonBase(func, tol=tol, remove_text=remove_text,
                                   savefig_kwargs=savefig_kwargs)
        matplotlib.testing.set_font_settings_for_testing()
        func(*args, **kwargs)
    
        # If the test is parametrized in any way other than applied via
        # this decorator, then we need to use a lock to prevent two
        # processes from touching the same output file.
        needs_lock = any(
            marker.args[0] != 'extension'
            for marker in request.node.iter_markers('parametrize'))
    
        if baseline_images is not None:
            our_baseline_images = baseline_images
        else:
            # Allow baseline image list to be produced on the fly based on
            # current parametrization.
            our_baseline_images = request.getfixturevalue(
                'baseline_images')
    
        assert len(plt.get_fignums()) == len(our_baseline_images), (
            "Test generated {} images but there are {} baseline images"
>           .format(len(plt.get_fignums()), len(our_baseline_images)))
E       AssertionError: Test generated 2 images but there are 1 baseline images

C:\Users\smirn\.conda\envs\gofish\lib\site-packages\matplotlib\testing\decorators.py:280: AssertionError
============================== warnings summary ===============================
C:\Users\smirn\.conda\envs\gofish\lib\site-packages\matplotlib\testing\decorators.py:248
  C:\Users\smirn\.conda\envs\gofish\lib\site-packages\matplotlib\testing\decorators.py:248: PytestUnknownMarkWarning: Unknown pytest.mark.style - is this a typo?  You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/latest/mark.html
    @pytest.mark.style(style)

-- Docs: https://docs.pytest.org/en/latest/warnings.html
=========================== short test summary info ===========================
FAILED test_mpl_sanity.py::test_line_dashes2[png] - AssertionError: Test gene...
=================== 1 failed, 1 passed, 1 warning in 1.14s ====================

Process finished with exit code 1

Assertion failed

@SmirnGreg perhaps discuss this at discourse.matplotlib.org ?

Dear @jklymak

I can discourse that on stack overflow or on some other forum as well, but I believe there is a severe issue in the image testing within matplotlib that might result into the library code change.

Let me explain my point. Matplotlib is the most heavily used plotting library in the Python community. In science and data science, it is definitely the most used. Groups of scientists rely on matplotlib, and since matplotlib turned to version 3 and dropped the python2 legacy, the community may demand stability. Current industry standard, as well as for serious scientific development, includes writing tests for the code, especially when using the rapidly changing frameworks. Matplotlib is trying to provide some testing environment, such as decorator image_comparison, and this is great. But the issue that forbids sequential runs of multiple test cases is here.

Currently, the documentation for image_comparison states that

Compare images generated by the test with those specified in
*baseline_images*, which must correspond, else an `ImageComparisonFailure`
exception will be raised.

But it is not comparing the images generated by test, rather all open figures.

As far as I can see (did not go to the source code yet), the decorator tries to compare all currently opened figures with the saved one. At the same time, the previous test leaves the figure open. I can propose three solutions for that:

1. The decorator first remembers all opened figures, then runs the test, then only compares the newly created figures with the saved images.
2. The decorator first remembers all opened figures, then runs the test, then compares all the currently opened figures, then closes the new ones.
3. Add plt.close('all') into the corresponding documentation example and change the docstring of image_comparison.

I am ready to implement one of the above if this PR is likely to be accepted, maybe with the support from the core team if possible.

Best wishes,
Greg.

I suggest discourse because I think you need help setting up your testing environment, and this is not the appropriate venue for that.

The way the test the work they have to collect all of the currently open figures (one could argue we should be returning the tuple of the figures we want tested but that is neither here nor there). The way we cope with this in our test suite (which uses these tools) is via

which is auto-used for all of the tests and takes care of making sure any global state that is touched is cleaned up and (implicitly) closing all of the figures via plt.switch_backend.

I think the fix here is to document importing that fixture into your conftest.py

Dear @tacaswell,

thanks a lot, that is what I was looking for! It seems to me that when testing not the matplotlib itself, but own packages, the following import makes everything fine (as the fixture is applied to every test case):

from matplotlib.testing.conftest import mpl_test_settings

No further setup is required with latest pip releases of pytest and matplotlib.

@SmirnGreg Where would you have expected to find that information (and can you open a PR putting it there?) ?

I'm wondering if our testing functions are intended to be public for the end user. How does our internal testing compare to pytest-mpl? A pytest plugin seems to be the cleaner approach compared to importing from conftest.

I would 100% say everyone else should use pytest-mpl and not matplotlib's own testing code. And honestly, if we had cycles to burn, I'd say we should migrate matplotlib itself to pytest-mpl.

@timhoffm @dopplershift
thank you for the discussion!

1. I wanted to find a way to write a test for the output figures I plot with matplotlib. I went to the matplotlib.org and typed "test" in the search box, and the first two link point to the matplotlib.testing package. You are right, it is written there that it is for developers, but users also might be interested in such functionality. Also, matplotlib.testing is provided alongside the main matplotlib package not only in the developer version. I assumed that it is a valid way for testing and tries to run an example, an failed.
2. 'pytest-mpl' has some issues, namely:
   a. It still depends on matplotlib.testing.compare.compare_images. It has some wrapper around it, but..
   b. I do find the original testing decorators more convenient. First, and the most important, I like the way they store the figures in the folder named after the file with the test case.
   c. I also like that the decorator saves the output figure even if the tolerance test has passed -- that is helpful if the tolerance value was set to high, but I still want to check by eye time to time.
   d. Its current release version has a leading 0. Does that mean that the API can change dramatically over time? I wanted a more stable solution.
   e. The decorator API is much richer than that of pytest-mpl.

Based on the comment by @tacaswell, I would propose one or more of the following:

• Add a note on testing that pytest-mpl is suggested for testing.
• Add a note here that this is how the test should be written in the matplotlib testing environment, and that one can import a fixture from matplotlib.testing.conftest import mpl_test_settings to make it close the figure after the test and roll back to the initial state.
• Add the similar note as previous in the decorator docstring
• Maybe it is better to add this (and other important) fixture being imported within the decorator, so it works as in the example without the need in changing the docs? I don't have an idea about which side effects this could have on the existing tests, maybe it is better to not touch that.

What would you say?

I'd go with the three documentation bullets.

@SmirnGreg just to be clear, adding

from matplotlib.testing.conftest import mpl_test_settings

fixes the problem with the tests failing? If so, I'll go ahead and add in this line to the example in the docs as well, in the places that you suggested.

Add a note on testing that pytest-mpl is suggested for testing.
Add a note here that this is how the test should be written in the matplotlib testing environment, and that one can import a ? fixture from matplotlib.testing.conftest import mpl_test_settings to make it close the figure after the test and roll back to the initial state.
Add the similar note as previous in the decorator docstring

@timhoffm Would that be alright?

Um, not the pytest expert here, but I think it's sufficient to have mpl_test_settings imported in one conftest.py per project. Since that's already done in the matplotlib codebase and matplotlib.testing is not to be used by external users, no documentation is needed concerning the import.

Maybe it is better to add this (and other important) fixture being imported within the decorator, so it works as in the example without the need in changing the docs? I don't have an idea about which side effects this could have on the existing tests, maybe it is better to not touch that.

This won't work as pytest needs to find the fixture before it finds the test. Might make sense to add a plt.close('all') to the test decorator though?

but I think it's sufficient to have mpl_test_settings imported in one conftest.py per project.

Yes, pytest can find fixtures either in a conftest.py or at the module level. If you want it everywhere in a package, put it in conftest.py, if you only want it in a single test module import it at the module level. Either will work, depends on how aggressively you want to reset Matplotlib's global state.

For better-or-worse I think users have discovered our testing module and are using it so we should document it, however I would not add it to the example as the example is about how to write tests for our test suite.

I do think that pytest-mpl is a better option (and I agree with @dopplershift that if we had effort we would migrate to depending on them).