Skip to content

check_figures_equal regression from 3.2.0 to 3.2.1 #16916

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
andrewzwicky opened this issue Mar 26, 2020 · 2 comments · Fixed by #16940
Closed

check_figures_equal regression from 3.2.0 to 3.2.1 #16916

andrewzwicky opened this issue Mar 26, 2020 · 2 comments · Fixed by #16940
Labels
Documentation
Milestone
v3.2.2

Comments

@andrewzwicky
Copy link

Bug report

Bug summary

Previously, tests using check_figures_equal worked as expected on V3.2.0. When upgrading to V3.2.1, with no other changes to system, tests now fail with the error message:

file c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\matplotlib\testing\decorators.py, line 393
          @pytest.mark.parametrize("ext", extensions)
          def wrapper(*args, **kwargs):
E       fixture 'test_figure' not found
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, get_preparsed_fingerprint_game, get_preparsed_timeline_games, get_test_events_folder, get_test_events_folder_in_progress, get_test_json_games_folder, get_test_replay_pickle_folder, get_test_unparsed_folder, get_unparsed_test_games, monkeypatch, no_cover, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory, worker_id
>       use 'pytest --fixtures [testpath]' for help on them.

Code for reproduction

@check_figures_equal(extensions=["png"])
def test_mpl_321(test_figure, reference_figure):

    reference_figure.set_size_inches(8, 8)
    ref_ax = reference_figure.subplots()
    ref_ax.set_title("Test Title")
    ref_ax.pie(
        [1, 2, 3, 4, 4],
        labels=["A", "B", "C", "D", "E"],
    )


    test_figure.set_size_inches(8, 8)
    test_ax = test_figure.subplots()
    test_ax.set_title("Test Title")
    test_ax.pie(
        [1, 2, 3, 4, 4],
        labels=["A", "B", "C", "D", "E"],
    )

Actual outcome

This is the console output showing:

  1. The test failing with V3.2.1
  2. Uninstall V3.2.1 and installing V3.2.0
  3. The test passing.
PS C:\Users\Andrew\Workspace\TripleAgent> pytest -k test_mpl_321
============================================================================================================================ test session starts =============================================================================================================================
platform win32 -- Python 3.7.5, pytest-4.5.0, py-1.8.0, pluggy-0.9.0
rootdir: C:\Users\Andrew\Workspace\TripleAgent, inifile: setup.cfg
plugins: xdist-1.29.0, forked-1.0.2, cov-2.7.1
collected 1223 items / 1222 deselected / 1 selected

triple_agent\tests\test_plot_types.py E                                                                                                                                                                                                                                 [100%]

=================================================================================================================================== ERRORS ===================================================================================================================================
____________________________________________________________________________________________________________________ ERROR at setup of test_mpl_321[png] _____________________________________________________________________________________________________________________
file c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\matplotlib\testing\decorators.py, line 393
          @pytest.mark.parametrize("ext", extensions)
          def wrapper(*args, **kwargs):
E       fixture 'test_figure' not found
>       available fixtures: cache, capfd, capfdbinary, caplog, capsys, capsysbinary, cov, doctest_namespace, get_preparsed_fingerprint_game, get_preparsed_timeline_games, get_test_events_folder, get_test_events_folder_in_progress, get_test_json_games_folder, get_test_replay_pickle_folder, get_test_unparsed_folder, get_unparsed_test_games, monkeypatch, no_cover, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory, worker_id
>       use 'pytest --fixtures [testpath]' for help on them.

c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\matplotlib\testing\decorators.py:393
============================================================================================================================== warnings summary ==============================================================================================================================
c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\nbformat\notebooknode.py:4
  c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\nbformat\notebooknode.py:4: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
    from collections import Mapping

c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\jinja2\utils.py:485
  c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\jinja2\utils.py:485: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
    from collections import MutableMapping

c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\nbconvert\exporters\exporter_locator.py:28
  c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\nbconvert\exporters\exporter_locator.py:28: DeprecationWarning: `nbconvert.exporters.exporter_locator` is deprecated in favor of `nbconvert.exporters.base` since nbconvert 5.0.
    DeprecationWarning)

-- Docs: https://docs.pytest.org/en/latest/warnings.html
=========================================================================================================== 1222 deselected, 3 warnings, 1 error in 17.47 seconds ============================================================================================================

Uninstall V3.2.1 -> Install V.3.2.0

PS C:\Users\Andrew\Workspace\TripleAgent> pip uninstall matplotlib
Found existing installation: matplotlib 3.2.1
Uninstalling matplotlib-3.2.1:
  Would remove:
    c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\matplotlib-3.2.1-py3.7-nspkg.pth
    c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\matplotlib-3.2.1.dist-info\*
    c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\matplotlib\*
    c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\mpl_toolkits\axes_grid1\*
    c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\mpl_toolkits\axes_grid\*
    c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\mpl_toolkits\axisartist\*
    c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\mpl_toolkits\mplot3d\*
    c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\mpl_toolkits\tests\*
    c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\pylab.py
Proceed (y/n)? y
  Successfully uninstalled matplotlib-3.2.1
PS C:\Users\Andrew\Workspace\TripleAgent> pip install matplotlib==3.2.0
Collecting matplotlib==3.2.0
  Using cached matplotlib-3.2.0-cp37-cp37m-win_amd64.whl (9.2 MB)
Requirement already satisfied: numpy>=1.11 in c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages (from matplotlib==3.2.0) (1.16.3)
Requirement already satisfied: python-dateutil>=2.1 in c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages (from matplotlib==3.2.0) (2.8.0)
Requirement already satisfied: kiwisolver>=1.0.1 in c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages (from matplotlib==3.2.0) (1.0.1)
Requirement already satisfied: cycler>=0.10 in c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages (from matplotlib==3.2.0) (0.10.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages (from matplotlib==3.2.0) (2.4.0)
Requirement already satisfied: six>=1.5 in c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages (from python-dateutil>=2.1->matplotlib==3.2.0) (1.12.0)
Requirement already satisfied: setuptools in c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages (from kiwisolver>=1.0.1->matplotlib==3.2.0) (40.8.0)
Installing collected packages: matplotlib
Successfully installed matplotlib-3.2.0

PS C:\Users\Andrew\Workspace\TripleAgent> pytest -k test_mpl_321
============================================================================================================================ test session starts =============================================================================================================================
platform win32 -- Python 3.7.5, pytest-4.5.0, py-1.8.0, pluggy-0.9.0
rootdir: C:\Users\Andrew\Workspace\TripleAgent, inifile: setup.cfg
plugins: xdist-1.29.0, forked-1.0.2, cov-2.7.1
collected 1223 items / 1222 deselected / 1 selected

triple_agent\tests\test_plot_types.py .                                                                                                                                                                                                                                 [100%]

============================================================================================================================== warnings summary ==============================================================================================================================
c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\nbformat\notebooknode.py:4
  c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\nbformat\notebooknode.py:4: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
    from collections import Mapping

c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\jinja2\utils.py:485
  c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\jinja2\utils.py:485: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
    from collections import MutableMapping

c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\nbconvert\exporters\exporter_locator.py:28
  c:\users\andrew\appdata\local\programs\python\python37\lib\site-packages\nbconvert\exporters\exporter_locator.py:28: DeprecationWarning: `nbconvert.exporters.exporter_locator` is deprecated in favor of `nbconvert.exporters.base` since nbconvert 5.0.
    DeprecationWarning)

-- Docs: https://docs.pytest.org/en/latest/warnings.html
=========================================================================================================== 1 passed, 1222 deselected, 3 warnings in 20.79 seconds ===========================================================================================================

Expected outcome

Tests that were previously working are not expected to start failing on version update? I might be missing something, but I didn't see anything in update notes about needing to refactor tests.

Matplotlib version

  • Operating system: WIN10
  • Matplotlib version: V3.2.1
  • Matplotlib backend (print(matplotlib.get_backend())): TkAgg
  • Python version: 3.7.5
  • Jupyter version (if applicable):
  • Other libraries:
absl-py               0.7.1
Adafruit-GPIO         1.0.3
Adafruit-PureIO       0.2.3
apipkg                1.5
appdirs               1.4.3
astor                 0.7.1
astroid               2.2.5
atomicwrites          1.3.0
attrs                 19.1.0
backcall              0.1.0
beautifulsoup4        4.7.1
black                 19.10b0
bleach                3.1.0
bs4                   0.0.1
certifi               2018.11.29
chardet               3.0.4
Click                 7.0
colorama              0.4.1
colored               1.3.93
coverage              4.5.3
cycler                0.10.0
decorator             4.3.0
defusedxml            0.5.0
entrypoints           0.3
execnet               1.6.0
future                0.17.1
gast                  0.2.2
grpcio                1.20.1
gym                   0.12.1
h5py                  2.9.0
idna                  2.8
ipykernel             5.1.0
ipython               7.2.0
ipython-genutils      0.2.0
ipywidgets            7.4.2
iso-639               0.4.5
iso3166               0.9
isodate               0.6.0
isort                 4.3.17
jedi                  0.13.2
Jinja2                2.10.1
json5                 0.9.3
jsonpickle            1.2
jsonschema            3.0.1
jupyter-client        5.2.4
jupyter-console       6.0.0
jupyter-core          4.4.0
jupyterlab            2.0.1
jupyterlab-server     1.0.7
Keras                 2.2.4
Keras-Applications    1.0.7
Keras-Preprocessing   1.0.9
kiwisolver            1.0.1
lazy-object-proxy     1.3.1
lxml                  4.3.2
Markdown              3.1
MarkupSafe            1.1.1
matplotlib            3.2.0
mccabe                0.6.1
mistune               0.8.4
mock                  2.0.0
more-itertools        7.0.0
mss                   4.0.3
nbconvert             5.4.1
nbformat              4.4.0
nose                  1.3.7
notebook              5.7.8
numpy                 1.16.3
opencv-contrib-python 4.0.0.21
opencv-python         4.1.0.25
pandas                0.25.0
pandocfilters         1.4.2
parso                 0.3.1
pathspec              0.6.0
pbr                   5.2.0
pickleshare           0.7.5
Pillow                5.4.1
pip                   20.0.2
pluggy                0.9.0
prometheus-client     0.6.0
prompt-toolkit        2.0.7
protobuf              3.7.1
py                    1.8.0
py-spy                0.1.10
PyAutoGUI             0.9.42
pycryptodome          3.7.2
PyGetWindow           0.0.4
pyglet                1.3.2
Pygments              2.3.1
pylint                2.3.1
PyMsgBox              1.0.6
pyparsing             2.4.0
pypiwin32             223
PyRect                0.1.4
pyrsistent            0.14.11
PyScreeze             0.1.20
PySocks               1.6.8
pytesseract           0.2.6
pytest                4.5.0
pytest-cov            2.7.1
pytest-forked         1.0.2
pytest-runner         4.4
pytest-xdist          1.29.0
python-dateutil       2.8.0
PyTweening            1.0.3
pytz                  2019.2
pywin32               224
pywinpty              0.5.5
PyYAML                5.1
pyzmq                 18.0.1
qtconsole             4.4.3
regex                 2019.8.19
requests              2.21.0
scipy                 1.3.0
selenium              3.141.0
Send2Trash            1.5.0
setuptools            40.8.0
simplejson            3.17.0
six                   1.12.0
snakeviz              2.0.0
soupsieve             1.8
spotipy               2.4.4
streamlink            1.3.0
tabulate              0.8.3
tensorboard           1.13.1
tensorflow            1.13.1
tensorflow-estimator  1.13.0
termcolor             1.1.0
terminado             0.8.2
testpath              0.4.2
toml                  0.10.0
tornado               6.0.4
traitlets             4.3.2
typed-ast             1.4.0
urllib3               1.24.1
vulture               1.0
wcwidth               0.1.7
webencodings          0.5.1
websocket-client      0.54.0
Werkzeug              0.15.2
wheel                 0.33.1
widgetsnbextension    3.4.2
wrapt                 1.11.1
@tacaswell
Copy link
Member

tacaswell commented Mar 26, 2020 •
edited by timhoffm
Loading

As part of #16795 (which was backport of #15199) we made it possible to stack other fixtures onto functions decorated wiith check_figures_equal but it changed to passing the figures in as keywords (and doing a bit of signature surgery on the wrapper function to let pytest find and feed it the required other fixtures.

We could tweak that surgery a bit and go back to passing the figures as positional, but that introduces a miss-match between how all of the other pytest fixtures work. I think is is better to have special names rather than special positions.

If you change the names to fig_test and fig_ref it should work on all versions of Matplotlib and we should document that.

@tacaswell tacaswell added this to the 3.2-doc milestone Mar 26, 2020
@andrewzwicky
Copy link
Author

That's correct, just tried with those renames and it works! Thanks for the quick reply.

@tacaswell tacaswell modified the milestones: 3.2-doc, v3.2.2 Mar 26, 2020
andrewzwicky added a commit to andrewzwicky/TripleAgent that referenced this issue Mar 26, 2020
@andrewzwicky
Resolve issue with matplotlib V3.2.1
e8a1579 
matplotlib/matplotlib#16916

fixes #13
tacaswell added a commit to tacaswell/matplotlib that referenced this issue Mar 27, 2020
@tacaswell
DOC/FIX: clarify the docs for check_figures_equal
d2328fe 
Clarify that the names matter and add a check with a better exception.

closes matplotlib#16916
@tacaswell tacaswell mentioned this issue Mar 27, 2020
Merged
4 tasks
tacaswell added a commit to tacaswell/matplotlib that referenced this issue Mar 30, 2020
@tacaswell
DOC/FIX: clarify the docs for check_figures_equal
b0d63b5 
Clarify that the names matter and add a check with a better exception.

closes matplotlib#16916
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Documentation
Projects
None yet
Milestone
v3.2.2
Development

Successfully merging a pull request may close this issue.

DOC/FIX: clarify the docs for check_figures_equal tacaswell/matplotlib
2 participants
@tacaswell @andrewzwicky