😢 This creates an inconsistent environment with pytest-5.4.1 and pytest-rerunfailures-8.0. pytest-5.4 compatibility was only provided via pytest-rerunfailures-9.0, which is not yet in conda forge. Hope this is a one-time thing (combination of pytest breaking change and plugins not updating on conda-forge). If this happens more often we'd need to install some things via pip.

Waiting for now if there is an update on conda-forge. Alternatively, we could also pin pytest<5.4.

For now, I'm pinning to pytest<5.4. pytest-rerunfailures 9.0 has been out for 3 weeks, but the conda-forge package hasn't been updated yet. So I'm not betting this will be updated in the next couple of days.

Removed pytest pinning as pytest-rerunfailures 9.0 is now available from conda-forge

I had a go with this as suggested at #6418 (comment). Following the instructions in the OP I created the environment and successfully ran most of the tests. However I did get several failures like this:

Traceback (most recent call last):
  File "[git-path]/matplotlib/lib/matplotlib/backends/backend_gtk3.py", line 24, in <module>
    gi.require_version("Gtk", "3.0")
  File "[conda-path]/envs/mpl-dev/lib/python3.7/site-packages/gi/__init__.py", line 130, in require_version
    (namespace, version))
ValueError: Namespace Gtk not available for version 3.0

and

  File "[git-path]/matplotlib/lib/matplotlib/backends/_backend_tk.py", line 22, in <module>
    from . import _tkagg
ImportError: [git-path]/matplotlib/lib/matplotlib/backends/_tkagg.cpython-37m-x86_64-linux-gnu.so: undefined symbol: MPL_matplotlib_backends__tkagg_ARRAY_API

Also a problem creating a webagg server as well as a couple of image comparison failures, though these aren't obviously to do with the environment.

Can you try adding gtk3 to the dependencies in environment.yml?

Added gtk3 to the yml file, and I'm now down to just 6 failures:

FAILED lib/matplotlib/tests/test_backend_pgf.py::test_mixedmode[pdf] - matplo...
FAILED lib/matplotlib/tests/test_backends_interactive.py::test_interactive_backend[toolbar2-tkagg]
FAILED lib/matplotlib/tests/test_backends_interactive.py::test_interactive_backend[toolmanager-tkagg]
FAILED lib/matplotlib/tests/test_backends_interactive.py::test_interactive_thread_safety[tkagg]
FAILED lib/matplotlib/tests/test_backends_interactive.py::test_webagg - Faile...
FAILED lib/matplotlib/tests/test_polar.py::test_polar_theta_limits[pdf] - mat...

The image comparison failures are very small and don't look different by eye. Could the fact that they are both pdfs point to something like the ImageMagick version? I don't have the authority to update that.

for the pdf tests we have the pdf under version control and then convert them both to png to compare. Because this can be a slow process we cache the results in ~/.cache/matplotlib/test_cache. It is possible that we are not including enough version information in the hashed names and due to a version difference from the first time you ran it may be the source of the problems. Can you try clearing that folder and trying again?

The tk failure is worrying, can you try a git clean -xfd and reinstalling?

What is the error you are getting with webagg?

Thanks @tacaswell, I deleted the test_cache directory, then did git clean -xfd in my working copy before creating a new environment. Unfortunately I get the same image comparison and tkagg failures.

The webagg failure is inconsistent: the first and (this) third time, I got
The webagg server could not be started because an available port could not be found

the second time, it looks like a server was spun up, but then
E urllib.error.HTTPError: HTTP Error 403: OK
which is caught and then
E Failed: Failed to connect to the webagg server.

All the pdf tests fail for me (on a mac). I assume it is just different versions of Ghostscript, and just ignore the errors.

I'm not sure it practical to give users a setup that will pass every test on every machine. I think the spirit of this is to get the users most of the way there with the minimum of steps.

I think the spirit of this is to get the users most of the way there with the minimum of steps.

It certainly does that very nicely.

I tried building the docs.

There are a couple of gallery examples that require scipy:

• Curve Error Band
• Double Pendulum

sphinxcontrib-svg2pdfconverter is also required, but doesn't seem to be available through conda-forge.

I added scipy to the yml, and installed sphinxcontrib-svg2pdfconverter with pip, and all seems well.

I just noticed that the installation guide specifies not using Tk 8.6.0 or 8.6.1. My environments have 8.6. Given that Tk is part of the standard library, how do you choose the version on that?

I think this is really helpful for getting a clean environment to start from! Here are some additional things I found to add to the sections.

Standard dependencies (This enables me to install without downloading any additional dependencies from pip):
fonttools>=4.22.0, setuptool_scm

Documentation:
scipy, pydata-sphinx-theme, sphinx-gallery>=0.9
The sphinx-gallery one is tough because we are currently pulling the version from their git repo. Depending on how they bump their version on the next release this will need to be updated (probably >=0.10)

Additionally, I needed the sphinxcontrib-svg2pdfconverter, which apparently isn't available through conda, so I think we need an additional pip area in the file

  - pip
  - pip:
    - sphinxcontrib-svg2pdfconverter

Testing:
gtk3
Adding poppler also got me the command line tool pdftocairo, which helped some pgf tests, but also caused other issues with my compiled libpng not being the same as the runtime libpng.

I'll add a description how to use this later to the dev docs. But basically following the instuctions in the file is sufficient.

Note: Failed check is due to a failing freetype download and unrelated to this PR.

I take the liberty to self-merge with one review, because this is not a code change. And even if a dependency should be missing it'S better to have this in the repo than nothing at all.