-
-
Notifications
You must be signed in to change notification settings - Fork 7.9k
Pytest documentation + build tweaks #8026
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
bdb8682
b71447f
c632e41
113d079
9d36e98
d9fbe8d
244806d
25192ea
50eaad6
4529002
5bc91c7
a4427a1
6a0c3ee
29c1308
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -20,17 +20,16 @@ Testing | |
|
||
After installation, you can launch the test suite:: | ||
|
||
python tests.py | ||
py.test | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. See previous comment about |
||
|
||
Or from the Python interpreter:: | ||
|
||
import matplotlib | ||
matplotlib.test() | ||
|
||
Consider reading http://matplotlib.org/devel/coding_guide.html#testing for | ||
more information. Note that the test suite requires nose and on Python 2.7 mock | ||
which are not installed by default. Please install with pip or your package | ||
manager of choice. | ||
more information. Note that the test suite requires pytest and, on Python 2.7, | ||
mock. Please install with pip or your package manager of choice. | ||
|
||
Contact | ||
======= | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -30,7 +30,7 @@ requirements: | |
- freetype 2.6* | ||
- msinttypes # [win] | ||
- cycler >=0.10 | ||
- nose | ||
- pytest >=3.0.0 | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 👍 |
||
- pyparsing | ||
- pytz | ||
# - py2cairo # [linux and py2k] | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -120,14 +120,16 @@ environment is set up properly:: | |
python tests.py | ||
|
||
|
||
.. _nose: https://nose.readthedocs.io/en/latest/ | ||
.. _pytest: http://doc.pytest.org/en/latest/ | ||
.. _pep8: https://pep8.readthedocs.io/en/latest/ | ||
.. _mock: https://docs.python.org/dev/library/unittest.mock.html | ||
.. _Ghostscript: https://www.ghostscript.com/ | ||
.. _Inkscape: https://inkscape.org> | ||
|
||
.. note:: | ||
|
||
**Additional dependencies for testing**: nose_ (version 1.0 or later), `mock | ||
<https://docs.python.org/dev/library/unittest.mock.html>`_ (if python < 3.3), `Ghostscript | ||
<https://www.ghostscript.com/>`_, `Inkscape <https://inkscape.org>`_ | ||
**Additional dependencies for testing**: pytest_ (version 3.0 or later), | ||
mock_ (if python < 3.3), Ghostscript_, Inkscape_ | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 👍 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Python 3 < 3.3 isn't supported but matplotlib, so might make more sense to change this to "if python 2.7" |
||
|
||
.. seealso:: | ||
|
||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -4,31 +4,36 @@ | |
Developer's tips for testing | ||
============================ | ||
|
||
Matplotlib has a testing infrastructure based on nose_, making it easy | ||
to write new tests. The tests are in :mod:`matplotlib.tests`, and | ||
customizations to the nose testing infrastructure are in | ||
:mod:`matplotlib.testing`. (There is other old testing cruft around, | ||
please ignore it while we consolidate our testing to these locations.) | ||
|
||
.. _nose: https://nose.readthedocs.io/en/latest/ | ||
Matplotlib's testing infrastructure depends on pytest_. The tests are in | ||
:file:`lib/matplotlib/tests`, and customizations to the pytest testing | ||
infrastructure are in :mod:`matplotlib.testing`. | ||
|
||
.. _pytest: http://doc.pytest.org/en/latest/ | ||
.. _mock: https://docs.python.org/dev/library/unittest.mock.html> | ||
.. _Ghostscript: https://www.ghostscript.com/ | ||
.. _Inkscape: https://inkscape.org | ||
.. _pytest-cov: https://pytest-cov.readthedocs.io/en/latest/ | ||
.. _pytest-pep8: https://pypi.python.org/pypi/pytest-pep8 | ||
.. _pytest-xdist: https://pypi.python.org/pypi/pytest-xdist | ||
.. _pytest-timeout: https://pypi.python.org/pypi/pytest-timeout | ||
|
||
Requirements | ||
------------ | ||
|
||
The following software is required to run the tests: | ||
|
||
- nose_, version 1.0 or later | ||
- `mock <https://docs.python.org/dev/library/unittest.mock.html>`_, when running python | ||
versions < 3.3 | ||
- `Ghostscript <https://www.ghostscript.com/>`_ (to render PDF | ||
files) | ||
- `Inkscape <https://inkscape.org>`_ (to render SVG files) | ||
- pytest_, version 3.0.0 or later | ||
- mock_, when running Python versions < 3.3 | ||
- Ghostscript_ (to render PDF files) | ||
- Inkscape_ (to render SVG files) | ||
|
||
Optionally you can install: | ||
|
||
- `coverage <https://coverage.readthedocs.io/en/latest/>`_ to collect coverage | ||
information | ||
- `pep8 <http://pep8.readthedocs.io/en/latest>`_ to test coding standards | ||
- pytest-cov_ to collect coverage information | ||
- pytest-pep8_ to test coding standards | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I would add all the optional dependencies here:
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Added both. |
||
- pytest-timeout_ to limit runtime in case of stuck tests | ||
- pytest-xdist_ to run tests in parallel | ||
|
||
|
||
Building matplotlib for image comparison tests | ||
---------------------------------------------- | ||
|
@@ -53,58 +58,64 @@ value. | |
Running the tests | ||
----------------- | ||
|
||
Running the tests is simple. Make sure you have nose installed and run:: | ||
Running the tests is simple. Make sure you have pytest installed and run:: | ||
|
||
py.test | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 👍 There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. See above comment about |
||
|
||
or:: | ||
|
||
python tests.py | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @tacaswell can we remove this from our documentation? :( There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I agree, attn @tacaswell. |
||
|
||
in the root directory of the distribution. The script takes a set of | ||
commands, such as: | ||
|
||
======================== =========== | ||
``--pep8`` pep8 checks | ||
``--no-pep8`` Do not perform pep8 checks | ||
``--no-network`` Disable tests that require network access | ||
``--pep8`` Perform pep8 checks (requires pytest-pep8_) | ||
``-m "not network"`` Disable tests that require network access | ||
======================== =========== | ||
|
||
Additional arguments are passed on to nosetests. See the nose | ||
documentation for supported arguments. Some of the more important ones are given | ||
here: | ||
Additional arguments are passed on to pytest. See the pytest documentation for | ||
`supported arguments`_. Some of the more important ones are given here: | ||
|
||
============================= =========== | ||
``--verbose`` Be more verbose | ||
``--processes=NUM`` Run tests in parallel over NUM processes | ||
``--process-timeout=SECONDS`` Set timeout for results from test runner process | ||
``--nocapture`` Do not capture stdout | ||
``--n NUM`` Run tests in parallel over NUM | ||
processes (requires pytest-xdist_) | ||
``--timeout=SECONDS`` Set timeout for results from each test | ||
process (requires pytest-timeout_) | ||
``--capture=no`` or ``-s`` Do not capture stdout | ||
============================= =========== | ||
|
||
To run a single test from the command line, you can provide a | ||
dot-separated path to the module followed by the function separated by | ||
a colon, e.g., (this is assuming the test is installed):: | ||
To run a single test from the command line, you can provide a file path, | ||
optionally followed by the function separated by two colons, e.g., (tests do | ||
not need to be installed, but Matplotlib should be):: | ||
|
||
python tests.py matplotlib.tests.test_simplification:test_clipping | ||
py.test lib/matplotlib/tests/test_simplification.py::test_clipping | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 👍 |
||
|
||
or, if tests are installed, a dot-separated path to the module, optionally | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I would remove this section. We only need to document one way to do this. |
||
followed by the function separated by two colons, such as:: | ||
|
||
py.test --pyargs matplotlib.tests.test_simplification::test_clipping | ||
|
||
If you want to run the full test suite, but want to save wall time try | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What is "wall time"? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Time on a clock on the wall, as opposed to cpu time. |
||
running the tests in parallel:: | ||
|
||
python tests.py --nocapture --verbose --processes=5 --process-timeout=300 | ||
py.test --verbose -n 5 | ||
|
||
Depending on your version of Python and pytest-xdist, you may need to set | ||
``PYTHONHASHSEED`` to a fixed value when running in parallel:: | ||
|
||
An alternative implementation that does not look at command line | ||
arguments works from within Python is to run the tests from the | ||
matplotlib library function :func:`matplotlib.test`:: | ||
PYTHONHASHSEED=0 py.test --verbose -n 5 | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can we just specify a minimum version of pytest-xdist where this is no longer an issue? It looks like it is related to https://bitbucket.org/pytest-dev/pytest/issues/346/pytest-xdist-and-python-33-is-sort-of, which has been resolved in 2013. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Unfortunately, it's not fixed yet: pytest-dev/pytest#920 I just PR'd the same fix to the pandas build, so I know that recent versions still don't work. |
||
|
||
An alternative implementation that does not look at command line arguments | ||
and works from within Python is to run the tests from the Matplotlib library | ||
function :func:`matplotlib.test`:: | ||
|
||
import matplotlib | ||
matplotlib.test() | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'd remove this section from our documentation. I think we should document (and have) only one way to run the tests. attn @tacaswell (as I believe Thomas will disagree with me on that one). |
||
|
||
.. hint:: | ||
|
||
To run the tests you need to install nose and mock if using python 2.7:: | ||
|
||
pip install nose | ||
pip install mock | ||
|
||
|
||
.. _`nosetest arguments`: http://nose.readthedocs.io/en/latest/usage.html | ||
.. _supported arguments: http://doc.pytest.org/en/latest/usage.html | ||
|
||
|
||
Writing a simple test | ||
|
@@ -113,30 +124,20 @@ Writing a simple test | |
Many elements of Matplotlib can be tested using standard tests. For | ||
example, here is a test from :mod:`matplotlib.tests.test_basic`:: | ||
|
||
from nose.tools import assert_equal | ||
|
||
def test_simple(): | ||
""" | ||
very simple example test | ||
""" | ||
assert_equal(1+1,2) | ||
|
||
Nose determines which functions are tests by searching for functions | ||
beginning with "test" in their name. | ||
assert 1 + 1 == 2 | ||
|
||
If the test has side effects that need to be cleaned up, such as | ||
creating figures using the pyplot interface, use the ``@cleanup`` | ||
decorator:: | ||
Pytest determines which functions are tests by searching for files whose names | ||
begin with ``"test_"`` and then within those files for functions beginning with | ||
``"test"`` or classes beginning with ``"Test"``. | ||
|
||
from matplotlib.testing.decorators import cleanup | ||
|
||
@cleanup | ||
def test_create_figure(): | ||
""" | ||
very simple example test that creates a figure using pyplot. | ||
""" | ||
fig = figure() | ||
... | ||
Some tests have internal side effects that need to be cleaned up after their | ||
execution (such as created figures or modified rc params). The pytest fixture | ||
:func:`~matplotlib.testing.conftest.mpl_test_settings` will automatically clean | ||
these up; there is no need to do anything further. | ||
|
||
|
||
Writing an image comparison test | ||
|
@@ -203,24 +204,22 @@ decorator: | |
Known failing tests | ||
------------------- | ||
|
||
If you're writing a test, you may mark it as a known failing test with | ||
the :func:`~matplotlib.testing.decorators.knownfailureif` | ||
decorator. This allows the test to be added to the test suite and run | ||
on the buildbots without causing undue alarm. For example, although | ||
the following test will fail, it is an expected failure:: | ||
If you're writing a test, you may mark it as a known failing test with the | ||
:func:`pytest.mark.xfail` decorator. This allows the test to be added to the | ||
test suite and run on the buildbots without causing undue alarm. For example, | ||
although the following test will fail, it is an expected failure:: | ||
|
||
from nose.tools import assert_equal | ||
from matplotlib.testing.decorators import knownfailureif | ||
import pytest | ||
|
||
@knownfailureif(True) | ||
@pytest.mark.xfail | ||
def test_simple_fail(): | ||
'''very simple example test that should fail''' | ||
assert_equal(1+1,3) | ||
assert 1 + 1 == 3 | ||
|
||
Note that the first argument to the | ||
:func:`~matplotlib.testing.decorators.knownfailureif` decorator is a | ||
fail condition, which can be a value such as True, False, or | ||
'indeterminate', or may be a dynamically evaluated expression. | ||
Note that the first argument to the :func:`~pytest.mark.xfail` decorator is a | ||
fail condition, which can be a value such as True, False, or may be a | ||
dynamically evaluated expression. If a condition is supplied, then a reason | ||
must also be supplied with the ``reason='message'`` keyword argument. | ||
|
||
Creating a new module in matplotlib.tests | ||
----------------------------------------- | ||
|
@@ -229,11 +228,6 @@ We try to keep the tests categorized by the primary module they are | |
testing. For example, the tests related to the ``mathtext.py`` module | ||
are in ``test_mathtext.py``. | ||
|
||
Let's say you've added a new module named ``whizbang.py`` and you want | ||
to add tests for it in ``matplotlib.tests.test_whizbang``. To add | ||
this module to the list of default tests, append its name to | ||
``default_test_modules`` in :file:`lib/matplotlib/__init__.py`. | ||
|
||
Using Travis CI | ||
--------------- | ||
|
||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -29,13 +29,13 @@ connect your function to the event manager, which is part of the | |
example that prints the location of the mouse click and which button | ||
was pressed:: | ||
|
||
fig = plt.figure() | ||
ax = fig.add_subplot(111) | ||
fig, ax = plt.subplots() | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 👍 |
||
ax.plot(np.random.rand(10)) | ||
|
||
def onclick(event): | ||
print('button=%d, x=%d, y=%d, xdata=%f, ydata=%f' % | ||
(event.button, event.x, event.y, event.xdata, event.ydata)) | ||
print('%s click: button=%d, x=%d, y=%d, xdata=%f, ydata=%f' % | ||
('double' if event.dblclick else 'single', event.button, | ||
event.x, event.y, event.xdata, event.ydata)) | ||
|
||
cid = fig.canvas.mpl_connect('button_press_event', onclick) | ||
|
||
|
This file was deleted.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
pytest
is preferred overpy.test
as of 3.0path\\to\\tests\\directory
bit; if you do then line 23 ofREADME.rst
should be made consistentThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is about separate uninstalled tests; line 23 in
README.rst
is about running from the full source directory.