Skip to content

Use some more pytest plugins: warnings & rerunfailures #8346

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Apr 29, 2017
+20 −7
Merged

Use some more pytest plugins: warnings & rerunfailures #8346

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
c0ef143
ci: Install pytest-warnings to report warnings.
QuLogic Mar 19, 2017
35b13b4
Add pytest-rerunfailures plugin.
QuLogic Mar 19, 2017
c310c08
ci: Error out of test script on first error.
QuLogic Mar 19, 2017
8a6b84e
TST: Mark mathtext tests as flaky.
QuLogic Mar 19, 2017
4c95844
TST: Mark test_invisible_Line_rendering as flaky.
QuLogic Mar 19, 2017
086e3d4
TST: Use only pip for pytest packages.
QuLogic Mar 20, 2017
28c8fef
TST: Mark mixedsubplots SVG test as xfail.
QuLogic Mar 23, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .appveyor.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ environment:
CMD_IN_ENV: cmd /E:ON /V:ON /C obvci_appveyor_python_build_env.cmd
# Workaround for https://github.com/conda/conda-build/issues/636
PYTHONIOENCODING: UTF-8
PYTEST_ARGS: -ra --timeout=300 --durations=25 -n %NUMBER_OF_PROCESSORS% --cov-report= --cov=lib -m "not network"
PYTEST_ARGS: -rawR --timeout=300 --durations=25 -n %NUMBER_OF_PROCESSORS% --cov-report= --cov=lib -m "not network"
PYTHONHASHSEED: 0 # Workaround for pytest-xdist flaky collection order
# https://github.com/pytest-dev/pytest/issues/920
# https://github.com/pytest-dev/pytest/issues/1075
Expand Down Expand Up @@ -94,7 +94,7 @@ install:
- echo %PYTHON_VERSION% %TARGET_ARCH%
- if %PYTHON_VERSION% == 2.7 conda install -q backports.functools_lru_cache
# pytest-cov>=2.3.1 due to https://github.com/pytest-dev/pytest-cov/issues/124
- conda install -q pytest "pytest-cov>=2.3.1" pytest-timeout pytest-xdist
- pip install -q pytest "pytest-cov>=2.3.1" pytest-rerunfailures pytest-timeout pytest-warnings pytest-xdist

# Let the install prefer the static builds of the libs
- set LIBRARY_LIB=%CONDA_PREFIX%\Library\lib
Expand Down
6 changes: 3 additions & 3 deletions .travis.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ env:
- INSTALL_PEP8=
- RUN_PEP8=
- NOSE=
- PYTEST_ARGS="-ra --maxfail=1 --timeout=300 --durations=25 --cov-report= --cov=lib -n $NPROC"
- PYTEST_ARGS="-rawR --maxfail=1 --timeout=300 --durations=25 --cov-report= --cov=lib -n $NPROC"
- PYTHON_ARGS=
- DELETE_FONT_CACHE=

Expand Down Expand Up @@ -119,7 +119,7 @@ install:
pip install $PRE -r doc-requirements.txt

# pytest-cov>=2.3.1 due to https://github.com/pytest-dev/pytest-cov/issues/124
pip install $PRE pytest 'pytest-cov>=2.3.1' pytest-timeout pytest-xdist pytest-faulthandler $INSTALL_PEP8
pip install $PRE pytest 'pytest-cov>=2.3.1' pytest-faulthandler pytest-rerunfailures pytest-timeout pytest-warnings pytest-xdist $INSTALL_PEP8

# We manually install humor sans using the package from Ubuntu 14.10. Unfortunatly humor sans is not
# availible in the Ubuntu version used by Travis but we can manually install the deb from a later
Expand All @@ -142,7 +142,7 @@ install:
# Install matplotlib
pip install -ve .

script: source ci/travis/test_script.sh
script: ci/travis/test_script.sh

before_cache:
- rm -rf $HOME/.cache/matplotlib/tex.cache
Expand Down
3 changes: 2 additions & 1 deletion ci/travis/test_script.sh
100644 → 100755
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
#! /bin/bash

set -e
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is causing the doc build to fail on line 39 below, which can just be deleted (I can't work out how to open a PR on your branch, but https://travis-ci.org/dstansby/matplotlib/builds/221288308 is my travis build)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should probably just drop line 39


# This script is meant to be called by the "script" step defined in
# .travis.yml. See http://docs.travis-ci.com/ for more details.
# The behavior of the script is controlled by environment variabled defined
Expand Down Expand Up @@ -34,7 +36,6 @@ else
# We don't build the LaTeX docs here, so linkchecker will complain
touch build/html/Matplotlib.pdf
# Linkchecker only works with python 2.7 for the time being
deactivate
source ~/virtualenv/python2.7/bin/activate
pip install pip --upgrade
# linkchecker is currently broken with requests 2.10.0 so force an earlier version
Expand Down
3 changes: 3 additions & 0 deletions lib/matplotlib/tests/test_lines.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,9 @@
from matplotlib.testing.decorators import image_comparison


# Runtimes on a loaded system are inherently flaky. Not so much that a rerun
# won't help, hopefully.
@pytest.mark.flaky(reruns=3)
def test_invisible_Line_rendering():
"""
Github issue #1256 identified a bug in Line.draw method
Expand Down
6 changes: 6 additions & 0 deletions lib/matplotlib/tests/test_mathtext.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -167,6 +167,9 @@ def baseline_images(request, fontset, index):
return ['%s_%s_%02d' % (request.param, fontset, index)]


# See #7911 for why these tests are flaky and #7107 for why they are not so
# easy to fix.
@pytest.mark.flaky(reruns=3)
@pytest.mark.parametrize('index, test', enumerate(math_tests),
ids=[str(index) for index in range(len(math_tests))])
@pytest.mark.parametrize('fontset',
Expand All @@ -181,6 +184,9 @@ def test_mathtext_rendering(baseline_images, fontset, index, test):
horizontalalignment='center', verticalalignment='center')


# See #7911 for why these tests are flaky and #7107 for why they are not so
# easy to fix.
@pytest.mark.flaky(reruns=3)
@pytest.mark.parametrize('index, test', enumerate(font_tests),
ids=[str(index) for index in range(len(font_tests))])
@pytest.mark.parametrize('fontset',
Expand Down
5 changes: 4 additions & 1 deletion lib/mpl_toolkits/tests/test_mplot3d.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,10 @@ def test_lines3d():
ax.plot(x, y, z)


@image_comparison(baseline_images=['mixedsubplot'], remove_text=True)
# Reason for flakiness of SVG test is still unknown.
@image_comparison(baseline_images=['mixedsubplot'], remove_text=True,
extensions=['png', 'pdf',
pytest.mark.xfail('svg', strict=False)])
def test_mixedsubplots():
def f(t):
s1 = np.cos(2*np.pi*t)
Expand Down