Switch testing to pytest completely #7974

QuLogic · 2017-01-28T21:38:04Z

I'm pre-opening this PR to verify that tests continue to build correctly. However, it still depends on #7935 and #7973.

This PR stops installing nose, and uses pytest everywhere. Note that I did not yet delete any redundant builds. I want to make sure everything works with the different methods first and will then see about pruning stuff.

tacaswell · 2017-01-29T06:14:44Z

attn @story645 I think I recognize this issue...

_________________________ TestUnitData.test_update_map _________________________
[gw0] linux2 -- Python 2.7.9 /home/travis/build/matplotlib/matplotlib/venv/bin/python
self = <matplotlib.tests.test_category.TestUnitData object at 0x7fffdfdb6910>
    def test_update_map(self):
        data = ['a', 'd']
        oseq = ['a', 'd']
        olocs = [0, 1]
    
        data_update = ['b', 'd', 'e', np.inf]
        useq = ['a', 'd', 'b', 'e', 'inf']
        ulocs = [0, 1, 2, 3, -2]
    
        unitdata = cat.UnitData(data)
        assert unitdata.seq == oseq
        assert unitdata.locs == olocs
    
        unitdata.update(data_update)
>       assert unitdata.seq == useq
E       assert [u'a', u'd', u'b', u'e', u'i'] == ['a', 'd', 'b', 'e', 'inf']
E         At index 4 diff: u'i' != u'inf'
E         Use -v to get the full diff
lib/matplotlib/tests/test_category.py:44: AssertionError

@QuLogic Might want to turn off the fast-fail while we debug this?

story645 · 2017-01-29T07:50:01Z

What version of numpy is that test using? that looks like the fun joy where numpy pre like 1.8 truncates inf and nan to the length of the largest string in the array, which in this case is 1.

QuLogic · 2017-01-29T07:52:09Z

It looks like NumPy 1.7.1.

Though I have a bigger question of why so few tests seem to be run compared to locally.

NelleV · 2017-01-29T18:02:07Z

Side note: that PR is going to conflict against #7970

NelleV · 2017-01-29T18:03:06Z

tests.py

@@ -4,10 +4,9 @@
 #


Can't we just remove this script?

Not against it, but also not going to make that decision unilaterally.

I strongly prefer this script stay and remain functional.

NelleV · 2017-01-29T18:03:37Z

lib/matplotlib/tests/test_ticker.py

@@ -240,7 +237,7 @@ def _sub_labels(axis, subs=()):
    assert label_test == label_expected


-@cleanup(style='default')
+@pytest.mark.style('default')


Why is the style classic being replaced by default?

Both lines say default here.

Indeed, they do… I thought I had seen some classic being converted to default, but I guess we'll see this once the tests run on travis anyways.

There are a few that do; see #7973 for an explanation.

NelleV · 2017-01-29T18:09:35Z

.travis.yml

@@ -155,20 +151,20 @@ script:
        rm -rf ~/.cache/matplotlib
      fi
      export MPL_REPO_DIR=$PWD  # needed for pep8-conformance test of the examples
+      # Workaround for pytest-xdist flaky colletion order


nitpick: There is a typo here: colletion -> collection

NelleV · 2017-01-29T18:10:45Z

@QuLogic We are setting the option to step testing after the first failure to speed up the tests.

QuLogic · 2017-01-29T22:12:35Z

I've marked this [WIP] so that people focus on #7973 before this one; plus I have to figure out AppVeyor.

QuLogic · 2017-02-01T07:59:21Z

I believe this should be working now, once CI catches up. It currently includes #8001 and #8002 as well as several bug fixes that were missed due to the CI not running everything:

Change stacklevel of warning in Figure.gca because it's triggered in two tests and on Python 2.7 it doesn't get triggered properly in the second test.
Skip broken categories tests temporarily (see poor categorical support w/ numpy<1.8 #7988)
Fix broken != comparison with SubplotSpec
Normalize paths when looking for fonts because Windows requires it.
Remove one last xfail that I missed in earlier PRs.

tacaswell · 2017-02-01T14:38:30Z

power-cycled as the code being run on travis (from reading the errors) was not matching the code in the PR. I saw something like this on one of my day-job projects yesterday too. I think the web-hook is firing before the +refs/PRNUMBER/merge references are updating or something...

It's really only needed to get the pep8 options.

This allows it to work when run via matplotlib.test().

Raising the warning on the current level breaks tests on Python 2.7 which doesn't reset the warning filter correctly.

In python3 `__ne__` automatically delegates to `__eq__`, in python2 it does not and falls back to the default `__ne__` test. The NumPy version of `assert_equal` actually checks `if a != b: raise`, whereas the version of `assert_equal` from `unittest` (where `nose` ultimately pulls from) checks `if not a == b: raise`

This is a problem on Windows where file paths are case-insensitive.

QuLogic · 2017-02-03T01:09:38Z

We're good to go here now.

tacaswell · 2017-02-03T03:06:45Z

.travis.yml

@@ -225,7 +220,7 @@ after_success:
    else
      echo "Will only deploy docs build from matplotlib master branch"
    fi
-    if [[ $NOSE_ARGS =~ "--with-coverage" || $USE_PYTEST == true ]]; then
+    if [[ $USE_PYTEST == true ]]; then


We should do this on all runs now?

I was leaving that to #8003, but I see now it doesn't change this condition. @dopplershift Should I turn on coverage for everything here or just leave it for you?

Go ahead and turn it on--I'll rebase on it.

NelleV

LGTM

dstansby · 2017-02-03T13:54:39Z

The change to pytest will need documenting, but I've opened up a new issue for that at #8015.

tacaswell · 2017-02-03T14:00:53Z

1: ======= 6470 passed, 10 skipped, 6 xfailed, 8 xpassed in 644.13 seconds ======== 6502
2: ======= 6477 passed, 12 skipped, 2 xfailed, 3 xpassed in 609.06 seconds ======== 6497
3: ======== 6485 passed, 4 skipped, 2 xfailed, 3 xpassed in 611.43 seconds ======== 6497
4: docs
5: ======== 6485 passed, 4 skipped, 2 xfailed, 3 xpassed in 617.37 seconds ======== 6497
6: ======= 7259 passed, 12 skipped, 2 xfailed, 3 xpassed in 610.41 seconds ======== 7276
7: ====== 5256 passed, 1227 skipped, 10 xfailed, 1 xpassed in 573.52 seconds ====== 6494
8: ======= 6477 passed, 12 skipped, 2 xfailed, 3 xpassed in 579.11 seconds ======== 6494

There seems to be some slight descrepencies in the number of tests each run finds.

dstansby · 2017-02-03T14:06:10Z

Adding up those numbers I get 6494 for everything but (6) for which I get 7276.

dstansby · 2017-02-03T14:09:44Z

And I'm guessing that the extra checks on (6) are PEP8 checks.

tacaswell · 2017-02-03T14:39:24Z

Apparently I can not use a calculator 🐑

QuLogic added the topic: testing label Jan 28, 2017

QuLogic added this to the 2.1 (next point release) milestone Jan 28, 2017

QuLogic force-pushed the pytest branch from aa11a84 to a44514d Compare January 28, 2017 23:52

QuLogic mentioned this pull request Jan 29, 2017

Convert test decorators to pytest fixtures #7973

Merged

NelleV reviewed Jan 29, 2017

View reviewed changes

QuLogic changed the title ~~Switch testing to pytest completely~~ [WIP] Switch testing to pytest completely Jan 29, 2017

story645 mentioned this pull request Jan 30, 2017

poor categorical support w/ numpy<1.8 #7988

Closed

QuLogic force-pushed the pytest branch from 7f9fa59 to 3bda4ce Compare January 31, 2017 01:41

QuLogic mentioned this pull request Jan 31, 2017

Switch to pytest-pep8. #8001

Merged

QuLogic force-pushed the pytest branch from 3bda4ce to 78866d5 Compare February 1, 2017 07:17

tacaswell closed this Feb 1, 2017

tacaswell reopened this Feb 1, 2017

QuLogic added 8 commits February 2, 2017 18:18

Use pip to install in AppVeyor too.

6a91b9a

Rename test_only to avoid pytest collection.

ab99e65

Remove pytest collection filter.

bd91b93

It's really only needed to get the pep8 options.

Move pytest_configure into installed area.

25fdef8

This allows it to work when run via matplotlib.test().

Enable pytest-xdist on AppVeyor.

c8fc318

Change stack level of Figure.gca warning.

02d7cf8

Raising the warning on the current level breaks tests on Python 2.7 which doesn't reset the warning filter correctly.

Skip broken category test with NumPy < 1.8.0.

30e40c6

Use pytest in matplotlib.test().

80281e9

QuLogic force-pushed the pytest branch from 78866d5 to 33a4a0e Compare February 2, 2017 23:22

Stop installing nose in CI.

c520c2c

tacaswell and others added 3 commits February 2, 2017 18:39

Normalize path case before checking for fonts.

ca255dd

This is a problem on Windows where file paths are case-insensitive.

Fix one last old xfail call.

3cb2752

QuLogic force-pushed the pytest branch from 33a4a0e to 3cb2752 Compare February 2, 2017 23:47

QuLogic changed the title ~~[WIP] Switch testing to pytest completely~~ [MRG] Switch testing to pytest completely Feb 3, 2017

Fix tests.py when passed explicit filenames.

857d846

QuLogic force-pushed the pytest branch from e332caa to 857d846 Compare February 3, 2017 02:19

tacaswell reviewed Feb 3, 2017

View reviewed changes

tacaswell approved these changes Feb 3, 2017

View reviewed changes

NelleV approved these changes Feb 3, 2017

View reviewed changes

NelleV changed the title ~~[MRG] Switch testing to pytest completely~~ [MRG+2] Switch testing to pytest completely Feb 3, 2017

Always upload coverage on Travis.

fe20a23

QuLogic force-pushed the pytest branch from 25724f7 to fe20a23 Compare February 3, 2017 06:39

dstansby approved these changes Feb 3, 2017

View reviewed changes

dstansby merged commit a517c60 into matplotlib:master Feb 3, 2017

dstansby changed the title ~~[MRG+2] Switch testing to pytest completely~~ Switch testing to pytest completely Feb 3, 2017

dstansby mentioned this pull request Feb 3, 2017

Document new testing procedure #8015

Closed

dopplershift mentioned this pull request Feb 3, 2017

Coverage config #8003

Merged

QuLogic deleted the pytest branch February 4, 2017 04:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Switch testing to pytest completely #7974

Switch testing to pytest completely #7974

QuLogic commented Jan 28, 2017 •

edited

Loading

tacaswell commented Jan 29, 2017

story645 commented Jan 29, 2017

QuLogic commented Jan 29, 2017

NelleV commented Jan 29, 2017

NelleV Jan 29, 2017

QuLogic Jan 29, 2017

tacaswell Jan 29, 2017

NelleV Jan 29, 2017

QuLogic Jan 29, 2017

NelleV Jan 29, 2017

QuLogic Jan 29, 2017

NelleV Jan 29, 2017

NelleV commented Jan 29, 2017

QuLogic commented Jan 29, 2017

QuLogic commented Feb 1, 2017 •

edited

Loading

tacaswell commented Feb 1, 2017

QuLogic commented Feb 3, 2017

tacaswell Feb 3, 2017

QuLogic Feb 3, 2017 •

edited

Loading

dopplershift Feb 3, 2017

NelleV left a comment

dstansby commented Feb 3, 2017

tacaswell commented Feb 3, 2017

dstansby commented Feb 3, 2017

dstansby commented Feb 3, 2017

tacaswell commented Feb 3, 2017

Switch testing to pytest completely #7974

Switch testing to pytest completely #7974

Conversation

QuLogic commented Jan 28, 2017 • edited Loading

tacaswell commented Jan 29, 2017

story645 commented Jan 29, 2017

QuLogic commented Jan 29, 2017

NelleV commented Jan 29, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NelleV commented Jan 29, 2017

QuLogic commented Jan 29, 2017

QuLogic commented Feb 1, 2017 • edited Loading

tacaswell commented Feb 1, 2017

QuLogic commented Feb 3, 2017

Choose a reason for hiding this comment

QuLogic Feb 3, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NelleV left a comment

Choose a reason for hiding this comment

dstansby commented Feb 3, 2017

tacaswell commented Feb 3, 2017

dstansby commented Feb 3, 2017

dstansby commented Feb 3, 2017

tacaswell commented Feb 3, 2017

QuLogic commented Jan 28, 2017 •

edited

Loading

QuLogic commented Feb 1, 2017 •

edited

Loading

QuLogic Feb 3, 2017 •

edited

Loading