Thanks for investigating this. I think the fundamental problem is that we have an opt in default_test_modules in matplotlib/__init__.py and new test modules have been added without adding them to that list. I suggest that we get rid of that mechanism. If tests need to be skipped they should be explicitly marked as such. This covers your first category.

I am not convinced that adding the tests to the default coverage is the right solution. To me it would bias the coverage towards an artificially high number which is not really representative of the real coverage of our code.

I suggest that we get rid of that mechanism.

I am for this too.

Currently for pytest I have developed two filters:

• --collect-filter=whitelist (default) uses the default_test_modules list
• --collect-filter=blacklist uses the list of disabled tests (actually this can be done with def setup_module: skip()), but I have done it to support backward comparability with nose
• --collect-filter=none do not use any filters

But I will drop them with pleasure.

it would bias the coverage towards an artificially high number which is not really representative of the real coverage of our code

Currently code coverage percentage is already faked (for example there are only 3 backends in coverage report, while other do not included, because they have zero coverage). I do not think anyone measures matplotlib by coverage number, but test code is a code too, and it is better to have overestimated score than tests which do not work.

I definitely support getting rid of the whitelisting as a option or at least not making it the default. I think our experience have shown that its very error prone.

The backends are definitely an issue for testing. That is the main driver behind deprecating the WX non AGG backend and the GDK backend. We do most definitely need more coverage for the Cairo, ps, pgf backends and the various interactive aspects of GUI toolkits.

That being said the backends are not excluded from coverage it just happens that the python code that implements most backends is a small fraction of the total code. You cannot imply that we need to run all examples with all backends to get full coverage since most of the tests are testing non backend specific code.

I cannot say that including tests in coverage report is a definitely good solution, but until there is no other ways (or I do not know about them) to prevent issues such like #6839 (and others mentioned above) I am for it.

So the way coverage is currently set up, anything using py.test tanks miserably: https://coveralls.io/jobs/17551007 (eta: it's an issue for #6889)

Fixed by #8003.