TST : enable coveralls #4367

tacaswell · 2015-04-22T05:26:59Z

Attempt to turn on coveralls to track how we are doing on test coverage.

tacaswell · 2015-04-22T06:11:58Z

I have no idea why this isn't working on travis. It runs on my local machine which makes me think there is some version issues going on here.

jenshnielsen · 2015-04-22T06:37:28Z

.travis.yml

@@ -24,7 +24,7 @@ matrix:
      env: BUILD_DOCS=true

 install:
-  - pip install -q --use-mirrors nose python-dateutil $NUMPY pep8 pyparsing pillow sphinx!=1.3.0
+  - pip install python-dateutil $NUMPY pep8 pyparsing pillow sphinx!=1.3.0 coveralls coverage


On purpose that nose is dropped here?

Yes, nose is installed by default on travis, I was looking for version conflict issues (there were a whole bunch of force-pushed attempts at this).

jenshnielsen · 2015-04-22T08:43:02Z

Perhaps try running the tests without gdb?

jenshnielsen · 2015-04-22T13:03:35Z

.coveragerc

+source=matplotlib
+[report]
+omit =
+    */python?.?/*


To actually get a cover report locally I had to remove this line (/python?.?/)

I don't think this is actually related to the Travis issue which I cannot reproduce.

tacaswell · 2015-04-22T14:10:28Z

Progress, I can reproduce this on my work machine.

Datadump for my self for later:

(dd_py2k)tcaswell@tcaswellpc1:~/other_source/matplotlib$ conda list
# packages in environment at /home/tcaswell/.virtualenvs/dd_py2k:
#
Could not extract name and version from: 'You are using pip version 6.0.8, however version 6.1.1 is available.'
Could not extract name and version from: "You should consider upgrading via the 'pip install --upgrade pip' command."
Could not extract name and version from: 'Error when trying to get requirement for VCS system Command "/home/tcaswell/local_installs/bin/git config remote.origin.url" failed with error code 1 in /home/tcaswell/bnl_source/fileStore, falling back to uneditable format'
Could not extract name and version from: 'Could not determine repository location of /home/tcaswell/bnl_source/fileStore'
Could not extract name and version from: 'Error when trying to get requirement for VCS system Command "/home/tcaswell/local_installs/bin/git config remote.origin.url" failed with error code 1 in /home/tcaswell/bnl_source/metadataStore, falling back to uneditable format'
Could not extract name and version from: 'Could not determine repository location of /home/tcaswell/bnl_source/metadataStore'
atom                      0.3.9                    py27_0  
backports.ssl-match-hostname 3.4.0.2                   <pip>
bubblegum                 v0.0.2.post2    2_g2a3d9ec_np19py27  
cairo                     1.12.18                       0  
channelarchiver           0.0.4.post3     3_g53bf081_np19py27  
coverage                  3.7.1                     <pip>
coveralls                 0.5                       <pip>
curl                      7.38.0                        0  
dateutil                  2.1                      py27_2  
decorator                 3.4.0                    py27_0  
docopt                    0.6.2                     <pip>
docutils                  0.12                     py27_0  
enaml                     0.9.8                    py27_0  
epics-base                3.14.12                      12  
filestore                 0.5.0.post0               <pip>
freetype                  2.4.10                        0  
h5py                      2.4.0                np19py27_0  
hdf5                      1.8.14                        0  
ipython                   2.4.1                    py27_0  
ipython-notebook          2.3.1                    py27_0  
jdcal                     1.0                      py27_0  
jinja2                    2.7.3                    py27_1  
jpeg                      8d                            0  
jsonschema                2.4.0                    py27_0  
keyring                   3.8                      py27_0  
kiwisolver                0.1.3                    py27_0  
libnetcdf                 4.3.2                         1  
libpng                    1.5.13                        1  
libsodium                 0.4.5                         0  
libtiff                   4.0.2                         1  
libxml2                   2.9.0                         0  
libxslt                   1.1.28                        0  
lmfit                     0.8.3rc1.post4  4_gac73a37_np19py27  
lxml                      3.4.2                    py27_0  
markupsafe                0.23                     py27_0  
matplotlib                1.4.3                np19py27_0  
metadatastore             0.1.0.dev0                <pip>
mongoengine               0.8.7                    py27_0  
ncurses                   5.9                           6  
netcdf4                   1.1.6                np19py27_0  
networkx                  1.9.1                    py27_0  
nose                      1.3.4                    py27_0  
numpy                     1.9.2                    py27_0  
numpydoc                  0.5                      py27_0  
openpyxl                  2.0.2                    py27_0  
openssl                   1.0.1k                        0  
ophyd                     v0.0.4                   py27_0  
pandas                    0.15.2               np19py27_0  
pcaspy                    0.5.1                    py27_0  
pep8                      1.5.7                    py27_0  
pillow                    2.7.0                    py27_0  
pims                      0.2.1                np19py27_0  
pip                       6.0.8                    py27_0  
pixman                    0.26.2                        0  
ply                       3.4                      py27_0  
py2cairo                  1.10.0                   py27_2  
pyepics                   3.2.4rc1.post20 20_g8da13e1_np19py27  
pyflakes                  0.8.1                    py27_0  
pygments                  2.0.2                    py27_0  
pymongo                   2.8                      py27_0  
pyolog                    v4.0.0.post0    0_ge5b0507_np19py27  
pyparsing                 2.0.3                    py27_0  
pyqt                      4.10.4                   py27_0  
python                    2.7.9                        10  
python-dateutil           1.5                       <pip>
pytz                      2014.9                   py27_0  
pyyaml                    3.11                     py27_0  
pyzmq                     14.5.0                   py27_0  
qt                        4.8.5                         0  
readline                  6.2                          12  
requests                  2.5.3                    py27_0  
scikit-image              0.11.2               np19py27_0  
scikit-xray               0.0.3.post117   117_g5ac2ba4_np19py27  
scipy                     0.15.1               np19py27_0  
setuptools                14.0                     py27_0  
sip                       4.15.5                   py27_0  
six                       1.9.0                    py27_0  
sphinx                    1.2.3                    py27_0  
sphinx-bootstrap-theme    0.4.5                    py27_0  
sphinxcontrib-napoleon    0.2.8                    py27_0  
sqlite                    3.8.4.1                       0  
ssl_match_hostname        3.4.0.2                  py27_0  
system                    5.8                           1  
tk                        8.5.15                        0  
tornado                   4.0.2                    py27_0  
tzlocal                   1.1.2                    py27_0  
xraylib                   master               np19py27_0  
yaml                      0.1.4                         0  
zeromq                    4.0.4                         0  
zlib                      1.2.8                         0

(dd_py2k)tcaswell@tcaswellpc1:~/other_source/matplotlib$ nosetests --plugins
Plugin capture
Plugin failuredetail
Plugin xunit
Plugin deprecated
Plugin skip
Plugin multiprocess
Plugin logcapture
Plugin coverage
Plugin attributeselector
Plugin doctest
Plugin profile
Plugin id
Plugin allmodules
Plugin collect-only
Plugin isolation
Plugin pdb

tacaswell · 2015-05-02T15:48:32Z

@jenshnielsen The issue seems to be that travis has nose 1.3.4 install and pip install nose helpfully says 'already installed, use --upgrade to upgarde'.

jenshnielsen · 2015-05-02T16:15:49Z

Makes sense I think it fails now because pip install nose>=1.3.6 doesn't upgrade nose. I guess we need to do either pip install nose==1.3.6 or pip install --upgrade nose?

coveralls · 2015-05-03T18:01:49Z

Changes Unknown when pulling 43c3d70 on tacaswell:tst_coveralls into * on matplotlib:master*.

- also update nose

We run tests out-of-repo which is to ensure that we pick up the installed version of mpl, not version which happens to be in the current directory (but I am not sure we strictly need this any more now that absolute imports are a thing). The coveralls scripts assume that you are running your tests _in_ the repo and are trying to extract git information from the directory they are run in, hence are failing for everything but the doc builds (which don't run nose so have no coverage). This exclude the doc build from uploading to coveralls and moves the cover output to be where coveralls expects it to be (and cds back into the repo).

tacaswell · 2015-05-03T19:46:13Z

woo, I think this is (finally) converging.

I have turned off the comment spam so that should not show up anywhere else again.

@jenshnielsen @mdboom @efiring @WeatherGod Thoughts? I am 👍 on this. The number we get back is depressing, but hopefully this will motivate use to fix it;) This is a great thing to work on at hackthons/sprints. I would propose that for scipy this year the focus should be on getting the test coverage up. There is a lot of low-hanging fruit (like functions that we never call in the test suite).

tacaswell · 2015-05-03T20:31:33Z

<rant>
Or disutils/setuptools/distribute/nose hates me.
</rant>

jenshnielsen · 2015-05-03T21:05:05Z

I am definitely 👍 I agree with all that you said about motivation, sprints etc And while I think we should avoid getting into to rigorous rules about coverage when merging PRs I think it is a useful tool to judge if additional tests are needed when writing a PR

tacaswell · 2015-05-03T21:16:18Z

There is a knob to set how much the coverage has to decrease to be a 'fail', once this is up and running, we should set that to be 1 or 2 % which will keep bug-fixes from failing, but will hopefully fail on new features with out tests.

efiring · 2015-05-03T21:17:32Z

I don't know how to use this effectively. I've only poked around a little bit on the "coverage" link, but what is returned appears to be quite a mish-mash--not a very good S/N ratio.
So as usual, I'm a bit skeptical. Is this where our sparse development energy needs to be directed? Certainly there are many places where more and better tests would help, but it is not clear to me that the barrage of stats provided by this tool will be much of a help in locating them, and in designing tests that are worth their increase in LOC.

WeatherGod · 2015-05-04T03:15:35Z

I would have it set not to fail. It is good information for us, and it can
certainly show were focus is needed. But it isn't all that useful to
non-regulars. We have been fairly attentive at adding tests as we go along,
but we need help directing our energies for the rest of the codebase.
On May 3, 2015 5:17 PM, "Eric Firing" notifications@github.com wrote:

I don't know how to use this effectively. I've only poked around a little
bit on the "coverage" link, but what is returned appears to be quite a
mish-mash--not a very good S/N ratio.
So as usual, I'm a bit skeptical. Is this where our sparse development
energy needs to be directed? Certainly there are many places where more and
better tests would help, but it is not clear to me that the barrage of
stats provided by this tool will be much of a help in locating them, and in
designing tests that are worth their increase in LOC.

—
Reply to this email directly or view it on GitHub
#4367 (comment)
.

jenshnielsen · 2015-11-19T08:58:48Z

replaced by #4678

tacaswell added the status: needs review label Apr 22, 2015

tacaswell force-pushed the tst_coveralls branch 3 times, most recently from d237b8d to 710bf5e Compare April 22, 2015 06:03

tacaswell added this to the proposed next point release milestone Apr 22, 2015

jenshnielsen reviewed Apr 22, 2015
View reviewed changes

tacaswell force-pushed the tst_coveralls branch from 858f796 to 4b01b40 Compare May 2, 2015 01:59

tacaswell force-pushed the tst_coveralls branch from 61f696f to 43c3d70 Compare May 3, 2015 17:40

tacaswell modified the milestones: next point release, proposed next point release May 3, 2015

tacaswell added 2 commits May 3, 2015 14:06

TST : enable coveralls

17ae8c9

- also update nose

tacaswell force-pushed the tst_coveralls branch 3 times, most recently from 30142c1 to f470e0d Compare May 3, 2015 18:22

tacaswell added 2 commits May 3, 2015 14:40

MNT : add coverage output to .gitignore

96a778a

TST : don't upload pep8 test to coveralls

d975243

tacaswell force-pushed the tst_coveralls branch from 0019fd6 to d975243 Compare May 3, 2015 18:40

tacaswell added 2 commits May 3, 2015 15:19

TST : tweak cover report filters

649702c

TST : don't report coverage on testing code

1a93e9d

TST : do not create an egg during installation

762a3bb

tacaswell force-pushed the tst_coveralls branch from bbddbc6 to 762a3bb Compare May 3, 2015 20:09

TST : send the record information to the void

52dd5bf

tacaswell modified the milestones: proposed next point release, next point release Jun 16, 2015

jenshnielsen mentioned this pull request Jul 13, 2015

TST: Enable coveralls/codecov code coverage #4678

Merged

jenshnielsen closed this Nov 19, 2015

tacaswell removed the status: needs review label Nov 19, 2015

tacaswell deleted the tst_coveralls branch January 1, 2016 22:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TST : enable coveralls #4367

TST : enable coveralls #4367

tacaswell commented Apr 22, 2015

tacaswell commented Apr 22, 2015

jenshnielsen Apr 22, 2015

tacaswell Apr 22, 2015

jenshnielsen commented Apr 22, 2015

jenshnielsen Apr 22, 2015

tacaswell commented Apr 22, 2015

tacaswell commented May 2, 2015

jenshnielsen commented May 2, 2015

coveralls commented May 3, 2015

tacaswell commented May 3, 2015

tacaswell commented May 3, 2015

jenshnielsen commented May 3, 2015

tacaswell commented May 3, 2015

efiring commented May 3, 2015

WeatherGod commented May 4, 2015

jenshnielsen commented Nov 19, 2015

TST : enable coveralls #4367

TST : enable coveralls #4367

Conversation

tacaswell commented Apr 22, 2015

tacaswell commented Apr 22, 2015

jenshnielsen Apr 22, 2015

Choose a reason for hiding this comment

tacaswell Apr 22, 2015

Choose a reason for hiding this comment

jenshnielsen commented Apr 22, 2015

jenshnielsen Apr 22, 2015

Choose a reason for hiding this comment

tacaswell commented Apr 22, 2015

tacaswell commented May 2, 2015

jenshnielsen commented May 2, 2015

coveralls commented May 3, 2015

tacaswell commented May 3, 2015

tacaswell commented May 3, 2015

jenshnielsen commented May 3, 2015

tacaswell commented May 3, 2015

efiring commented May 3, 2015

WeatherGod commented May 4, 2015

jenshnielsen commented Nov 19, 2015