Skip to content

TST : enable coveralls #4367

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 8 commits into from
Closed

TST : enable coveralls #4367

wants to merge 8 commits into from

Conversation

tacaswell
Copy link
Member

Attempt to turn on coveralls to track how we are doing on test coverage.

@tacaswell tacaswell force-pushed the tst_coveralls branch 3 times, most recently from d237b8d to 710bf5e Compare April 22, 2015 06:03
@tacaswell
Copy link
Member Author

I have no idea why this isn't working on travis. It runs on my local machine which makes me think there is some version issues going on here.

@tacaswell tacaswell added this to the proposed next point release milestone Apr 22, 2015
jenshnielsen
jenshnielsen reviewed Apr 22, 2015
View reviewed changes
.travis.yml
@@ -24,7 +24,7 @@ matrix:
env: BUILD_DOCS=true

install:
- pip install -q --use-mirrors nose python-dateutil $NUMPY pep8 pyparsing pillow sphinx!=1.3.0
- pip install python-dateutil $NUMPY pep8 pyparsing pillow sphinx!=1.3.0 coveralls coverage
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On purpose that nose is dropped here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, nose is installed by default on travis, I was looking for version conflict issues (there were a whole bunch of force-pushed attempts at this).

@jenshnielsen
Copy link
Member

Perhaps try running the tests without gdb?

jenshnielsen
jenshnielsen reviewed Apr 22, 2015
View reviewed changes
.coveragerc
source=matplotlib
[report]
omit =
*/python?.?/*
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To actually get a cover report locally I had to remove this line (/python?.?/)

I don't think this is actually related to the Travis issue which I cannot reproduce.

@tacaswell
Copy link
Member Author

Progress, I can reproduce this on my work machine.

Datadump for my self for later:

(dd_py2k)tcaswell@tcaswellpc1:~/other_source/matplotlib$ conda list
# packages in environment at /home/tcaswell/.virtualenvs/dd_py2k:
#
Could not extract name and version from: 'You are using pip version 6.0.8, however version 6.1.1 is available.'
Could not extract name and version from: "You should consider upgrading via the 'pip install --upgrade pip' command."
Could not extract name and version from: 'Error when trying to get requirement for VCS system Command "/home/tcaswell/local_installs/bin/git config remote.origin.url" failed with error code 1 in /home/tcaswell/bnl_source/fileStore, falling back to uneditable format'
Could not extract name and version from: 'Could not determine repository location of /home/tcaswell/bnl_source/fileStore'
Could not extract name and version from: 'Error when trying to get requirement for VCS system Command "/home/tcaswell/local_installs/bin/git config remote.origin.url" failed with error code 1 in /home/tcaswell/bnl_source/metadataStore, falling back to uneditable format'
Could not extract name and version from: 'Could not determine repository location of /home/tcaswell/bnl_source/metadataStore'
atom                      0.3.9                    py27_0  
backports.ssl-match-hostname 3.4.0.2                   <pip>
bubblegum                 v0.0.2.post2    2_g2a3d9ec_np19py27  
cairo                     1.12.18                       0  
channelarchiver           0.0.4.post3     3_g53bf081_np19py27  
coverage                  3.7.1                     <pip>
coveralls                 0.5                       <pip>
curl                      7.38.0                        0  
dateutil                  2.1                      py27_2  
decorator                 3.4.0                    py27_0  
docopt                    0.6.2                     <pip>
docutils                  0.12                     py27_0  
enaml                     0.9.8                    py27_0  
epics-base                3.14.12                      12  
filestore                 0.5.0.post0               <pip>
freetype                  2.4.10                        0  
h5py                      2.4.0                np19py27_0  
hdf5                      1.8.14                        0  
ipython                   2.4.1                    py27_0  
ipython-notebook          2.3.1                    py27_0  
jdcal                     1.0                      py27_0  
jinja2                    2.7.3                    py27_1  
jpeg                      8d                            0  
jsonschema                2.4.0                    py27_0  
keyring                   3.8                      py27_0  
kiwisolver                0.1.3                    py27_0  
libnetcdf                 4.3.2                         1  
libpng                    1.5.13                        1  
libsodium                 0.4.5                         0  
libtiff                   4.0.2                         1  
libxml2                   2.9.0                         0  
libxslt                   1.1.28                        0  
lmfit                     0.8.3rc1.post4  4_gac73a37_np19py27  
lxml                      3.4.2                    py27_0  
markupsafe                0.23                     py27_0  
matplotlib                1.4.3                np19py27_0  
metadatastore             0.1.0.dev0                <pip>
mongoengine               0.8.7                    py27_0  
ncurses                   5.9                           6  
netcdf4                   1.1.6                np19py27_0  
networkx                  1.9.1                    py27_0  
nose                      1.3.4                    py27_0  
numpy                     1.9.2                    py27_0  
numpydoc                  0.5                      py27_0  
openpyxl                  2.0.2                    py27_0  
openssl                   1.0.1k                        0  
ophyd                     v0.0.4                   py27_0  
pandas                    0.15.2               np19py27_0  
pcaspy                    0.5.1                    py27_0  
pep8                      1.5.7                    py27_0  
pillow                    2.7.0                    py27_0  
pims                      0.2.1                np19py27_0  
pip                       6.0.8                    py27_0  
pixman                    0.26.2                        0  
ply                       3.4                      py27_0  
py2cairo                  1.10.0                   py27_2  
pyepics                   3.2.4rc1.post20 20_g8da13e1_np19py27  
pyflakes                  0.8.1                    py27_0  
pygments                  2.0.2                    py27_0  
pymongo                   2.8                      py27_0  
pyolog                    v4.0.0.post0    0_ge5b0507_np19py27  
pyparsing                 2.0.3                    py27_0  
pyqt                      4.10.4                   py27_0  
python                    2.7.9                        10  
python-dateutil           1.5                       <pip>
pytz                      2014.9                   py27_0  
pyyaml                    3.11                     py27_0  
pyzmq                     14.5.0                   py27_0  
qt                        4.8.5                         0  
readline                  6.2                          12  
requests                  2.5.3                    py27_0  
scikit-image              0.11.2               np19py27_0  
scikit-xray               0.0.3.post117   117_g5ac2ba4_np19py27  
scipy                     0.15.1               np19py27_0  
setuptools                14.0                     py27_0  
sip                       4.15.5                   py27_0  
six                       1.9.0                    py27_0  
sphinx                    1.2.3                    py27_0  
sphinx-bootstrap-theme    0.4.5                    py27_0  
sphinxcontrib-napoleon    0.2.8                    py27_0  
sqlite                    3.8.4.1                       0  
ssl_match_hostname        3.4.0.2                  py27_0  
system                    5.8                           1  
tk                        8.5.15                        0  
tornado                   4.0.2                    py27_0  
tzlocal                   1.1.2                    py27_0  
xraylib                   master               np19py27_0  
yaml                      0.1.4                         0  
zeromq                    4.0.4                         0  
zlib                      1.2.8                         0  

(dd_py2k)tcaswell@tcaswellpc1:~/other_source/matplotlib$ nosetests --plugins
Plugin capture
Plugin failuredetail
Plugin xunit
Plugin deprecated
Plugin skip
Plugin multiprocess
Plugin logcapture
Plugin coverage
Plugin attributeselector
Plugin doctest
Plugin profile
Plugin id
Plugin allmodules
Plugin collect-only
Plugin isolation
Plugin pdb

@tacaswell
Copy link
Member Author

@jenshnielsen The issue seems to be that travis has nose 1.3.4 install and pip install nose helpfully says 'already installed, use --upgrade to upgarde'.

@jenshnielsen
Copy link
Member

Makes sense I think it fails now because pip install nose>=1.3.6 doesn't upgrade nose. I guess we need to do either pip install nose==1.3.6 or pip install --upgrade nose?

@tacaswell tacaswell modified the milestones: next point release, proposed next point release May 3, 2015
@coveralls
Copy link

Coverage Status

Changes Unknown when pulling 43c3d70 on tacaswell:tst_coveralls into * on matplotlib:master*.

tacaswell added 2 commits May 3, 2015 14:06
@tacaswell
TST : enable coveralls
17ae8c9 
 - also update nose
@tacaswell
TST : move coverage output where coveralls expects
983e1be 
We run tests out-of-repo which is to ensure that we pick up
the installed version of mpl, not version which happens to be
in the current directory (but I am not sure we strictly need this
any more now that absolute imports are a thing).  The coveralls scripts
assume that you are running your tests _in_ the repo and are trying to
extract git information from the directory they are run in, hence are
failing for everything but the doc builds (which don't run nose so
have no coverage).

This exclude the doc build from uploading to coveralls and moves
the cover output to be where coveralls expects it to be (and cds
back into the repo).
@tacaswell tacaswell force-pushed the tst_coveralls branch 3 times, most recently from 30142c1 to f470e0d Compare May 3, 2015 18:22
@tacaswell
Copy link
Member Author

woo, I think this is (finally) converging.

I have turned off the comment spam so that should not show up anywhere else again.

@jenshnielsen @mdboom @efiring @WeatherGod Thoughts? I am 👍 on this. The number we get back is depressing, but hopefully this will motivate use to fix it;) This is a great thing to work on at hackthons/sprints. I would propose that for scipy this year the focus should be on getting the test coverage up. There is a lot of low-hanging fruit (like functions that we never call in the test suite).

@tacaswell
Copy link
Member Author

<rant>
Or disutils/setuptools/distribute/nose hates me.
</rant>

@jenshnielsen
Copy link
Member

I am definitely 👍 I agree with all that you said about motivation, sprints etc And while I think we should avoid getting into to rigorous rules about coverage when merging PRs I think it is a useful tool to judge if additional tests are needed when writing a PR

@tacaswell
Copy link
Member Author

There is a knob to set how much the coverage has to decrease to be a 'fail', once this is up and running, we should set that to be 1 or 2 % which will keep bug-fixes from failing, but will hopefully fail on new features with out tests.

@efiring
Copy link
Member

efiring commented May 3, 2015

I don't know how to use this effectively. I've only poked around a little bit on the "coverage" link, but what is returned appears to be quite a mish-mash--not a very good S/N ratio.
So as usual, I'm a bit skeptical. Is this where our sparse development energy needs to be directed? Certainly there are many places where more and better tests would help, but it is not clear to me that the barrage of stats provided by this tool will be much of a help in locating them, and in designing tests that are worth their increase in LOC.

@WeatherGod
Copy link
Member

I would have it set not to fail. It is good information for us, and it can
certainly show were focus is needed. But it isn't all that useful to
non-regulars. We have been fairly attentive at adding tests as we go along,
but we need help directing our energies for the rest of the codebase.
On May 3, 2015 5:17 PM, "Eric Firing" notifications@github.com wrote:

I don't know how to use this effectively. I've only poked around a little
bit on the "coverage" link, but what is returned appears to be quite a
mish-mash--not a very good S/N ratio.
So as usual, I'm a bit skeptical. Is this where our sparse development
energy needs to be directed? Certainly there are many places where more and
better tests would help, but it is not clear to me that the barrage of
stats provided by this tool will be much of a help in locating them, and in
designing tests that are worth their increase in LOC.

—
Reply to this email directly or view it on GitHub
#4367 (comment)
.

@tacaswell tacaswell modified the milestones: proposed next point release, next point release Jun 16, 2015
@jenshnielsen jenshnielsen mentioned this pull request Jul 13, 2015
Merged
@jenshnielsen
Copy link
Member

replaced by #4678

@tacaswell tacaswell deleted the tst_coveralls branch January 1, 2016 22:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
v2.1
Development

Successfully merging this pull request may close these issues.
5 participants
@tacaswell @jenshnielsen @coveralls @efiring @WeatherGod