@mattip seemed to suggest that #10042 fixes the pypy tests, but maybe I misunderstood that.

IMO this should just be merged since it doesn't actually change any code and allows reviewers to see the results of the pypy tests. #10042 isn't merged yet, so the pypy tests would tend to fail (I can't personally vouch for that PR though).

Is there any way to make that test give more useful output? There's not much I can do to track down the problem in this output

nose version 1.3.7
.......................E....E................S.............................................................................................................................................S.................................E.................S..................................................................................................................................E..............................................................................KKK.EE.EEE.EEEE.................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................KK.........................E........S.......................................................F..................................................................................................................................................................................................................................................................................RPython traceback:
  File "pypy_module_pypyjit.c", line 183, in portal_12
  File "pypy_interpreter_2.c", line 35363, in handle_bytecode__AccessDirect_None
  File "pypy_interpreter_2.c", line 46582, in dispatch_bytecode__AccessDirect_None
*** Invalid usage of a dying CPython object ***
cpyext, the emulation layer, detected that while it is calling
an object's tp_dealloc, the C code calls back a function that
tries to recreate the PyPy version of the object.  Usually it
means that tp_dealloc calls some general PyXxx() API.  It is
a dangerous and potentially buggy thing to do: even in CPython
the PyXxx() function could, in theory, cause a reference to the
object to be taken and stored somewhere, for an amount of time
exceeding tp_dealloc itself.  Afterwards, the object will be
freed, making that reference point to garbage.
>>> PyPy could contain some workaround to still work if
you are lucky, but it is not done so far; better fix the bug in
the CPython extension.
./tools/travis-test.sh: line 105:  6158 Aborted                 (core dumped) $PYTHON ../tools/test-installed-numpy.py --mode=full

Perhaps run the tests in verbose mode or something so that the actual test names print out?

That test failure appears to be non-deterministic because I couldn't see it in the last test run. Running the tests in verbose mode is an option but not sure that should be done on the CI. Do you want verbose mode for pypy only?

Ooh, another approach would be to just enable faulthandler in the tests, assuming that PyPy has that. That would be generally useful on all test setups.

Beat me to it, was about to link to nose-faulthandler!

Note that the faulthandler module is only available on python 3, so I think nose-faulthandler needs to be installed conditionally

That's not the problem. nose-faulthandler should always be installed, but it's triggering a dependency.

@larsoner Please update nose-faulthandler on pypi. In the mean time, I will work around the problem.

Hmm, now it seems to hang

Do you want to disable faulthandler then?

Don't look now, but it might be unhung.

faulthandler seems pretty handy for all the other modes, so I'd be inclined to keep that.

Maybe we should not run the full tests for pypy yet, if that's what's causing it to be slow? Having tests take longer to complete is a pain, and it seems travis won't mark the job as complete even if all the required tests pass.

No, the fact is that pypy is slow for running tests. We could skip it for PRs. Would you prefer that?

I should be the last one to criticize, but:

• Travis is running an old version of PyPy, 5.8.0 which is considered "old". Version 5.9.0 released in October fixes many issues related to running pandas and numpy. Is there a way to make travis use the latest released PyPy like the portable one (runs on most versions of linux) available here?
• PyPy3 is still beta-quality, for instance virtualenv does not install on it yet
• doctests will still fail, even after pr ENH: Nditer as context manager #9998 and BUG: Allow int to be called on nested object arrays, fix np.str_.__int__ #10042. Fixing that will require assiging the C-level pointers before calling PyType_Ready, but that is a whole `nother story.

Travis is running an old version of PyPy, 5.8.0 which is considered "old". Version 5.9.0 released in October fixes many issues related to running pandas and numpy. Is there a way to make travis use the latest released PyPy like the portable one (runs on most versions of linux) available here?

File an issue with Travis CI.

PyPy3 is still beta-quality, for instance virtualenv does not install on it yet

It passes most of the tests, and I think it should pass nearly all of them when nditer is merged. The goal is to discover bugs in the tests. If it's more likely to fail, that's good.

I don't see this being merged though due to the timing issue.

FTR, there's an example here of fetching and setting up the latest pypy3 release in a travis build script: https://github.com/python-trio/trio/blob/3edfafeedef4071646a9015e28be01f83dc02f94/ci/travis.sh#L52-L62

@mattip not sure what you mean about virtualenv not working on pypy3, you can see in the snippet above that I've been using it for months...

@njsmith Just to be clear, I closed this because people are not going to tolerate pypy taking 30 minutes to finish the tests. That's not unique to numpy; every project that I've seen takes at least two times as long to test.

@njsmith virtualenv fails on our 32 bit linux chdir buildbot which uses Ubuntu 12.04, and this travis build also fails.

@xoviat ah, true, that's a discussion we'll probably need to have at some point. 30 minutes seems excessive when most of our tests runs are ~5 minutes -- though I guess you have it running all the slow tests too? The next pypy release should have speedups for cpyext calls that might make a significant difference... or not. Another trick that can be useful is to convince Travis to kick off the pypy run first, since the travis run limit means we're doing 2-3 separate passes to get all the other tests finished.

@mattip ah weird -- it's been fine for me on travis's 64-bit 14.04. Not sure what's going on with that travis build -- the error messages are about cython?

You need to open the install step. This is the correct link to failing virtualenv

@mattip Huh, lots of weird stuff going on there. That's using a pinned old version of virtualenv -- could that be the problem? I wonder if we even still need to do that. And then why doesn't our script notice when the virtualenv call fails. And why do we then pip install -U virtualenv after we've already set up a virtualenv using an old version...

Please update nose-faulthandler on pypi.

In principle this is possible. nose itself is unmaintained, though, so I am disinclined to maintain the plugin for it. Installing from GitHub should be doable with pip, though, and it sounds like you got to that solution anyway.