Occurs with numpy 1.7.1 and 1.6

Occurs on '1.9.0.dev-ec3603f' too.

when its done comparing the objects which always fail (and set PyErr) it starts comparing the integers which returns -1, 0, 1. But the PyErr is still set which causes python warn on the 1 return.
looks like the OBJECT_compare should clear the python error after its done.

also happens on linux if one reverses the integer list (mac quicksort probably compares in reverse order sometimes)

clearing the error in OBJECT_compare obviously can't be done just for quicksort, maybe one should disable runtime warnings in before npy_quicksort and restore later, or add a special OBJECT_compare which clears python errors to be used by only by quicksort.

Why do we want to clear the error? Shouldn't we just bail out early if a
comparison fails?
On 7 Oct 2013 13:52, "Julian Taylor" notifications@github.com wrote:

clearing the error in OBJECT_compare obviously can't be done just for
quicksort, maybe one should disable runtime warnings in before
npy_quicksort and restore later, or add a special OBJECT_compare which
clears python errors to be used by only by quicksort.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3879#issuecomment-25805639
.

Your comment made me wonder if sorting that causes an error causes the array to be left in a different state than it was before (at least on Windows with numpy 1.6, it does). Not sure if that's an issue.

Apologies that it's a bit hard to read - just note that the integers move.

import numpy as np
import random
class Raiser(object):
    def raises_anything(*args, **kwargs):
        raise TypeError("SOMETHING ERRORED")
    __eq__ = __ne__ = __lt__ = __gt__ = __ge__ = __le__ = raises_anything
arr = np.array([Raiser() for n in range(10)] + list(range(10)))
random.shuffle(arr)


arr
Out[6]: 
array([<__main__.Raiser object at 0x0000000007F7BA20>,
       <__main__.Raiser object at 0x0000000007F7B908>, 5, 8,
       <__main__.Raiser object at 0x0000000007F7B940>, 9,
       <__main__.Raiser object at 0x0000000007F7B978>,
       <__main__.Raiser object at 0x0000000007F7B9E8>, 3,
       <__main__.Raiser object at 0x0000000007F7BA58>, 2,
       <__main__.Raiser object at 0x0000000007F7BA90>,
       <__main__.Raiser object at 0x0000000007F7B9B0>, 1,
       <__main__.Raiser object at 0x0000000007F7B898>, 6, 7, 4, 0,
       <__main__.Raiser object at 0x0000000007F7BAC8>], dtype=object)

original = arr.copy()

arr.sort()
-c:1: RuntimeWarning: tp_compare didn't return -1 or -2 for exception
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-8-f2555d8c7d46> in <module>()
----> 1 arr.sort()

<ipython-input-1-1f1f20d5336e> in raises_anything(*args, **kwargs)
      2 class Raiser(object):
      3     def raises_anything(*args, **kwargs):
----> 4         raise TypeError("SOMETHING ERRORED")
      5     __eq__ = __ne__ = __lt__ = __gt__ = __ge__ = __le__ = raises_anything
      6 arr = np.array([Raiser() for n in range(10)] + list(range(10)))

TypeError: SOMETHING ERRORED

arr
Out[9]: 
array([<__main__.Raiser object at 0x0000000007F7B908>, 5, 8,
       <__main__.Raiser object at 0x0000000007F7B940>, 9,
       <__main__.Raiser object at 0x0000000007F7B978>,
       <__main__.Raiser object at 0x0000000007F7B9E8>,
       <__main__.Raiser object at 0x0000000007F7BA20>, 3,
       <__main__.Raiser object at 0x0000000007F7BA58>, 2,
       <__main__.Raiser object at 0x0000000007F7BA90>,
       <__main__.Raiser object at 0x0000000007F7B9B0>, 1,
       <__main__.Raiser object at 0x0000000007F7B898>, 6, 7, 4, 0,
       <__main__.Raiser object at 0x0000000007F7BAC8>], dtype=object)

original
Out[10]: 
array([<__main__.Raiser object at 0x0000000007F7BA20>,
       <__main__.Raiser object at 0x0000000007F7B908>, 5, 8,
       <__main__.Raiser object at 0x0000000007F7B940>, 9,
       <__main__.Raiser object at 0x0000000007F7B978>,
       <__main__.Raiser object at 0x0000000007F7B9E8>, 3,
       <__main__.Raiser object at 0x0000000007F7BA58>, 2,
       <__main__.Raiser object at 0x0000000007F7BA90>,
       <__main__.Raiser object at 0x0000000007F7B9B0>, 1,
       <__main__.Raiser object at 0x0000000007F7B898>, 6, 7, 4, 0,
       <__main__.Raiser object at 0x0000000007F7BAC8>], dtype=object)

Why do we want to clear the error? Shouldn't we just bail out early if a comparison fails?

we can't as we are using C quicksort for object arrays which has no early bailout option. We could reimplement it of course.

The only way to prevent objects moving is to do a copy before sorting, which is explicitly what one does not want to do with inplace sorting. If the array should be unchanged after a failure the user must use a out of place sort (np.sort)
It is same with pure python inplace sorting.

The only way to prevent objects moving is to do a copy before sorting,
which is explicitly what one does not want to do with inplace sorting. If
the array should be unchanged after a failure the user must use a out of
place sort (np.sort)
It is same with pure python inplace sorting.

Okay, just wanted to check. Thanks for clarifying!

@juliantaylor Is that the reason why npy_quicksort always returns 0? Does that matter?

Yeah, I think objects moving is a non-issue, this isn't an ACID database
and failing compares are pretty weird.

Re: lack of bailout: okay, but then we should still propagate the error
sooner or later? Or is the issue that as soon as we re-enter the
interpreter on the next comparison function, it gets annoyed when it
discovers the lingering exception?

On Mon, Oct 7, 2013 at 3:56 PM, Julian Taylor notifications@github.comwrote:

Why do we want to clear the error? Shouldn't we just bail out early if a
comparison fails?

we can't as we are using C quicksort for object arrays which has no early
bailout option. We could reimplement it of course.

The only way to prevent objects moving is to do a copy before sorting,
which is explicitly what one does not want to do with inplace sorting. If
the array should be unchanged after a failure the user must use a out of
place sort (np.sort)
It is same with pure python inplace sorting.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/3879#issuecomment-25815223
.

@cpclout yes it uses qsort which does not have a return value, the error is propagated via pythons global error state instead.

@njsmith the latter, after each comparison cpython goes into adjust_tp_compare(comparison_result)
the error from the previous failing comparison still lingers causing it to warn about correct results from the current non failing comparison. It is assuming you bail out on error or reset it instead of continuing comparing stuff.
From what I can tell nothing is actually broken by this, you just get a spurious warning.
python3 doesn't seem to care about it (numpy there it treats the error cases as compared equal and errors out later)

python3 doesn't seem to care about it

I think this is because tp_compare isn't allowed in Py3. There are uses of the convenience PyObject_Cmp function but tp_compare seems to be only used in with numpy scalars in Py2

We could reimplement it of course.

I've been planning to do that for the sort functions for several years now ;) All as part of cleaning up the ugly calling code for those sorting functions that get passed a comparison function.

we can't as we are using C quicksort for object arrays

I do have a numpy version of quicksort that has been sitting in a branch for the last 18 mos that, IIRC, does fix that problem.