Skip to content

Scipy sparse inplace fails after __array_ufunc__ merge #9019

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
charris opened this issue Apr 28, 2017 · 11 comments · Fixed by #9021
Closed

Scipy sparse inplace fails after __array_ufunc__ merge #9019

charris opened this issue Apr 28, 2017 · 11 comments · Fixed by #9021

Comments

@charris
Copy link
Member

charris commented Apr 28, 2017

The following works before merge of #8247.

In [1]: import scipy.sparse as sp

In [2]: a = np.array((2,3))

In [3]: a += sp.lil_matrix(a)

After merge

In [1]: import scipy.sparse as sp

In [2]: a = np.array((2,3))

In [3]: a += sp.lil_matrix(a)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-3-4472ce1a04df> in <module>()
----> 1 a += sp.lil_matrix(a)

TypeError: ufunc 'add' output (typecode 'O') could not be coerced to provided output parameter (typecode 'l') according to the casting rule ''same_kind''

The sparse matrices depend on the fact that they are cast to objects, so this is a function of how object arrays are handled,

@charris
Copy link
Member Author

charris commented Apr 28, 2017

Probably missing priority forwarding for inplace operators.

@pv
Copy link
Member

pv commented Apr 28, 2017

Probably indeed __array_priority__ handling. In previous numpy:

>>> a.__iadd__(sp.lil_matrix(a))
NotImplemented

current master:

>>> a.__iadd__(sp.lil_matrix(a))
Traceback (most recent call last):
  File "<ipython-input-3-c4acac29d15e>", line 1, in <module>
    a.__iadd__(sp.lil_matrix(a))
TypeError: ufunc 'add' output (typecode 'O') could not be coerced to provided output parameter (typecode 'l') according to the casting rule ''same_kind''

@charris
Copy link
Member Author

charris commented Apr 28, 2017

Hmm, we are still depending on ufuncs returning NotImplemented, that doesn't seem right. This seems like the sort of thing that should be handled at the ndarray level.

@charris
Copy link
Member Author

charris commented Apr 28, 2017

This fixes the inplace add

diff --git a/numpy/core/src/multiarray/number.c b/numpy/core/src/multiarray/number.c
index 0d667ac..2ee2c2d 100644
--- a/numpy/core/src/multiarray/number.c
+++ b/numpy/core/src/multiarray/number.c
@@ -628,6 +628,7 @@ array_bitwise_xor(PyArrayObject *m1, PyObject *m2)
 static PyObject *
 array_inplace_add(PyArrayObject *m1, PyObject *m2)
 {
+    BINOP_GIVE_UP_IF_NEEDED(m1, m2, nb_add, array_add);
     return PyArray_GenericInplaceBinaryFunction(m1, m2, n_ops.add);
 }

The result is that a is replaced by a matrix, python rules, but that is as before.

I seems that all the inplace operators suffer the same omission. Major test failure ;)

@charris
Copy link
Member Author

charris commented Apr 28, 2017

@juliantaylor What happens to the elision code if the inplace function returns NotImplemented?

EDIT: Looks like that is already checked before elision is attempted, so should not be a problem. However, the same check is now repeated twice in the path if elision is done, so might want to bypass that at some point.

@eric-wieser
Copy link
Member

Called it ;)

@eric-wieser
Copy link
Member

eric-wieser commented Apr 28, 2017

There's a hidden discussion here with some rationale

The result is that a is replaced by a matrix, python rules, but that is as before.

I think this was behaviour that was ruled as undesirable in that discussion. Evidently, it's too late for us to decide that.

This needs a doc fix as well as @charris's patch

@charris
Copy link
Member Author

charris commented Apr 29, 2017

See #9021.

@charris
Copy link
Member Author

charris commented Apr 29, 2017

@pv Would it be appropriate to raise a DeprecationWarning in this case while preserving past behavior? The return value really isn't consistent with the ndarray inplace behavior.

@pv
Copy link
Member

pv commented Apr 29, 2017 via email

@pv
Copy link
Member

pv commented Apr 29, 2017 via email

charris added a commit to charris/numpy that referenced this issue Apr 29, 2017
For backward compatibility, we need to let `__array_priority__`
determine the override for inplace ops. If `__array_ufunc__ = None`
on the right hand side it is still the case that TypeError will be
raised.

This fixes SciPy test failures of the type

   ndarray += sparse_array

Closes numpy#9019.
mherkazandjian pushed a commit to mherkazandjian/numpy that referenced this issue May 30, 2017
For backward compatibility, we need to let `__array_priority__`
determine the override for inplace ops. If `__array_ufunc__ = None`
on the right hand side it is still the case that TypeError will be
raised.

This fixes SciPy test failures of the type

   ndarray += sparse_array

Closes numpy#9019.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants