Skip to content

NEP: Add initial draft of NEP-31: Context-local and global overrides of the NumPy API #14389

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 27 commits into from
Oct 18, 2019

Conversation

hameerabbasi
Copy link
Contributor

@hameerabbasi hameerabbasi commented Aug 28, 2019

This PR adds the initial draft version of NEP-31. Posted to the mailing list: https://mail.python.org/pipermail/numpy-discussion/2019-August/079952.html

@hameerabbasi hameerabbasi force-pushed the uarray branch 2 times, most recently from f5da97b to 58ea2bd Compare August 28, 2019 12:49
@hameerabbasi
Copy link
Contributor Author

this, depending on the user's style::

import numpy.overridable as unumpy
import numpy as np
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the difference from simply import unumpy? Also, I think it's very confusing that import numpy.overridable is an alias to a different top-level package with a completely different name.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What we're proposing here is that a completed version of unumpy be vendored into NumPy as numpy.overridable, with a soft dependency on unumpy in case it exists.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

then it should be explicit where this comes from and not a top-level import: from numpy.uarray import overridable as unumpy or even better from numpy.uarray import unumpy.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My reasoning was that people reach for np whenever using NumPy, so importing it as np keeps familiarity as the API is mirrored.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've changed the import to from numpy import overridable as unumpy.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SciPy FFT already uses it, but I did not notice too much discussion on scipy (maybe I missed it). And the use case is probably quite a lot simpler.

simpler in some ways, but not strictly simpler. for example, there are multiple backends there that all consume numpy arrays. a protocol like __array_function__ that's attached to the array itself cannot deal with that.

It might be worth noting that agreeing on one protocol/library (for the most part) means that it is OK if the need for new overriding protocols is not within numpy itself, but within the ecosystem.

I'm not sure I got your point here. can you expand on this a bit?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

simpler in some ways, but not strictly simpler. for example, there are multiple backends there that all consume numpy arrays. a protocol like array_function that's attached to the array itself cannot deal with that.

Right, it is different. I am actually very much interested in how the two would/can interact (but I already asked that a few times in the other comments I think).

I'm not sure I got your point here. can you expand on this a bit?

Well, if Scipy uses this, and say scikit-learn uses uarray then you at least have the nice effect that library author can learn the same tool for many different things (and more users means better docs, etc.). Maybe not much of a point.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yes. Indeed, uarray is meant as a generic backend creation mechanism. There's not so many of those to choose from in Python land. Most backends are ad-hoc and specific to one library.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am actually very much interested in how the two would/can interact

It'll be nested - if there's a unumpy function that's also covered by __array_ufunc__ or __array_function__, then those protocols will be used like normal when the numpy backend to unumpy is used. In case another backend is used (dask/sparse/etc), then of course the protocols do nothing.

Copy link
Contributor Author

@hameerabbasi hameerabbasi Sep 4, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, there has to be an opt-in/new API somewhere, but what are the real advantages of having it inside numpy proper? As far as I can see:

1. `uarray` (really more general library than "array", which is fine) is the toolchain, a `unumpy` may want to vendor it to ensure stable API.
   
   * If the proposal is to try import the non-vendored version, I am actually a bit less sure about that.

2. `unumpy` a reduced numpy-like API, which provides backend switching possibilities based on `uarray`.

An explicit import with matching API is of course the cleanest (and only reasonable thing). What I am missing is an argument why it should live within the numpy namespace or even within numpy at all and not just in the numpy organization (with documentation pointing it out).

Also maybe to be clear. Unlike scipy.fft the main target of unumpy I think are downstream packages such as scikit-learn or so, to allow them to easier support multiple array-likes with a single code base. These already have a big number of dependencies, so it is worth vendoring in the numpy namespace?

While I don't mind either way, I'm +1 on moving unumpy into the NumPy repo org and +0.7 on vendoring.

@jakirkham
Copy link
Contributor

cc @pentschev

@jakirkham
Copy link
Contributor

Just a note, EuroSciPy is this week. So some people might be absorbed by that and probably won't have time to look at this more closely until next week.

Copy link
Member

@seberg seberg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please do not take my comments very serious. I am not very good at writing this type of documents. I think the Motivation could be much nicer/clearer. Right now I read it as: "This NEP solves a bit more than NEP-30" (at this time, but with more complexity). But I think unumpy actually covers most of the NumPy API, so that you actually do have a secondary override mechanism next to __array_function__. Should unumpy still use __array_function__ if it ends up with the numpy backend, or should it coerce?

@hameerabbasi
Copy link
Contributor Author

Thanks for all the feedback, @seberg! Would you like me to add you to the list of authors?

@hameerabbasi hameerabbasi force-pushed the uarray branch 3 times, most recently from 7559259 to 419b42c Compare September 5, 2019 11:06
Co-authored-by: Ralf Gommers <ralf.gommers@gmail.com>
Co-authored-by: Peter Bell <peterbell10@live.co.uk>
@rgommers
Copy link
Member

rgommers commented Sep 8, 2019

I think this version has stabilized a little, and discussion is focused more on the bigger picture than on the details in this PR at the moment. So this may be a good point to merge it (with status Draft).

@seberg
Copy link
Member

seberg commented Sep 8, 2019

To be honest, the big picture discussion is important, in the sense that at least to me it is still impossible to judge how the NEP fits into it, and without it I am still having difficulties to even start make up my mind seriously. Mind you, if library authors started to signal intent of using this, that would probably remove my doubts.

I do think it reads much better now. But it still makes vague promises in the sense that I have no clue that it can deliver them. I just tried to write a function myfunc = lambda x: unumpy.exp(x), and I cannot use it without setting a backend. But the backend has to be set by the end-user. In other words, a library cannot use unumpy without forcing the end-users to specificy a backend. Of course that is probably a simple matter of defaulting to the numpy backend when first importing it, but it raises question as to whether I can write a single code base supporting both unumpy users and current usage. If that is not trivial, we have a transition that is worse than python2 to python3...

@hameerabbasi
Copy link
Contributor Author

I just tried to write a function myfunc = lambda x: unumpy.exp(x), and I cannot use it without setting a backend. But the backend has to be set by the end-user. In other words, a library cannot use unumpy without forcing the end-users to specificy a backend.

A backend can be easily set as default at import-time. See this documentation page

@rgommers
Copy link
Member

The motivation and usage section is now more extensive than for most other NEPs, even though it still could use a bigger overhaul at some point. I think we should merge this with draft status. I don't expect it to change much soon, we need a separate document first that covers all dispatching we want/need to offer users, independent of the whole protocols vs multimethods discussion.

@shoyer
Copy link
Member

shoyer commented Oct 18, 2019

Sounds good to me. @rgommers feel free to merge!

@rgommers rgommers merged commit c1f56c4 into numpy:master Oct 18, 2019
@rgommers
Copy link
Member

Okay, in it goes. Thanks for the feedback and discussion everyone!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants