-
-
Notifications
You must be signed in to change notification settings - Fork 10.8k
NEP: Add initial draft of NEP-31: Context-local and global overrides of the NumPy API #14389
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
DOC: update NEP 31 for vendoring and being complementary to NEP-13/18
f5da97b
to
58ea2bd
Compare
this, depending on the user's style:: | ||
|
||
import numpy.overridable as unumpy | ||
import numpy as np |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What's the difference from simply import unumpy
? Also, I think it's very confusing that import numpy.overridable
is an alias to a different top-level package with a completely different name.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What we're proposing here is that a completed version of unumpy
be vendored into NumPy as numpy.overridable
, with a soft dependency on unumpy
in case it exists.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
then it should be explicit where this comes from and not a top-level import: from numpy.uarray import overridable as unumpy
or even better from numpy.uarray import unumpy
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My reasoning was that people reach for np
whenever using NumPy, so importing it as np
keeps familiarity as the API is mirrored.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've changed the import to from numpy import overridable as unumpy
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
SciPy FFT already uses it, but I did not notice too much discussion on scipy (maybe I missed it). And the use case is probably quite a lot simpler.
simpler in some ways, but not strictly simpler. for example, there are multiple backends there that all consume numpy arrays. a protocol like __array_function__
that's attached to the array itself cannot deal with that.
It might be worth noting that agreeing on one protocol/library (for the most part) means that it is OK if the need for new overriding protocols is not within numpy itself, but within the ecosystem.
I'm not sure I got your point here. can you expand on this a bit?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
simpler in some ways, but not strictly simpler. for example, there are multiple backends there that all consume numpy arrays. a protocol like array_function that's attached to the array itself cannot deal with that.
Right, it is different. I am actually very much interested in how the two would/can interact (but I already asked that a few times in the other comments I think).
I'm not sure I got your point here. can you expand on this a bit?
Well, if Scipy uses this, and say scikit-learn uses uarray
then you at least have the nice effect that library author can learn the same tool for many different things (and more users means better docs, etc.). Maybe not much of a point.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah yes. Indeed, uarray
is meant as a generic backend creation mechanism. There's not so many of those to choose from in Python land. Most backends are ad-hoc and specific to one library.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am actually very much interested in how the two would/can interact
It'll be nested - if there's a unumpy
function that's also covered by __array_ufunc__
or __array_function__
, then those protocols will be used like normal when the numpy
backend to unumpy
is used. In case another backend is used (dask/sparse/etc), then of course the protocols do nothing.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, there has to be an opt-in/new API somewhere, but what are the real advantages of having it inside numpy proper? As far as I can see:
1. `uarray` (really more general library than "array", which is fine) is the toolchain, a `unumpy` may want to vendor it to ensure stable API. * If the proposal is to try import the non-vendored version, I am actually a bit less sure about that. 2. `unumpy` a reduced numpy-like API, which provides backend switching possibilities based on `uarray`.
An explicit import with matching API is of course the cleanest (and only reasonable thing). What I am missing is an argument why it should live within the numpy namespace or even within numpy at all and not just in the numpy organization (with documentation pointing it out).
Also maybe to be clear. Unlike
scipy.fft
the main target ofunumpy
I think are downstream packages such asscikit-learn
or so, to allow them to easier support multiple array-likes with a single code base. These already have a big number of dependencies, so it is worth vendoring in the numpy namespace?
While I don't mind either way, I'm +1 on moving unumpy
into the NumPy repo org and +0.7 on vendoring.
cc @pentschev |
Just a note, EuroSciPy is this week. So some people might be absorbed by that and probably won't have time to look at this more closely until next week. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please do not take my comments very serious. I am not very good at writing this type of documents. I think the Motivation could be much nicer/clearer. Right now I read it as: "This NEP solves a bit more than NEP-30" (at this time, but with more complexity). But I think unumpy actually covers most of the NumPy API, so that you actually do have a secondary override mechanism next to __array_function__
. Should unumpy
still use __array_function__
if it ends up with the numpy backend, or should it coerce?
Thanks for all the feedback, @seberg! Would you like me to add you to the list of authors? |
7559259
to
419b42c
Compare
Co-authored-by: Ralf Gommers <ralf.gommers@gmail.com> Co-authored-by: Peter Bell <peterbell10@live.co.uk>
I think this version has stabilized a little, and discussion is focused more on the bigger picture than on the details in this PR at the moment. So this may be a good point to merge it (with status Draft). |
To be honest, the big picture discussion is important, in the sense that at least to me it is still impossible to judge how the NEP fits into it, and without it I am still having difficulties to even start make up my mind seriously. Mind you, if library authors started to signal intent of using this, that would probably remove my doubts. I do think it reads much better now. But it still makes vague promises in the sense that I have no clue that it can deliver them. I just tried to write a function |
A backend can be easily set as default at import-time. See this documentation page |
The motivation and usage section is now more extensive than for most other NEPs, even though it still could use a bigger overhaul at some point. I think we should merge this with draft status. I don't expect it to change much soon, we need a separate document first that covers all dispatching we want/need to offer users, independent of the whole protocols vs multimethods discussion. |
Sounds good to me. @rgommers feel free to merge! |
Okay, in it goes. Thanks for the feedback and discussion everyone! |
This PR adds the initial draft version of NEP-31. Posted to the mailing list: https://mail.python.org/pipermail/numpy-discussion/2019-August/079952.html