Mostly looks good. But there is the bike shed: I propose to just call it __implementation__ - partially because our other methods do not refer to numpy either, and partially because I think we would like to encourage eventually, e.g., scipy to wrap its functions similarly (already works for all its special functions with __array_ufunc__).

What I wanted to invoke with the name __numpy_implementation__ was a notion of "the implementation for NumPy's arrays", not "the implementation by NumPy the library". These mean the same thing within NumPy, but this is also compatible overrides for outside functions, e.g., scipy.linalg.solve.__numpy_implementation__. You could even imagine SciPy defining its own first class implementations for other array backends, e.g., scipy.linalg.solve.__cupy_implementation__.

(Either way, this is probably worth clarifying in the text.)

I'm not 100% sure that is right: suppose DASK wants to make its functions more general, is its default implementation a __numpy_implementation__?

On the other hand, I guess that is what you are getting add: with __numpy_implementation__ it is immediately obvious that one is trying to get the one that is relevant for arrays, not just for whatever DASK array. I think I'd still prefer __ndarray_implementation__ then, since that makes clear it is an implementation for the class, not for the project.

I do wonder if for this case we should recommend to cast oneself to ndarray explicitly and use ndarray.__array_function__(...).

I'm not 100% sure that is right: suppose DASK wants to make its functions more general, is its default implementation a __numpy_implementation__?

Dask defines it's own data structures, so it's generic implementations would be something like __dask_implementation__.

On the other hand, I guess that is what you are getting add: with __numpy_implementation__ it is immediately obvious that one is trying to get the one that is relevant for arrays, not just for whatever DASK array. I think I'd still prefer __ndarray_implementation__ then, since that makes clear it is an implementation for the class, not for the project.

Maybe __numpy_array_implementation__? Or the slightly abbreviated __numpy_array_impl__? I think including the name "numpy" is important. Informally, people think of numpy.ndarray as a "numpy array".t __ndarray_implementation__ on its own is too generic, especially when it's an attribute on a function that isn't defined by NumPy, e.g., what does scipy.linalg.solve.__ndarray_implementation__ mean?

Let me try adding some clarifying language about the intent of __numpy_implementation__. But certainly my intention is that this should not preclude adding more generic implementations to NumPy functions, e.g., we can imagine also defining __generic_implementation__ as something that works with duck typing.

I do wonder if for this case we should recommend to cast oneself to ndarray explicitly and use ndarray.__array_function__(...).

We've discussed this before, but this doesn't work as a fall-back in in __array_function__ because we don't have an easy way to find "this array" in *args and **kwargs.

OK, yes, should have remembered the problem with ndarray.__array_function__.

If we're abbreviating, probably __np_array_implementation__, but I think I prefer original suggestion over that.

I prefer __array_implementation__... NumPy uses all of the __array_*__ anyway...

This should probably be accompanied by consensus on the mailing list, as per governance rules.

I prefer __array_implementation__... NumPy uses all of the __array_*__ anyway...

I'm not sure this comparison is entirely apt. So far (until this PR) the __array_function__ protocol has been entirely agnostic as to the underlying array library -- you could just as easily use it with arbitrary functions defined on any arbitrary array type.

I guess the critical question is whether it makes sense to distinguish between array function implementations intended for different types. For example, consider a scipy function that is implemented by coercing its arguments to sparse arrays, e.g., scipy.sparse.linalg.splu. What should the implementation be?

1. splu.__array_implementation__
2. splu.__numpy_implementation__
3. splu.__sparse_implementation__

(1) would preserve existing behavior for splu() on NumPy arrays, assuming we use __array_implementation__ inside np.ndarray.__array_function__.

(2) feels a little strange -- this implementation is not NumPy array specific. (Though it is what should be used on NumPy arrays.)

(3) could make sense, but there's no way NumPy is going to know to call a __sparse_implementation__ method. So at best, using a NumPy array would result in an error, unless we defined both __sparse_implementation__ and __numpy_implementation__.

The advantage of __sparse_implementation__ is that we can imagine it being used inside SparseArray.__array_function__, e.g.,

class SparseArray:
    def __array_function__(self, func, types, args, kwargs):
        if not all(issubclass(t, (SparseArray, np.ndarray) for t in types):
            return NotImplemented
        sparse_func = getattr(func, '__sparse_implementation__', None)
        if sparse_func is not None:
            return sparse_func(*args, **kwargs)
        # look elsewhere for implementations, or return NotImplemented

In contrast, SparseArray.__array_function__ would definitely not call __array_function__ or __numpy_implementation__ internally in a generic way, because densifying a sparse array is often catastrophic. I don't know if this is a real use-case, though.

Thinking more about this, I'm pretty firmly of the opinion that we should stick with __numpy_implementation__ rather than __array_implementation__.

This attribute is current used for three things:

• providing the hook that allows np.ndarray.__array_function__ to call back out to an external implementation
• providing a hook that other __array_function__ methods can use to call the implemented intended for NumPy arrays.
• fast access to a non-dispatching version of NumPy functions, e.g., np.concatenate.__numpy_implementation__(list_of_all_numpy_arrays).

For all these use cases, the name "numpy" is more descriptive than "array." The situation reminds me of the mistake that is the name of the __array__ method. __array_function__ and __array_ufunc__ are also not great names, but least they could conceivably refer to functions/ufuncs defined by libraries that aren't NumPy. Also, unlike these __array* methods, __numpy_implementation__ will be on functions, not array types, so the consistently argument is weaker.

Yes, assuming the existence of a __numpy_implementation__ attribute means that dispatching with __array_function__ will bake in special treatment for NumPy arrays, but I think that is basically inevitable with our current design approach. __array_function__ is a dispatching protocol for the NumPy ecosystem.

The generic alternative would be to guard access to __numpy_implementation__ even inside ndarray.__array_function__, e.g.,

class ndarray:
    def __array_function__(self, func, types, args, kwargs):
        if not all(issubclass(t, ndarray) for t in types):
            # Defer to any non-subclasses that implement __array_function__
            return NotImplemented
        implementation = getattr(func, '__numpy_implementation__', None)
        if implementation is None:
            return NotImplemented
        return implementation(*args, **kwargs)

But I think this would be a little much. It is not too onerous to expect that libraries that reuse NumPy's dispatching protocol could also at least implement a dummy version of __numpy_implementation__.

Thinking a bit more, I agree with @shoyer. It does make sense to "separate the interface from implementation" in a sense.

My original suggestion was plain __implementation__ or __ndarray_implementation__, but I see the disadvantages of the former in not being clear what implementation it is, and of the latter of most people not thinking of ndarray as the class, but rather as array (since that's the constructor). But __array_implementation__ is worse for the reasons given, so let's stick with __numpy_implementation__.

I emailed the mailing list: https://mail.python.org/pipermail/numpy-discussion/2019-April/079317.html

I think it was agreed that this is ready after the name change. Comments?

LGTM

Are NEPs still being automatically updated on numpy.org?

The old version of NEP-18 does not seem to have been updated yet: http://www.numpy.org/neps/nep-0018-array-function-protocol.html

Very strange. The built documents are pushed to a repo, and the nep there is the updated version.