I would suggest we discuss this here (or in a new topic there):

• https://discuss.scientific-python.org/t/a-proposed-design-for-supporting-multiple-array-types-across-scipy-scikit-learn-scikit-image-and-beyond/131/31
• https://discuss.scientific-python.org/t/default-dispatching-behavior-for-supporting-multiple-array-types-across-scipy-scikit-learn-scikit-image/135/7
• https://discuss.scientific-python.org/t/requirements-and-discussion-of-a-type-dispatcher-for-the-ecosystem/157/37

I think the transition problem should be clear enough. __array_function__ should return whatever makes most sense for the inputs (which usually means dask input, dask output, etc.), I simply don't see a point in returning numpy arrays.

In my personal opinion, the question is not about what we want, but about how to transition users. There are three ways to do that, which I noted in the second point in the above discussion.

For this issue: I doubt we should bother here for documentation, this needs to be solved ecosystem wide in those discussions first. Note that IMO, NEP 37 "moves" the responsibility a bit, but doesn't remove the problem itself entirely.

Thanks a lot @seberg. I went over those posts during the weekend - it's quite a lot to digest and I don't think I'll easily wrap my head around all of it, but in any case, I'll try to see what things we can distill from those discussions now (even by adding notes or warnings about possible caveats), and what is still in flux and needs wider ecosystem consensus first.

To give one particular example, it seems that the idea of subclassing ndarrays is frowned upon, but I couldn't find detailed explanations, only breadcrumbs. For example, @rgommers in one of the threads you linked:

The only reason we’re even thinking about this is because of NumPy being overly flexible and accepting too much, and as a result the proliferation of asarray throughout the ecosystem, which then resulted in us recommending against using ndarray subclasses etc.

Subclasses are tangentially mentioned in NEP 47 when discussing the asarray / asanyarray (anti)pattern:

Many existing libraries use the same asarray (or asanyarray) pattern as NumPy itself does; accepting any object that can be coerced into a np.ndarray. We consider this design pattern problematic - keeping in mind the Zen of Python, “explicit is better than implicit”, as well as the pattern being historically problematic in the SciPy ecosystem for ndarray subclasses and with over-eager object creation. All other array/tensor libraries are more strict, and that works out fine in practice. We would advise authors of new libraries to avoid the asarray pattern. Instead they should either accept just NumPy arrays or, if they want to support multiple kinds of arrays, check if the incoming array object supports the array API standard by checking for __array_namespace__ as shown in the example above.

And the readme of your precodita:

The second restriction is implemented because in the world of arrays, subclasses usually add behaviour in a way that means implementations cannot be expected to return useful results anymore. This makes subclassing less convenient (because sometimes things work out) but protects from those surprises common to NumPy's masked arrays and matrices (why did I get a normal array back and the mask got ignored?).

In summary, there seems to be some "tribal knowledge" that I would like to help properly capture in written form. It doesn't have to be "everybody agrees subclassing is bad", but at least we could describe the kind of surprises that might appear, how to avoid them, etc.

I'll be honest - I'd be out of my depth here. But as long as someone can point me in the right direction as far as what should actually be documented, or even pair up on this doc, I'd be happy to work on it.

there seems to be some "tribal knowledge"

I prefer "culture" ;). More serious, subclasses are problematic for exactly the same reason that subclasses are always problematic: If you break Liskov's substitution principle, you have to live with the consequences.

The problem is that:

• Almost all interesting subclasses break Liskov's substitution principle
• Arrays have a large API surface and most of that is functions and not methods. Just like math.<function> won't work for a "masked scalar" a lot of NumPy doesn't actually work right for "masked arrays". These functions will often return base-class arrays and may convert the input silently to a base-class array (which again should be OK for a "proper" subclass!).
• And finally, there are some bad subclasses that look like they should be fine (masked arrays, matrix).

but while such notes may make sense somewhere, I think they would belong on the doc page for "subclassing" which should probably point to the interop stuff.

There is nothing wrong with subclassing (memmaps are fine, except that they should demote more aggressively IMO), but almost everyone breaks Liskov's. That is OK, but there are clear limits and most of the time the gains (a few things "just work") are not worth the downsides (the stuff that doesn't work may surprise you).

EDIT: OK, to be clear, I am not sure masked arrays break "Liskov" explicitly – matrix does. But they "enrich" the base class in ways that mean you lose vital information if you go back to the base-class.