When you pickle or np.save this view, does the dtype metadata get serialized as well?

pickle:

In [1]: import numpy as np

In [2]: a = np.array([1])

In [4]: m_dtype = np.dtype(a.dtype, metadata={"unique": np.unique(a)})

In [5]: b = a.view(dtype=m_dtype)

In [7]: from pickle import loads, dumps

In [8]: loads(dumps(b)).dtype.metadata
Out[8]: mappingproxy({'unique': array([1])})

However, np.save complains:

In [10]: from io import BytesIO

In [11]: f = BytesIO()

In [13]: np.save(f, b)
/home/adrin/micromamba/envs/sklearn/lib/python3.12/site-packages/numpy/lib/format.py:380: UserWarning: metadata on a dtype is not saved to an npy/npz. Use another format (such as pickle) to store it.
  d['descr'] = dtype_to_descr(array.dtype)

We're not gonna save these anyway, but interesting. cc @seberg

Yes, we had to put this in because of h5py, you can't store metadata in npy format.

We are not saving it, but a user may end up saving an ndarray with the metadata attached. In this case:

• If the unique values end up to be long, then it would increase their pickle size.
• For np.save, they will get a warning that they can ignore. (But it's hard to tell if it's safe to ignore)

We are not modifying user's arrays. We create a new view and these arrays are not returned.

Looking through the code now, I see that nothing is returned. Moving forward, we'll need to be careful that the output of _check_targets is not returned by a public function.

That's a very good point, and I think it's a good idea to not modify y in any function that returns them. So I've moved attach_unique to the callers of _check_targets instead, so that in the future we don't have to worry about this point.

Can we include a comment here that starts that the output of attach_unique should never be returned from a public function?

Yep, added a comment for this.