How did you pick between 'result_type' and not 'common_type'? It would be nice to justify the choice between them, even if the rationale is arbitrary

common_type will convert a dtype with all-integer fields to float, which doesn't seem right. result_type preserves the int type in that case.

I'd prefer an implementation that just does arr.astype(replace_all_fields(arr.dtype, dtype)).view(dtype), since then it could take copy and casting parameters to forward to astype

Good idea, done.

As a bonus, the function now returns a view instead of a copy if all the fields have the same dtype, which is the use-case people said they used on the mailing list.

I think that this needs a warning that fields are all cast to the same type

To work on 0d arrays, this needs to be ...

Good point.

This name strikes me as a little odd, but I also can't think of a better one.

It might be handy to use the word "require" in the description somewhere, to make the name easier to remember.

What happens if dtype is itself a structured array? Eg, consider:

point = np.dtype([('x'. int), ('y', int)])
triangle = np.dtype([('p_a', point), ('p_b', point), ('p_c', point)]

I'd expect to be able to do

arr = np.zeros(10, triangle)

structured_to_unstructured(arr, dtype=point)

This is accounted for, however your particular example shows there is a bug in this code because it can't account for repeated field names in the nested structures. Will fix.

On second examination, I also missed that your output was structured. structured_to_unstructured doesn't account for that the way you expected, and I'm not sure there is a good "rule" for how it should work. Your example makes sense because each field can be unambiguously cast to the new structured type, so you expect the output shape to be (10, 3) with point dtype. My implementation currently casts all nested fields to the new dtype, resulting in a (10, 6) array of points: It casts the x and y of each point to a point individually.

Generating "field_{}".format(i) as the name for each field is probably the safest bet - if you control all the names, you don't need to worry about escape sequences.

Yup, that's already fixed/implemented in #12446.

I'd rather not attempt to account for structured dtypes in the output though: That's not a pre-existing use-case we're trying to fix, and the best behavior to implement is unclear to me at the moment. Any users who previously did something like that can still do it using repack_fields instead of structured_to_unstructured, though without the added safety the latter has added.

That's not a pre-existing use-case we're trying to fix,

As I understand it, the purpose of structured_to_unstructured is to replace the arr[fields].view(dt) idiom. But it sounds like it doesn't work as a universal replacement:

• arr[['f_a', 'f_b']].view(float) → structured_to_unstructured(arr['f_a', 'f_c'], float)
• arr[['p_a', 'p_b']].view(point) → ???

Is there a way to spell the second case with these functions?

though without the added safety the latter has added.

Can you give an example of that safety, maybe even in the docs?

Here's my understanding:

All code of the form arr[['field1', 'field2, ...]].view(dt) in 1.15 can be replaced by repack_fields(arr[['field1', 'field2, ...]]).view(dt) in 1.16, without exceptions. It should be identical performance, since in both cases a copy is made.

Additionally, we have implemented a new function structured_to_unstructured. Although this can't be used as a replacement in all cases as you point out, in the many cases where it can be used it is better because it avoids a copy for multifield-indexes, it is safer since it is memory-layout-agnostic, and it better documents the user's intent. It is safer because it saves the user from bugs when doing the view: If the user tries to do the view themselves it is quite easy to forget to account for padding bytes, endianness, dtype, or misunderstand the memory layout, but structured_to_unstructured is written in a way to guarantee the view is "safe" for any memory layout and dtype (the fields are always viewed with the right offsets). structured_to_unstructured also casts the fields if needed, which the view above does not do.

I added a description based on my last comment in #12447

I think there are places we learnt that unsafe was a bad default, but ended up stuck with it, leaving users surprised by the conversion.

Should we apply that learning here, and pick a more conservative default?

I may have missed that. Maybe in #8733? But that was about assignment using unsafe casting, with no option to specify otherwise, unlike here where there is a keyword the user can specify.

Yeah, that issue was one of the ones I was thinking of - thanks for linking it, I was looking for that for unrelated reasons too!

Unsafe just feels like an... unsafe default to me. In my opinion, unsafe behavior should be something you ask for, not something you get by default. You're picking between:

• as is: f(...) vs f(..., casting='safe')
• proposed: f(..., casting='unsafe') vs f(...)

I'd much rather see the word 'unsafe' to tell me I need to think more carefully about that line of code, rather than having to look for the absence of it.

I don't have a good memory of how the other casting modes behave. I'd be inclined to pick same_kind to match the default value of the casting argument for for ufuncs

Here's an argument for unsafe:

First, it matches the default for the same keyword in astype, and so its easier for the user to remember if they are used to using astype.

Second, it seems like most of the time the user wants unsafe, because there are many common casts that are ruled out otherwise. For instance casts from f8 to i8 are disallowed with same_kind, but I expect this is a very common cast.

Actually, for reasons I don't understand, ufuncs seem to allow casts from f8 to i8 even though they supposedly use same_kind:

>>> np.arange(3, dtype='f8').astype('i8', casting='same_kind')                 
TypeError: Cannot cast array from dtype('float64') to dtype('int64') according to the rule 'same_kind'
>>> np.add(np.arange(3, dtype='f8'), np.arange(3, dtype='i8'))                 
array([0., 2., 4.])
>>> np.can_cast('f8', 'i8', casting='same_kind')
False

So ufuncs appear to use unsafe casting despite the keyword default??