This is a try at improving structured array assignment, aiming for 1.11. The goal is to remove various inconsistencies in structure assignment, and to allow multi-field indexing to return a view.

The biggest change is that assignment from one structured array to another now works "by position" instead of "by field name", as in this example:

>>> arr1 = np.array([(1,2,3)], dtype=[('a', 'i4'), ('b', 'i4'), ('c', 'i4')])
>>> arr2 = np.array([(4,5,6)], dtype=[('b', 'i4'), ('a', 'i4'), ('c', 'i4')])
>>> arr1[:] = arr2

Previously, arr1 would be set to "(5,4,6)" but with this PR it is set to "(4,5,6)".

This fixes the issues in structured array assignment outlined below, and allows casting/assignment in new situations. Then I allow multi-field indexing (like arr1[['b', 'c']]) to return a view instead of a copy (issue #5994), since assignment to this view will now work more sensibly.

This PR also fixes #2353 #6085 #3351.

Problems in structure assignment

There are two code paths used to assign structured items, 1. using setitem or 2. using the dtype_transfer functions. The two paths behave differently from each other, and each sometimes have unexpected behaviors. Consider this setup:

def testassign(arr, v):
    c = arr.copy()
    c[0] = v  # assign using setitem
    c[1:] = v # assign using "dtype_transfer"
    print(c)

arr = ones(2, dtype=[('foo', 'i8'), ('bar', 'i8')])
v1 = array([(2,3)], dtype=[('foo', 'i8'), ('bar', 'i8')])
v2 = array([(2,3)], dtype=[('bar', 'i8'), ('foo', 'i8')])
v3 = array([(2,3)], dtype=[('bar', 'i8'), ('baz', 'i8')])
v4 = array([(2,)],  dtype=[('bar', 'i8')])
v5 = array([(2,3)], dtype=[('foo', 'f8'), ('bar', 'f8')])
w = arr.view({'names': ['bar'], 'formats': ['i8'], 'offsets': [8]})

testassign(arr, v1)
testassign(arr, v2)
testassign(arr, v3)
testassign(arr, v4)
testassign(arr, v5)
w[:] = 4
print arr

Output:

[(2, 3) (2, 3)]
[(2, 3) (3, 2)]
[(2, 3) (0, 2)]
[(2, 0) (0, 2)]
[(4607182418800017408, 4611686018427387904) (2, 3)]
[(140514438937232, 4) (140514438937232, 4)]

In summary, only the assignment using exactly the same dtype (v1) works, all other cases are broken in some way. For v2, setitem effectively assigns the fields "by position", but dtype_transfer code assigns "by field name". For v3, dtype_transfer converts "missing" fields to 0. In v4 dtype_transfer sets missing fields to 0, while setitem pads short buffers with 0. For v5, setitem copies the bytes directly (no casting). For the "partial view" w, we get uninitialized values at unviewed fields in both paths.

Proposed Solution

In this PR I made all field assignment occur "by position", and cast if needed. That is, if you order the fields from left to right in dst and src arrays, dst field 0 gets set to src field 0 and so on, regardless of field name. This makes sense since the "python representation" of a structured element (in VOID_getitem) is a tuple, and tuple assignment to a structure works "by position" too:

>>> a[0] = (1,2)
>>> a[0] = a[1].item() #works the same, essentially "a.setitem(0, a.getitem(1))"

Assignment to a different number of fields (v4 above) is no longer allowed. With this behavior, the output of the example will be:

[(2, 3) (2, 3)]
[(2, 3) (2, 3)]
[(2, 3) (2, 3)]
# v4 raises exception
[(2, 3) (2, 3)]
[(1, 4) (1, 4)]

This is a backward incompatible change, but I suspect it's not a big problem for most people given how inconsistently structure assignment worked. The most common case - assigning using an identical dtype (v1) - behaves the same as before. I ran unit tests for scipy, pandas and astropy with the PR on my system (python2, x64 linux). Scipy and pandas are unaffected, and astropy fails 1 new test (this one).

One alternative is to make all assignment work "by field name", but that also involves a compatibility break for all of the setitem cases. The fact that dtype_transfer code assigns by field name is currently undocumented (as far as I see), however there are a couple unit tests that test whether "reordering" of fields is allowed (like in v2), which depends on assignment by field name. I removed/modified these tests. I think assignment by field name is confusing and bug prone though, and without it I was able to get rid of a number of unit tests for edge cases we no longer have to worry about (eg, checking that "unviewed" fields of various kinds got set to 0 or None).

Historical note: Based on some comments in unit tests it looks like numpy 1.5 and before assigned fields "by position", but this was changed in 1.6 to assignment "by field name" in PR #36, though this was not noted anywhere in docs or release notes.

Implementation involved rewriting parts of VOID_setitem and get_fields_transfer_function.

Multi-Field Assignment

With these changes it is now reasonable to allow writeable multi-field views, which allows assignment like:

>>> a = np.ones(4, dtype=[('foo', 'i8'), ('bar', 'i8'), ('baz', 'f4')])
>>> a[['bar', 'baz']] = (2, 3.0)
>>> a[['baz', 'foo']][:2] = (4.0, 5)
>>> a
array([(5, 2, 4.0), (5, 2, 4.0), (1, 2, 3.0), (1, 2, 3.0)], 
      dtype=[('foo', '<i8'), ('bar', '<i8'), ('baz', '<f4')])

Note that with assignment "by position" reordering the fields (like in v2) would no longer work. However, multi-field assignment makes up for this, since fields will be assigned in the order they are subscripted, allowing reordering in a different way. For example,

>>> a[['foo', 'bar']] = a[['bar', 'foo']]

swaps the field data (while it was a no-op/broken in the old assignment "by field name").

Note that multi-field views (whether by this PR or another) introduce another backward incompatible change, since the byte-arrangement of a multi-field copy is different from the byte-arrangement of a multi-field view:

>>> a = ones(10, dtype='i8,i8,i8')
>>> a[['f0','f2']].dtype
# old ouput, for the multi-field copy:
dtype([('f0', '<i8'), ('f2', '<i8')])
# this PR's output, for multi-field view
dtype({'names':['f0','f2'], 'formats':['<i8','<i8'], 'offsets':[0,16], 'itemsize':24})

This is a problem for code which does things like a[['f0','f2']].view('i4') (like in unit test test_field_names), which would end up viewing different values in the old vs new code. In the new code this particular view raises an error as a result of my other PR #4488 to add safety checks to views. Similarly, np.recarray code uses such a view internally, and needed to be fixed. There might be a similar problem for the planned conversion to np.diagonal to a view.

I suggest we add a deprecation warning for code that attempts such a view in 1.10. Multi-field views have been planned for a long time and there is already a deprecation warning that copies will become views, but the warning only appears if you try to assign to the copy, and not if you try to view the copy. I've made another PR, #6054, which adds the appropriate deprecation warnings.

This change required a small edit to _internals.py, and removing warnings in mapping.c.

Other changes

1. One difference between setitem and dtype_transfer was that dtype_transfer allows assignment of a scalar to a structure (a[ind] = 1), which sets all fields to that value, while setitem did not. This is important for eg np.ones, which initializes the array with such a statement. I made it so both paths now allow assignment using a scalar.

2. Relatedly, got rid of undocumented special case for assignment from a structure to a scalar array of boolean type: If the scalar was of boolean type it would be set to "True" regardless of the values in the structured array, with the explanation that "the existence of fields implies True". This broke a MaskedArray test, which I fixed by updating make_mask to do the equivalent.

3. Previously, if you tried to assign a structured array to a non-structured array just the first field was copied over. Eg

 >>> a = np.array([1,2,3])
 >>> b = np.array([(10,20,3.0)], dtype='i4,i8,f4')
 >>> a[:] = b
 >>> a
 array([10, 10, 10])

Now, I made it raise an exception unless the structured type has only one field, since I found it confusing.

4. Slightly changed PyArray_EquivTypes for structured arrays. Now it checks that the fields are in the same order in both dtypes. (previously only cared that the dtype.fields dicts had the same values). In other words, field order is now important!

5. I rewrote most of the docs on structured arrays. A possibly iffy part of the new docs is the section "obsolete features", in which I discourage use of field titles and use the "second" dictionary based form of dtype specification.