Skip to content

Masked arrays with nested structured multidimensional dtypes inconsistent between data and mask #6760

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
gerritholl opened this issue Dec 3, 2015 · 2 comments
Closed

Masked arrays with nested structured multidimensional dtypes inconsistent between data and mask #6760

gerritholl opened this issue Dec 3, 2015 · 2 comments

Comments

@gerritholl
Copy link
Contributor

This may or may not be caused by the same problem as other numpy.ma-related issues. When a dtype contains a nested structured multidimensional sub-dtype, the dtype of the mask is inconsistent with the dtype of the data:

In [244]: Z = ma.zeros(2, dtype([("A", "(2,2)i1,(2,2)i1", (2,2))]))

In [245]: print(Z.data.dtype)
[('A', [('f0', 'i1', (2, 2)), ('f1', 'i1', (2, 2))], (2, 2))] 

In [246]: print(Z.mask.dtype)
[('A', '?', (2, 2))]

In [277]: Z.fill_value.dtype
[('A', [('f0', 'i1', (2, 2)), ('f1', 'i1', (2, 2))], (2, 2))]

For somewhat simpler dtypes, this problem does not occur:

In [260]: Z = ma.zeros(2, "(2,2)i2,(2,2)i2")

In [261]: print(Z.data.dtype)
[('f0', '<i2', (2, 2)), ('f1', '<i2', (2, 2))]

In [262]: print(Z.mask.dtype)
[('f0', '?', (2, 2)), ('f1', '?', (2, 2))]

In [281]: print(Z.fill_value.dtype)
[('f0', '<i2', (2, 2)), ('f1', '<i2', (2, 2))]

This is bound to lead to problems somewhere. It does appear to lead to a failure when calling Z.filled, for example (IndexError: only integers, slices (:), ellipsis (...), numpy.newaxis (None) and integer or boolean arrays are valid), although it is also possible that the dtype I constructed wreaks havoc elsewhere as well.
@gerritholl
Copy link
Contributor Author

This is related to: #3404 #3145 #2572 #5203 and others.

@gerritholl
Copy link
Contributor Author

This is probably due to a bug in numpy.ma.core.make_mask_descr and its helper numpy.ma.core._recursive_make_descr.

@gerritholl gerritholl mentioned this issue Dec 3, 2015
Merged
gerritholl added a commit to gerritholl/numpy that referenced this issue Dec 3, 2015
@gerritholl
BUG/TST: Fix numpy#6760 by correctly describing mask on nested subdtypes
c8a0982 
Fix numpy#6760.  In ma.core._recursive_make_descr, consider the case where a
subdtype does itself have named fields.  This ensures the correct mask for
an array like `ma.zeros(2, dtype([("A", "(2,2)i1,(2,2)i1", (2,2))]))`.
charris added a commit that referenced this issue Dec 3, 2015
@charris
Merge pull request #6761 from gerritholl/structured_nested_masked_arr…
d69d51c 
…ay_maskfill

BUG/TST: Fix #6760 by correctly describing mask on nested subdtypes
colbych added a commit to colbych/numpy that referenced this issue Dec 6, 2015
@colbych
Merge branch 'master' of git://github.com/numpy/numpy into first_patch
441f1a5 
* 'master' of git://github.com/numpy/numpy: (24 commits)
  BENCH: allow benchmark suite to run on Python 3
  TST: test f2py, fallback on f2py2.7 etc., fixes numpy#6718
  BUG: link cblas library if cblas is detected
  BUG/TST: Fix for numpy#6724, make numpy.ma.mvoid consistent with numpy.void
  BUG/TST: Fix numpy#6760 by correctly describing mask on nested subdtypes
  BUG: resizing empty array with complex dtype failed
  DOC: Add changelog for numpy#6734 and numpy#6748.
  Use integer division to avoid casting to int.
  Allow to change the maximum width with a class variable.
  Add some tests for mask creation with mask=True or False.
  Test that the mask dtype if MaskType before using np.zeros/ones
  BUG/TST: Fix for numpy#6729
  ENH: Avoid memory peak and useless computations when printing a MaskedArray.
  ENH: Avoid memory peak when creating a MaskedArray with mask=True/False (numpy#6732).
  BUG: Readd fallback CBLAS detection on linux.
  TST: Fix travis-ci test for numpy wheels.
  MAINT: Localize variables only used with relaxed stride checking.
  BUG: Fix for numpy#6719
  MAINT: enable Werror=vla in travis
  BUG: Include relevant files from numpy/linalg/lapack_lite in sdist.
  ...
jaimefrio pushed a commit to jaimefrio/numpy that referenced this issue Mar 22, 2016
@gerritholl @jaimefrio
BUG/TST: Fix numpy#6760 by correctly describing mask on nested subdtypes
23e4ce2 
Fix numpy#6760.  In ma.core._recursive_make_descr, consider the case where a
subdtype does itself have named fields.  This ensures the correct mask for
an array like `ma.zeros(2, dtype([("A", "(2,2)i1,(2,2)i1", (2,2))]))`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@gerritholl