Skip to content

numpy.ma.tests.test_regression.TestRegression.test_var_sets_maskedarray_scalar is racy #10270

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ghost opened this issue Dec 24, 2017 · 12 comments
Labels
component: numpy.ma masked arrays

Comments

@ghost
Copy link

ghost commented Dec 24, 2017

This test has failed a few times with the following message:

AttributeError: 'numpy.float64' object has no attribute '_mask'

And stack trace

self = <numpy.ma.tests.test_regression.TestRegression object at 0x0000000012B264E0>

    def test_var_sets_maskedarray_scalar(self):
        # Issue gh-2757
        a = np.ma.array(np.arange(5), mask=True)
        mout = np.ma.array(-1, dtype=float)
>       a.var(out=mout)

numpy\ma\tests\test_regression.py:64: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
numpy\ma\core.py:5219: in var
    dvar = divide(danom.sum(axis, **kwargs), cnt).view(type(self))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <numpy.ma.core._DomainedBinaryOperation object at 0x0000000006CA1048>
a = masked, b = 0, args = (), kwargs = {}, da = array(0.)
db = array(0, dtype=int64), result = nan, m = array([ True])
domain = <numpy.ma.core._DomainSafeDivide object at 0x0000000006CA1240>
masked_result = nan

    def __call__(self, a, b, *args, **kwargs):
        "Execute the call behavior."
        # Get the data
        (da, db) = (getdata(a), getdata(b))
        # Get the result
        with np.errstate(divide='ignore', invalid='ignore'):
            result = self.f(da, db, *args, **kwargs)
        # Get the mask as a combination of the source masks and invalid
        m = ~umath.isfinite(result)
        m |= getmask(a)
        m |= getmask(b)
        # Apply the domain
        domain = ufunc_domain.get(self.f, None)
        if domain is not None:
            m |= domain(da, db)
        # Take care of the scalar case first
        if (not m.ndim):
            if m:
                return masked
            else:
                return result
        # When the mask is True, put back da if possible
        # any errors, just abort; impossible to guarantee masked values
        try:
            np.copyto(result, 0, casting='unsafe', where=m)
            # avoid using "*" since this may be overlaid
            masked_da = umath.multiply(m, da)
            # only add back if it can be cast safely
            if np.can_cast(masked_da.dtype, result.dtype, casting='safe'):
                result += masked_da
        except Exception:
            pass
    
        # Transforms to a (subclass of) MaskedArray
        masked_result = result.view(get_masked_subclass(a, b))
>       masked_result._mask = m
E       AttributeError: 'numpy.float64' object has no attribute '_mask'

Link to failures: https://ci.appveyor.com/project/charris/numpy/build/1.0.8371/job/ewtkpj7ss3beoky3/tests

@eric-wieser
Copy link
Member

Since when did appveyor start listing the tests?

@eric-wieser
Copy link
Member

It alarms me that m = array([ True]), given it is computed as

        m = ~umath.isfinite(result)
        m |= getmask(a)
        m |= getmask(b)

And a, b, and result all appear to be scalars.

@ghost
Copy link
Author

ghost commented Dec 28, 2017

Since when did appveyor start listing the tests?

I enabled that functionality. I hope you don't mind?

@eric-wieser
Copy link
Member

I'm just curious how you did it

eric-wieser added a commit to eric-wieser/numpy that referenced this issue Jan 1, 2018
It's possible this is the cause of numpygh-10270, where it seems something is wrong with the shape
@mhvk
Copy link
Contributor

mhvk commented Jan 2, 2018

@eric-wieser - I see why in #10308 you thought maybe to avoid this type of stuff with making the mask shape effectively unsettable. But looking at the code, the only shape that could possibly be relevant would seem to be the one on masked - in which case #10292 should be the more relevant one. Though you wrote there that it does not stop assigning the mask of masked - why is that?

@mhvk
Copy link
Contributor

mhvk commented Jan 2, 2018

Ah, sorry, I see now: #10292 stopped from assigning attributes of masked, but not of masked.mask, i.e., masked.mask.shape is settable. Ideally we find out where one is trying to assign its shape in numpy -- I suspect one of the matrix work-arounds (grrrrr) - and also make that impossible. Might it be a solution to just define a MaskedConstant.mask property that always returns np.array(True)?

@eric-wieser
Copy link
Member

From your comment there, I think you hit the nail on the head with

And I see that __array_finalize__ happily changes the shape of the mask it got from obj

This looks very likely to be the cause, which of course #10308 will do nothing to prevent

@eric-wieser
Copy link
Member

Misclick, sorry

@eric-wieser
Copy link
Member

hanjohn pushed a commit to hanjohn/numpy that referenced this issue Feb 15, 2018
It's possible this is the cause of numpygh-10270, where it seems something is wrong with the shape
@eric-wieser
Copy link
Member

Turns out that np.tile(np.ma.masked, (2, 1)) ends up reshaping the masked constant, which is likely the cause of this bug.

@eric-wieser
Copy link
Member

And a simpler test-case:

>>> np.array(np.ma.masked, subok=True, ndmin=2, copy=False)
>>> np.ma.masked.mask.ndim
2

@mhvk
Copy link
Contributor

mhvk commented May 16, 2018

I reproduce the bug with np.tile (and it is solved by #10314), but not your simpler test case (on numpy 1.13.3).

eric-wieser added a commit to eric-wieser/numpy that referenced this issue May 11, 2019
…mask

This prevents consumers from reshaping the mask in place, which breaks things

As a result, `x.mask is x.mask` returns `False`, but this was already true of `x.data is x.data`.

May also be related to numpygh-10270
@WarrenWeckesser WarrenWeckesser added the component: numpy.ma masked arrays label Nov 18, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component: numpy.ma masked arrays
Projects
None yet
Development

No branches or pull requests

3 participants