Skip to content

BUG: Sum of ndarray subclassess incorrectly typed as np.ndarray #20072

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
rdbisme opened this issue Oct 8, 2021 · 7 comments
Closed

BUG: Sum of ndarray subclassess incorrectly typed as np.ndarray #20072

rdbisme opened this issue Oct 8, 2021 · 7 comments

Comments

@rdbisme
Copy link

rdbisme commented Oct 8, 2021

Describe the issue:

In my code I'm using a slightly customized numpy subclass. When I sum two numpy arrays viewed as this class, the result has a different type. I'd expect the sum to be of the same type as the two addends.

Reproduce the code example:

import numpy as np


class State(np.ndarray):
    pass


a: State = np.random.rand(10, 5).view(State)
b: State = np.random.rand(10, 5).view(State)

reveal_type(a)
reveal_type(b)
reveal_type(a + b)

Error message:

test-mypy-numpy.py:12: note: Revealed type is "test-mypy-numpy.State"
test-mypy-numpy.py:13: note: Revealed type is "test-mypy-numpy.State"
test-mypy-numpy.py:14: note: Revealed type is "numpy.ndarray[Any, numpy.dtype[numpy.bool_]]"

NumPy/Python version information:

Python 3.9.5
Numpy: 1.21.2
Mypy: 0.910

@BvB93
Copy link
Member

BvB93 commented Oct 9, 2021

It's unfortunate, but without higher-kinded typevars (python/typing#548) I see no way of fixing this issue.

The problem is that, when working with typevars, we've no way of decoupling the type from its parameters
(i.e. shape and dtype). So we can currently type that the addition of a float and int array returns a float array,
but we cannot do the same for ndarray subclasses (unless you explicitly override the subclasses' methods,
that is).

test-mypy-numpy.py:14: note: Revealed type is "numpy.ndarray[Any, numpy.dtype[numpy.bool_]]"

Interesting, the appearance of the boolean dtype most definitely looks like a mypy bug.
I would expect the type here to be ndarray[Any, Any] due to overload ambiguity, which is exactly what
happens with a: ndarray[Any, Any]; reveal_type(a + a).

@rdbisme
Copy link
Author

rdbisme commented Oct 9, 2021

Thanks @BvB936! So should I open an issue on the mypy tracker for the bool problem?

@BvB93
Copy link
Member

BvB93 commented Oct 11, 2021

Thanks @BvB936! So should I open an issue on the mypy tracker for the bool problem?

Yes, though I'd recommend narrowing it down to a minimal reproducible example first.

@vnmabus
Copy link

vnmabus commented Oct 15, 2021

Thanks @BvB936! So should I open an issue on the mypy tracker for the bool problem?

I think it is the same as #20099.

@BvB93
Copy link
Member

BvB93 commented Oct 16, 2021

I think it is the same as #20099.

@vnmabus it seems to be the case, yes.

After some trial and error I've managed to narrow the issue down (see below for a minimal example).
In short, there seem to be three requirements for triggering this mypy bug:

  • The function must be a method.
  • The method must be overloaded.
  • self must be annotated with a parametrized generic.
from typing import Any, Generic, overload, TypeVar, TYPE_CHECKING
import numpy as np

_CharType = TypeVar("_CharType", np.str_, np.bytes_)

class CharArray(Generic[_CharType]):
    @overload
    def strip(self: CharArray[np.str_], chars: str = ...) -> CharArray[np.str_]: ...
    @overload
    def strip(self: CharArray[np.bytes_], chars: bytes = ...) -> CharArray[np.bytes_]: ...

ar: CharArray[Any]
ar_U: CharArray[np.str_]
ar_S: CharArray[np.bytes_]

if TYPE_CHECKING:
    reveal_type(ar.strip())  # E: Revealed type is "__main__.CharArray[numpy.str_]"
    reveal_type(ar_U.strip())  # E: Revealed type is "__main__.CharArray[numpy.str_]"
    reveal_type(ar_S.strip())  # E: Revealed type is "__main__.CharArray[numpy.bytes_]"

@BvB93
Copy link
Member

BvB93 commented Oct 17, 2021

FYI, I just opened an issue on the mypy tracker (python/mypy#11347).

@jorenham
Copy link
Member

jorenham commented Jan 6, 2025

This is impossible without higher kinded typing, which isn't supported in Python:
python/typing#548

@jorenham jorenham closed this as completed Jan 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants