Narrow type variable bounds in binder #19183

ilevkivskyi · 2025-05-31T14:02:45Z

Fixes #5720
Fixes #8556
Fixes #9778
Fixes #10003
Fixes #10817
Fixes #11163
Fixes #11664
Fixes #12882
Fixes #13426
Fixes #13462
Fixes #14941
Fixes #15151
Fixes #19166

This handles a (surprisingly) common edge case. The charges in bind_self() and bind_self_fast() are tricky. I got a few "Redundant cast" errors there, which seemed good, but then I realized that attribute access etc. on a type variable go through slow PyObject paths, so I am actually forcing CallableType instead of F(bound=CallableType) there, since these are performance-critical functions.

ilevkivskyi · 2025-05-31T21:25:35Z

OK, the mypy_primer results are good, and although it looks like a niche use case, this PR fixes more than dozen issues (some with many upvotes).

hauntsaninja · 2025-06-01T00:27:57Z

Hooray!

…unds

github-actions · 2025-06-03T18:36:39Z

Diff from mypy_primer, showing the effect of this PR on open source code:

PyGithub (https://github.com/PyGithub/PyGithub)
+ github/MainClass.py:988: error: Unused "type: ignore" comment  [unused-ignore]

antidote (https://github.com/Finistere/antidote): 1.47x slower (65.5s -> 96.3s in single noisy sample)

prefect (https://github.com/PrefectHQ/prefect)
- src/prefect/states.py:369: error: Argument 1 to "StateGroup" has incompatible type "Collection[Union[State[Any], Any]]"; expected "list[State[Any]]"  [arg-type]
+ src/prefect/states.py:369: error: Argument 1 to "StateGroup" has incompatible type "Collection[Never]"; expected "list[State[Any]]"  [arg-type]
+ src/prefect/futures.py:222: note:     def result(self, raise_on_failure: Literal[True] = ..., retry_result_failure: bool = ...) -> Any
+ src/prefect/futures.py:222: note:     def result(self, raise_on_failure: Literal[False] = ..., retry_result_failure: bool = ...) -> Union[Any, Exception]
+ src/prefect/futures.py:222: note:     def result(self, raise_on_failure: bool = ..., retry_result_failure: bool = ...) -> Union[Any, Exception]
- src/prefect/task_engine.py:467: error: Item "type" of "Union[ResultRecord[Any], type[NotSet]]" has no attribute "result"  [union-attr]
+ src/prefect/task_engine.py:467: error: Item "type" of "Union[R, type[NotSet]]" has no attribute "result"  [union-attr]
- src/prefect/task_engine.py:1022: error: Item "type" of "Union[ResultRecord[Any], type[NotSet]]" has no attribute "result"  [union-attr]
+ src/prefect/task_engine.py:1022: error: Item "type" of "Union[R, type[NotSet]]" has no attribute "result"  [union-attr]
- src/prefect/flow_runs.py:292: error: Item "type" of "Union[type[AutomaticRunInput[Never]], type[T]]" has no attribute "load"  [union-attr]
+ src/prefect/flow_runs.py:292: error: "type[T]" has no attribute "load"  [attr-defined]
- src/prefect/flow_runs.py:307: error: Item "type" of "Union[type[AutomaticRunInput[Never]], type[T]]" has no attribute "save"  [union-attr]
+ src/prefect/flow_runs.py:307: error: "type[T]" has no attribute "save"  [attr-defined]
- src/prefect/flow_runs.py:319: error: Item "type" of "Union[type[AutomaticRunInput[Never]], type[T]]" has no attribute "load"  [union-attr]
+ src/prefect/flow_runs.py:319: error: "type[T]" has no attribute "load"  [attr-defined]
- src/prefect/flow_runs.py:446: error: Item "type" of "Union[type[AutomaticRunInput[Never]], type[T]]" has no attribute "load"  [union-attr]
+ src/prefect/flow_runs.py:446: error: "type[T]" has no attribute "load"  [attr-defined]
- src/prefect/flow_runs.py:456: error: Item "type" of "Union[type[AutomaticRunInput[Never]], type[T]]" has no attribute "save"  [union-attr]
+ src/prefect/flow_runs.py:456: error: "type[T]" has no attribute "save"  [attr-defined]

kopf (https://github.com/nolar/kopf)
+ kopf/_cogs/structs/dicts.py:243: error: Unused "type: ignore" comment  [unused-ignore]
+ kopf/_core/intents/handlers.py:36: error: Redundant cast to "execution.CauseT"  [redundant-cast]

python-chess (https://github.com/niklasf/python-chess)
- chess/pgn.py:1315: error: Returning Any from function declared to return "HeadersT"  [no-any-return]

kornia (https://github.com/kornia/kornia)
+ kornia/utils/helpers.py:387: error: Unused "type: ignore" comment  [unused-ignore]

mypy (https://github.com/python/mypy)
+ mypy/expandtype.py:125: error: Redundant cast to "F"  [redundant-cast]
+ mypy/expandtype.py:125: note: See https://mypy.rtfd.io/en/stable/_refs.html#code-redundant-cast for more info
+ mypy/typeops.py:421: error: Redundant cast to "F"  [redundant-cast]
+ mypy/typeops.py:430: error: Redundant cast to "F"  [redundant-cast]
+ mypy/typeops.py:469: error: Incompatible types in assignment (expression has type "CallableType", variable has type "F")  [assignment]
+ mypy/typeops.py:484: error: Redundant cast to "F"  [redundant-cast]
+ mypy/checkmember.py:1489: error: Redundant cast to "F"  [redundant-cast]
+ mypy/checkmember.py:1492: error: Redundant cast to "F"  [redundant-cast]
+ mypy/checkmember.py:1502: error: Redundant cast to "F"  [redundant-cast]

hydra-zen (https://github.com/mit-ll-responsible-ai/hydra-zen)
- src/hydra_zen/third_party/beartype.py:128: error: Incompatible types in assignment (expression has type "_T", variable has type "type[Any]")  [assignment]
+ src/hydra_zen/third_party/beartype.py:128: error: Incompatible types in assignment (expression has type "_T", variable has type "_T")  [assignment]
+ src/hydra_zen/third_party/beartype.py:130: error: Redundant cast to "_T"  [redundant-cast]
+ src/hydra_zen/third_party/pydantic.py:111: error: Redundant cast to "_T"  [redundant-cast]
+ src/hydra_zen/third_party/pydantic.py:115: error: Redundant cast to "_T"  [redundant-cast]
- src/hydra_zen/third_party/pydantic.py:123: error: Argument 1 has incompatible type "_T | type[object]"; expected "_T"  [arg-type]

pytest (https://github.com/pytest-dev/pytest)
- src/_pytest/fixtures.py:1132: error: Returning Any from function declared to return "FixtureValue"  [no-any-return]

pandas-stubs (https://github.com/pandas-dev/pandas-stubs)
+ tests/__init__.py:67: error: Unused "type: ignore" comment  [unused-ignore]
+ tests/__init__.py:85: error: Unused "type: ignore" comment  [unused-ignore]

beartype (https://github.com/beartype/beartype)
+ beartype/_decor/_type/decortype.py:133: error: Unused "type: ignore" comment  [unused-ignore]
+ beartype/_decor/_type/decortype.py:295: error: Unused "type: ignore" comment  [unused-ignore]
+ beartype/_decor/_nontype/decornontype.py:294: error: Unused "type: ignore" comment  [unused-ignore]
+ beartype/_decor/_nontype/decornontype.py:301: error: Unused "type: ignore" comment  [unused-ignore]
+ beartype/_decor/_nontype/decornontype.py:380: error: Unused "type: ignore" comment  [unused-ignore]
+ beartype/_decor/_nontype/decornontype.py:395: error: Unused "type: ignore" comment  [unused-ignore]
+ beartype/_decor/_nontype/decornontype.py:567: error: Unused "type: ignore" comment  [unused-ignore]
+ beartype/_decor/decorcore.py:133: error: Unused "type: ignore" comment  [unused-ignore]

xarray (https://github.com/pydata/xarray)
+ xarray/structure/alignment.py: note: In function "_broadcast_helper":
+ xarray/structure/alignment.py:1140: error: Redundant cast to "T_Alignable"  [redundant-cast]
+ xarray/structure/alignment.py:1142: error: Redundant cast to "T_Alignable"  [redundant-cast]

steam.py (https://github.com/Gobot1234/steam.py)
- steam/ext/commands/commands.py:284: error: Argument 1 to "append" of "list" has incompatible type "Command[Any, Any, Any]"; expected "Self"  [arg-type]
- steam/ext/commands/cog.py:182: error: Argument 1 to "decorator" has incompatible type "Callable[[Any | None, VarArg(Any), KwArg(Any)], Coroutine[Any, Any, None]]"; expected "F"  [arg-type]
- steam/ext/commands/bot.py:415: error: Argument 1 to "decorator" has incompatible type "Callable[..., Coroutine[Any, Any, Any]]"; expected "F"  [arg-type]

netomi · 2025-06-11T19:16:11Z

ftr: this commit fixes a bug we encountered with mypy 1.16.0 (and not in 1.15.0) in different places with a stacktrace like:

asfyaml/feature/github/pull_requests.py: error: INTERNAL ERROR -- Please try using mypy master on GitHub:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 1.16.0
Traceback (most recent call last):
  File "/home/tn/workspace/apache/infrastructure-asfyaml/.venv/bin/mypy", line 8, in <module>
    sys.exit(console_entry())
  File "mypy/build.py", line 2062, in wrap_context
RuntimeError: ('Not implemented', <class 'mypy.types.TypeGuardedType'>)
asfyaml/feature/github/pull_requests.py: : note: use --pdb to drop into pdb

ilevkivskyi · 2025-06-11T19:57:13Z

@netomi Hm, this is interesting, could you please post a full traceback? (use --show-traceback)

netomi · 2025-06-11T20:14:27Z

that is already with this option:

tn@proteus:~/workspace/apache/infrastructure-asfyaml$ poetry run mypy --show-traceback asfyaml/feature/github/pull_requests.py 
asfyaml/feature/github/pull_requests.py: error: INTERNAL ERROR -- Please try using mypy master on GitHub:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 1.16.0
Traceback (most recent call last):
  File "/home/tn/workspace/apache/infrastructure-asfyaml/.venv/bin/mypy", line 8, in <module>
    sys.exit(console_entry())
  File "mypy/build.py", line 2062, in wrap_context
RuntimeError: ('Not implemented', <class 'mypy.types.TypeGuardedType'>)
asfyaml/feature/github/pull_requests.py: : note: use --pdb to drop into pdb

the project can be found here: https://github.com/apache/infrastructure-asfyaml

it is now using 1.15.0, but updating to 1.16.0 shows the bug.

Narrow type variable bounds in binder

fd8d1cd

This comment has been minimized.

Sign in to view

Reorder branches in narrowing logic

e2fadec

This comment has been minimized.

Sign in to view

Use a bit more principled handling of unions

fcc9af6

This comment has been minimized.

Sign in to view

This was referenced May 31, 2025

Checking Generic Type at Runtime #13053

Closed

Narrowing doesn't work on a typevar with a union bound #15631

Closed

Narrow TypeVars in the negative case + narrow them through upper bounds #15711

Closed

Narrow TypeVar to bounded TypeVar #15377

Closed

ilevkivskyi requested review from JukkaL and hauntsaninja May 31, 2025 21:25

hauntsaninja approved these changes Jun 1, 2025

View reviewed changes

sterliakov approved these changes Jun 1, 2025

View reviewed changes

sterliakov mentioned this pull request Jun 3, 2025

Problems with narrowing generic T | Sequence[T] #19202

Closed

ilevkivskyi mentioned this pull request Jun 3, 2025

1.16 Release Planning #18739

Open

ilevkivskyi added 2 commits June 3, 2025 18:29

Merge remote-tracking branch 'upstream/master' into narrow-typevar-bo…

21a2153

…unds

Address offline feedback

54a80bb

ilevkivskyi merged commit 5a0fa55 into python:master Jun 3, 2025
19 checks passed

ilevkivskyi deleted the narrow-typevar-bounds branch June 3, 2025 18:37

This was referenced Jun 5, 2025

mypy too strict in enforcing generic return types when they are unions #19239

Closed

An instance of a final class should be treated as satisfying a TypeVar bound to it, after being narrowed #19255

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Narrow type variable bounds in binder #19183

Narrow type variable bounds in binder #19183

Uh oh!

ilevkivskyi commented May 31, 2025 •

edited

Loading

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

ilevkivskyi commented May 31, 2025

Uh oh!

hauntsaninja commented Jun 1, 2025

Uh oh!

github-actions bot commented Jun 3, 2025

Uh oh!

Uh oh!

netomi commented Jun 11, 2025 •

edited

Loading

Uh oh!

ilevkivskyi commented Jun 11, 2025

Uh oh!

netomi commented Jun 11, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Narrow type variable bounds in binder #19183

Narrow type variable bounds in binder #19183

Uh oh!

Conversation

ilevkivskyi commented May 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

ilevkivskyi commented May 31, 2025

Uh oh!

hauntsaninja commented Jun 1, 2025

Uh oh!

github-actions bot commented Jun 3, 2025

Uh oh!

Uh oh!

netomi commented Jun 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ilevkivskyi commented Jun 11, 2025

Uh oh!

netomi commented Jun 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

ilevkivskyi commented May 31, 2025 •

edited

Loading

netomi commented Jun 11, 2025 •

edited

Loading

netomi commented Jun 11, 2025 •

edited

Loading