Rework starargs with union argument #19651

randolf-scholz · 2025-08-13T12:32:35Z

Makes *args inference smarter by special casing UnionType

*(tuple[A, B, C] | tuple[None|None|None]) gets treated like tuple[A | None, B | None, C | None] inside ArgTypeExpander (applies if all union member are fixed size tuples of equal length
*(Iterable[X] | Iterable[Y]) gets treated like *Iterable[X | Y] inside ArgTypeExpander (applies if ① is not triggered)

This gives some better inference in some cases:

x: list[str | None] | list[str]
reveal_type([*x])      # master: list[Any]        PR: list[str | None]
x2: tuple[int, int] | tuple[None, None]
reveal_type( (*x2,) )  # master: tuple[Any, ...]  PR: tuple[int | None, int | None]
x3: tuple[int, int] | tuple[None, None, None]
reveal_type( (*x3,) )  # master: tuple[Any, ...]  PR: tuple[int | None, ...]

See added unit tests for more examples.

Dev notes:

If we detect that the union has non-tuple elements or tuples of different sizes, we upcast-reinterpret every union member as some Iterable[T], then apply each union member, and then return the union of the results.
Handling unions of finite size tuples with different sizes in infeasible, due to combinatoric explosion¹
map_actuals_to_formals added special case for union of same sized tuples.
ExpressionChecker.check_arg: added some extra logic that applies if the callee_type is UnpackType, but the actual_type got expanded from Unpack_type to something else. Previously, this only worked "by accident" because the expander was guaranteed to return AnyType(TypeOfAny.from_errors) in this case.
ArgTypeExpander:
- Create a NewType alias IterableType for Iteralble[T], and helper functions is_iterable, is_iterable_instance_type, _make_iterable_instance_type.
- added _solve_as_iterable function that uses the solver to interpret a type as Iterable[T]
- added as_iterable_type that applies a few special cases (UnionType, etc. before calling the solver)
- added parse_star_args_type that converts the argument of *args to one of TupleType | IterableType | ParamSpecType | AnyType
visit_tuple_expr: Applies ArgTypeExpander(*parse_star_args_type) to ensure consistent behavior across [*args], {*args} and (*args).

consider f(*x1, *x2, ..., *xn). If each $x_k$ is comprised of a union of $m_k$ differently sized tuples, then there are $m_1⋅m_2⋅…⋅m_k$ possible paths. ↩

randolf-scholz · 2025-08-13T14:19:33Z

Hm there is still some issue to be solved with Unpack. The flatten_nested_tuples method helps./

I think one could determine the correct upcast by using the solver, but I discovered some inconsistencies: #19652

randolf-scholz · 2025-08-13T14:29:28Z

This really does catch quite a few false negatives, for example:

Repro of homeassistant/components/govee_light_local https://mypy-play.net/?mypy=latest&python=3.12&gist=3bf7d56a2bfbc1fc23be62ef521dc613

def set_rgb_color(red: int, green: int, blue: int) -> None: ...

_last_color_state: tuple[
    str | None,
    int | None,
    tuple[int, int, int] | tuple[int | None] | None,
]

color_mode, brightness, color = _last_color_state
if color:
    set_rgb_color(*color)  # MASTER: no error ❌, PR: raises [arg-type] ✅

Repro of xarray/tests/test_namedarray: https://mypy-play.net/?mypy=latest&python=3.12&gist=9999ca2cb89e849978c979c1e4cc9bca

from typing import Iterable, Hashable

type _Dim = Hashable
type _Dims = tuple[_Dim, ...]
type _DimsLike = str | Iterable[_Dim]
def permute_dims(*dim: Iterable[_Dim]) -> None: ...

def test(dims: _DimsLike) -> None:
    permute_dims(*dims)  # master: no error ❌, PR: raises [arg-type] ✅

Repro of src/bokeh/server/tornado.py:450: https://mypy-play.net/?mypy=latest&python=3.12&gist=0df919e6986b3fcbdb9d7c8727fb8d48

from typing import Any

class RequestHandler: ...
type RouteContext = dict[str, Any]

type URLRoutes = list[
    tuple[str, type[RequestHandler]] |
    tuple[str, type[RequestHandler], RouteContext],
]

def demo(
    all_patterns: URLRoutes,
    extra_patterns: URLRoutes,
    toplevel_patterns: URLRoutes,
    prefix: str,
    data: dict[str, Any],
) -> None:
    for p in extra_patterns + toplevel_patterns:
        prefixed_pat = (prefix + p[0], *p[1:], data)
        all_patterns.append(prefixed_pat)  # MASTER: false negative, PR: [arg-type]

randolf-scholz · 2025-08-15T14:01:59Z

I think this is ready for initial review.

mypy/argmap.py

test-data/unit/lib-stub/builtins.pyi

mypy/argmap.py

This reverts commit e8dcf88.

github-actions · 2025-08-15T17:41:57Z

Diff from mypy_primer, showing the effect of this PR on open source code:

tornado (https://github.com/tornadoweb/tornado)
+ tornado/routing.py:355: error: Argument 2 to "Rule" has incompatible type "*Union[list[Any], tuple[Any], tuple[Any, dict[str, Any]], tuple[Any, dict[str, Any], str]]"; expected "Optional[dict[str, Any]]"  [arg-type]
+ tornado/routing.py:355: error: Argument 2 to "Rule" has incompatible type "*Union[list[Any], tuple[Any], tuple[Any, dict[str, Any]], tuple[Any, dict[str, Any], str]]"; expected "Optional[str]"  [arg-type]
+ tornado/routing.py:357: error: Argument 1 to "Rule" has incompatible type "*Union[list[Any], tuple[Union[str, Matcher], Any], tuple[Union[str, Matcher], Any, dict[str, Any]], tuple[Union[str, Matcher], Any, dict[str, Any], str]]"; expected "Matcher"  [arg-type]
+ tornado/routing.py:357: error: Argument 1 to "Rule" has incompatible type "*Union[list[Any], tuple[Union[str, Matcher], Any], tuple[Union[str, Matcher], Any, dict[str, Any]], tuple[Union[str, Matcher], Any, dict[str, Any], str]]"; expected "Optional[dict[str, Any]]"  [arg-type]
+ tornado/routing.py:357: error: Argument 1 to "Rule" has incompatible type "*Union[list[Any], tuple[Union[str, Matcher], Any], tuple[Union[str, Matcher], Any, dict[str, Any]], tuple[Union[str, Matcher], Any, dict[str, Any], str]]"; expected "Optional[str]"  [arg-type]

hydpy (https://github.com/hydpy-dev/hydpy)
+ hydpy/core/selectiontools.py:370: error: Argument 1 to "intersection" of "Devices" has incompatible type "*Elements"; expected "Elements"  [arg-type]
+ hydpy/core/pubtools.py:163: error: Argument 1 to "Timegrids" has incompatible type "*Timegrids | Timegrid | tuple[datetime | str | Date, datetime | str | Date, timedelta | str | Period]"; expected "Timegrid"  [arg-type]
+ hydpy/core/pubtools.py:163: error: Argument 1 to "Timegrids" has incompatible type "*Timegrids | Timegrid | tuple[datetime | str | Date, datetime | str | Date, timedelta | str | Period]"; expected "Timegrid | None"  [arg-type]

aiohttp (https://github.com/aio-libs/aiohttp)
+ aiohttp/web_urldispatcher.py:671:16: error: Exception type must be derived from BaseException (or be a tuple of exception classes)  [misc]

colour (https://github.com/colour-science/colour)
+ colour/plotting/models.py:1986: error: Argument 2 to "_linear_equation" has incompatible type "*ndarray[tuple[int], dtype[float64]]"; expected "ndarray[tuple[Any, ...], dtype[floating[_16Bit] | floating[_32Bit] | float64]]"  [arg-type]

core (https://github.com/home-assistant/core)
+ homeassistant/components/govee_light_local/light.py:216: error: Argument 2 to "set_rgb_color" of "GoveeLocalApiCoordinator" has incompatible type "*tuple[int, int, int] | tuple[int | None]"; expected "int"  [arg-type]
+ homeassistant/components/govee_light_local/light.py:218: error: Argument 2 to "set_temperature" of "GoveeLocalApiCoordinator" has incompatible type "*tuple[int, int, int] | tuple[int | None]"; expected "int"  [arg-type]

comtypes (https://github.com/enthought/comtypes)
- comtypes/_memberspec.py:85: error: Unused "type: ignore" comment  [unused-ignore]

werkzeug (https://github.com/pallets/werkzeug)
+ src/werkzeug/test.py:436: error: Argument 2 to "add_file" of "FileMultiDict" has incompatible type "*Union[tuple[IO[bytes], str], tuple[IO[bytes], str, str]]"; expected "Optional[str]"  [arg-type]

dd-trace-py (https://github.com/DataDog/dd-trace-py)
+ ddtrace/llmobs/_integrations/bedrock_agents.py:237: error: Argument 1 to "set_exc_info" of "Span" has incompatible type "*tuple[type[BaseException], BaseException, TracebackType] | tuple[None, None, None]"; expected "type[BaseException]"  [arg-type]
+ ddtrace/llmobs/_integrations/bedrock_agents.py:237: error: Argument 1 to "set_exc_info" of "Span" has incompatible type "*tuple[type[BaseException], BaseException, TracebackType] | tuple[None, None, None]"; expected "BaseException"  [arg-type]
+ ddtrace/llmobs/_integrations/bedrock_agents.py:286: error: Argument 1 to "set_exc_info" of "Span" has incompatible type "*tuple[type[BaseException], BaseException, TracebackType] | tuple[None, None, None]"; expected "type[BaseException]"  [arg-type]
+ ddtrace/llmobs/_integrations/bedrock_agents.py:286: error: Argument 1 to "set_exc_info" of "Span" has incompatible type "*tuple[type[BaseException], BaseException, TracebackType] | tuple[None, None, None]"; expected "BaseException"  [arg-type]
+ ddtrace/llmobs/_experiment.py:367: error: Argument 1 to "set_exc_info" of "Span" has incompatible type "*tuple[type[BaseException], BaseException, TracebackType] | tuple[None, None, None]"; expected "type[BaseException]"  [arg-type]
+ ddtrace/llmobs/_experiment.py:367: error: Argument 1 to "set_exc_info" of "Span" has incompatible type "*tuple[type[BaseException], BaseException, TracebackType] | tuple[None, None, None]"; expected "BaseException"  [arg-type]

dedupe (https://github.com/dedupeio/dedupe)
+ dedupe/core.py:196: error: Argument 1 to "zip" has incompatible type "*tuple[int, Mapping[str, Any]] | tuple[str, Mapping[str, Any]]"; expected "Iterable[str]"  [arg-type]

materialize (https://github.com/MaterializeInc/materialize)
+ misc/python/materialize/mzcompose/composition.py:382: error: Argument 1 to "Popen" has incompatible type "list[object]"; expected "str | bytes | PathLike[str] | PathLike[bytes] | Sequence[str | bytes | PathLike[str] | PathLike[bytes]]"  [arg-type]
+ misc/python/materialize/mzcompose/composition.py:433: error: Not all union combinations were tried because there are too many unions  [misc]
+ misc/python/materialize/mzcompose/composition.py:434: error: Argument 1 to "run" has incompatible type "list[object]"; expected "str | bytes | PathLike[str] | PathLike[bytes] | Sequence[str | bytes | PathLike[str] | PathLike[bytes]]"  [arg-type]

ignite (https://github.com/pytorch/ignite)
+ ignite/engine/engine.py:156: error: Argument 1 to "register_events" of "Engine" has incompatible type "*type[Events]"; expected "list[str] | list[EventEnum]"  [arg-type]
+ ignite/contrib/engines/tbptt.py:120: error: Argument 1 to "register_events" of "Engine" has incompatible type "*type[Tbptt_Events]"; expected "list[str] | list[EventEnum]"  [arg-type]

xarray (https://github.com/pydata/xarray)
+ xarray/tests/test_namedarray.py: note: In member "test_permute_dims" of class "TestNamedArray":
+ xarray/tests/test_namedarray.py:545: error: Argument 1 to "permute_dims" of "NamedArray" has incompatible type "*str | Iterable[Hashable]"; expected "Iterable[Hashable] | EllipsisType"  [arg-type]
+ xarray/tests/test_namedarray.py: note: In class "TestNamedArray":

bokeh (https://github.com/bokeh/bokeh)
+ src/bokeh/server/tornado.py: note: In member "__init__" of class "BokehTornado":
+ src/bokeh/server/tornado.py:450:41: error: Argument 1 to "append" of "list" has incompatible type "tuple[dict[str, bool | dict[str, ApplicationContext] | str | None] | str | type[RequestHandler] | dict[str, Any], ...]"; expected "tuple[str, type[RequestHandler]] | tuple[str, type[RequestHandler], dict[str, Any]]"  [arg-type]
+ src/bokeh/server/tornado.py:453:37: error: Argument 1 to "append" of "list" has incompatible type "tuple[dict[str, bool | dict[str, ApplicationContext] | str | None] | str | type[RequestHandler] | dict[str, Any], ...]"; expected "tuple[str, type[RequestHandler]] | tuple[str, type[RequestHandler], dict[str, Any]]"  [arg-type]
+ src/bokeh/server/tornado.py: note: At top level:

randolf-scholz force-pushed the fix_list_comprehension branch from 7146a29 to 049afbc Compare August 13, 2025 12:38

This comment has been minimized.

Sign in to view

randolf-scholz marked this pull request as ready for review August 14, 2025 19:00

randolf-scholz marked this pull request as draft August 14, 2025 19:01

randolf-scholz added 4 commits August 15, 2025 09:34

rework starargs with union argument

245cba3

Use typing.Unpack in tests

f2d254c

reworked argmapper with star args

df7b3f0

always use the solver meachnism

5e239a3

randolf-scholz force-pushed the fix_list_comprehension branch from b341226 to 5e239a3 Compare August 15, 2025 08:13

This comment has been minimized.

Sign in to view

randolf-scholz added 3 commits August 15, 2025 11:55

fixed infer_constraints

6cbb237

cleanup

115bf00

Merge branch 'master' into fix_list_comprehension

08db3da

This comment has been minimized.

Sign in to view

randolf-scholz marked this pull request as ready for review August 15, 2025 14:01

randolf-scholz mentioned this pull request Aug 15, 2025

Fix inference when unpacking union type #19650

Open

sterliakov reviewed Aug 15, 2025

View reviewed changes

randolf-scholz added 2 commits August 15, 2025 17:41

address review

fac1e07

Merge branch 'python:master' into fix_list_comprehension

68bc671

This comment has been minimized.

Sign in to view

randolf-scholz mentioned this pull request Aug 15, 2025

make ParamSpecFlavor an IntEnum and ParamSpecType generic in that type. #19670

Closed

check paramspec flavor

e8dcf88

This comment has been minimized.

Sign in to view

randolf-scholz added 2 commits August 15, 2025 19:24

Revert "check paramspec flavor"

fe4289a

This reverts commit e8dcf88.

update parse_star_args_type docstring

0139c12

update parse_star_args_type docstring

109b4f3

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Rework starargs with union argument #19651

Rework starargs with union argument #19651

randolf-scholz commented Aug 13, 2025 •

edited

Loading

Uh oh!

This comment has been minimized.

randolf-scholz commented Aug 13, 2025

Uh oh!

randolf-scholz commented Aug 13, 2025 •

edited

Loading

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

randolf-scholz commented Aug 15, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Aug 15, 2025

Uh oh!

Uh oh!

Uh oh!

Rework starargs with union argument #19651

Are you sure you want to change the base?

Rework starargs with union argument #19651

Conversation

randolf-scholz commented Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dev notes:

Footnotes

Uh oh!

This comment has been minimized.

randolf-scholz commented Aug 13, 2025

Uh oh!

randolf-scholz commented Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

randolf-scholz commented Aug 15, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

github-actions bot commented Aug 15, 2025

Uh oh!

Uh oh!

randolf-scholz commented Aug 13, 2025 •

edited

Loading

randolf-scholz commented Aug 13, 2025 •

edited

Loading