Attempt to fix Distribute Tuple False Positive bug #19005
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…into cleanDistributeTuple
|
Diff from mypy_primer, showing the effect of this PR on open source code: pydantic (https://github.com/pydantic/pydantic)
+ pydantic/json_schema.py:2478: error: INTERNAL ERROR -- Please try using mypy master on GitHub:
+ https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
+ Please report a bug at https://github.com/python/mypy/issues
+ version: 1.16.0+dev.eaaff6f835b5e045c7f1cac969ef9a240a490236
+ pydantic/json_schema.py:2478: : note: use --pdb to drop into pdb
- pydantic/errors.py:128: error: Unused "type: ignore" comment [unused-ignore]
- pydantic/_internal/_utils.py:182: error: Unused "type: ignore" comment [unused-ignore]
- pydantic/_internal/_validators.py:134: error: Unused "type: ignore" comment [unused-ignore]
- pydantic/_internal/_mock_val_ser.py:137: error: Incompatible types in assignment (expression has type "MockValSer[SchemaValidator | PluggableSchemaValidator | SchemaSerializer]", variable has type "SchemaValidator | PluggableSchemaValidator") [assignment]
- pydantic/_internal/_mock_val_ser.py:143: error: Incompatible types in assignment (expression has type "MockValSer[SchemaSerializer]", variable has type "SchemaSerializer") [assignment]
- pydantic/_internal/_generics.py:63: error: Unused "type: ignore" comment [unused-ignore]
- pydantic/_internal/_generics.py:223: error: Unused "type: ignore" comment [unused-ignore]
- pydantic/_internal/_decorators.py:301: error: Unused "type: ignore" comment [unused-ignore]
- pydantic/_internal/_decorators.py:759: error: Unused "type: ignore" comment [unused-ignore]
- pydantic/_internal/_config.py:142: error: TypedDict key must be a string literal; expected one of ("title", "model_title_generator", "field_title_generator", "str_to_lower", "str_to_upper", ...) [literal-required]
- pydantic/json_schema.py:451: error: Unused "type: ignore" comment [unused-ignore]
- pydantic/json_schema.py:1961: error: Unused "type: ignore" comment [unused-ignore]
+ Traceback (most recent call last):
+ File "", line 8, in <module>
+ sys.exit(console_entry())
+ File "/__main__.py", line 15, in console_entry
+ main()
+ File "/main.py", line 127, in main
+ res, messages, blockers = run_build(sources, options, fscache, t0, stdout, stderr)
+ File "/main.py", line 211, in run_build
+ res = build.build(sources, options, None, flush_errors, fscache, stdout, stderr)
+ File "/build.py", line 191, in build
+ result = _build(
+ File "/build.py", line 267, in _build
+ graph = dispatch(sources, manager, stdout)
+ File "/build.py", line 2939, in dispatch
+ process_graph(graph, manager)
+ File "/build.py", line 3337, in process_graph
+ process_stale_scc(graph, scc, manager)
+ File "/build.py", line 3438, in process_stale_scc
+ graph[id].type_check_first_pass()
+ File "/build.py", line 2311, in type_check_first_pass
+ self.type_checker().check_first_pass()
+ File "/checker.py", line 472, in check_first_pass
+ self.accept(d)
+ File "/checker.py", line 579, in accept
+ stmt.accept(self)
+ File "/nodes.py", line 813, in accept
+ return visitor.visit_func_def(self)
+ File "/checker.py", line 1129, in visit_func_def
+ self._visit_func_def(defn)
+ File "/checker.py", line 1133, in _visit_func_def
+ self.check_func_item(defn, name=defn.name)
+ File "/checker.py", line 1168, in check_func_item
+ self.check_func_def(defn, typ, name, allow_empty)
+ File "/checker.py", line 1455, in check_func_def
+ self.accept(item.body)
+ File "/checker.py", line 579, in accept
+ stmt.accept(self)
+ File "/nodes.py", line 1281, in accept
+ return visitor.visit_block(self)
+ File "/checker.py", line 3025, in visit_block
+ self.accept(s)
+ File "/checker.py", line 579, in accept
+ stmt.accept(self)
+ File "/nodes.py", line 1549, in accept
+ return visitor.visit_if_stmt(self)
+ File "/checker.py", line 4846, in visit_if_stmt
+ self.accept(b)
+ File "/checker.py", line 579, in accept
+ stmt.accept(self)
+ File "/nodes.py", line 1281, in accept
+ return visitor.visit_block(self)
+ File "/checker.py", line 3025, in visit_block
+ self.accept(s)
+ File "/checker.py", line 579, in accept
+ stmt.accept(self)
+ File "/nodes.py", line 1477, in accept
+ return visitor.visit_return_stmt(self)
+ File "/checker.py", line 4724, in visit_return_stmt
+ self.check_return_stmt(s)
+ File "/checker.py", line 4760, in check_return_stmt
+ self.expr_checker.accept(
+ File "/checkexpr.py", line 5970, in accept
+ typ = node.accept(self)
+ File "/nodes.py", line 1990, in accept
+ return visitor.visit_call_expr(self)
+ ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
+ File "/checkexpr.py", line 488, in visit_call_expr
+ return self.visit_call_expr_inner(e, allow_none_return=allow_none_return)
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+ File "/checkexpr.py", line 622, in visit_call_expr_inner
+ ret_type = self.check_call_expr_with_callee_type(
+ callee_type, e, fullname, object_type, member
+ )
+ File "/checkexpr.py", line 1478, in check_call_expr_with_callee_type
+ ret_type, callee_type = self.check_call(
+ ~~~~~~~~~~~~~~~^
+ callee_type,
+ ^^^^^^^^^^^^
+ ...<6 lines>...
+ object_type=object_type,
+ ^^^^^^^^^^^^^^^^^^^^^^^^
+ )
+ ^
+ File "/checkexpr.py", line 1580, in check_call
+ return self.check_callable_call(
+ ~~~~~~~~~~~~~~~~~~~~~~~~^
+ callee,
+ ^^^^^^^
+ ...<6 lines>...
+ object_type,
+ ^^^^^^^^^^^^
+ )
+ ^
+ File "/checkexpr.py", line 1752, in check_callable_call
+ callee = self.infer_function_type_arguments_using_context(callee, context)
+ File "/checkexpr.py", line 2042, in infer_function_type_arguments_using_context
+ args = infer_type_arguments(
+ callable.variables, ret_type, erased_ctx, skip_unsatisfied=True
+ )
+ File "/infer.py", line 75, in infer_type_arguments
+ constraints = infer_constraints(template, actual, SUPERTYPE_OF if is_supertype else SUBTYPE_OF)
+ File "/constraints.py", line 317, in infer_constraints
+ res = _infer_constraints(template, actual, direction, skip_neg_op)
+ File "/constraints.py", line 336, in _infer_constraints
+ actual = mypy.typeops.make_simplified_union(actual.items, keep_erased=True)
+ File "/typeops.py", line 622, in make_simplified_union
+ simplified_set: Sequence[Type] = _remove_redundant_union_items(items, keep_erased)
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
+ File "/typeops.py", line 708, in _remove_redundant_union_items
+ if is_proper_subtype(
+ ~~~~~~~~~~~~~~~~~^
+ ti, tj, keep_erased_types=keep_erased, ignore_promotions=True
+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+ ):
+ ^
+ File "/subtypes.py", line 227, in is_proper_subtype
+ return _is_subtype(left, right, subtype_context, proper_subtype=True)
+ File "/subtypes.py", line 395, in _is_subtype
+ return left.accept(SubtypeVisitor(orig_right, subtype_context, proper_subtype))
+ ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+ File "/types.py", line 1475, in accept
+ return visitor.visit_instance(self)
+ ~~~~~~~~~~~~~~~~~~~~~~^^^^^^
+ File "/subtypes.py", line 622, in visit_instance
+ if not check_type_parameter(
+ ~~~~~~~~~~~~~~~~~~~~^
+ lefta, righta, variance, self.proper_subtype, self.subtype_context
+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+ ):
+ ^
+ File "/subtypes.py", line 412, in check_type_parameter
+ return is_proper_subtype(left, right, subtype_context=subtype_context)
+ File "/subtypes.py", line 227, in is_proper_subtype
+ return _is_subtype(left, right, subtype_context, proper_subtype=True)
+ File "/subtypes.py", line 342, in _is_subtype
+ _is_subtype(t, right, subtype_context, proper_subtype=False)
+ ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+ File "/subtypes.py", line 307, in _is_subtype
+ subtype_context.check_context(proper_subtype)
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
+ File "/subtypes.py", line 120, in check_context
+ assert not self.erase_instances and not self.keep_erased_types
+ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+ AssertionError:
- pydantic/json_schema.py:2056: error: Unused "type: ignore" comment [unused-ignore]
- pydantic/json_schema.py:2698: error: Missing return statement [return]
- pydantic/_internal/_fields.py:69: error: Unused "type: ignore" comment [unused-ignore]
- pydantic/types.py:230: error: Incompatible return value type (got "<typing special form>", expected "type[int]") [return-value]
- pydantic/types.py:491: error: Incompatible return value type (got "<typing special form>", expected "type[float]") [return-value]
- pydantic/types.py:679: error: Incompatible return value type (got "<typing special form>", expected "type[bytes]") [return-value]
- pydantic/types.py:817: error: Incompatible return value type (got "<typing special form>", expected "type[str]") [return-value]
- pydantic/types.py:852: error: Incompatible return value type (got "<typing special form>", expected "type[set[HashableItemType]]") [return-value]
- pydantic/types.py:868: error: Incompatible return value type (got "<typing special form>", expected "type[frozenset[HashableItemType]]") [return-value]
- pydantic/types.py:903: error: Incompatible return value type (got "<typing special form>", expected "type[list[AnyItemType]]") [return-value]
- pydantic/types.py:1125: error: Incompatible return value type (got "<typing special form>", expected "type[Decimal]") [return-value]
- pydantic/types.py:1704: error: Unused "type: ignore" comment [unused-ignore]
- pydantic/types.py:1736: error: Unused "type: ignore" comment [unused-ignore]
- pydantic/types.py:2262: error: Incompatible return value type (got "<typing special form>", expected "type[date]") [return-value]
- pydantic/types.py:2974: error: "Mapping[str, Any]" has no attribute "setdefault" [attr-defined]
- pydantic/color.py:104: error: Need type annotation for "field_schema" (hint: "field_schema: dict[<type>, <type>] = ...") [var-annotated]
- pydantic/_internal/_known_annotated_metadata.py:191: error: "Mapping[str, Any]" has no attribute "copy" [attr-defined]
- pydantic/_internal/_known_annotated_metadata.py:222: error: Unsupported target for indexed assignment ("Mapping[str, Any]") [index]
- pydantic/_internal/_known_annotated_metadata.py:222: error: Unsupported target for indexed assignment ("Mapping[str, Any] | AfterValidatorFunctionSchema") [index]
- pydantic/_internal/_known_annotated_metadata.py:222: error: TypedDict key must be a string literal; expected one of ("function", "schema", "ref", "metadata", "serialization", ...) [literal-required]
- pydantic/_internal/_known_annotated_metadata.py:255: error: Unused "type: ignore" comment [unused-ignore]
- pydantic/_internal/_known_annotated_metadata.py:309: error: Unused "type: ignore" comment [unused-ignore]
- pydantic/_internal/_known_annotated_metadata.py:315: error: Unused "type: ignore" comment [unused-ignore]
- pydantic/fields.py:493: error: Unused "type: ignore" comment [unused-ignore]
- pydantic/fields.py:561: error: No overload variant of "Field" matches argument types "Any", "Any", "bool", "dict[Any, Any]" [call-overload]
- pydantic/fields.py:561: note: Possible overload variants:
- pydantic/fields.py:561: note: def Field(default, default: EllipsisType, *, alias: str | None = ..., alias_priority: int | None = ..., validation_alias: str | AliasPath | AliasChoices | None = ..., serialization_alias: str | None = ..., title: str | None = ..., field_title_generator: Callable[[str, FieldInfo], str] | None = ..., description: str | None = ..., examples: list[Any] | None = ..., exclude: bool | None = ..., discriminator: str | Discriminator | None = ..., deprecated: deprecated | str | bool | None = ..., json_schema_extra: JsonDict | Callable[[JsonDict], None] | None = ..., frozen: bool | None = ..., validate_default: bool | None = ..., repr: bool = ..., init: bool | None = ..., init_var: bool | None = ..., kw_only: bool | None = ..., pattern: str | Pattern[str] | None = ..., strict: bool | None = ..., coerce_numbers_to_str: bool | None = ..., gt: SupportsGt | None = ..., ge: SupportsGe | None = ..., lt: SupportsLt | None = ..., le: SupportsLe | None = ..., multiple_of: float | None = ..., allow_inf_nan: bool | None = ..., max_digits: int | None = ..., decimal_places: int | None = ..., min_length: int | None = ..., max_length: int | None = ..., union_mode: Literal['smart', 'left_to_right'] = ..., fail_fast: bool | None = ...) -> Any
- pydantic/fields.py:561: note: def [_T] Field(default, default: _T, *, alias: str | None = ..., alias_priority: int | None = ..., validation_alias: str | AliasPath | AliasChoices | None = ..., serialization_alias: str | None = ..., title: str | None = ..., field_title_generator: Callable[[str, FieldInfo], str] | None = ..., description: str | None = ..., examples: list[Any] | None = ..., exclude: bool | None = ..., discriminator: str | Discriminator | None = ..., deprecated: deprecated | str | bool | None = ..., json_schema_extra: JsonDict | Callable[[JsonDict], None] | None = ..., frozen: bool | None = ..., validate_default: bool | None = ..., repr: bool = ..., init: bool | None = ..., init_var: bool | None = ..., kw_only: bool | None = ..., pattern: str | Pattern[str] | None = ..., strict: bool | None = ..., coerce_numbers_to_str: bool | None = ..., gt: SupportsGt | None = ..., ge: SupportsGe | None = ..., lt: SupportsLt | None = ..., le: SupportsLe | None = ..., multiple_of: float | None = ..., allow_inf_nan: bool | None = ..., max_digits: int | None = ..., decimal_places: int | None = ..., min_length: int | None = ..., max_length: int | None = ..., union_mode: Literal['smart', 'left_to_right'] = ..., fail_fast: bool | None = ...) -> _T
- pydantic/fields.py:561: note: def [_T] Field(default_factory, *, default_factory: Callable[[], _T] | Callable[[dict[str, Any]], _T], alias: str | None = ..., alias_priority: int | None = ..., validation_alias: str | AliasPath | AliasChoices | None = ..., serialization_alias: str | None = ..., title: str | None = ..., field_title_generator: Callable[[str, FieldInfo], str] | None = ..., description: str | None = ..., examples: list[Any] | None = ..., exclude: bool | None = ..., discriminator: str | Discriminator | None = ..., deprecated: deprecated | str | bool | None = ..., json_schema_extra: JsonDict | Callable[[JsonDict], None] | None = ..., frozen: bool | None = ..., validate_default: bool | None = ..., repr: bool = ..., init: bool | None = ..., init_var: bool | None = ..., kw_only: bool | None = ..., pattern: str | Pattern[str] | None = ..., strict: bool | None = ..., coerce_numbers_to_str: bool | None = ..., gt: SupportsGt | None = ..., ge: SupportsGe | None = ..., lt: SupportsLt | None = ..., le: SupportsLe | None = ..., multiple_of: float | None = ..., allow_inf_nan: bool | None = ..., max_digits: int | None = ..., decimal_places: int | None = ..., min_length: int | None = ..., max_length: int | None = ..., union_mode: Literal['smart', 'left_to_right'] = ..., fail_fast: bool | None = ...) -> _T
... (truncated 185 lines) ...
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is being changed?⁉️
This PR is an attempt to fix issue #18922 from the myPy repo.
Issue link: #18922 (comment)
How is it implemented? 🛠️
The approach addresses the bug report by explicitly distributing union types across tuple elements when a TupleType is compared against a UnionType. It first checks whether any elements of the tuple are themselves union types, and if so, computes the Cartesian product of all combinations of those element types. For each combination, it constructs a new TupleType representing one possible instantiation of the original tuple. These are then combined into a single UnionType using make_simplified_union, effectively transforming something like Tuple[float, Optional[float]] into Union[Tuple[float, float], Tuple[float, None]]. Finally, this expanded union is compared to the right-hand side type for subtyping.
Results 📊
This approaches successful identify there is no issue with:
and identify there is an issue with:
Limitation of this approach 🔄
However, this approach leads to an maximum recursive depth error when running on recursive data structure, more specifically on the test testRecursiveDoubleUnionNoCrash in check-recursive-types.test. The following example is two recursive data type that are trying to get checked whether they are a subtype of another. However, they don't fall under
is_recursive_paircase, so they continue on to get checked, since they are recursive, the check goes on forever. An attempt with cache was also tried but failed due to these datatypes don't have anlast_knownproperty.Example:
Some important notes 📝
Since the implemented solution leads to an maximum recursive depth error when running on recursive data structure, some higher level recursive tests were not able to be added. Therefore, the below test cases only target certain cases that the implementation takes care of. In the future, once the maximum recursive depth error is fixed, more tests should be added.
For example, the following test case should succeed but it currently fails with the error
Incompatible return value type (got "Tuple[int, Tuple[Union[str, bytes], float]]", expected "Union[Tuple[int, Tuple[str, float]], Tuple[int, Tuple[bytes, float]]]")Test cases 🧪
Success cases (testTupleUnionDistributionSuccess)
f1: Distribute Optional[float] in second position
f2: Distribute Union[int,str] in first position
f3: Distribute Union[str,None] in second position
f4: Single-element tuple with Union[int,float]
f5: Two positions both containing unions
Failure cases (testTupleUnionDistributionFail)
g1: Mismatched first branch (str vs. expected float)
g2: Mismatched second branch (str vs. expected float)
Note: Each case ends with [builtins fixtures/tuple.pyi] to import the tuple stubs required by the harness.