Skip to content

min()/max() with default value cause issues with key function type #17536

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
pfaion opened this issue Jul 18, 2024 · 3 comments · May be fixed by #18976
Open

min()/max() with default value cause issues with key function type #17536

pfaion opened this issue Jul 18, 2024 · 3 comments · May be fixed by #18976
Labels
bug mypy got something wrong

Comments

@pfaion
Copy link

pfaion commented Jul 18, 2024

Bug Report

Hi everyone, thanks for all the work on mypy, I love this tool! 👏

I run into an issue with this code:

def foo(values: list[dict]) -> dict | None:
    return max(values, default=None, key=lambda entry: entry.get("value", 0))

See playground: https://mypy-play.net/?mypy=latest&python=3.12&gist=3daadb92c263a8447595aa7640a04599

Expected Behavior:

mypy should not report any errors

Actual Behavior

mypy reports an error:

Item "None" of "dict[Any, Any] | None" has no attribute "get"

Environment

  • Mypy version used: mypy 1.10.1 (compiled: yes)
  • Mypy command-line flags: none
  • Mypy configuration options from mypy.ini (and other config files): none
  • Python version used: Python 3.12.2

Additional Details

I found this issue from 2019, which seems to be the same issue: #6460

Here people reported that it was an issue with typeshed, which was fixed in typeshed with this PR: python/typeshed#2833

I checked current trunk for typeshed and cannot spot any issue with the stubs for max(), the type correctly seems to indicate that the key function has nothing to do with the type of the default parameter: https://github.com/python/typeshed/blob/d482d4e83c9d247e79c87d954a9ccb11f85bfc37/stdlib/builtins.pyi#L1515-L1516

So while it seems this issue was once fixed it the past, it looks like it might have reappeared through a different cause.

@pfaion pfaion added the bug mypy got something wrong label Jul 18, 2024
@pfaion
Copy link
Author

pfaion commented Sep 29, 2024

I looked a bit into this and managed to create an example that's isolated from the max type stub. The type stub is correct!

from typing import List, Union, Callable, TypeVar, reveal_type

_T1 = TypeVar("_T1")
_T2 = TypeVar("_T2")

def bar(a: _T1, b: _T2, function: Callable[[_T2], None]) -> Union[_T1, _T2]:
    raise NotImplementedError()

# this does not cause an issue,
result_1 = bar(42, [42], lambda a_list: a_list.append(42))
reveal_type(result_1)  # note: Revealed type is "Union[builtins.int, builtins.list[builtins.int]]"

# this is the same call, we set the same type, but it causes an issue with the lambda body
result_2: Union[int, List[int]] = bar(
    42,
    [42],
    lambda a_list: a_list.append(42)  # error: Item "int" of "int | list[int]" has no attribute "append"  [union-attr]
)

Playground link

Looking at the code, this seems to be because ExpressionChecker.infer_function_type_arguments_using_context() somehow infers the type variables from the return type, not from known passed argument values. I'll see if I can put up an MR for that.

@brianschubert
Copy link
Collaborator

Looks like a duplicate of #14664

Thanks for taking a deeper look into this @pfaion! Fair warning, this may be a tricky issue.

This has to do with the way mypy infers the types of generic callables, and in particular the relative "trust" given to the type context vs the argument types.

Currently, mypy infers the type of generic callables in two steps: first using the type context to infer against type variables in the return type, and second using the argument types to infer any remaining type variables.

The issue in this case is that, during the first step, mypy can use the type context to fully resolve the type variables into a broad supertype, which then gets propagated down into the argument checks, which then fail due to the inferred type being too broad.

The tricky bit is that, on its own, this "trusting the type context" behavior is a intended feature. It's what lets mypy use return types and explicit type annotations (which are generally very trustworthy) to type check inner expressions (which is often where the actual problems are). For example, it's what makes this possible:

def foo(x: list[int]) -> None: ...

foo([1, 2, 3, "4", 5])  # List item 3 has incompatible type "str"; expected "int"

Here, mypy needs to infer the type of the list being passed to foo. From the type context, it can see that the ultimate expected type is list[int]. This lets mypy resolve the type of the list constructor from def [T] (*T) -> list[T] to def (*int) -> list[int] (via inferring the against the return type using the type context). It can then check the individual arguments against the expected type int, allowing it to catch the unexpected str argument. In the alternative case, mypy could infer the type of the list just from the argument types. This would result in it deducing the type list[object], leading to the more opaque error Argument 1 to "foo" has incompatible type "list[object]"; expected "list[int]".

The fix to this issue will probably involve reworking the way that mypy uses the type context during type inference. I'm guessing this will invovle some major code changes, and probably has a high risk of breaking existing behavior. Good luck to whoever attempts it! :-)

@pfaion
Copy link
Author

pfaion commented Sep 29, 2024

Hmmm... ok, that is unfortunate.

I think if you'd infer the type from the arguments, you might be able to provide something like: Argument 1 to "foo" has incompatible type "list[int | str]"; expected "list[int]". Wouldn't that be somewhat useful?

Or maybe an exception needs to be made for "constructors" in this context? The example you gave sounded like you're interested in errors coming from inner expressions before errors coming from outer expressions. I'm wondering if that really is generally the case, or if that's an exception for when the inner expression is about constructing a type, where that would be more helpful?

I might try understanding the surrounding code a bit more. But thanks for the heads-up, I understand now how tricky this is. I was a bit confused why the type inference was started from the return type, but I thought there must be some explanation behind that!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug mypy got something wrong
Projects
None yet
2 participants