Skip to content

@overloading an @lru_cache'd function crashes mypy #9112

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
akdor1154 opened this issue Jul 9, 2020 · 3 comments · Fixed by #11630
Closed

@overloading an @lru_cache'd function crashes mypy #9112

akdor1154 opened this issue Jul 9, 2020 · 3 comments · Fixed by #11630

Comments

@akdor1154
Copy link

akdor1154 commented Jul 9, 2020 •
edited
Loading

Simple to reproduce, mypy 0.782.

#!/usr/bin/env python
from typing import *
from functools import lru_cache

@overload
def f(a: str, b: str) -> int: ...

@overload
def f(a: str) -> str: ...

@lru_cache
def f(a: str, b: Optional[str] = None) -> Union[str, int]:
  if b is not None:
    return 123
  else:
    return 'abc'
mypy --strict --show-traceback .
./example.py:5: error: INTERNAL ERROR -- Please try using mypy master on Github:
https://mypy.rtfd.io/en/latest/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 0.782
Traceback (most recent call last):
  File "mypy/checker.py", line 401, in accept
  File "mypy/nodes.py", line 515, in accept
  File "mypy/checker.py", line 434, in visit_overloaded_func_def
  File "mypy/checker.py", line 461, in _visit_overloaded_func_def
  File "mypy/checker.py", line 482, in check_overlapping_overloads
AssertionError: 
./example.py:5: : note: use --pdb to drop into pdb

Also happens with master: version: 0.790+dev.ffdbeb3d47ccb2d9691b36993be0a18e0718d997
@hauntsaninja
Copy link
Collaborator

Similar to #8356

@thomasgilgenast
Copy link

thomasgilgenast commented Jul 11, 2020 •
edited
Loading

Full traceback in case it is helpful:

Traceback (most recent call last):
  File ".../bin/mypy", line 33, in <module>
    sys.exit(load_entry_point('mypy', 'console_scripts', 'mypy')())
  File ".../git/mypy/mypy/__main__.py", line 8, in console_entry
    main(None, sys.stdout, sys.stderr)
  File ".../git/mypy/mypy/main.py", line 89, in main
    res = build.build(sources, options, None, flush_errors, fscache, stdout, stderr)
  File ".../git/mypy/mypy/build.py", line 181, in build
    sources, options, alt_lib_path, flush_errors, fscache, stdout, stderr, extra_plugins
  File ".../git/mypy/mypy/build.py", line 252, in _build
    graph = dispatch(sources, manager, stdout)
  File ".../git/mypy/mypy/build.py", line 2626, in dispatch
    process_graph(graph, manager)
  File ".../git/mypy/mypy/build.py", line 2949, in process_graph
    process_stale_scc(graph, scc, manager)
  File ".../git/mypy/mypy/build.py", line 3047, in process_stale_scc
    graph[id].type_check_first_pass()
  File ".../git/mypy/mypy/build.py", line 2107, in type_check_first_pass
    self.type_checker().check_first_pass()
  File ".../git/mypy/mypy/checker.py", line 294, in check_first_pass
    self.accept(d)
  File ".../git/mypy/mypy/checker.py", line 401, in accept
    stmt.accept(self)
  File ".../git/mypy/mypy/nodes.py", line 516, in accept
    return visitor.visit_overloaded_func_def(self)
  File ".../git/mypy/mypy/checker.py", line 434, in visit_overloaded_func_def
    self._visit_overloaded_func_def(defn)
  File ".../git/mypy/mypy/checker.py", line 461, in _visit_overloaded_func_def
    self.check_overlapping_overloads(defn)
  File ".../git/mypy/mypy/checker.py", line 482, in check_overlapping_overloads
    assert isinstance(inner_type, CallableType)
AssertionError:

In my case:

  • defn.impl is a mypy.nodes.Decorator object, so inner_type is taken from defn.impl.var.type
  • inner_type is functools._lru_cache_wrapper[Any] and has type <class 'mypy.types.Instance'>

Simply returning if inner_type is not an instance of CallableType solves the problem for me:

$ diff --git a/mypy/checker.py b/mypy/checker.py
index 25584401..cf178c94 100644
--- a/mypy/checker.py
+++ b/mypy/checker.py
@@ -479,7 +479,8 @@ class TypeChecker(NodeVisitor[None], CheckerPluginInterface):
             # decorator or if the implementation is untyped -- we gave up on the types.
             inner_type = get_proper_type(inner_type)
             if inner_type is not None and not isinstance(inner_type, AnyType):
-                assert isinstance(inner_type, CallableType)
+                if not isinstance(inner_type, CallableType):
+                    return
                 impl_type = inner_type

         is_descriptor_get = defn.info and defn.name == "__get__"

My understanding is very limited here, but at first glance it seems to me like mypy is simply not equipped to reason about this situation, so letting this check pass seems better than crashing. Hoping someone with more experience can chime in and explain a bit more of the context here, though.

After this change, mypy correctly reasons about the return types of the overloaded decorated function.

@Phlogistique
Copy link
Contributor

This happens with any decorator which does not return a callable, not only @lru_cache. For example this produces the same crash:

from typing import overload

def silly_decorator(x) -> None: ...


@overload
def silly_function(x: int):
    ...


@overload
def silly_function(x: str):
    ...


@silly_decorator
def silly_function(x):
    ...

@jakebailey jakebailey mentioned this issue Oct 12, 2021
Closed
@hauntsaninja hauntsaninja mentioned this issue Nov 27, 2021
Merged
hauntsaninja added a commit that referenced this issue Nov 28, 2021
@hauntsaninja
Fix crash with overload and callable object decorators (#11630)
7b13ff8 
Fixes #8356, as identified by @pranavrajpal
Fixes #9112
Fixes #9967

Note that we still don't fully support the singledispatch pattern in #8356,
since we get 'overloaded function has no attribute "register"', but that's
much easier to work around than a crash.

Co-authored-by: hauntsaninja <>
ilevkivskyi pushed a commit that referenced this issue Dec 3, 2021
@hauntsaninja
Fix crash with overload and callable object decorators (#11630)
dd086bc 
Fixes #8356, as identified by @pranavrajpal
Fixes #9112
Fixes #9967

Note that we still don't fully support the singledispatch pattern in #8356,
since we get 'overloaded function has no attribute "register"', but that's
much easier to work around than a crash.

Co-authored-by: hauntsaninja <>
tushar-deepsource pushed a commit to DeepSourceCorp/mypy that referenced this issue Jan 20, 2022
@hauntsaninja @tushar-deepsource
Fix crash with overload and callable object decorators (python#11630)
713df5e 
Fixes python#8356, as identified by @pranavrajpal
Fixes python#9112
Fixes python#9967

Note that we still don't fully support the singledispatch pattern in python#8356,
since we get 'overloaded function has no attribute "register"', but that's
much easier to work around than a crash.

Co-authored-by: hauntsaninja <>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix crash with overload and callable object decorators hauntsaninja/mypy
4 participants
@Phlogistique @thomasgilgenast @akdor1154 @hauntsaninja