Skip to content

Replacing Callable[[T], T] with callback protocol in click #4045

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jun 26, 2020
Merged

Replacing Callable[[T], T] with callback protocol in click #4045

merged 2 commits into from
Jun 26, 2020

Conversation

mrkmndz
Copy link
Contributor

@mrkmndz mrkmndz commented May 19, 2020

We are trying to strictify the type variable scoping semantics in Pyre such that classes and the parameters of defs are the only places in which new type variable scopes can be created.

This would mean that things like

from typing import Callable, TypeVar
T = TypeVar("T")
def produce_no_op_decorator() -> Callable[[T], T]:...

would be rejected with Invalid type variable [34]: The type variable `Variable[T]` isn't present in the function's parameters.

We currently don't error here, but don't handle this consistently correctly.

MyPy currently supports this (mod python/mypy#3924), and we don't necessarily need that behavior to change. Instead we'd just like typeshed to reflect the lowest common denominator between our two semantics: callback protocols.

For the above example, both MyPy and Pyre support this alternative spelling:

from typing import Callable, TypeVar, Protocol
T = TypeVar("T")
class IdentityFunction(Protocol):
    def __call__(self, __x: T) -> T: ...
def produce_no_op_decorator() -> IdentityFunction...

This PR converts a series of decorators in click that look like the first example into spellings like the second example. If this is agreeable to the community, we'd like to replicate this throughout typeshed.

@mrkmndz mrkmndz marked this pull request as ready for review May 20, 2020 00:19
Copy link
Member

@JelleZijlstra JelleZijlstra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds sensible.

@srittau
Copy link
Collaborator

srittau commented May 20, 2020

I am not sure I like this change and I am unsure why pyre would like to introduce this arbitrary restriction. I find Callable[[_F], _F] more readable and convenient than introducing a new name by using a callback protocol. I am -1 on this change.

(Also, per our conventions, this would need to be called _IdentityFunction as it doesn't exist at runtime.)

@srittau
Copy link
Collaborator

srittau commented May 20, 2020

As this is quite a significant change, I think it's best discussed at typing-sig.

@mrkmndz
Copy link
Contributor Author

mrkmndz commented May 20, 2020

I am unsure why pyre would like to introduce this arbitrary restriction.

This isn't quite introducing a new restriction as much as codifying our pre-existing lack of support for a MyPy-specific special case. Pyre has always interpreted all free type variables in a def or class as being in the scope of that define/class. MyPy also does this in most circumstances with Callable[[T], T], as both checkers have identical interpretations of

T = TypeVar("T")
def f(x: T) -> Callable[[T], T]: ... # def <T> (T) -> Callable[[T], T]
# and
def f(f: Callable[[T], T]) -> T: ... # def <T> (Callable[[T], T]) -> T
#and
def i() -> Tuple[Callable[[T], T], T]: ...  # def <T>  () -> Tuple[Callable[[T], T], T]: ... 
# and
class Child(Base[Callable[[T], T]]): ... # class <T> Child(Base[Callable[[T], T]])

Pyre and MyPy only differ here in that MyPy introduces a unique special case that alters the behavior for free variables:

  • inside a Callable type
  • in the return type of a def
  • not present in any parameter of the def
  • not present anywhere else in the return type that isn't also inside of a Callable type.

I can definitely see how this special case is helpful in making definitions for decorator factories more compact, but for us the inconsistency of this behavior from a user's perspective outweighs the concision.

Happy to forward this discussion onto typing-sig if you think it would be helpful.

@mrkmndz
Copy link
Contributor Author

mrkmndz commented Jun 1, 2020

@srittau checking back in: do you still think that this needs to be discussed on typing-sig, or are you satisfied with the explanation above?

@srittau
Copy link
Collaborator

srittau commented Jun 1, 2020

@mrkmndz Sorry for forgetting to answer. Yes, considering theconsequences this has for the typing infrastructure, I think this should be discussed on typing-sig.

@mrkmndz
Copy link
Contributor Author

mrkmndz commented Jun 2, 2020

@mrkmndz
Copy link
Contributor Author

mrkmndz commented Jun 11, 2020

Hi @srittau , would you mind responding to @dkgi 's question of you on typing-sig?

I think we're both still a bit confused as to what precisely is blocking this PR.

Thanks!

@srittau
Copy link
Collaborator

srittau commented Jun 11, 2020

I am honestly not sure what else there is to add. So far, Callable[[_T], _T] is a construct that's accepted by typeshed. There seemed to be no support for changing this outside of the pyre team. I am opposed to replacing this construct with protocols all over typeshed, although I could live with a central IdentityFunction (or similar) alias in _typeshed. I'd be interested what the other typeshed maintainers think.

@JelleZijlstra
Copy link
Member

I just replied on the email thread; I think we should go ahead with this change.

Copy link
Collaborator

@srittau srittau left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's do this for now, although, as mentioned here and on typing-sig, I'd like _IdentityFunction to be moved to _typeshed.pyi.

@srittau srittau merged commit 1d16e6c into python:master Jun 26, 2020
vishalkuo pushed a commit to vishalkuo/typeshed that referenced this pull request Jun 26, 2020
facebook-github-bot pushed a commit to facebook/pyre-check that referenced this pull request Aug 6, 2020
Summary:
This behavior is confusing and slightly non-standard (python/typeshed#4045), thus it ought to be explained in detail on this page.

Pull Request resolved: #297

Reviewed By: shannonzhu

Differential Revision: D22964776

Pulled By: mrkmndz

fbshipit-source-id: ae3b168049d84cb721762ce6ac6d2c8a0c05e187
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants