Skip to content

bpo-44975: [typing] Support issubclass for ClassVar data members #27883

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 9 commits into from
Closed

bpo-44975: [typing] Support issubclass for ClassVar data members #27883

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
a613af6
Support issubclass for ClassVar data members
Fidget-Spinner Aug 22, 2021
f045d2e
make it work with string annotations and PEP 563
Fidget-Spinner Aug 23, 2021
58c69e1
Apply Lukasz' suggestions
Fidget-Spinner Aug 24, 2021
2ed2ee5
fix test
Fidget-Spinner Aug 24, 2021
c842370
Use identity check instead of equality
Fidget-Spinner Aug 26, 2021
f44d3b6
Merge remote-tracking branch 'upstream/main' into issubclass_data_pro…
Fidget-Spinner Dec 5, 2021
9278a34
fix merge issues
Fidget-Spinner Dec 5, 2021
ba57bb1
Add more tests and comments
Fidget-Spinner Dec 5, 2021
43ec2ea
support value checking classvars
Fidget-Spinner Dec 5, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions Doc/library/typing.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1284,6 +1284,11 @@ These are not used in annotations. They are building blocks for creating generic

.. versionadded:: 3.8

.. versionchanged:: 3.11
Protocols with data members annotated with :data:`ClassVar` now support
:func:`issubclass` checks. Subclasses must set these data members to pass.


Other special directives
""""""""""""""""""""""""

Expand Down
7 changes: 7 additions & 0 deletions Doc/whatsnew/3.11.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -284,6 +284,13 @@ sqlite3
experience.
(Contributed by Erlend E. Aasland in :issue:`45828`.)

typing
------

* Runtime protocols with data members now support :func:`issubclass` as long
as those members are annotated with :data:`typing.ClassVar`.
(Contributed by Ken Jin in :issue:`44975`).


sys
---
Expand Down
49 changes: 49 additions & 0 deletions Lib/test/test_typing.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1622,6 +1622,55 @@ def __init__(self):

Foo() # Previously triggered RecursionError

def test_runtime_issubclass_with_classvar_data_members(self):
@runtime_checkable
class P(Protocol):
x: ClassVar[int] = 1

class C: pass

class D:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's also add a case with the same class prop, but different value. Example:

class E:
    x = 2

Because right now all names / values always match. It is not clear whether value is a part of the protocol or not.

Copy link
Member Author

@Fidget-Spinner Fidget-Spinner Dec 5, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed.

It is not clear whether value is a part of the protocol or not.

It shouldn't be. IMO, we should only care about the "shape". Trying to runtime-check the value is difficult and should be left to third-party libs to do.

Nevermind, I changed my mind. Let's do it!

x = 1

class E:
x = 2

class F:
x = 'foo'

self.assertNotIsSubclass(C, P)
self.assertIsSubclass(D, P)
self.assertIsSubclass(E, P)
self.assertNotIsSubclass(F, P)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's also test these classes:

class G:
   x: ClassVar[int] = 1
class H:
   x: 'ClassVar[int]' = 1

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These aren't testing more code paths, the code doesn't look at the annotations of the first class argument in issubclass.

# String annotations (forward references).
@runtime_checkable
class P(Protocol):
# Special case, bare ClassVar, our checks should
# just skip these.
w: "ClassVar"
x: "ClassVar[int]" = 1
y: "typing.ClassVar[int]" = 2
z: "t.ClassVar[int]" = 3

class D:
w = 0
x = 1
y = 2
z = 3

self.assertNotIsSubclass(C, P)
self.assertIsSubclass(D, P)

# Make sure mixed are forbidden.
@runtime_checkable
class P(Protocol):
x: "ClassVar[int]" = 1
y = 2

self.assertRaises(TypeError, issubclass, C, P)
self.assertRaises(TypeError, issubclass, D, P)


class GenericTests(BaseTestCase):

Expand Down
52 changes: 43 additions & 9 deletions Lib/typing.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1404,10 +1404,43 @@ def _get_protocol_attrs(cls):
attrs.add(attr)
return attrs

_CLASSVAR_PREFIXES = ("typing.ClassVar", "t.ClassVar", "ClassVar")

def _is_callable_members_only(cls):
def _is_callable_or_classvar_members_only(cls, instance):
"""Returns a 2-tuple signalling two things:
(Valid protocol?, If not valid protocol, was it due to ClassVar value mismatch?)
"""
attr_names = _get_protocol_attrs(cls)
annotations = getattr(cls, '__annotations__', {})
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about using typing.get_type_hints to resolve annotations for a class?

It will allow having ClassVar annotated as str:

@runtime_checkable
class P(Protocol):
    x: 'ClassVar[int]' = 1
Suggested change
annotations = getattr(cls, '__annotations__', {})
annotations = get_type_hints(cls)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question! I'd considered that too but decided against it for two main reasons:

  1. get_type_hints is slow
  2. get_type_hints won't work with forward references/ PEP 563 if ClassVar is not defined in the caller's namespace

Consider the following:

# foo.py
from __future__ import annotations
from typing import *

@runtime_checkable
class X(Protocol):
 x: ClassVar[int] = 1
 y: SomeUndeclaredType = None
# bar.py
from .foo import X
class Y: ...

# Error! get_type_hints cannot resolve 'ClassVar' and 'SomeUndeclaredType'
issubclass(X, Y)

Nonetheless, your suggestion has reminded me of a basic workaround for string annotations. Thanks.

# PEP 544 prohibits using issubclass() with protocols that have non-method members.
return all(callable(getattr(cls, attr, None)) for attr in _get_protocol_attrs(cls))
# bpo-44975: Relaxing that restriction to allow for runtime-checkable
# protocols with class variables since those should be available at class
# definition time.
for attr_name in attr_names:
attr = getattr(cls, attr_name, None)
# Method-like.
if callable(attr):
continue
annotation = annotations.get(attr_name)
# ClassVar member
if getattr(annotation, '__origin__', None) is ClassVar:
instance_attr = getattr(instance, attr_name, None)
# If we couldn't find anything, don't bother checking value types.
if (instance_attr is not None
and attr is not None
and type(instance_attr) != type(attr)):
return False, True
continue
# ClassVar string annotations (forward references).
if isinstance(annotation, str) and annotation.startswith(_CLASSVAR_PREFIXES):
instance_attr = getattr(instance, attr_name, None)
if (instance_attr is not None
and attr is not None
and type(instance_attr) != type(attr)):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is incorrect. Consider this example:

class P(Protocol):
     x: ClassVar[object] = object()

class Y:
     x: ClassVar[object] = 1

Y implements protocol P, but your code rejects it. There are also problems with None here that could produce an incorrect result.

I'd recommend removing this whole check, and just checking that the attribute exists. Full runtime type checking is hard and the standard library shouldn't try to do it.

return False, True
continue
return False, False
return True, False


def _no_init_or_replace_init(self, *args, **kwargs):
Expand Down Expand Up @@ -1477,10 +1510,9 @@ def __instancecheck__(cls, instance):
):
raise TypeError("Instance and class checks can only be used with"
" @runtime_checkable protocols")

if ((not getattr(cls, '_is_protocol', False) or
_is_callable_members_only(cls)) and
issubclass(instance.__class__, cls)):
_is_callable_or_classvar_members_only(cls, instance)[0]) and
issubclass(instance.__class__, cls)):
return True
if cls._is_protocol:
if all(hasattr(instance, attr) and
Expand Down Expand Up @@ -1544,11 +1576,13 @@ def _proto_hook(other):
return NotImplemented
raise TypeError("Instance and class checks can only be used with"
" @runtime_checkable protocols")
if not _is_callable_members_only(cls):
if _allow_reckless_class_checks():
ok_members, classvar_mismatch = _is_callable_or_classvar_members_only(cls, other)
if not ok_members:
if _allow_reckless_class_checks() or classvar_mismatch:
return NotImplemented
raise TypeError("Protocols with non-method members"
" don't support issubclass()")
raise TypeError("Protocol members must be methods or data"
" attributes annotated with ClassVar to support"
" issubclass()")
if not isinstance(other, type):
# Same error message as for issubclass(1, int).
raise TypeError('issubclass() arg 1 must be a class')
Expand Down
2 changes: 2 additions & 0 deletions Misc/NEWS.d/next/Library/2021-08-22-16-58-02.bpo-44975.ub3Vxk.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Runtime protocols with data members now support :func:`issubclass` as long
as those members are annotated with :data:`typing.ClassVar`.