Skip to content

Add importlib.machinery.NamespaceLoader #11074

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Nov 28, 2023

Conversation

AlexWaygood
Copy link
Member

@AlexWaygood AlexWaygood commented Nov 27, 2023

This was added in Python 3.11; it's documented in https://docs.python.org/3/library/importlib.html. Stubtest didn't alert us to it being missing because it's actually defined in importlib._bootstrap_external at runtime, so stubtest didn't realise it's meant to be publicly re-exported in importlib.machinery.

Comment on lines 171 to 174
# Actually get_resource_reader() returns importlib.readers.NamespaceReader.
# Unfortunately, importing from importlib.readers in importlib.machinery
# appears to break mypy's brain a little (probably due to a massive import cycle)?
def get_resource_reader(self, module: types.ModuleType) -> Incomplete: ...
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Specifically, if I apply this diff to my PR branch:

+    import importlib.readers
+
     class NamespaceLoader(importlib.abc.InspectLoader):
         def __init__(
             self, name: str, path: MutableSequence[str], path_finder: Callable[[str, tuple[str, ...]], ModuleSpec]
@@ -171,7 +173,7 @@ if sys.version_info >= (3, 11):
         # Actually get_resource_reader() returns importlib.readers.NamespaceReader.
         # Unfortunately, importing from importlib.readers in importlib.machinery
         # appears to break mypy's brain a little (probably due to a massive import cycle)?
-        def get_resource_reader(self, module: types.ModuleType) -> Incomplete: ...
+        def get_resource_reader(self, module: types.ModuleType) -> importlib.readers.NamespaceReader: ...
         if sys.version_info < (3, 12):
             @staticmethod
             def module_repr(module: types.ModuleType) -> str: ...

Then mypy starts issuing this (bizarre) complaint:

stdlib\zipfile.pyi:200: error: Variable "zipfile._DateTuple" is not valid as a type  [valid-type]
stdlib\zipfile.pyi:200: note: See https://mypy.readthedocs.io/en/stable/common_issues.html#variables-vs-type-aliases

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After some investigation, it seems like the cause of the false positive is an import cycle between importlib, zipfile, typing_extensions and types:

  • importlib.machinery imports imprtlib.resources.readers
  • importlib.reasources.readers imports zipfile
  • zipfile imports typing_extensions
  • typing_extensions imports types
  • types imports importlib.machinery

The false positive can be avoided if we don't explicitly mark zipfile._DateTuple as a type alias using typing_extensions.TypeAlias. So I've used that workaround instead, in 277ba41.

This comment has been minimized.

This comment has been minimized.

@AlexWaygood
Copy link
Member Author

@AlexWaygood AlexWaygood requested a review from srittau November 28, 2023 18:41

This comment has been minimized.

Copy link
Collaborator

@srittau srittau left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! A few completely optional thoughts below.

if sys.version_info >= (3, 11):
import importlib.readers

class NamespaceLoader(importlib.abc.InspectLoader):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Off-topic, and outside the scope of this PR, but I wonder whether it would make sense to change the classes in importlib.abc to protocols. Although that's probably not possible, because they are also abcs.

It might also make sense to include create_module(), exec_module(), and load_module() here. While they are inherited in the stubs, they aren't in the implementation (since NamespaceLoader inherits directly from object).

Copy link
Member Author

@AlexWaygood AlexWaygood Nov 28, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might also make sense to include create_module(), exec_module(), and load_module() here. While they are inherited in the stubs, they aren't in the implementation (since NamespaceLoader inherits directly from object).

Good shout. Done in a15d3cc!

Copy link
Contributor

According to mypy_primer, this change has no effect on the checked open source code. 🤖🎉

@AlexWaygood AlexWaygood merged commit 9bbfa33 into python:main Nov 28, 2023
@AlexWaygood AlexWaygood deleted the namespaceloader branch November 28, 2023 20:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants