Three inconsistent module attributes #81700

maggyero · 2019-07-07T12:30:52Z

BPO	37519
Nosy	@warsaw, @brettcannon, @ncoghlan, @ericvsmith, @ericsnowcurrently, @maggyero

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-07-07.12:30:52.364>
labels = ['interpreter-core', 'type-bug', 'library', '3.7']
title = 'Three inconsistent module attributes'
updated_at = <Date 2019-07-15.18:59:48.443>
user = 'https://github.com/maggyero'

bugs.python.org fields:

activity = <Date 2019-07-15.18:59:48.443>
actor = 'eric.smith'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core', 'Library (Lib)']
creation = <Date 2019-07-07.12:30:52.364>
creator = 'maggyero'
dependencies = []
files = []
hgrepos = []
issue_num = 37519
keywords = []
message_count = 4.0
messages = ['347470', '347507', '347960', '347988']
nosy_count = 6.0
nosy_names = ['barry', 'brett.cannon', 'ncoghlan', 'eric.smith', 'eric.snow', 'maggyero']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue37519'
versions = ['Python 3.7']

maggyero · 2019-07-07T12:30:51Z

Analysis
========

In the next two sections showing the module attributes and corresponding spec attributes of imported modules and run modules, we notice the following rule (which is in accordance with this PEP-451 section <https://www.python.org/dev/peps/pep-0451/#attributes>_):

If a *spec* attribute is either ``None``, ``'built-in'`` or ``'frozen'``, then the corresponding *module* attribute is not set.

However we also notice three exceptions to this rule, that I think are unintended inconsistencies that should be corrected:

module.__file__ is None (instead of being not set) for imported namespace packages;
module.__cached__ is None (instead of being not set) for non-package modules run from the file system and run from standard input;
module.__package__ is None (instead of being '') for non-package modules run from the file system and run from standard input.

The first exception was introduced recently (26 February 2018) by this pull request <https://github.com/python/cpython/pull/5481>_, which changed the module.__spec__.origin attribute from namespace to None (which I agree with as it avoids conflicting non-namespace-package modules named namespace with namespace packages) and the module.__file__ attribute from being unset to None (which I disagree with as it introduces an inconsistency and contradicts PEP-451).

Environment: CPython 3.7, MacOS 10.14.

Imported modules
================

Running the following code::

    import module

    print("MODULE")

for attr in ["__name__", "__file__", "__cached__", "__path__", "__package__", "__loader__"]:
    print(f"{attr}:", repr(getattr(module, attr, "not set")))

    print("SPEC")

    if hasattr(module, "__spec__"):
        if module.__spec__ is None:
            print("__spec__:", repr(module.__spec__))
        else:
            for attr in ["name", "origin", "cached", "submodule_search_locations", "parent", "loader"]:
                print(f"__spec__.{attr}:", repr(getattr(module.__spec__, attr)))
    else:
        print("__spec__: not set")

where module refers to:

a non-package module (e.g., pathlib);
a regular package (e.g., json);
a namespace package;
a built-in module (e.g., time);
a frozen module (e.g., _frozen_importlib_external)

prints the following module attributes and corresponding spec attributes of the imported module:

for a non-package module:

| MODULE
| __name__: 'pathlib'
| __file__: '/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pathlib.py'
| __cached__: '/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pycache/pathlib.cpython-37.pyc'
| __path__: 'not set'
| __package__: ''
| __loader__: <frozen_importlib_external.SourceFileLoader object at 0x1018896d8>
| SPEC
| __spec_.name: 'pathlib'
| __spec__.origin: '/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pathlib.py'
| __spec__.cached: '/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pycache/pathlib.cpython-37.pyc'
| __spec__.submodule_search_locations: None
| __spec__.parent: ''
| __spec__.loader: <_frozen_importlib_external.SourceFileLoader object at 0x1018896d8>

for a regular package:

| MODULE
| __name__: 'json'
| __file__: '/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/init.py'
| __cached__: '/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/pycache/init.cpython-37.pyc'
| __path__: ['/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json']
| __package__: 'json'
| __loader__: <frozen_importlib_external.SourceFileLoader object at 0x10f9aa6d8>
| SPEC
| __spec_.name: 'json'
| __spec__.origin: '/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/init.py'
| __spec__.cached: '/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/pycache/init.cpython-37.pyc'
| __spec__.submodule_search_locations: ['/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json']
| __spec__.parent: 'json'
| __spec__.loader: <_frozen_importlib_external.SourceFileLoader object at 0x10f9aa6d8>

for a namespace package:

for a built-in module:

for a frozen module:

Run modules
===========

Putting the following code::

    import sys

    print("MODULE")

for attr in ["__name__", "__file__", "__cached__", "__path__", "__package__", "__loader__"]:
    print(f"{attr}:", repr(getattr(sys.modules[__name__], attr, "not set")))

    print("SPEC")

    if hasattr(sys.modules[__name__], "__spec__"):
        if sys.modules[__name__].__spec__ is None:
            print("__spec__:", repr(sys.modules[__name__].__spec__))
        else:
            for attr in ["name", "origin", "cached", "submodule_search_locations", "parent", "loader"]:
                print(f"__spec__.{attr}:", repr(getattr(sys.modules[__name__].__spec__, attr)))
    else:
        print("__spec__: not set")

in:

a module.py file for getting a module non-package module;
a main.py file in a module directory with an init.py file for getting a module regular package;
a main.py file in a module directory without an init.py file for getting a module namespace package

and running the code:

from the file system (python3 module.py for a non-package module, python3 module/ for a package module);
from standard input (cat module.py | python3 for a non-package module);
from the module namespace (python3 -m module)

prints the following module attributes and corresponding spec attributes of the run module:

for a non-package module:

| $ python3 module.py
| MODULE
| __name__: '__main__'
| __file__: 'module.py'
| __cached__: None
| __path__: 'not set'
| __package__: None
| __loader__: <frozen_importlib_external.SourceFileLoader object at 0x1051970b8>
| SPEC
| __spec_: None
|
| $ cat module.py | python3
| MODULE
| __name__: '__main__'
| __file__: '<stdin>'
| __cached__: None
| __path__: 'not set'
| __package__: None
| __loader__: <class '_frozen_importlib.BuiltinImporter'>
| SPEC
| __spec__: None
|
| $ python3 -m module
| MODULE
| __name__: '__main__'
| __file__: '/Users/maggyero/module.py'
| __cached__: '/Users/maggyero/pycache/module.cpython-37.pyc'
| __path__: 'not set'
| __package__: ''
| __loader__: <frozen_importlib_external.SourceFileLoader object at 0x1056b16d8>
| SPEC
| __spec_.name: 'module'
| __spec__.origin: '/Users/maggyero/module.py'
| __spec__.cached: '/Users/maggyero/pycache/module.cpython-37.pyc'
| __spec__.submodule_search_locations: None
| __spec__.parent: ''
| __spec__.loader: <_frozen_importlib_external.SourceFileLoader object at 0x1056b16d8>

for a regular package:

| $ python3 module/
| MODULE
| __name__: '__main__'
| __file__: 'module/main.py'
| __cached__: 'module/pycache/main.cpython-37.pyc'
| __path__: 'not set'
| __package__: ''
| __loader__: <frozen_importlib_external.SourceFileLoader object at 0x10826a550>
| SPEC
| __spec_.name: '__main__'
| __spec__.origin: 'module/main.py'
| __spec__.cached: 'module/pycache/main.cpython-37.pyc'
| __spec__.submodule_search_locations: None
| __spec__.parent: ''
| __spec__.loader: <frozen_importlib_external.SourceFileLoader object at 0x10826a550>
|
| $ python3 -m module
| MODULE
| __name_: '__main__'
| __file__: '/Users/maggyero/module/main.py'
| __cached__: '/Users/maggyero/module/pycache/main.cpython-37.pyc'
| __path__: 'not set'
| __package__: 'module'
| __loader__: <frozen_importlib_external.SourceFileLoader object at 0x10832d278>
| SPEC
| __spec_.name: 'module.__main__'
| __spec__.origin: '/Users/maggyero/module/main.py'
| __spec__.cached: '/Users/maggyero/module/pycache/main.cpython-37.pyc'
| __spec__.submodule_search_locations: None
| __spec__.parent: 'module'
| __spec__.loader: <_frozen_importlib_external.SourceFileLoader object at 0x10832d278>

for a namespace package:

| $ python3 module/
| MODULE
| __name__: '__main__'
| __file__: 'module/main.py'
| __cached__: 'module/pycache/main.cpython-37.pyc'
| __path__: 'not set'
| __package__: ''
| __loader__: <frozen_importlib_external.SourceFileLoader object at 0x107a06518>
| SPEC
| __spec_.name: '__main__'
| __spec__.origin: 'module/main.py'
| __spec__.cached: 'module/pycache/main.cpython-37.pyc'
| __spec__.submodule_search_locations: None
| __spec__.parent: ''
| __spec__.loader: <frozen_importlib_external.SourceFileLoader object at 0x107a06518>
|
| $ python3 -m module
| MODULE
| __name_: '__main__'
| __file__: '/Users/maggyero/module/main.py'
| __cached__: '/Users/maggyero/module/pycache/main.cpython-37.pyc'
| __path__: 'not set'
| __package__: 'module'
| __loader__: <frozen_importlib_external.SourceFileLoader object at 0x10fb69240>
| SPEC
| __spec_.name: 'module.__main__'
| __spec__.origin: '/Users/maggyero/module/main.py'
| __spec__.cached: '/Users/maggyero/module/pycache/main.cpython-37.pyc'
| __spec__.submodule_search_locations: None
| __spec__.parent: 'module'
| __spec__.loader: <_frozen_importlib_external.SourceFileLoader object at 0x10fb69240>

brettcannon · 2019-07-08T19:12:54Z

PEPs actually become historical documents once they are implemented, so could you please check what the official docs say in regards to this to see if there is an inconsistency in the semantics?

maggyero · 2019-07-15T14:09:52Z

@brett Cannon

PEPs actually become historical documents once they are implemented

Actually the inconsistency of the values of the 3 module attributes (__file__, __cached__ and __package__) is with respect to the other values within the current implementation (not only with respect to the values specified in PEP-451). Sorry if I did not explain this well. Let me detail:

For __file__, if you look at the current output of the above "Imported modules" section, you have:

__file__: None, for an imported namespace package;
__file__: 'not set', for an imported built-in module;
__file__: 'not set', for an imported frozen module,

which is inconsistent: it should always be 'not set' when __file__ has no meaning.

For __cached__, if you look at the current output of the above "Run modules" section, you have:

cached: None, for a non-package module run from the file system (python3 module.py) or run from standard input (cat module.py | python3);
path: 'not set', for a non-package module run from the file system (python3 module.py) or run from standard input (cat module.py | python3),

which is inconsistent: it should always be 'not set' when __cached__ has no meaning, like it is already the case for __path__ and other module attributes.

For __package__, if you look at the current output of the above "Run modules" section, you have:

package: None, for a non-package module run from the file system (python3 module.py) or run from standard input (cat module.py | python3);
package: '', for a non-package module run from the module namespace (python3 -m module) or a package run from the file system (python3 module/).

which is inconsistent: it should always be '' when there is no parent package for __package__ to refer to.

ericvsmith · 2019-07-15T18:59:48Z

While some of these might be inconsistent (I haven't really looked at it thoroughly yet), I think it might be problematic to change them at this point, since there's no doubt code out there that depends on the current behavior.

picnixz · 2024-09-07T12:52:49Z

Related: #64019

maggyero mannequin added 3.7 (EOL) end of life interpreter-core (Objects, Python, Grammar, and Parser dirs) stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error labels Jul 7, 2019

ezio-melotti transferred this issue from another repository Apr 10, 2022

serhiy-storchaka added the topic-importlib label May 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Three inconsistent module attributes #81700

Three inconsistent module attributes #81700

maggyero mannequin commented Jul 7, 2019

maggyero mannequin commented Jul 7, 2019

brettcannon commented Jul 8, 2019

maggyero mannequin commented Jul 15, 2019

ericvsmith commented Jul 15, 2019

picnixz commented Sep 7, 2024

Three inconsistent module attributes #81700

Three inconsistent module attributes #81700

Comments

maggyero mannequin commented Jul 7, 2019

maggyero mannequin commented Jul 7, 2019

brettcannon commented Jul 8, 2019

maggyero mannequin commented Jul 15, 2019

ericvsmith commented Jul 15, 2019

picnixz commented Sep 7, 2024