From 7f962237525a7c2f662b836015d00d2965e88bbd Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Tue, 27 May 2025 08:40:25 +0200 Subject: [PATCH 1/4] gh-134160: Improve note on multi-phase init & subinterpreters --- Doc/c-api/module.rst | 9 +++++++-- Doc/howto/isolating-extensions.rst | 4 +++- 2 files changed, 10 insertions(+), 3 deletions(-) diff --git a/Doc/c-api/module.rst b/Doc/c-api/module.rst index f7f4d37d4c721f..5ea3b5c518f529 100644 --- a/Doc/c-api/module.rst +++ b/Doc/c-api/module.rst @@ -302,8 +302,13 @@ using :c:func:`PyModule_GetState`), or its contents (such as the module's :attr:`~object.__dict__` or individual classes created with :c:func:`PyType_FromSpec`). All modules created using multi-phase initialization are expected to support -:ref:`sub-interpreters `. Making sure multiple modules -are independent is typically enough to achieve this. +:ref:`sub-interpreters `. +Typically, extensions ensure this in one of these ways: + +- :ref:`isolating module instances `, +- :ref:`raising an error on repeated initialization `, or +- limiting a module to the main interpreter using + :c:data:`Py_mod_multiple_interpreters`. To request multi-phase initialization, the initialization function (PyInit_modulename) returns a :c:type:`PyModuleDef` instance with non-empty diff --git a/Doc/howto/isolating-extensions.rst b/Doc/howto/isolating-extensions.rst index 5513cd7367519f..b2109b1503992b 100644 --- a/Doc/howto/isolating-extensions.rst +++ b/Doc/howto/isolating-extensions.rst @@ -168,7 +168,7 @@ possible, consider explicit locking. If it is necessary to use process-global state, the simplest way to avoid issues with multiple interpreters is to explicitly prevent a module from being loaded more than once per process—see -`Opt-Out: Limiting to One Module Object per Process`_. +:ref:`isolating-extensions-optout`. Managing Per-Module State @@ -207,6 +207,8 @@ An example of a module with per-module state is currently available as example module initialization shown at the bottom of the file. +.. _isolating-extensions-optout: + Opt-Out: Limiting to One Module Object per Process -------------------------------------------------- From 3fc9447465fe0c543c26231e9aaefa2df1f11c17 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Wed, 28 May 2025 09:07:27 +0200 Subject: [PATCH 2/4] Rewrite the whole section --- Doc/c-api/module.rst | 47 ++++++++++++++++++++++++++++---------------- 1 file changed, 30 insertions(+), 17 deletions(-) diff --git a/Doc/c-api/module.rst b/Doc/c-api/module.rst index 5ea3b5c518f529..a2d4f704e403c4 100644 --- a/Doc/c-api/module.rst +++ b/Doc/c-api/module.rst @@ -288,27 +288,40 @@ An alternate way to specify extensions is to request "multi-phase initialization Extension modules created this way behave more like Python modules: the initialization is split between the *creation phase*, when the module object is created, and the *execution phase*, when it is populated. -The distinction is similar to the :py:meth:`!__new__` and :py:meth:`!__init__` methods -of classes. +The distinction is similar to the :py:meth:`~object.__new__` and +:py:meth:`~object.__init__` methods of classes. Unlike modules created using single-phase initialization, these modules are not -singletons: if the *sys.modules* entry is removed and the module is re-imported, -a new module object is created, and the old module is subject to normal garbage -collection -- as with Python modules. -By default, multiple modules created from the same definition should be -independent: changes to one should not affect the others. -This means that all state should be specific to the module object (using e.g. -using :c:func:`PyModule_GetState`), or its contents (such as the module's -:attr:`~object.__dict__` or individual classes created with :c:func:`PyType_FromSpec`). +singletons. +For example, if the :py:attr:`sys.modules` entry is removed and the module +is re-imported, a new module object is created, and, typically, populated with +fresh method and class objects. +The old module is subject to normal garbage collection. +This mirrors the behavior of pure-Python modules. + +Additional module instances may also be created in +:ref:`sub-interpreters ` +or after after Python runtime reinitialization +(:c:func:`Py_Finalize` and :c:func:`Py_Initialize`). +In these cases, sharing Python objects between module instances would likely +cause crashes or undefined behavior. + +To avoid such issues, multiple modules created from the same definition should +be *isolated*: changes to one should not affect the others, +and all state, including references to Python objects, should be specific to +a particular module instance. +See :ref:`isolating-extensions-howto` for more details and a practical guide. + +A simpler way to avoid the issues is +:ref:`raising an error on repeated initialization `. All modules created using multi-phase initialization are expected to support -:ref:`sub-interpreters `. -Typically, extensions ensure this in one of these ways: - -- :ref:`isolating module instances `, -- :ref:`raising an error on repeated initialization `, or -- limiting a module to the main interpreter using - :c:data:`Py_mod_multiple_interpreters`. +:ref:`sub-interpreters `, or otherwise explicitly +signal a lack of support. +This is usually achieved by isolation or blocking repeated initialization, +as above. +A module may also be limited to the main interpreter using +the :c:data:`Py_mod_multiple_interpreters` slot. To request multi-phase initialization, the initialization function (PyInit_modulename) returns a :c:type:`PyModuleDef` instance with non-empty From 3b0514d5d34bcd012318c3951c8ec24e287751e0 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Wed, 28 May 2025 09:46:38 +0200 Subject: [PATCH 3/4] Apply suggestions from code review Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> --- Doc/c-api/module.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/Doc/c-api/module.rst b/Doc/c-api/module.rst index a2d4f704e403c4..d5957d6a51629d 100644 --- a/Doc/c-api/module.rst +++ b/Doc/c-api/module.rst @@ -294,12 +294,12 @@ The distinction is similar to the :py:meth:`~object.__new__` and Unlike modules created using single-phase initialization, these modules are not singletons. For example, if the :py:attr:`sys.modules` entry is removed and the module -is re-imported, a new module object is created, and, typically, populated with -fresh method and class objects. +is re-imported, a new module object is created, and typically populated with +fresh method and type objects. The old module is subject to normal garbage collection. This mirrors the behavior of pure-Python modules. -Additional module instances may also be created in +Additional module instances may be created in :ref:`sub-interpreters ` or after after Python runtime reinitialization (:c:func:`Py_Finalize` and :c:func:`Py_Initialize`). @@ -312,7 +312,7 @@ and all state, including references to Python objects, should be specific to a particular module instance. See :ref:`isolating-extensions-howto` for more details and a practical guide. -A simpler way to avoid the issues is +A simpler way to avoid these issues is :ref:`raising an error on repeated initialization `. All modules created using multi-phase initialization are expected to support From 11f8675f676ab46dc0831233fc4b7ccb137ec25d Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Wed, 28 May 2025 09:53:11 +0200 Subject: [PATCH 4/4] Apply suggestions from code review Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> --- Doc/c-api/module.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/c-api/module.rst b/Doc/c-api/module.rst index d5957d6a51629d..710135dca89eda 100644 --- a/Doc/c-api/module.rst +++ b/Doc/c-api/module.rst @@ -306,8 +306,8 @@ or after after Python runtime reinitialization In these cases, sharing Python objects between module instances would likely cause crashes or undefined behavior. -To avoid such issues, multiple modules created from the same definition should -be *isolated*: changes to one should not affect the others, +To avoid such issues, each instance of an extension module should +be *isolated*: changes to one instance should not implicitly affect the others, and all state, including references to Python objects, should be specific to a particular module instance. See :ref:`isolating-extensions-howto` for more details and a practical guide.