Skip to content

gh-134160: Improve multi-phase init note on isolation & subinterpreters #134775

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
May 30, 2025
Merged

gh-134160: Improve multi-phase init note on isolation & subinterpreters #134775

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
7f96223
gh-134160: Improve note on multi-phase init & subinterpreters
encukou May 27, 2025
3fc9447
Rewrite the whole section
encukou May 28, 2025
3b0514d
Apply suggestions from code review
encukou May 28, 2025
11f8675
Apply suggestions from code review
encukou May 28, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 30 additions & 12 deletions Doc/c-api/module.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -288,22 +288,40 @@ An alternate way to specify extensions is to request "multi-phase initialization
Extension modules created this way behave more like Python modules: the
initialization is split between the *creation phase*, when the module object
is created, and the *execution phase*, when it is populated.
The distinction is similar to the :py:meth:`!__new__` and :py:meth:`!__init__` methods
of classes.
The distinction is similar to the :py:meth:`~object.__new__` and
:py:meth:`~object.__init__` methods of classes.

Unlike modules created using single-phase initialization, these modules are not
singletons: if the *sys.modules* entry is removed and the module is re-imported,
a new module object is created, and the old module is subject to normal garbage
collection -- as with Python modules.
By default, multiple modules created from the same definition should be
independent: changes to one should not affect the others.
This means that all state should be specific to the module object (using e.g.
using :c:func:`PyModule_GetState`), or its contents (such as the module's
:attr:`~object.__dict__` or individual classes created with :c:func:`PyType_FromSpec`).
singletons.
For example, if the :py:attr:`sys.modules` entry is removed and the module
is re-imported, a new module object is created, and typically populated with
fresh method and type objects.
The old module is subject to normal garbage collection.
This mirrors the behavior of pure-Python modules.

Additional module instances may be created in
:ref:`sub-interpreters <sub-interpreter-support>`
or after after Python runtime reinitialization
(:c:func:`Py_Finalize` and :c:func:`Py_Initialize`).
In these cases, sharing Python objects between module instances would likely
cause crashes or undefined behavior.

To avoid such issues, each instance of an extension module should
be *isolated*: changes to one instance should not implicitly affect the others,
and all state, including references to Python objects, should be specific to
a particular module instance.
See :ref:`isolating-extensions-howto` for more details and a practical guide.

A simpler way to avoid these issues is
:ref:`raising an error on repeated initialization <isolating-extensions-optout>`.

All modules created using multi-phase initialization are expected to support
:ref:`sub-interpreters <sub-interpreter-support>`. Making sure multiple modules
are independent is typically enough to achieve this.
:ref:`sub-interpreters <sub-interpreter-support>`, or otherwise explicitly
signal a lack of support.
This is usually achieved by isolation or blocking repeated initialization,
as above.
A module may also be limited to the main interpreter using
the :c:data:`Py_mod_multiple_interpreters` slot.

To request multi-phase initialization, the initialization function
(PyInit_modulename) returns a :c:type:`PyModuleDef` instance with non-empty
Expand Down
4 changes: 3 additions & 1 deletion Doc/howto/isolating-extensions.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,7 +168,7 @@ possible, consider explicit locking.
If it is necessary to use process-global state, the simplest way to
avoid issues with multiple interpreters is to explicitly prevent a
module from being loaded more than once per process—see
`Opt-Out: Limiting to One Module Object per Process`_.
:ref:`isolating-extensions-optout`.


Managing Per-Module State
Expand Down Expand Up @@ -207,6 +207,8 @@ An example of a module with per-module state is currently available as
example module initialization shown at the bottom of the file.


.. _isolating-extensions-optout:

Opt-Out: Limiting to One Module Object per Process
--------------------------------------------------

Expand Down
Loading