gh-134160: Improve multi-phase init note on isolation & subinterpreters #134775

encukou · 2025-05-27T06:42:06Z

(This isn't strictly related to #134160, but I think it's more useful to group this PR with the other ones there.)

Issue: Docs Examples Focus on Outdated Single-Phase Init Modules and Static Types #134160

📚 Documentation preview 📚: https://cpython-previews--134775.org.readthedocs.build/en/134775/c-api/module.html

pitrou · 2025-05-27T07:36:38Z

Doc/c-api/module.rst

+- :ref:`raising an error on repeated initialization <isolating-extensions-optout>`, or
+- limiting a module to the main interpreter using
+  :c:data:`Py_mod_multiple_interpreters`.


It's a bit weird to present this as a way of supporting sub-interpreters, no?

Yeah. The original wording from PEP 489 would work a bit better:

All modules created using multi-phase initialization are expected to support :ref:`sub-interpreters <sub-interpreter-support>` and multiple :c:func:`Py_Initialize`/:c:func:`Py_Finalize` cycles correctly. Typically, extensions ensure this in one of these ways:

or should we expand the “correctly“ to something like “work as expected or raise exceptions, rather than crash”?

I would definitely not call this "support". So perhaps something like:

Modules created using multi-phase initialization are expected to handle sub-interpreters in either of the following two ways: * either they fully support multiple interpreters by :ref:`isolating module instances <isolating-extensions-howto>`; * or they signal that multiple interpreters are not supported by :ref:`raising an error on repeated initialization <isolating-extensions-optout>` and/or limiting a module to the main interpreter using :c:data:`Py_mod_multiple_interpreters`. This way, users will not encounter crashes or undefined behavior when trying to use these modules from multiple interpreters.

I guess right thing would be to take step back and not make this so much about subinterpreters:

Multi-phase initialization allows creating multiple independent instances of the module object. Extensions are expected to handle this in one of the following ways: * either by :ref:`isolating module instances <isolating-extensions-howto>`; * and/or by :ref:`raising an error on repeated initialization <isolating-extensions-optout>`; * and/or limiting a module to the main interpreter using :c:data:`Py_mod_multiple_interpreters`. This way, users will not encounter crashes or undefined behavior when trying to use these modules from :ref:`sub-interpreters <sub-interpreter-support>`. The first two ways also prevent crashes when reusing the module after Python runtime reinitialization (:c:func:`Py_Finalize` and :c:func:`Py_Initialize`) and reimporting a module after it is removed from :py:attr:`sys.modules`.

Thanks for suggesting the effects on the users! It helps to make this less of a thou shalt commandment and more like something where you can consider the trade-offs.

This makes me wonder: why is there a third way if the second is better (and not more difficult to implement)?

Not that it is critical for this PR, but I've been thinking about something related that might at least a little insight: gh-132861.

@pitrou: Let's say there's subtle nuance that doesn't really belong on this page of the docs :)
It would be good to explain it in the HOWTO, of course.

AFAIK, the main interpreter is special; sometimes, being loadable once but in any interpreter isn't enough.

The extant prior paragraph covers much of this, so here's an alternative:

When using per-module state, module instances should be :ref:`isolated <isolating-extensions-howto>` to ensure that they are free of unwanted side-effects. If the module still uses global state, or it otherwise has not yet been isolated, either or both of of the following two options should be used: * :ref:`raising an error on repeated initialization <isolating-extensions-optout>` * limiting a module to the main interpreter using :c:data:`Py_mod_multiple_interpreters`. This way, users will not encounter crashes or undefined behavior when trying to use these modules from :ref:`sub-interpreters <sub-interpreter-support>`. Ensuring a module is properly isolated or raising an explicit error also prevent crashes when reusing the module after Python runtime reinitialization (:c:func:`Py_Finalize` and :c:func:`Py_Initialize`) or reimporting a module after it has been removed from :py:attr:`sys.modules`.

Doc/c-api/module.rst

ericsnowcurrently · 2025-05-27T14:21:55Z

Doc/c-api/module.rst

+Typically, extensions ensure this in one of these ways:
+
+- :ref:`isolating module instances <isolating-extensions-howto>`,
+- :ref:`raising an error on repeated initialization <isolating-extensions-optout>`, or


FWIW, I just noticed that the referenced section does not mention opting out by setting the Py_mod_multiple_interpreters slot to Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED. (I didn't think to update the docs when I added that value.) Fixing that doesn't need to be part of this PR, but it is relevant to this bullet (and the third bullet).

(Please read this in a friendly voice; I can't find a way to reword this so it's clear and doesn't have a terrible sarcastic interpretation.)

Yeah. I'm not a good person to write that, as I never fully understood why a slot is better than explicitly raising an exception when you're about to clobber shared state.
It would be good to record that in the HOWTO, could you do it?

Doc/c-api/module.rst

AA-Turner · 2025-05-27T23:42:26Z

Doc/c-api/module.rst

@@ -302,8 +302,13 @@ using :c:func:`PyModule_GetState`), or its contents (such as the module's
 :attr:`~object.__dict__` or individual classes created with :c:func:`PyType_FromSpec`).

 All modules created using multi-phase initialization are expected to support
-:ref:`sub-interpreters <sub-interpreter-support>`. Making sure multiple modules
-are independent is typically enough to achieve this.
+:ref:`sub-interpreters <sub-interpreter-support>`.


Suggested change

:ref:`sub-interpreters <sub-interpreter-support>`.

:ref:`sub-interpreters <sub-interpreter-support>`, or otherwise explicitly

signal a lack of support.

AA-Turner · 2025-05-28T00:05:37Z

Doc/c-api/module.rst

+- :ref:`raising an error on repeated initialization <isolating-extensions-optout>`, or
+- limiting a module to the main interpreter using
+  :c:data:`Py_mod_multiple_interpreters`.


The extant prior paragraph covers much of this, so here's an alternative:

When using per-module state, module instances should be :ref:`isolated <isolating-extensions-howto>` to ensure that they are free of unwanted side-effects. If the module still uses global state, or it otherwise has not yet been isolated, either or both of of the following two options should be used: * :ref:`raising an error on repeated initialization <isolating-extensions-optout>` * limiting a module to the main interpreter using :c:data:`Py_mod_multiple_interpreters`. This way, users will not encounter crashes or undefined behavior when trying to use these modules from :ref:`sub-interpreters <sub-interpreter-support>`. Ensuring a module is properly isolated or raising an explicit error also prevent crashes when reusing the module after Python runtime reinitialization (:c:func:`Py_Finalize` and :c:func:`Py_Initialize`) or reimporting a module after it has been removed from :py:attr:`sys.modules`.

encukou · 2025-05-28T07:11:45Z

I integrated this into the existing section. It's a bigger change now, but hopefully for the better :)

@ericsnowcurrently, I'm specifically interested in suggestions to make Py_mod_multiple_interpreters look more important.

AA-Turner

Wordsmithing:

Doc/c-api/module.rst

Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

AA-Turner · 2025-05-28T11:19:19Z

@ericsnowcurrently, I'm specifically interested in suggestions to make Py_mod_multiple_interpreters look more important.

Is this a blocker for merging this, or can we follow up later?

encukou · 2025-05-28T12:33:18Z

@ericsnowcurrently @pitrou, does this look good?
I plan to merge this on Friday if I don't hear back. Shout if you need more review time.

miss-islington-app · 2025-05-30T14:28:19Z

Thanks @encukou for the PR 🌮🎉.. I'm working now to backport this PR to: 3.14.
🐍🍒⛏🤖

…rpreters (pythonGH-134775) (cherry picked from commit eb145fa) Co-authored-by: Petr Viktorin <encukou@gmail.com> Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

bedevere-app · 2025-05-30T14:28:28Z

GH-134932 is a backport of this pull request to the 3.14 branch.

…erpreters (GH-134775) (#134932) gh-134160: Improve multi-phase init note on isolation & subinterpreters (GH-134775) (cherry picked from commit eb145fa) Co-authored-by: Petr Viktorin <encukou@gmail.com> Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

miss-islington-app · 2025-05-31T17:41:14Z

Thanks @encukou for the PR 🌮🎉.. I'm working now to backport this PR to: 3.13.
🐍🍒⛏🤖

…rpreters (pythonGH-134775) (cherry picked from commit eb145fa) Co-authored-by: Petr Viktorin <encukou@gmail.com> Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

bedevere-app · 2025-05-31T17:41:26Z

GH-134983 is a backport of this pull request to the 3.13 branch.

…erpreters (GH-134775) (GH-134983) (cherry picked from commit eb145fa) Co-authored-by: Petr Viktorin <encukou@gmail.com> Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

pythongh-134160: Improve note on multi-phase init & subinterpreters

7f96223

encukou requested a review from ericsnowcurrently as a code owner May 27, 2025 06:42

bedevere-app bot added the awaiting core review label May 27, 2025

bedevere-app bot mentioned this pull request May 27, 2025

Docs Examples Focus on Outdated Single-Phase Init Modules and Static Types #134160

Open

bedevere-app bot added docs Documentation in the Doc dir skip news labels May 27, 2025

github-project-automation bot added this to Docs PRs May 27, 2025

github-project-automation bot moved this to Todo in Docs PRs May 27, 2025

pitrou reviewed May 27, 2025

View reviewed changes

ericsnowcurrently reviewed May 27, 2025

View reviewed changes

AA-Turner reviewed May 28, 2025

View reviewed changes

Rewrite the whole section

3fc9447

AA-Turner approved these changes May 28, 2025

View reviewed changes

bedevere-app bot added awaiting merge and removed awaiting core review labels May 28, 2025

AA-Turner mentioned this pull request May 28, 2025

MAINT: Convert umath_linalg to multi-phase init (PEP 489) numpy/numpy#29030

Merged

encukou commented May 28, 2025

View reviewed changes

Doc/c-api/module.rst Outdated Show resolved Hide resolved

encukou and others added 2 commits May 28, 2025 09:46

Apply suggestions from code review

3b0514d

Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

Apply suggestions from code review

11f8675

Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

encukou changed the title ~~gh-134160: Improve note on multi-phase init & subinterpreters~~ gh-134160: Improve multi-phase init note on isolation & subinterpreters May 28, 2025

encukou merged commit eb145fa into python:main May 30, 2025
24 checks passed

bedevere-app bot removed the awaiting merge label May 30, 2025

github-project-automation bot moved this from Todo to Done in Docs PRs May 30, 2025

encukou deleted the multiphase-init-doc branch May 30, 2025 14:27

encukou added the needs backport to 3.14 bugs and security fixes label May 30, 2025

bedevere-app bot removed the needs backport to 3.14 bugs and security fixes label May 30, 2025

AA-Turner added the needs backport to 3.13 bugs and security fixes label May 31, 2025

bedevere-app bot removed the needs backport to 3.13 bugs and security fixes label May 31, 2025

mostafaammer added this to lavitaconnect@MOSTAFAAMMER May 31, 2025

	:ref:`sub-interpreters <sub-interpreter-support>`.
	:ref:`sub-interpreters <sub-interpreter-support>`, or otherwise explicitly
	signal a lack of support.

Uh oh!

gh-134160: Improve multi-phase init note on isolation & subinterpreters #134775

gh-134160: Improve multi-phase init note on isolation & subinterpreters #134775

Uh oh!

Conversation

encukou commented May 27, 2025 • edited by AA-Turner Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

encukou commented May 28, 2025

Uh oh!

AA-Turner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AA-Turner commented May 28, 2025

Uh oh!

encukou commented May 28, 2025

Uh oh!

Uh oh!

miss-islington-app bot commented May 30, 2025

Uh oh!

bedevere-app bot commented May 30, 2025

Uh oh!

miss-islington-app bot commented May 31, 2025

Uh oh!

bedevere-app bot commented May 31, 2025

Uh oh!

Uh oh!

encukou commented May 27, 2025 •

edited by AA-Turner

Loading