From 12c4dbcdbede9955ab86c92b7dbf014a0d468430 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Mon, 7 Mar 2022 18:41:46 -0700 Subject: [PATCH 01/17] Add a PEP for making the GIL per-interpreter. --- pep-06XX-per-interpreter-gil.rst | 642 +++++++++++++++++++++++++++++++ 1 file changed, 642 insertions(+) create mode 100644 pep-06XX-per-interpreter-gil.rst diff --git a/pep-06XX-per-interpreter-gil.rst b/pep-06XX-per-interpreter-gil.rst new file mode 100644 index 00000000000..a78329b4bbe --- /dev/null +++ b/pep-06XX-per-interpreter-gil.rst @@ -0,0 +1,642 @@ +PEP: 6XX +Title: A Per-Interpreter GIL +Author: Eric Snow +Discussions-To: python-dev@python.org +Status: Draft +Type: Standards Track +Content-Type: text/x-rst +Created: 03-08-2022 +Python-Version: 3.11 +Post-History: 03-08-2022 +Resolution: + +.. XXX Split out an informational PEP with all the relevant info, + based on the "Consolidating Runtime Global State" section? + +Abstract +======== + +Since Python 1.5 (1997), CPython users can run multiple interpreters +in the same process. However, they have always shared a significant +amount of global state. This is a source of bugs, with a growing +impact as more and more people use the feature. Furthermore, +sufficient isolation would facilitate true multi-core parallelism, +where interpreters no longer share the GIL. The changes outlined in +this proposal will result in that level of interpreter isolation. + + +High-Level Summary +================== + +As a high level, this proposal changes CPython in the following ways: + +* stops sharing the GIL between interpreters, given sufficient isolation +* adds several new interpreter config options for isolation settings +* adds public C-API for fine-grain control when creating interpreters +* keeps incompatible extensions from causing problems + +The GIL +------- + +The GIL protects concurrent access of most of CPython's runtime state. +So all that GIL-protected global state must move to each interpreter +before the GIL can. + +(In a handful of cases, other mechanisms can be used to ensure +thread-safe sharing instead, such as locks or "immortal" objects.) + +CPython Runtime State +--------------------- + +Properly isolating interpreters requires that most of CPython's +runtime state be stored in the ``PyInterpreterState`` struct. Currently +only a portion of it is. The rest is found either in global variables +or in ``_PyRuntimeState``. Most of that will have to be moved. + +This directly coincides with an ongoing effort of many years to greatly +reduce internal use of C global variables and consolidate the runtime +state into ``_PyRuntimeState`` and ``PyInterpreterState``. +(See `Consolidating Runtime Global State`_ below.) That project has +`significant merit on its own `_ +and has faced little controversy. So, while a per-interpreter GIL +relies on the completion of that effort, that project should not be +considered a part of this proposal, only a dependency. + +Other Isolation Considerations +------------------------------ + +CPython's interpreters must be strictly isolated from each other, with +few exceptions. To a large extent they already are. Each interpreter +has its own copy of all modules, classes, functions, and variables. +The CPython C-API docs `explain further `_. + +.. _caveats: https://docs.python.org/3/c-api/init.html#bugs-and-caveats + +However, aside from what has already been mentioned (e.g. the GIL), +there are a couple of ways in which interpreters still share some state. + +First of all, some process-global resources (e.g. memory, +file descriptors, environment variables) are shared. There are no +plans to change this. + +Second, some isolation is faulty due to bugs or implementations that +did not take multiple interpreters into account. This includes +CPython's runtime and the stdlib, as well as extension modules that +rely on globals variables. In these cases bugs should be opened. +(Some are already.) + +Depending on Immortal Objects +----------------------------- + +:pep:`683` introduces immortal objects as a CPython-internal feature. +With immortal objects we can share any otherwise immutable global +objects between all interpreters. Consequently, this PEP does not +need to address how to deal with the various objects +`exposed in the public C-API `_. +It also simplifies the question of what to do about the builtin +static types. (See `Global Objects`_ below.) + +Both matters have alternate solutions but everything is simpler with +immortal objects. If PEP 683 is not accepted then this one will be +updated with the alternatives. This lets us reduce noise in this +proposal. + + +Motivation +========== + +The fundamental problem we're solving here is a lack of true multi-core +parallelism (for Python code) in the CPython runtime. The GIL is the +cause. While it usually isn't a problem in practice, at the very least +it makes Python's multi-core story murky, which makes the GIL +a consistent distraction. + +Isolated interpreters are also an effective mechanism to support +certain concurrency models. :pep:`554` discusses this in more detail. + +Indirect Benefits +----------------- + +Most of the effort needed for a per-interpreter GIL has benefits that +make those tasks worth doing anyway: + +* makes multiple-interpreter behavior more reliable +* has lead to fixes for long-standing runtime bugs that otherwise + hadn't been prioritized +* has been exposing (and inspiring fixes for) previously unknown runtime bugs +* has driven cleaner runtime initialization (:pep:`432`, :pep:`587`) +* has driven cleaner and more complete runtime finalization +* led to structural layering of the C-API (e.g. ``Include/internal``) +* also see `Benefits to Consolidation`_ below + +Furthermore, much of that work benefits other CPython-related projects: + +* performance improvements ("faster-cpython") +* pre-fork application deployment (e.g. Instagram) +* extension module isolation (see :pep:`630`, etc.) +* embedding CPython + +Existing Use of Multiple Iinterpreters +-------------------------------------- + +The C-API for multiple interpreters has been used for many years. +However, until relatively recently the feature wasn't widely known, +nor extensively used (with the exception of mod_wsgi). + +In the last few years the feature has been gaining traction. Here are +some of the public projects using multiple interpreters currently: + +* `mod_wsgi `_ +* `OpenStack Ceph `_ +* `JEP `_ +* `Kodi `_ + +Note that, with :pep:`554`, multiple interpreter usage would likely +grow significantly (via Python code rather than the C-API). + +PEP 554 +------- + +:pep:`554` is strictly about providing a minimal stdlib module +to give users access to multiple interpreters from Python code. +It specifically avoids the question of a per-interpreter GIL but +users of that module would benefit, making PEP 554 more appealing. + + +Rationale +========= + +During initial investigations in 2014, a variety of possible solutions +for multi-core Python were explored, including: + +* the existing practice of releasing the GIL in extension modules +* other Python implementations (e.g. Jython, IronPython) +* remove the GIL (e.g. gilectomy, "no-gil") +* Trent Nelson's "PyParallel" project +* ``multiprocessing`` +* other parallelism tools (e.g. dask, ray, MPI) +* give up on multi-core (e.g. async, do nothing) + +Each had drawbacks without simple solutions: + +* extensions: doesn't help with Python code +* alt. implementations: CPython dominates the community +* get rid of the GIL: too much technical risk (at the time) +* PyParallel: incomplete; Windows-only at the time +* multiprocessing: too much work to make it effective enough; + high penalties in some situations (at large scale, Windows) +* other tools: not a fit for the stdlib +* give up: this can only end in tears + +Even in 2014 it was fairly clear that a solution using isolated +interpreters did not have a high level of technical risk and that +most of the work was worth doing anyway. +(The downside was the volume of work to be done.) + + +Impact +====== + +Backwards Compatibility +----------------------- + +No behavior or APIs are intended to change due to this proposal, +with one exception noted in `the next section `_. +The existing C-API for managing interpreters will preserve its current +behavior, with new behavior exposed through new API. No other APIs +or runtime behavior is meant to change, including compatibility with +the stable ABI. + +(See `Objects Exposed in the C-API`_ below for related discussion.) + +Extension Modules +''''''''''''''''' + +Currently the most common usage of Python, by far, is with the main +interpreter running by itself. This proposal has zero impact on +extension modules in that scenario. Likewise, for better or worse, +there is no change in behavior under multiple interpreters created +using the existing ``Py_NewInterpreter()``. + +Keep in mind that some extensions already break when used in multiple +interpreters, due to keeping module state in global variables. They +may crash or, worse, experience inconsistent behavior. That was part +of the motivation for :pep:`630` and friends. So this is not a new +situation nor a consequence of this proposal. + +In contrast, when the `proposed API `_ is used to +create multiple interpreters, the default behavior will change for +some extensions. In that case, importing an extension will fail +(outside the main interpreter) if it doesn't indicate support for +multiple interpreters. For extensions that already break in +multiple interpreters, this will be an improvement. + +Now we get to the break in compatibility mentioned above. Some +extensions are safe under multiple interpreters, even though they +haven't indicated that. Unfortunately, there is no reliable way for +the import system to infer that such an extension is safe. So +importing them will still fail. This case is addressed in +`Extension Module Compatibility`_ below. + +Extension Module Maintainers +---------------------------- + +One related consideration is that a per-interpreter GIL will likely +drive increased use of multiple interpreters, particularly if :pep:`554` +is accepted. Some maintainers of large extension modules have expressed +concern about the increased burden they anticipate due to increased +use of multiple interpreters. + +Specifically, enabling support for multiple interpreters will require +substantial work for some extension modules. To add that support, +the maintainer(s) of such a module (often volunteers) would have to +set aside their normal priorities and interests to focus on +compatibility (see :pep:`630`). + +Certainly extension maintainers are free to not add support for use +in multiple interpreters. However, users will increasingly demand +support for multiple interpreters, especially if the feature grows +in popularity. + +Either way the situation can be stressful for maintainers of such +extensions, particularly when they are doing the work in their spare +time. The concerns they have expressed are understandable. We address +the partial solution in `Restricting Extension Modules`_ below. + +Alternate Python Implementations +-------------------------------- + +Other Python implementation are not required to provide support for +multiple interpreters in the same process (though some do already). + +Security Implications +--------------------- + +There is no known impact to security with this proposal. + +Maintainability +--------------- + +On the one hand, this proposal has already motivated a number of +improvements that make CPython *more* maintainable. That is expected +to continue. On the other hand, the underlying work has already +exposed various pre-existing defects in the runtime that have had +to be fixed. That is also expected continue as multiple interpreters +receive more use. Otherwise there shouldn't be a significant impact +on maintainability. The effect should be net positive. + +Performance +----------- + +The work to consolidate globals has already provided a number of +improvements to CPython's performance, both speeding it up and using +less memory. This should continue. Performance benefits to a +per-interpreter GIL have not been explored. At the least it is +not expected to make CPython slower (as long as interpreters are +sufficiently isolated). + + +Specification +============= + +As summarized `above `_, this proposal involves the +following changes, in the order they must happen: + +1. `consolidate global runtime state `_ + (including objects) into ``_PyRuntimeState`` +2. move nearly all of the state down into ``PyInterpreterState`` +3. finally, move the GIL down into ``PyInterpreterState`` +4. everything else + + * add to the public C-API + * implement restrictions in ``ExtensionFileLoader`` + * work with popular extension maintainers to help with multi-interpreter support + +Per-Interpreter State +--------------------- + +The following runtime state will be moved to ``PyInterpreterState``: + +* all global objects that are not safely shareable (fully immutable) +* the GIL +* mutable, currently protected by the GIL +* mutable, currently protected by some other per-interpreter lock +* mutable, may be used independently in different interpreters +* all other mutable (or effectively mutable) state not otherwise excluded below + +The following state will not be moved: + +* global objects that are safely shareable, if any +* immutable, often ``const`` +* treated as immutable +* related to CPython's ``main()`` execution +* related to the REPL +* set during runtime init, then treated as immutable +* mutable, protected by some global lock +* mutable, atomic + +Also, a number of parts of the global state were already moved to the +interpreter, such as GC, warnings, and atexit hooks. + +Memory Allocators +----------------- + + + +.. _proposed capi: + +C-API +----- + +The following private API will be made public: + +* ``_PyInterpreterConfig`` +* ``_Py_NewInterpreter()`` (as ``Py_NewInterpreterEx()``) + +The following fields will be added to ``PyInterpreterConfig``: + +* ``own_gil`` - (bool) create a new interpreter lock + (instead of sharing with the main interpreter) +* ``strict_extensions`` - fail import in this interpreter for + incompatible extensions (see `Restricting Extension Modules`_) + +Restricting Extension Modules +----------------------------- + +Extension modules have many of the same problems as the runtime when +state is stored in global variables. :pep:`630` covers all the details +of what extensions must do to support isolation, and thus safely run in +multiple interpreters at once. This includes dealing with their globals. + +Extension modules that do not implement isolation will only run in +the main interpreter. In all other interpreters the import will +raise ``ImportError``. This will be done through +``importlib._bootstrap_external.ExtensionFileLoader``. + +We will work with popular extensions to help them support use in +multiple interpreters. This may involve adding new public APIs, which +we will address on a case-by-case basis. + +Extension Module Compatibility +'''''''''''''''''''''''''''''' + +An extension may work fine in multiple interpreters even if it doesn't +explicitly indicate support. Importing such modules will fail outside +the main interpreter. + + + +.. XXX What to do about it? + +Documentation +------------- + +The "Sub-interpreter support" section of Doc/c-api/init.rst will be +updated with the added API. + + +How to Teach This +================= + +This is an advanced feature for users of the C-API. There is no +expectation that this will be taught. + +That said, if it were taught then it would boil down to the following:: + + In addition to Py_NewInterpreter(), you can use Py_NewInterpreterEx() + to create an interpreter. The config you pass it indicates how you + want that interpreter to behave. + + +Extra Context +============= + +Sharing Global Objects +---------------------- + +We are sharing some global objects between interpreters. +This is an implementation detail and relates more to +`globals consolidation `_ +than to this proposal, but it is a significant enough detail +to explain here. + +The alternative is to share no objects between interpreters ever. +To accomplish that we'd have to sort out the fate of all our static +types, as well as deal with compatibility issues for the many objects +`exposed in the public C-API `_. + +That approach introduces a meaningful amount of extra complexity +and higher risk, though prototyping has demonstrated valid solutions. +Also, it would likely result in a performance penalty. + +`Immortal objects `_ allow us to +share the otherwise immutable global objects. That way we avoid +the extra costs. + +.. _capi objects: + +Objects Exposed in the C-API +'''''''''''''''''''''''''''' + +The C-API (including the limited API) exposes all the builtin types, +including the builtin exceptions, as well as the builtin singletons. +The exceptions are exposed as ``PyObject *`` but the rest are exposed +as the static values rather than pointers. This was one of the few +non-trivial problems we had to solve for per-interpreter GIL. + +With immortal objects this is a non-issue. + + +Consolidating Runtime Global State +================================== + +As noted in `CPython Runtime State`_ above, there is an active effort +(separate from this PEP) to consolidate CPython's global state into the +``_PyRuntimeState`` struct. Nearly all the work involves moving that +state from global variables. The project is particularly relevant to +this proposal, so below is some extra detail. + +Benefits to Consolidation +------------------------- + + + +Most of the effort needed for a per-interpreter GIL has benefits that +make those tasks worth doing anyway: + +* greatly reduces the number of C globals (best practice for C code) +* draws attention to runtime state that is unstable +* encourages more consistency in how runtime state is used +* makes multiple-interpreter behavior more reliable +* leads to fixes for long-standing runtime bugs that otherwise + haven't been prioritized +* exposes (and inspires fixes for) previously unknown runtime bugs +* facilitates cleaner runtime initialization and finalization +* makes it easier to discover/identify CPython's runtime state +* makes it easier to statically allocate runtime state in a consistent way +* increases consistency between objects created in C vs. in Python code +* better memory locality for runtime state +* structural layering of the C-API (e.g. ``Include/internal``) + +Furthermore, much of that work benefits other CPython-related projects: + +* performance improvements ("faster-cpython") +* pre-fork application deployment (e.g. Instagram) +* extension module isolation (see :pep:`630`, etc.) +* embedding CPython + +Scale of Work +------------- + +The number of global variables to be moved is large enough to matter, +but most are Python objects that can be dealt with in large groups +(like ``Py_IDENTIFIER``). In nearly all cases, moving these globals +to the interpreter is highly mechanical. That doesn't require +cleverness but instead requires someone to put in the time. + +State To Be Moved +----------------- + +The remaining global variables can be categorized as follows:: + + global objects + static types (incl. exception types) + non-static types (incl. heap types, structseq types) + singletons (static) + singletons (initialized once) + cached objects + non-objects + will not (or unlikely to) change after init + only used in the main thread + initialized lazily + pre-allocated buffers + state + +Those globals are spread between the core runtime, the builtin modules, +and the stdlib extension modules. + +For a breakdown of the remaining globals, run:: + + ./python Tools/c-analyzer/table-file.py Tools/c-analyzer/cpython/globals-to-fix.tsv + +Already Completed Work +---------------------- + +As mentioned, this work has been going on for many years. Here are some +of the things that have already been done. + +* cleanup of runtime initialization (see :pep:`432` / :pep:`587`) +* extension module isolation machinery (see :pep:`384` / :pep:`3121` / :pep:`489`) +* isolation for many builtin modules +* isolation for many stdlib extension modules +* addition of ``_PyRuntimeState`` +* no more ``_Py_IDENTIFIER()`` +* statically allocated: + + * empty string + * string literals + * identifiers + * latin-1 strings + * length-1 bytes + * empty tuple + +Tooling +------- + +As already indicated, there are several tools to help identify the +globals and reason about them. + +* Tools/c-analyzer/cpython/globals-to-fix.tsv - the list of remaining globals +* Tools/c-analyzer/c-analyzer.py + + + analyze - identify all the globals + + check - fail if there are any unsupported globals that aren't ignored + +* Tools/c-analyzer/table-file.py - summarize the known globals + +As well, the check for unsupported globals is incorporated into CI so that +not new globals are accidentally added. + +Global Objects +-------------- + +Global objects that are safe to be shared (without a GIL) between +interpreters can stay on ``_PyRuntimeState``. Not only must the object +be effectively immutable (e.g. singletons, strings), but not even the +refcount can change for it to be safe. Immortality (:pep:`683`) +provides that. + +Builtin static types are a special case. They are effectively immutable +except for one part: ``__subclasses__`` (AKA ``tp_subclasses``). We +expect that nothing else on a builtin type will change, even the content +of ``__dict__`` (AKA ``tp_dict``). + +``__subclasses__`` for the builtin types will be dealt with by making +it a getter that does a lookup on the current ``PyInterpreterState`` +for that type. + + +Reference Implementation +======================== + + + + +Open Issues +=========== + +Implementation details: + +* explain sharing some global objects vs. sharing none of them? +* mention sharing the memory allocators? + +Other: + +* add a helper to allow importing incompatible extensions? + + +Deferred Functionality +====================== + +* ``PyInterpreterConfig`` option to always run the interpreter in a new thread +* ``PyInterpreterConfig`` option to assign a "main" thread to the interpreter + and only run in that thread + + +Rejected Ideas +============== + + + + +References +========== + +Related: + +* :pep:`384` +* :pep:`432` +* :pep:`489` +* :pep:`554` +* :pep:`573` +* :pep:`587` +* :pep:`630` +* :pep:`683` +* :pep:`3121` + + +Copyright +========= + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive. + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: From f8f32ffcd448c0ffcf51e751d52b499dfb0ebb15 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Mon, 7 Mar 2022 19:10:01 -0700 Subject: [PATCH 02/17] Fix dates. --- pep-06XX-per-interpreter-gil.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pep-06XX-per-interpreter-gil.rst b/pep-06XX-per-interpreter-gil.rst index a78329b4bbe..b98e702fd15 100644 --- a/pep-06XX-per-interpreter-gil.rst +++ b/pep-06XX-per-interpreter-gil.rst @@ -5,9 +5,9 @@ Discussions-To: python-dev@python.org Status: Draft Type: Standards Track Content-Type: text/x-rst -Created: 03-08-2022 +Created: 08-Mar-2022 Python-Version: 3.11 -Post-History: 03-08-2022 +Post-History: 08-Mar-2022 Resolution: .. XXX Split out an informational PEP with all the relevant info, From 5d7131bfe623a68ab5811b7386506fe7df0837d9 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Mon, 7 Mar 2022 19:10:23 -0700 Subject: [PATCH 03/17] Set the PEP number. --- pep-06XX-per-interpreter-gil.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pep-06XX-per-interpreter-gil.rst b/pep-06XX-per-interpreter-gil.rst index b98e702fd15..685b3165056 100644 --- a/pep-06XX-per-interpreter-gil.rst +++ b/pep-06XX-per-interpreter-gil.rst @@ -1,4 +1,4 @@ -PEP: 6XX +PEP: 684 Title: A Per-Interpreter GIL Author: Eric Snow Discussions-To: python-dev@python.org From 324c9d7d11bc2b7a948afab1a18a8c6e0f44607e Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Mon, 7 Mar 2022 19:11:23 -0700 Subject: [PATCH 04/17] Change the filenme. --- ...XX-per-interpreter-gil.rst => pep-0684-per-interpreter-gil.rst | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename pep-06XX-per-interpreter-gil.rst => pep-0684-per-interpreter-gil.rst (100%) diff --git a/pep-06XX-per-interpreter-gil.rst b/pep-0684-per-interpreter-gil.rst similarity index 100% rename from pep-06XX-per-interpreter-gil.rst rename to pep-0684-per-interpreter-gil.rst From 40f7da77562e6f98707dc542c53e5eeb1cc233b1 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 8 Mar 2022 11:05:19 -0700 Subject: [PATCH 05/17] Fix typos. --- pep-0684-per-interpreter-gil.rst | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/pep-0684-per-interpreter-gil.rst b/pep-0684-per-interpreter-gil.rst index 685b3165056..794f11807a1 100644 --- a/pep-0684-per-interpreter-gil.rst +++ b/pep-0684-per-interpreter-gil.rst @@ -32,7 +32,7 @@ As a high level, this proposal changes CPython in the following ways: * stops sharing the GIL between interpreters, given sufficient isolation * adds several new interpreter config options for isolation settings -* adds public C-API for fine-grain control when creating interpreters +* adds public C-API for fine-grained control when creating interpreters * keeps incompatible extensions from causing problems The GIL @@ -82,7 +82,7 @@ plans to change this. Second, some isolation is faulty due to bugs or implementations that did not take multiple interpreters into account. This includes CPython's runtime and the stdlib, as well as extension modules that -rely on globals variables. In these cases bugs should be opened. +rely on global variables. In these cases bugs should be opened. (Some are already.) Depending on Immortal Objects @@ -121,7 +121,7 @@ Most of the effort needed for a per-interpreter GIL has benefits that make those tasks worth doing anyway: * makes multiple-interpreter behavior more reliable -* has lead to fixes for long-standing runtime bugs that otherwise +* has led to fixes for long-standing runtime bugs that otherwise hadn't been prioritized * has been exposing (and inspiring fixes for) previously unknown runtime bugs * has driven cleaner runtime initialization (:pep:`432`, :pep:`587`) @@ -136,8 +136,8 @@ Furthermore, much of that work benefits other CPython-related projects: * extension module isolation (see :pep:`630`, etc.) * embedding CPython -Existing Use of Multiple Iinterpreters --------------------------------------- +Existing Use of Multiple Interpreters +------------------------------------- The C-API for multiple interpreters has been used for many years. However, until relatively recently the feature wasn't widely known, @@ -203,7 +203,7 @@ Backwards Compatibility No behavior or APIs are intended to change due to this proposal, with one exception noted in `the next section `_. The existing C-API for managing interpreters will preserve its current -behavior, with new behavior exposed through new API. No other APIs +behavior, with new behavior exposed through new API. No other API or runtime behavior is meant to change, including compatibility with the stable ABI. @@ -281,7 +281,7 @@ On the one hand, this proposal has already motivated a number of improvements that make CPython *more* maintainable. That is expected to continue. On the other hand, the underlying work has already exposed various pre-existing defects in the runtime that have had -to be fixed. That is also expected continue as multiple interpreters +to be fixed. That is also expected to continue as multiple interpreters receive more use. Otherwise there shouldn't be a significant impact on maintainability. The effect should be net positive. @@ -555,7 +555,7 @@ globals and reason about them. * Tools/c-analyzer/table-file.py - summarize the known globals As well, the check for unsupported globals is incorporated into CI so that -not new globals are accidentally added. +no new globals are accidentally added. Global Objects -------------- From 9291a8ae0154792ea234269afc0dc42463137e07 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 8 Mar 2022 11:07:03 -0700 Subject: [PATCH 06/17] Fix the filename. --- pep-0684-per-interpreter-gil.rst => pep-0684.rst | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename pep-0684-per-interpreter-gil.rst => pep-0684.rst (100%) diff --git a/pep-0684-per-interpreter-gil.rst b/pep-0684.rst similarity index 100% rename from pep-0684-per-interpreter-gil.rst rename to pep-0684.rst From ed4e05d18fe988f9075132128e8e8c44522002ee Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 8 Mar 2022 13:30:47 -0700 Subject: [PATCH 07/17] Add a note about the allocators. --- pep-0684.rst | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/pep-0684.rst b/pep-0684.rst index 794f11807a1..f687903def3 100644 --- a/pep-0684.rst +++ b/pep-0684.rst @@ -338,10 +338,13 @@ The following state will not be moved: Also, a number of parts of the global state were already moved to the interpreter, such as GC, warnings, and atexit hooks. -Memory Allocators ------------------ - - +Note that currently the allocators (see Objects/obmalloc.c) are shared +between all interpreters, protected by the GIL. They will need to move +to each interpreter (or a global lock will be needed). This is the +highest risk part of the work to isolate interpreters and may require +more than just moving fields down from ``_PyRuntimeState``. Some of +the complexity is reduced if CPython switches to a thread-safe +allocator like mimalloc. .. _proposed capi: @@ -585,14 +588,12 @@ Reference Implementation Open Issues =========== +* what are the risks/hurdles involved with moving the allocators? +* add a helper to allow importing incompatible extensions? + Implementation details: * explain sharing some global objects vs. sharing none of them? -* mention sharing the memory allocators? - -Other: - -* add a helper to allow importing incompatible extensions? Deferred Functionality From ab72d4f7e16d695ea52ff8499ac5bb6bb90f7885 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 8 Mar 2022 13:57:34 -0700 Subject: [PATCH 08/17] Add importlib.util.allow_all_extensions(). --- pep-0684.rst | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/pep-0684.rst b/pep-0684.rst index f687903def3..6953d4ff31c 100644 --- a/pep-0684.rst +++ b/pep-0684.rst @@ -383,13 +383,15 @@ we will address on a case-by-case basis. Extension Module Compatibility '''''''''''''''''''''''''''''' -An extension may work fine in multiple interpreters even if it doesn't -explicitly indicate support. Importing such modules will fail outside -the main interpreter. +As noted in `Extension Modules`_, many extensions work fine in multiple +interpreters without needing any changes. The import system will still +fail if such a module doesn't explicitly indicate support. At first, +not as many extension modules will do so, so this is a potential source +of frustration. - - -.. XXX What to do about it? +We will address this by adding a context manager to temporarily disable +the check on multiple interpreter support: +``importlib.util.allow_all_extensions()``. Documentation ------------- @@ -589,7 +591,7 @@ Open Issues =========== * what are the risks/hurdles involved with moving the allocators? -* add a helper to allow importing incompatible extensions? +* is "allow_all_extensions" the best name for the context manager? Implementation details: From 306bf96b7117735ab3f541e8b73540675358e206 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 8 Mar 2022 14:00:32 -0700 Subject: [PATCH 09/17] Add a note about the possibility of not sharing any objects at all. --- pep-0684.rst | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/pep-0684.rst b/pep-0684.rst index 6953d4ff31c..0a695cc3a75 100644 --- a/pep-0684.rst +++ b/pep-0684.rst @@ -569,11 +569,14 @@ Global objects that are safe to be shared (without a GIL) between interpreters can stay on ``_PyRuntimeState``. Not only must the object be effectively immutable (e.g. singletons, strings), but not even the refcount can change for it to be safe. Immortality (:pep:`683`) -provides that. - -Builtin static types are a special case. They are effectively immutable -except for one part: ``__subclasses__`` (AKA ``tp_subclasses``). We -expect that nothing else on a builtin type will change, even the content +provides that. (The alternative is that no objects are shared, which +adds significant complexity to the solution, particularly for the +objects `exposed in the public C-API `_.) + +Builtin static types are a special case of global objects that will be +shared. They are effectively immutable except for one part: +``__subclasses__`` (AKA ``tp_subclasses``). We expect that nothing +else on a builtin type will change, even the content of ``__dict__`` (AKA ``tp_dict``). ``__subclasses__`` for the builtin types will be dealt with by making @@ -593,10 +596,6 @@ Open Issues * what are the risks/hurdles involved with moving the allocators? * is "allow_all_extensions" the best name for the context manager? -Implementation details: - -* explain sharing some global objects vs. sharing none of them? - Deferred Functionality ====================== From 4a67da77b7ea96dd7c75fddbd769607f7c7c78a4 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 8 Mar 2022 14:15:08 -0700 Subject: [PATCH 10/17] Clean up "Benefits to Consolidation". --- pep-0684.rst | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/pep-0684.rst b/pep-0684.rst index 0a695cc3a75..cdb74b3a002 100644 --- a/pep-0684.rst +++ b/pep-0684.rst @@ -464,13 +464,10 @@ this proposal, so below is some extra detail. Benefits to Consolidation ------------------------- - - -Most of the effort needed for a per-interpreter GIL has benefits that -make those tasks worth doing anyway: +Consolidating the globals has a variety of benefits: * greatly reduces the number of C globals (best practice for C code) -* draws attention to runtime state that is unstable +* the move draws attention to runtime state that is unstable or broken * encourages more consistency in how runtime state is used * makes multiple-interpreter behavior more reliable * leads to fixes for long-standing runtime bugs that otherwise @@ -479,7 +476,6 @@ make those tasks worth doing anyway: * facilitates cleaner runtime initialization and finalization * makes it easier to discover/identify CPython's runtime state * makes it easier to statically allocate runtime state in a consistent way -* increases consistency between objects created in C vs. in Python code * better memory locality for runtime state * structural layering of the C-API (e.g. ``Include/internal``) From 7c10d47c9f164d05e3a02768c5a38de7a7ef03bf Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 8 Mar 2022 19:14:34 -0700 Subject: [PATCH 11/17] typos and formatting Co-authored-by: CAM Gerlach --- pep-0684.rst | 52 ++++++++++++++++++++++------------------------------ 1 file changed, 22 insertions(+), 30 deletions(-) diff --git a/pep-0684.rst b/pep-0684.rst index cdb74b3a002..c1516543e75 100644 --- a/pep-0684.rst +++ b/pep-0684.rst @@ -28,17 +28,17 @@ this proposal will result in that level of interpreter isolation. High-Level Summary ================== -As a high level, this proposal changes CPython in the following ways: +At a high level, this proposal changes CPython in the following ways: * stops sharing the GIL between interpreters, given sufficient isolation * adds several new interpreter config options for isolation settings -* adds public C-API for fine-grained control when creating interpreters +* adds some public C-API for fine-grained control when creating interpreters * keeps incompatible extensions from causing problems The GIL ------- -The GIL protects concurrent access of most of CPython's runtime state. +The GIL protects concurrent access to most of CPython's runtime state. So all that GIL-protected global state must move to each interpreter before the GIL can. @@ -49,8 +49,8 @@ CPython Runtime State --------------------- Properly isolating interpreters requires that most of CPython's -runtime state be stored in the ``PyInterpreterState`` struct. Currently -only a portion of it is. The rest is found either in global variables +runtime state be stored in the ``PyInterpreterState`` struct. Currently, +only a portion of it is; the rest is found either in global variables or in ``_PyRuntimeState``. Most of that will have to be moved. This directly coincides with an ongoing effort of many years to greatly @@ -221,7 +221,7 @@ using the existing ``Py_NewInterpreter()``. Keep in mind that some extensions already break when used in multiple interpreters, due to keeping module state in global variables. They may crash or, worse, experience inconsistent behavior. That was part -of the motivation for :pep:`630` and friends. So this is not a new +of the motivation for :pep:`630` and friends, so this is not a new situation nor a consequence of this proposal. In contrast, when the `proposed API `_ is used to @@ -234,7 +234,7 @@ multiple interpreters, this will be an improvement. Now we get to the break in compatibility mentioned above. Some extensions are safe under multiple interpreters, even though they haven't indicated that. Unfortunately, there is no reliable way for -the import system to infer that such an extension is safe. So +the import system to infer that such an extension is safe, so importing them will still fail. This case is addressed in `Extension Module Compatibility`_ below. @@ -258,7 +258,7 @@ in multiple interpreters. However, users will increasingly demand support for multiple interpreters, especially if the feature grows in popularity. -Either way the situation can be stressful for maintainers of such +Either way, the situation can be stressful for maintainers of such extensions, particularly when they are doing the work in their spare time. The concerns they have expressed are understandable. We address the partial solution in `Restricting Extension Modules`_ below. @@ -290,8 +290,8 @@ Performance The work to consolidate globals has already provided a number of improvements to CPython's performance, both speeding it up and using -less memory. This should continue. Performance benefits to a -per-interpreter GIL have not been explored. At the least it is +less memory, and this should continue. Performance benefits to a +per-interpreter GIL have not been explored. At the very least, it is not expected to make CPython slower (as long as interpreters are sufficiently isolated). @@ -299,7 +299,7 @@ sufficiently isolated). Specification ============= -As summarized `above `_, this proposal involves the +As `summarized above `__, this proposal involves the following changes, in the order they must happen: 1. `consolidate global runtime state `_ @@ -372,7 +372,7 @@ of what extensions must do to support isolation, and thus safely run in multiple interpreters at once. This includes dealing with their globals. Extension modules that do not implement isolation will only run in -the main interpreter. In all other interpreters the import will +the main interpreter. In all other interpreters, the import will raise ``ImportError``. This will be done through ``importlib._bootstrap_external.ExtensionFileLoader``. @@ -396,7 +396,7 @@ the check on multiple interpreter support: Documentation ------------- -The "Sub-interpreter support" section of Doc/c-api/init.rst will be +The "Sub-interpreter support" section of ``Doc/c-api/init.rst`` will be updated with the added API. @@ -406,7 +406,7 @@ How to Teach This This is an advanced feature for users of the C-API. There is no expectation that this will be taught. -That said, if it were taught then it would boil down to the following:: +That said, if it were taught then it would boil down to the following: In addition to Py_NewInterpreter(), you can use Py_NewInterpreterEx() to create an interpreter. The config you pass it indicates how you @@ -425,8 +425,8 @@ This is an implementation detail and relates more to than to this proposal, but it is a significant enough detail to explain here. -The alternative is to share no objects between interpreters ever. -To accomplish that we'd have to sort out the fate of all our static +The alternative is to share no objects between interpreters, ever. +To accomplish that, we'd have to sort out the fate of all our static types, as well as deal with compatibility issues for the many objects `exposed in the public C-API `_. @@ -516,7 +516,9 @@ The remaining global variables can be categorized as follows:: Those globals are spread between the core runtime, the builtin modules, and the stdlib extension modules. -For a breakdown of the remaining globals, run:: +For a breakdown of the remaining globals, run: + +.. code-block:: bash ./python Tools/c-analyzer/table-file.py Tools/c-analyzer/cpython/globals-to-fix.tsv @@ -524,7 +526,7 @@ Already Completed Work ---------------------- As mentioned, this work has been going on for many years. Here are some -of the things that have already been done. +of the things that have already been done: * cleanup of runtime initialization (see :pep:`432` / :pep:`587`) * extension module isolation machinery (see :pep:`384` / :pep:`3121` / :pep:`489`) @@ -589,8 +591,8 @@ Reference Implementation Open Issues =========== -* what are the risks/hurdles involved with moving the allocators? -* is "allow_all_extensions" the best name for the context manager? +* What are the risks/hurdles involved with moving the allocators? +* Is ``allow_all_extensions`` the best name for the context manager? Deferred Functionality @@ -628,13 +630,3 @@ Copyright This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. - - -.. - Local Variables: - mode: indented-text - indent-tabs-mode: nil - sentence-end-double-space: t - fill-column: 70 - coding: utf-8 - End: From 4e820219ee0a8a8725e1d3d8a55e371a0a237701 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 8 Mar 2022 19:58:09 -0700 Subject: [PATCH 12/17] More formatting and wording. Co-authored-by: CAM Gerlach --- pep-0684.rst | 71 +++++++++++++++++++++++++--------------------------- 1 file changed, 34 insertions(+), 37 deletions(-) diff --git a/pep-0684.rst b/pep-0684.rst index c1516543e75..da86625b8f1 100644 --- a/pep-0684.rst +++ b/pep-0684.rst @@ -17,7 +17,8 @@ Abstract ======== Since Python 1.5 (1997), CPython users can run multiple interpreters -in the same process. However, they have always shared a significant +in the same process. However, interpreters in the same process +have always shared a significant amount of global state. This is a source of bugs, with a growing impact as more and more people use the feature. Furthermore, sufficient isolation would facilitate true multi-core parallelism, @@ -53,14 +54,14 @@ runtime state be stored in the ``PyInterpreterState`` struct. Currently, only a portion of it is; the rest is found either in global variables or in ``_PyRuntimeState``. Most of that will have to be moved. -This directly coincides with an ongoing effort of many years to greatly +This directly coincides with an ongoing effort (of many years) to greatly reduce internal use of C global variables and consolidate the runtime state into ``_PyRuntimeState`` and ``PyInterpreterState``. (See `Consolidating Runtime Global State`_ below.) That project has `significant merit on its own `_ and has faced little controversy. So, while a per-interpreter GIL relies on the completion of that effort, that project should not be -considered a part of this proposal, only a dependency. +considered a part of this proposal--only a dependency. Other Isolation Considerations ------------------------------ @@ -82,14 +83,14 @@ plans to change this. Second, some isolation is faulty due to bugs or implementations that did not take multiple interpreters into account. This includes CPython's runtime and the stdlib, as well as extension modules that -rely on global variables. In these cases bugs should be opened. -(Some are already.) +rely on global variables. Bugs should be opened in these cases, +as some already have been. Depending on Immortal Objects ----------------------------- :pep:`683` introduces immortal objects as a CPython-internal feature. -With immortal objects we can share any otherwise immutable global +With immortal objects, we can share any otherwise immutable global objects between all interpreters. Consequently, this PEP does not need to address how to deal with the various objects `exposed in the public C-API `_. @@ -167,28 +168,26 @@ Rationale ========= During initial investigations in 2014, a variety of possible solutions -for multi-core Python were explored, including: +for multi-core Python were explored, but each had its drawbacks +without simple solutions: * the existing practice of releasing the GIL in extension modules + * doesn't help with Python code * other Python implementations (e.g. Jython, IronPython) + * CPython dominates the community * remove the GIL (e.g. gilectomy, "no-gil") + * too much technical risk (at the time) * Trent Nelson's "PyParallel" project + * incomplete; Windows-only at the time * ``multiprocessing`` + * too much work to make it effective enough; + high penalties in some situations (at large scale, Windows) * other parallelism tools (e.g. dask, ray, MPI) + * not a fit for the stdlib * give up on multi-core (e.g. async, do nothing) + * this can only end in tears -Each had drawbacks without simple solutions: - -* extensions: doesn't help with Python code -* alt. implementations: CPython dominates the community -* get rid of the GIL: too much technical risk (at the time) -* PyParallel: incomplete; Windows-only at the time -* multiprocessing: too much work to make it effective enough; - high penalties in some situations (at large scale, Windows) -* other tools: not a fit for the stdlib -* give up: this can only end in tears - -Even in 2014 it was fairly clear that a solution using isolated +Even in 2014, it was fairly clear that a solution using isolated interpreters did not have a high level of technical risk and that most of the work was worth doing anyway. (The downside was the volume of work to be done.) @@ -207,7 +206,7 @@ behavior, with new behavior exposed through new API. No other API or runtime behavior is meant to change, including compatibility with the stable ABI. -(See `Objects Exposed in the C-API`_ below for related discussion.) +See `Objects Exposed in the C-API`_ below for related discussion. Extension Modules ''''''''''''''''' @@ -253,14 +252,14 @@ the maintainer(s) of such a module (often volunteers) would have to set aside their normal priorities and interests to focus on compatibility (see :pep:`630`). -Certainly extension maintainers are free to not add support for use +Of course, extension maintainers are free to not add support for use in multiple interpreters. However, users will increasingly demand -support for multiple interpreters, especially if the feature grows +such support, especially if the feature grows in popularity. Either way, the situation can be stressful for maintainers of such extensions, particularly when they are doing the work in their spare -time. The concerns they have expressed are understandable. We address +time. The concerns they have expressed are understandable, and we address the partial solution in `Restricting Extension Modules`_ below. Alternate Python Implementations @@ -282,8 +281,8 @@ improvements that make CPython *more* maintainable. That is expected to continue. On the other hand, the underlying work has already exposed various pre-existing defects in the runtime that have had to be fixed. That is also expected to continue as multiple interpreters -receive more use. Otherwise there shouldn't be a significant impact -on maintainability. The effect should be net positive. +receive more use. Otherwise, there shouldn't be a significant impact +on maintainability, so the net effect should be positive. Performance ----------- @@ -338,7 +337,7 @@ The following state will not be moved: Also, a number of parts of the global state were already moved to the interpreter, such as GC, warnings, and atexit hooks. -Note that currently the allocators (see Objects/obmalloc.c) are shared +Note that currently the allocators (see ``Objects/obmalloc.c``) are shared between all interpreters, protected by the GIL. They will need to move to each interpreter (or a global lock will be needed). This is the highest risk part of the work to isolate interpreters and may require @@ -377,8 +376,8 @@ raise ``ImportError``. This will be done through ``importlib._bootstrap_external.ExtensionFileLoader``. We will work with popular extensions to help them support use in -multiple interpreters. This may involve adding new public APIs, which -we will address on a case-by-case basis. +multiple interpreters. This may involve adding to CPython's public C-API, +which we will address on a case-by-case basis. Extension Module Compatibility '''''''''''''''''''''''''''''' @@ -386,7 +385,7 @@ Extension Module Compatibility As noted in `Extension Modules`_, many extensions work fine in multiple interpreters without needing any changes. The import system will still fail if such a module doesn't explicitly indicate support. At first, -not as many extension modules will do so, so this is a potential source +not many extension modules will, so this is a potential source of frustration. We will address this by adding a context manager to temporarily disable @@ -549,15 +548,13 @@ Tooling As already indicated, there are several tools to help identify the globals and reason about them. -* Tools/c-analyzer/cpython/globals-to-fix.tsv - the list of remaining globals -* Tools/c-analyzer/c-analyzer.py - - + analyze - identify all the globals - + check - fail if there are any unsupported globals that aren't ignored - -* Tools/c-analyzer/table-file.py - summarize the known globals +* ``Tools/c-analyzer/cpython/globals-to-fix.tsv`` - the list of remaining globals +* ``Tools/c-analyzer/c-analyzer.py`` + * ``analyze`` - identify all the globals + * ``check`` - fail if there are any unsupported globals that aren't ignored +* ``Tools/c-analyzer/table-file.py`` - summarize the known globals -As well, the check for unsupported globals is incorporated into CI so that +Also, the check for unsupported globals is incorporated into CI so that no new globals are accidentally added. Global Objects From 04c7888ce092ee9bde3b946ed840a9c99dd76d56 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 8 Mar 2022 20:14:21 -0700 Subject: [PATCH 13/17] More formatting and wording. --- pep-0684.rst | 308 ++++++++++++++++++++++++++------------------------- 1 file changed, 155 insertions(+), 153 deletions(-) diff --git a/pep-0684.rst b/pep-0684.rst index da86625b8f1..4a21ac89400 100644 --- a/pep-0684.rst +++ b/pep-0684.rst @@ -97,7 +97,7 @@ need to address how to deal with the various objects It also simplifies the question of what to do about the builtin static types. (See `Global Objects`_ below.) -Both matters have alternate solutions but everything is simpler with +Both issues have alternate solutions, but everything is simpler with immortal objects. If PEP 683 is not accepted then this one will be updated with the alternatives. This lets us reduce noise in this proposal. @@ -144,8 +144,8 @@ The C-API for multiple interpreters has been used for many years. However, until relatively recently the feature wasn't widely known, nor extensively used (with the exception of mod_wsgi). -In the last few years the feature has been gaining traction. Here are -some of the public projects using multiple interpreters currently: +In the last few years use of multiple interpreters has been increasing. +Here are some of the public projects using the feature currently: * `mod_wsgi `_ * `OpenStack Ceph `_ @@ -160,8 +160,9 @@ PEP 554 :pep:`554` is strictly about providing a minimal stdlib module to give users access to multiple interpreters from Python code. -It specifically avoids the question of a per-interpreter GIL but -users of that module would benefit, making PEP 554 more appealing. +In fact, it specifically avoids proposing any changes related to +the GIL. Consider, however, that users of that module would benefit +from a per-interpreter GIL, which makes PEP 554 more appealing. Rationale @@ -193,6 +194,111 @@ most of the work was worth doing anyway. (The downside was the volume of work to be done.) +Specification +============= + +As `summarized above `__, this proposal involves the +following changes, in the order they must happen: + +1. `consolidate global runtime state `_ + (including objects) into ``_PyRuntimeState`` +2. move nearly all of the state down into ``PyInterpreterState`` +3. finally, move the GIL down into ``PyInterpreterState`` +4. everything else + * add to the public C-API + * implement restrictions in ``ExtensionFileLoader`` + * work with popular extension maintainers to help + with multi-interpreter support + +Per-Interpreter State +--------------------- + +The following runtime state will be moved to ``PyInterpreterState``: + +* all global objects that are not safely shareable (fully immutable) +* the GIL +* mutable, currently protected by the GIL +* mutable, currently protected by some other per-interpreter lock +* mutable, may be used independently in different interpreters +* all other mutable (or effectively mutable) state + not otherwise excluded below + +Furthermore, a number of parts of the global state have already been +moved to the interpreter, such as GC, warnings, and atexit hooks. + +The following state will not be moved: + +* global objects that are safely shareable, if any +* immutable, often ``const`` +* treated as immutable +* related to CPython's ``main()`` execution +* related to the REPL +* set during runtime init, then treated as immutable +* mutable, protected by some global lock +* mutable, atomic + +Note that currently the allocators (see ``Objects/obmalloc.c``) are shared +between all interpreters, protected by the GIL. They will need to move +to each interpreter (or a global lock will be needed). This is the +highest risk part of the work to isolate interpreters and may require +more than just moving fields down from ``_PyRuntimeState``. Some of +the complexity is reduced if CPython switches to a thread-safe +allocator like mimalloc. + +.. _proposed capi: + +C-API +----- + +The following private API will be made public: + +* ``_PyInterpreterConfig`` +* ``_Py_NewInterpreter()`` (as ``Py_NewInterpreterEx()``) + +The following fields will be added to ``PyInterpreterConfig``: + +* ``own_gil`` - (bool) create a new interpreter lock + (instead of sharing with the main interpreter) +* ``strict_extensions`` - fail import in this interpreter for + incompatible extensions (see `Restricting Extension Modules`_) + +Restricting Extension Modules +----------------------------- + +Extension modules have many of the same problems as the runtime when +state is stored in global variables. :pep:`630` covers all the details +of what extensions must do to support isolation, and thus safely run in +multiple interpreters at once. This includes dealing with their globals. + +Extension modules that do not implement isolation will only run in +the main interpreter. In all other interpreters, the import will +raise ``ImportError``. This will be done through +``importlib._bootstrap_external.ExtensionFileLoader``. + +We will work with popular extensions to help them support use in +multiple interpreters. This may involve adding to CPython's public C-API, +which we will address on a case-by-case basis. + +Extension Module Compatibility +'''''''''''''''''''''''''''''' + +As noted in `Extension Modules`_, many extensions work fine in multiple +interpreters without needing any changes. The import system will still +fail if such a module doesn't explicitly indicate support. At first, +not many extension modules will, so this is a potential source +of frustration. + +We will address this by adding a context manager to temporarily disable +the check on multiple interpreter support: +``importlib.util.allow_all_extensions()``. + +Documentation +------------- + +The "Sub-interpreter support" section of ``Doc/c-api/init.rst`` will be +updated with the added API. + + Impact ====== @@ -295,121 +401,44 @@ not expected to make CPython slower (as long as interpreters are sufficiently isolated). -Specification -============= - -As `summarized above `__, this proposal involves the -following changes, in the order they must happen: - -1. `consolidate global runtime state `_ - (including objects) into ``_PyRuntimeState`` -2. move nearly all of the state down into ``PyInterpreterState`` -3. finally, move the GIL down into ``PyInterpreterState`` -4. everything else - - * add to the public C-API - * implement restrictions in ``ExtensionFileLoader`` - * work with popular extension maintainers to help with multi-interpreter support - -Per-Interpreter State ---------------------- - -The following runtime state will be moved to ``PyInterpreterState``: - -* all global objects that are not safely shareable (fully immutable) -* the GIL -* mutable, currently protected by the GIL -* mutable, currently protected by some other per-interpreter lock -* mutable, may be used independently in different interpreters -* all other mutable (or effectively mutable) state not otherwise excluded below - -The following state will not be moved: - -* global objects that are safely shareable, if any -* immutable, often ``const`` -* treated as immutable -* related to CPython's ``main()`` execution -* related to the REPL -* set during runtime init, then treated as immutable -* mutable, protected by some global lock -* mutable, atomic - -Also, a number of parts of the global state were already moved to the -interpreter, such as GC, warnings, and atexit hooks. - -Note that currently the allocators (see ``Objects/obmalloc.c``) are shared -between all interpreters, protected by the GIL. They will need to move -to each interpreter (or a global lock will be needed). This is the -highest risk part of the work to isolate interpreters and may require -more than just moving fields down from ``_PyRuntimeState``. Some of -the complexity is reduced if CPython switches to a thread-safe -allocator like mimalloc. - -.. _proposed capi: - -C-API ------ - -The following private API will be made public: - -* ``_PyInterpreterConfig`` -* ``_Py_NewInterpreter()`` (as ``Py_NewInterpreterEx()``) +How to Teach This +================= -The following fields will be added to ``PyInterpreterConfig``: +This is an advanced feature for users of the C-API. There is no +expectation that this will be taught. -* ``own_gil`` - (bool) create a new interpreter lock - (instead of sharing with the main interpreter) -* ``strict_extensions`` - fail import in this interpreter for - incompatible extensions (see `Restricting Extension Modules`_) +That said, if it were taught then it would boil down to the following: -Restricting Extension Modules ------------------------------ + In addition to Py_NewInterpreter(), you can use Py_NewInterpreterEx() + to create an interpreter. The config you pass it indicates how you + want that interpreter to behave. -Extension modules have many of the same problems as the runtime when -state is stored in global variables. :pep:`630` covers all the details -of what extensions must do to support isolation, and thus safely run in -multiple interpreters at once. This includes dealing with their globals. -Extension modules that do not implement isolation will only run in -the main interpreter. In all other interpreters, the import will -raise ``ImportError``. This will be done through -``importlib._bootstrap_external.ExtensionFileLoader``. +Reference Implementation +======================== -We will work with popular extensions to help them support use in -multiple interpreters. This may involve adding to CPython's public C-API, -which we will address on a case-by-case basis. + -Extension Module Compatibility -'''''''''''''''''''''''''''''' -As noted in `Extension Modules`_, many extensions work fine in multiple -interpreters without needing any changes. The import system will still -fail if such a module doesn't explicitly indicate support. At first, -not many extension modules will, so this is a potential source -of frustration. - -We will address this by adding a context manager to temporarily disable -the check on multiple interpreter support: -``importlib.util.allow_all_extensions()``. +Open Issues +=========== -Documentation -------------- +* What are the risks/hurdles involved with moving the allocators? +* Is ``allow_all_extensions`` the best name for the context manager? -The "Sub-interpreter support" section of ``Doc/c-api/init.rst`` will be -updated with the added API. +Deferred Functionality +====================== -How to Teach This -================= +* ``PyInterpreterConfig`` option to always run the interpreter in a new thread +* ``PyInterpreterConfig`` option to assign a "main" thread to the interpreter + and only run in that thread -This is an advanced feature for users of the C-API. There is no -expectation that this will be taught. -That said, if it were taught then it would boil down to the following: +Rejected Ideas +============== - In addition to Py_NewInterpreter(), you can use Py_NewInterpreterEx() - to create an interpreter. The config you pass it indicates how you - want that interpreter to behave. + Extra Context @@ -452,7 +481,7 @@ With immortal objects this is a non-issue. Consolidating Runtime Global State -================================== +---------------------------------- As noted in `CPython Runtime State`_ above, there is an active effort (separate from this PEP) to consolidate CPython's global state into the @@ -461,7 +490,7 @@ state from global variables. The project is particularly relevant to this proposal, so below is some extra detail. Benefits to Consolidation -------------------------- +''''''''''''''''''''''''' Consolidating the globals has a variety of benefits: @@ -486,7 +515,7 @@ Furthermore, much of that work benefits other CPython-related projects: * embedding CPython Scale of Work -------------- +''''''''''''' The number of global variables to be moved is large enough to matter, but most are Python objects that can be dealt with in large groups @@ -495,22 +524,22 @@ to the interpreter is highly mechanical. That doesn't require cleverness but instead requires someone to put in the time. State To Be Moved ------------------ +''''''''''''''''' -The remaining global variables can be categorized as follows:: - - global objects - static types (incl. exception types) - non-static types (incl. heap types, structseq types) - singletons (static) - singletons (initialized once) - cached objects - non-objects - will not (or unlikely to) change after init - only used in the main thread - initialized lazily - pre-allocated buffers - state +The remaining global variables can be categorized as follows: + +* global objects + * static types (incl. exception types) + * non-static types (incl. heap types, structseq types) + * singletons (static) + * singletons (initialized once) + * cached objects +* non-objects + * will not (or unlikely to) change after init + * only used in the main thread + * initialized lazily + * pre-allocated buffers + * state Those globals are spread between the core runtime, the builtin modules, and the stdlib extension modules. @@ -522,7 +551,7 @@ For a breakdown of the remaining globals, run: ./python Tools/c-analyzer/table-file.py Tools/c-analyzer/cpython/globals-to-fix.tsv Already Completed Work ----------------------- +'''''''''''''''''''''' As mentioned, this work has been going on for many years. Here are some of the things that have already been done: @@ -543,7 +572,7 @@ of the things that have already been done: * empty tuple Tooling -------- +''''''' As already indicated, there are several tools to help identify the globals and reason about them. @@ -558,7 +587,7 @@ Also, the check for unsupported globals is incorporated into CI so that no new globals are accidentally added. Global Objects --------------- +'''''''''''''' Global objects that are safe to be shared (without a GIL) between interpreters can stay on ``_PyRuntimeState``. Not only must the object @@ -579,33 +608,6 @@ it a getter that does a lookup on the current ``PyInterpreterState`` for that type. -Reference Implementation -======================== - - - - -Open Issues -=========== - -* What are the risks/hurdles involved with moving the allocators? -* Is ``allow_all_extensions`` the best name for the context manager? - - -Deferred Functionality -====================== - -* ``PyInterpreterConfig`` option to always run the interpreter in a new thread -* ``PyInterpreterConfig`` option to assign a "main" thread to the interpreter - and only run in that thread - - -Rejected Ideas -============== - - - - References ========== From 622a84e2e83f979c7b79156ee9a2acd24a6581ca Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 8 Mar 2022 20:17:07 -0700 Subject: [PATCH 14/17] Update CODEOWNERS. --- .github/CODEOWNERS | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index d9c21f2ae21..3b2a3e70a43 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -567,7 +567,8 @@ pep-0679.rst @pablogsal pep-0680.rst @encukou pep-0681.rst @jellezijlstra pep-0682.rst @mdickinson -pep-0683.rst @ericsnowcurrently +pep-0683.rst @ericsnowcurrently @eduardo-elizondo +pep-0684.rst @ericsnowcurrently # pep-0684.rst pep-0685.rst @brettcannon # ... From 2d56b70ca0fc9aa7a878992cdbaab5dee04ea24a Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 8 Mar 2022 20:20:02 -0700 Subject: [PATCH 15/17] Fix indent --- pep-0684.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pep-0684.rst b/pep-0684.rst index 4a21ac89400..e7809cf1a10 100644 --- a/pep-0684.rst +++ b/pep-0684.rst @@ -182,7 +182,7 @@ without simple solutions: * incomplete; Windows-only at the time * ``multiprocessing`` * too much work to make it effective enough; - high penalties in some situations (at large scale, Windows) + high penalties in some situations (at large scale, Windows) * other parallelism tools (e.g. dask, ray, MPI) * not a fit for the stdlib * give up on multi-core (e.g. async, do nothing) From c31d9e890a925c852c880eb8d28c6bf57e8365ec Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 8 Mar 2022 20:22:19 -0700 Subject: [PATCH 16/17] Separate the problematic list item. --- pep-0684.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/pep-0684.rst b/pep-0684.rst index e7809cf1a10..c991b8d791d 100644 --- a/pep-0684.rst +++ b/pep-0684.rst @@ -181,8 +181,10 @@ without simple solutions: * Trent Nelson's "PyParallel" project * incomplete; Windows-only at the time * ``multiprocessing`` + * too much work to make it effective enough; high penalties in some situations (at large scale, Windows) + * other parallelism tools (e.g. dask, ray, MPI) * not a fit for the stdlib * give up on multi-core (e.g. async, do nothing) From cf3bae100be70f4c5908157eda0c0e9829f6ccf6 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Tue, 8 Mar 2022 20:25:04 -0700 Subject: [PATCH 17/17] More nested list pain... --- pep-0684.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/pep-0684.rst b/pep-0684.rst index c991b8d791d..c7b1268cd1b 100644 --- a/pep-0684.rst +++ b/pep-0684.rst @@ -209,6 +209,7 @@ following changes, in the order they must happen: 4. everything else * add to the public C-API * implement restrictions in ``ExtensionFileLoader`` + * work with popular extension maintainers to help with multi-interpreter support