From c64de22e76aa621a4e168b21f43b8b8aa10b2940 Mon Sep 17 00:00:00 2001 From: "Yves.Duprat" Date: Wed, 23 Jul 2025 10:05:20 +0200 Subject: [PATCH 1/7] initial commit --- Doc/library/multiprocessing.rst | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst index c80f78e614818e..78aef7a2f9ffc4 100644 --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -544,6 +544,21 @@ The :mod:`multiprocessing` package mostly replicates the API of the .. versionchanged:: 3.3 Added the *daemon* parameter. + .. note:: + + Starting with Python 3.14, ``'fork'`` is no longer the default start + method on any operating system. When creating a new ``Process`` object + in a REPL session, with a start method such as ``'spawn'`` or ``'forkserver'`` + (other than ``'fork'``), the *target* argument must be + a callable object **mandatorily** defined in a module. + + Using a callable object defined in the current REPL session raises + an :exc:`AttributeError` exception when starting the process, + although this is still possible when the start method is ``'fork'``. + + This also applies to the use of the + :class:`concurrent.futures.ProcessPoolExecutor` class. + .. method:: run() Method representing the process's activity. From 607ae4a30a51027058e675188e4c916bd9a9af17 Mon Sep 17 00:00:00 2001 From: "Gregory P. Smith" Date: Sun, 27 Jul 2025 17:25:20 +0000 Subject: [PATCH 2/7] Reword, expand, and clarify the limitation, highlighting the REPL case. --- Doc/library/concurrent.futures.rst | 5 ++++ Doc/library/multiprocessing.rst | 44 +++++++++++++++++++++--------- 2 files changed, 36 insertions(+), 13 deletions(-) diff --git a/Doc/library/concurrent.futures.rst b/Doc/library/concurrent.futures.rst index dd92765038c4f7..9e81b8d25c99d0 100644 --- a/Doc/library/concurrent.futures.rst +++ b/Doc/library/concurrent.futures.rst @@ -342,6 +342,11 @@ that :class:`ProcessPoolExecutor` will not work in the interactive interpreter. Calling :class:`Executor` or :class:`Future` methods from a callable submitted to a :class:`ProcessPoolExecutor` will result in deadlock. +Note that the restrictions on functions and arguments needing to picklable as +per :class:`multiprocessing.Process` apply when using :meth:`~Executor.submit` +and :meth:`~Executor.map` on a :class:`ProcessPoolExecutor`. A function defined +in a REPL or a lambda should not be expected to work. + .. class:: ProcessPoolExecutor(max_workers=None, mp_context=None, initializer=None, initargs=(), max_tasks_per_child=None) An :class:`Executor` subclass that executes calls asynchronously using a pool diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst index 78aef7a2f9ffc4..caf78c41d05dd9 100644 --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -233,9 +233,12 @@ processes for a different context. In particular, locks created using the *fork* context cannot be passed to processes started using the *spawn* or *forkserver* start methods. -A library which wants to use a particular start method should probably -use :func:`get_context` to avoid interfering with the choice of the -library user. +Libraries using :mod:`multiprocessing` or +:class:`~concurrent.futures.ProcessPoolExecutor` should be designed to allow +their users to provide their own multiprocessing context. Using a specific +context of your own within a library can lead to incompatibilities with the +rest of the library user's application. Always document if your library +requires a specific start method. .. warning:: @@ -546,18 +549,33 @@ The :mod:`multiprocessing` package mostly replicates the API of the .. note:: - Starting with Python 3.14, ``'fork'`` is no longer the default start - method on any operating system. When creating a new ``Process`` object - in a REPL session, with a start method such as ``'spawn'`` or ``'forkserver'`` - (other than ``'fork'``), the *target* argument must be - a callable object **mandatorily** defined in a module. + In general, all arguments to :meth:`Process.__init__` must be picklable. + This is particularly notable when trying to create a :class:`Process` or + use a :class:`~concurrent.futures.ProcessPoolExecutor` from a REPL with a + locally defined *target* function. - Using a callable object defined in the current REPL session raises - an :exc:`AttributeError` exception when starting the process, - although this is still possible when the start method is ``'fork'``. + Passing a callable object defined in the current REPL session raises an + :exc:`AttributeError` exception when starting the process as such as + *target* must have been defined within an importable module to under to be + unpickled. - This also applies to the use of the - :class:`concurrent.futures.ProcessPoolExecutor` class. + Example:: + + >>> import multiprocessing as mp + >>> def knigit(): + ... print("knee!") + ... + >>> mp.Process(target=knigit).start() + >>> Traceback (most recent call last): + File ".../multiprocessing/spawn.py", line ..., in spawn_main + File ".../multiprocessing/spawn.py", line ..., in _main + AttributeError: module '__main__' has no attribute 'knigit' + + See :ref:`multiprocessing-programming-spawn`. + + While this restriction is not true if using the ``"fork"`` start method, + as of Python ``3.14`` that is no longer the default on any platform. See + :ref:`multiprocessing-start-methods`. .. method:: run() From b9773c737fbac45d1b0ffcc0554f2ebd790bd560 Mon Sep 17 00:00:00 2001 From: "Gregory P. Smith" Date: Sun, 27 Jul 2025 18:56:33 +0000 Subject: [PATCH 3/7] Move out of a .. note into the main text. mention in the high level Process description. --- Doc/library/multiprocessing.rst | 54 +++++++++++++++++---------------- 1 file changed, 28 insertions(+), 26 deletions(-) diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst index caf78c41d05dd9..ca3e0bca76c2d0 100644 --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -97,6 +97,10 @@ To show the individual process IDs involved, here is an expanded example:: For an explanation of why the ``if __name__ == '__main__'`` part is necessary, see :ref:`multiprocessing-programming`. +The arguments to :class:`Process` usually need to be unpickleable from within +the child process. If you tried typing the above example directly into a REPL it +could lead to an :exc:`AttributeError` in the child process trying to locate the +*f* function in the ``__main__`` module. .. _multiprocessing-start-methods: @@ -544,38 +548,36 @@ The :mod:`multiprocessing` package mostly replicates the API of the base class constructor (:meth:`Process.__init__`) before doing anything else to the process. - .. versionchanged:: 3.3 - Added the *daemon* parameter. + In general, all arguments to :meth:`Process.__init__` must be picklable. + This is particularly notable when trying to create a :class:`Process` or + use a :class:`~concurrent.futures.ProcessPoolExecutor` from a REPL with a + locally defined *target* function. - .. note:: + Passing a callable object defined in the current REPL session raises an + :exc:`AttributeError` exception when starting the process as such as + *target* must have been defined within an importable module to under to be + unpickled. - In general, all arguments to :meth:`Process.__init__` must be picklable. - This is particularly notable when trying to create a :class:`Process` or - use a :class:`~concurrent.futures.ProcessPoolExecutor` from a REPL with a - locally defined *target* function. + Example:: - Passing a callable object defined in the current REPL session raises an - :exc:`AttributeError` exception when starting the process as such as - *target* must have been defined within an importable module to under to be - unpickled. - - Example:: + >>> import multiprocessing as mp + >>> def knigit(): + ... print("knee!") + ... + >>> mp.Process(target=knigit).start() + >>> Traceback (most recent call last): + File ".../multiprocessing/spawn.py", line ..., in spawn_main + File ".../multiprocessing/spawn.py", line ..., in _main + AttributeError: module '__main__' has no attribute 'knigit' - >>> import multiprocessing as mp - >>> def knigit(): - ... print("knee!") - ... - >>> mp.Process(target=knigit).start() - >>> Traceback (most recent call last): - File ".../multiprocessing/spawn.py", line ..., in spawn_main - File ".../multiprocessing/spawn.py", line ..., in _main - AttributeError: module '__main__' has no attribute 'knigit' + See :ref:`multiprocessing-programming-spawn`. - See :ref:`multiprocessing-programming-spawn`. + While this restriction is not true if using the ``"fork"`` start method, + as of Python ``3.14`` that is no longer the default on any platform. See + :ref:`multiprocessing-start-methods`. - While this restriction is not true if using the ``"fork"`` start method, - as of Python ``3.14`` that is no longer the default on any platform. See - :ref:`multiprocessing-start-methods`. + .. versionchanged:: 3.3 + Added the *daemon* parameter. .. method:: run() From 4ad605aa1e3bee48da97ed134c354acd8dcf2467 Mon Sep 17 00:00:00 2001 From: "Gregory P. Smith" Date: Sun, 27 Jul 2025 19:07:37 +0000 Subject: [PATCH 4/7] move back to a note, it keeps it clear that this is "special" --- Doc/library/multiprocessing.rst | 46 +++++++++++++++++---------------- 1 file changed, 24 insertions(+), 22 deletions(-) diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst index ca3e0bca76c2d0..9df60627ef763f 100644 --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -548,33 +548,35 @@ The :mod:`multiprocessing` package mostly replicates the API of the base class constructor (:meth:`Process.__init__`) before doing anything else to the process. - In general, all arguments to :meth:`Process.__init__` must be picklable. - This is particularly notable when trying to create a :class:`Process` or - use a :class:`~concurrent.futures.ProcessPoolExecutor` from a REPL with a - locally defined *target* function. + .. note:: - Passing a callable object defined in the current REPL session raises an - :exc:`AttributeError` exception when starting the process as such as - *target* must have been defined within an importable module to under to be - unpickled. + In general, all arguments to :meth:`Process.__init__` must be picklable. + This is particularly notable when trying to create a :class:`Process` or use + a :class:`concurrent.futures.ProcessPoolExecutor` from a REPL with a locally + defined *target* function. - Example:: + Passing a callable object defined in the current REPL session raises an + :exc:`AttributeError` exception when starting the process as such as + *target* must have been defined within an importable module to under to be + unpickled. - >>> import multiprocessing as mp - >>> def knigit(): - ... print("knee!") - ... - >>> mp.Process(target=knigit).start() - >>> Traceback (most recent call last): - File ".../multiprocessing/spawn.py", line ..., in spawn_main - File ".../multiprocessing/spawn.py", line ..., in _main - AttributeError: module '__main__' has no attribute 'knigit' + Example:: + + >>> import multiprocessing as mp + >>> def knigit(): + ... print("knee!") + ... + >>> mp.Process(target=knigit).start() + >>> Traceback (most recent call last): + File ".../multiprocessing/spawn.py", line ..., in spawn_main + File ".../multiprocessing/spawn.py", line ..., in _main + AttributeError: module '__main__' has no attribute 'knigit' - See :ref:`multiprocessing-programming-spawn`. + See :ref:`multiprocessing-programming-spawn`. - While this restriction is not true if using the ``"fork"`` start method, - as of Python ``3.14`` that is no longer the default on any platform. See - :ref:`multiprocessing-start-methods`. + While this restriction is not true if using the ``"fork"`` start method, + as of Python ``3.14`` that is no longer the default on any platform. See + :ref:`multiprocessing-start-methods`. .. versionchanged:: 3.3 Added the *daemon* parameter. From 2a115a31475647b01b04d401bc2cb9d48883d9be Mon Sep 17 00:00:00 2001 From: "Gregory P. Smith" Date: Sun, 27 Jul 2025 19:23:34 +0000 Subject: [PATCH 5/7] editing for clarity --- Doc/library/multiprocessing.rst | 36 +++++++++++++++++---------------- 1 file changed, 19 insertions(+), 17 deletions(-) diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst index 9df60627ef763f..3f940782382c83 100644 --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -545,37 +545,39 @@ The :mod:`multiprocessing` package mostly replicates the API of the to pass to *target*. If a subclass overrides the constructor, it must make sure it invokes the - base class constructor (:meth:`Process.__init__`) before doing anything else + base class constructor (``super().__init__()``) before doing anything else to the process. .. note:: - In general, all arguments to :meth:`Process.__init__` must be picklable. - This is particularly notable when trying to create a :class:`Process` or use - a :class:`concurrent.futures.ProcessPoolExecutor` from a REPL with a locally - defined *target* function. + In general, all arguments to :class:`Process` must be picklable. This is + frequently observed when trying to create a :class:`Process` or use a + :class:`concurrent.futures.ProcessPoolExecutor` from a REPL with a + locally defined *target* function. - Passing a callable object defined in the current REPL session raises an - :exc:`AttributeError` exception when starting the process as such as - *target* must have been defined within an importable module to under to be - unpickled. + Passing a callable object defined in the current REPL session causes the + child process to die via an uncaught :exc:`AttributeError` exception when + starting as *target* must have been defined within an importable module + in order to be loaded during unpickling. - Example:: + Example of this uncatchable error from the child:: >>> import multiprocessing as mp >>> def knigit(): - ... print("knee!") + ... print("Ni!") ... - >>> mp.Process(target=knigit).start() + >>> process = mp.Process(target=knigit) + >>> process.start() >>> Traceback (most recent call last): File ".../multiprocessing/spawn.py", line ..., in spawn_main File ".../multiprocessing/spawn.py", line ..., in _main AttributeError: module '__main__' has no attribute 'knigit' + >>> process + - See :ref:`multiprocessing-programming-spawn`. - - While this restriction is not true if using the ``"fork"`` start method, - as of Python ``3.14`` that is no longer the default on any platform. See + See :ref:`multiprocessing-programming-spawn`. While this restriction is + not true if using the ``"fork"`` start method, as of Python ``3.14`` that + is no longer the default on any platform. See :ref:`multiprocessing-start-methods`. .. versionchanged:: 3.3 @@ -3107,7 +3109,7 @@ start method. More picklability - Ensure that all arguments to :meth:`Process.__init__` are picklable. + Ensure that all arguments to :class:`Process` are picklable. Also, if you subclass :class:`~multiprocessing.Process` then make sure that instances will be picklable when the :meth:`Process.start ` method is called. From 5f36fbff8c5c0f524ec2d799bc40c191cf1b7bd6 Mon Sep 17 00:00:00 2001 From: "Gregory P. Smith" Date: Sun, 27 Jul 2025 19:30:54 +0000 Subject: [PATCH 6/7] formatting --- Doc/library/multiprocessing.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst index 3f940782382c83..dd6591043fbb23 100644 --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -3109,10 +3109,10 @@ start method. More picklability - Ensure that all arguments to :class:`Process` are picklable. - Also, if you subclass :class:`~multiprocessing.Process` then make sure that - instances will be picklable when the :meth:`Process.start - ` method is called. + Ensure that all arguments to :class:`~multiprocessing.Process` are + picklable. Also, if you subclass ``Process.__init__``, you must make sure + that instances will be picklable when the + :meth:`Process.start ` method is called. Global variables From 4b848a9ff95806c55fcb3ea52c4119795f802bbd Mon Sep 17 00:00:00 2001 From: "Gregory P. Smith" Date: Sun, 27 Jul 2025 19:39:53 +0000 Subject: [PATCH 7/7] add a pointer to the GH issue from the doc note --- Doc/library/multiprocessing.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst index dd6591043fbb23..d18ada3511d891 100644 --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -579,6 +579,7 @@ The :mod:`multiprocessing` package mostly replicates the API of the not true if using the ``"fork"`` start method, as of Python ``3.14`` that is no longer the default on any platform. See :ref:`multiprocessing-start-methods`. + See also :gh:`132898`. .. versionchanged:: 3.3 Added the *daemon* parameter.