Skip to content

gh-87135: Raise PythonFinalizationError when joining a blocked daemon thread #130402

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Apr 28, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Doc/c-api/init.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1112,7 +1112,7 @@ Cautions regarding runtime finalization
In the late stage of :term:`interpreter shutdown`, after attempting to wait for
non-daemon threads to exit (though this can be interrupted by
:class:`KeyboardInterrupt`) and running the :mod:`atexit` functions, the runtime
is marked as *finalizing*: :c:func:`_Py_IsFinalizing` and
is marked as *finalizing*: :c:func:`Py_IsFinalizing` and
:func:`sys.is_finalizing` return true. At this point, only the *finalization
thread* that initiated finalization (typically the main thread) is allowed to
acquire the :term:`GIL`.
Expand Down
4 changes: 4 additions & 0 deletions Doc/library/exceptions.rst
Original file line number Diff line number Diff line change
Expand Up @@ -428,13 +428,17 @@ The following exceptions are the exceptions that are usually raised.
:exc:`PythonFinalizationError` during the Python finalization:

* Creating a new Python thread.
* :meth:`Joining <threading.Thread.join>` a running daemon thread.
* :func:`os.fork`.

See also the :func:`sys.is_finalizing` function.

.. versionadded:: 3.13
Previously, a plain :exc:`RuntimeError` was raised.

.. versionchanged:: next

:meth:`threading.Thread.join` can now raise this exception.

.. exception:: RecursionError

Expand Down
8 changes: 8 additions & 0 deletions Doc/library/threading.rst
Original file line number Diff line number Diff line change
Expand Up @@ -435,6 +435,14 @@ since it is impossible to detect the termination of alien threads.
an error to :meth:`~Thread.join` a thread before it has been started
and attempts to do so raise the same exception.

If an attempt is made to join a running daemonic thread in in late stages
of :term:`Python finalization <interpreter shutdown>` :meth:`!join`
raises a :exc:`PythonFinalizationError`.

.. versionchanged:: next

May raise :exc:`PythonFinalizationError`.

.. attribute:: name

A string used for identification purposes only. It has no semantics.
Expand Down
71 changes: 71 additions & 0 deletions Lib/test/test_threading.py
Original file line number Diff line number Diff line change
Expand Up @@ -1171,6 +1171,77 @@ def __del__(self):
self.assertEqual(out.strip(), b"OK")
self.assertIn(b"can't create new thread at interpreter shutdown", err)

def test_join_daemon_thread_in_finalization(self):
# gh-123940: Py_Finalize() prevents other threads from running Python
# code, so join() can not succeed unless the thread is already done.
# (Non-Python threads, that is `threading._DummyThread`, can't be
# joined at all.)
# We raise an exception rather than hang.
for timeout in (None, 10):
with self.subTest(timeout=timeout):
code = textwrap.dedent(f"""
import threading


def loop():
while True:
pass


class Cycle:
def __init__(self):
self.self_ref = self
self.thr = threading.Thread(
target=loop, daemon=True)
self.thr.start()

def __del__(self):
assert self.thr.is_alive()
try:
self.thr.join(timeout={timeout})
except PythonFinalizationError:
assert self.thr.is_alive()
print('got the correct exception!')

# Cycle holds a reference to itself, which ensures it is
# cleaned up during the GC that runs after daemon threads
# have been forced to exit during finalization.
Cycle()
""")
rc, out, err = assert_python_ok("-c", code)
self.assertEqual(err, b"")
self.assertIn(b"got the correct exception", out)

def test_join_finished_daemon_thread_in_finalization(self):
# (see previous test)
# If the thread is already finished, join() succeeds.
code = textwrap.dedent("""
import threading
done = threading.Event()

def loop():
done.set()


class Cycle:
def __init__(self):
self.self_ref = self
self.thr = threading.Thread(target=loop, daemon=True)
self.thr.start()
done.wait()

def __del__(self):
assert not self.thr.is_alive()
self.thr.join()
assert not self.thr.is_alive()
print('all clear!')

Cycle()
""")
rc, out, err = assert_python_ok("-c", code)
self.assertEqual(err, b"")
self.assertIn(b"all clear", out)

def test_start_new_thread_failed(self):
# gh-109746: if Python fails to start newly created thread
# due to failure of underlying PyThread_start_new_thread() call,
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Joining running daemon threads during interpreter shutdown
now raises :exc:`PythonFinalizationError`.
20 changes: 15 additions & 5 deletions Modules/_threadmodule.c
Original file line number Diff line number Diff line change
Expand Up @@ -509,11 +509,21 @@ ThreadHandle_join(ThreadHandle *self, PyTime_t timeout_ns)
// To work around this, we set `thread_is_exiting` immediately before
// `thread_run` returns. We can be sure that we are not attempting to join
// ourselves if the handle's thread is about to exit.
if (!_PyEvent_IsSet(&self->thread_is_exiting) &&
ThreadHandle_ident(self) == PyThread_get_thread_ident_ex()) {
// PyThread_join_thread() would deadlock or error out.
PyErr_SetString(ThreadError, "Cannot join current thread");
return -1;
if (!_PyEvent_IsSet(&self->thread_is_exiting)) {
if (ThreadHandle_ident(self) == PyThread_get_thread_ident_ex()) {
// PyThread_join_thread() would deadlock or error out.
PyErr_SetString(ThreadError, "Cannot join current thread");
return -1;
}
if (Py_IsFinalizing()) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this code path taken by all threads, or only daemon threads?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's taken by the thread that called Py_FinalizeEx.
When Py_IsFinalizing is true, all other threads than the one that called Py_FinalizeEx are daemonic and they cannot call Python API (including ThreadHandle_join).
So, self must be a daemon thread.

// gh-123940: On finalization, other threads are prevented from
// running Python code. They cannot finalize themselves,
// so join() would hang forever (or until timeout).
// We raise instead.
PyErr_SetString(PyExc_PythonFinalizationError,
"cannot join thread at interpreter shutdown");
return -1;
}
}

// Wait until the deadline for the thread to exit.
Expand Down
Loading