Skip to content

Race on _PyRuntime.signals.unhandled_keyboard_interrupt when calling eval() concurrently in free-threading mode #128130

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
hawkinsp opened this issue Dec 20, 2024 · 10 comments
Closed

Race on _PyRuntime.signals.unhandled_keyboard_interrupt when calling eval() concurrently in free-threading mode #128130

hawkinsp opened this issue Dec 20, 2024 · 10 comments
Labels
3.13 bugs and security fixes 3.14 new features, bugs and security fixes extension-modules C modules in the Modules dir topic-free-threading type-bug An unexpected behavior, bug, or error

Comments

@hawkinsp
Copy link
Contributor

hawkinsp commented Dec 20, 2024 •
edited by bedevere-app bot
Loading

Bug report

Bug description:

Run the following code under Python 3.13.1t with thread-sanitizer enabled:

from collections import namedtuple
import concurrent.futures
import functools
import threading

num_threads = 8


def closure(b):
  b.wait()
  namedtuple('mytuple', ['f'])

with concurrent.futures.ThreadPoolExecutor(max_workers=num_threads) as executor:
  for _ in range(100):
    b = threading.Barrier(num_threads)
    for _ in range(num_threads):
      executor.submit(functools.partial(closure, b))

Get:

WARNING: ThreadSanitizer: data race (pid=682761)
  Write of size 4 at 0x56312bb5f168 by thread T6:
    #0 run_eval_code_obj /usr/local/google/home/phawkins/p/cpython/Python/pythonrun.c:1318:53 (python3.13+0x4a02a6) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #1 run_mod /usr/local/google/home/phawkins/p/cpython/Python/pythonrun.c:1422:19 (python3.13+0x49fa95) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #2 _PyRun_StringFlagsWithName /usr/local/google/home/phawkins/p/cpython/Python/pythonrun.c:1221:15 (python3.13+0x49cfe4) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #3 PyRun_StringFlags /usr/local/google/home/phawkins/p/cpython/Python/pythonrun.c:1233:12 (python3.13+0x49cfe4)
    #4 builtin_eval_impl /usr/local/google/home/phawkins/p/cpython/Python/bltinmodule.c:1027:18 (python3.13+0x3d8189) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #5 builtin_eval /usr/local/google/home/phawkins/p/cpython/Python/clinic/bltinmodule.c.h:467:20 (python3.13+0x3d8189)
    #6 cfunction_vectorcall_FASTCALL_KEYWORDS /usr/local/google/home/phawkins/p/cpython/Objects/methodobject.c:441:24 (python3.13+0x28a010) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #7 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1eafaa) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #8 PyObject_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c:327:12 (python3.13+0x1eafaa)
    #9 _PyEval_EvalFrameDefault /usr/local/google/home/phawkins/p/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e24fb) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #10 _PyEval_EvalFrame /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de62a) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #11 _PyEval_Vector /usr/local/google/home/phawkins/p/cpython/Python/ceval.c:1811:12 (python3.13+0x3de62a)
    #12 _PyFunction_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c (python3.13+0x1eb61f) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #13 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x571bb2) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #14 partial_vectorcall /usr/local/google/home/phawkins/p/cpython/./Modules/_functoolsmodule.c:252:16 (python3.13+0x571bb2)
    #15 _PyVectorcall_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:273:16 (python3.13+0x1eb293) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #16 _PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:348:16 (python3.13+0x1eb293)
    #17 PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:373:12 (python3.13+0x1eb315) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #18 _PyEval_EvalFrameDefault /usr/local/google/home/phawkins/p/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e46e2) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #19 _PyEval_EvalFrame /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de62a) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #20 _PyEval_Vector /usr/local/google/home/phawkins/p/cpython/Python/ceval.c:1811:12 (python3.13+0x3de62a)
    #21 _PyFunction_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c (python3.13+0x1eb61f) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #22 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef5ef) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #23 method_vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/classobject.c:70:20 (python3.13+0x1ef5ef)
    #24 _PyVectorcall_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:273:16 (python3.13+0x1eb293) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #25 _PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:348:16 (python3.13+0x1eb293)
    #26 PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:373:12 (python3.13+0x1eb315) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #27 thread_run /usr/local/google/home/phawkins/p/cpython/./Modules/_threadmodule.c:337:21 (python3.13+0x564292) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #28 pythread_wrapper /usr/local/google/home/phawkins/p/cpython/Python/thread_pthread.h:243:5 (python3.13+0x4bd637) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)

  Previous write of size 4 at 0x56312bb5f168 by thread T8:
    #0 run_eval_code_obj /usr/local/google/home/phawkins/p/cpython/Python/pythonrun.c:1318:53 (python3.13+0x4a02a6) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #1 run_mod /usr/local/google/home/phawkins/p/cpython/Python/pythonrun.c:1422:19 (python3.13+0x49fa95) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #2 _PyRun_StringFlagsWithName /usr/local/google/home/phawkins/p/cpython/Python/pythonrun.c:1221:15 (python3.13+0x49cfe4) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #3 PyRun_StringFlags /usr/local/google/home/phawkins/p/cpython/Python/pythonrun.c:1233:12 (python3.13+0x49cfe4)
    #4 builtin_eval_impl /usr/local/google/home/phawkins/p/cpython/Python/bltinmodule.c:1027:18 (python3.13+0x3d8189) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #5 builtin_eval /usr/local/google/home/phawkins/p/cpython/Python/clinic/bltinmodule.c.h:467:20 (python3.13+0x3d8189)
    #6 cfunction_vectorcall_FASTCALL_KEYWORDS /usr/local/google/home/phawkins/p/cpython/Objects/methodobject.c:441:24 (python3.13+0x28a010) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #7 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1eafaa) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #8 PyObject_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c:327:12 (python3.13+0x1eafaa)
    #9 _PyEval_EvalFrameDefault /usr/local/google/home/phawkins/p/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e24fb) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #10 _PyEval_EvalFrame /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de62a) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #11 _PyEval_Vector /usr/local/google/home/phawkins/p/cpython/Python/ceval.c:1811:12 (python3.13+0x3de62a)
    #12 _PyFunction_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c (python3.13+0x1eb61f) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #13 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x571bb2) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #14 partial_vectorcall /usr/local/google/home/phawkins/p/cpython/./Modules/_functoolsmodule.c:252:16 (python3.13+0x571bb2)
    #15 _PyVectorcall_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:273:16 (python3.13+0x1eb293) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #16 _PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:348:16 (python3.13+0x1eb293)
    #17 PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:373:12 (python3.13+0x1eb315) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #18 _PyEval_EvalFrameDefault /usr/local/google/home/phawkins/p/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e46e2) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #19 _PyEval_EvalFrame /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3de62a) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #20 _PyEval_Vector /usr/local/google/home/phawkins/p/cpython/Python/ceval.c:1811:12 (python3.13+0x3de62a)
    #21 _PyFunction_Vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/call.c (python3.13+0x1eb61f) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #22 _PyObject_VectorcallTstate /usr/local/google/home/phawkins/p/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef5ef) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #23 method_vectorcall /usr/local/google/home/phawkins/p/cpython/Objects/classobject.c:70:20 (python3.13+0x1ef5ef)
    #24 _PyVectorcall_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:273:16 (python3.13+0x1eb293) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #25 _PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:348:16 (python3.13+0x1eb293)
    #26 PyObject_Call /usr/local/google/home/phawkins/p/cpython/Objects/call.c:373:12 (python3.13+0x1eb315) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #27 thread_run /usr/local/google/home/phawkins/p/cpython/./Modules/_threadmodule.c:337:21 (python3.13+0x564292) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)
    #28 pythread_wrapper /usr/local/google/home/phawkins/p/cpython/Python/thread_pthread.h:243:5 (python3.13+0x4bd637) (BuildId: 9c1c16fb1bb8a435fa6fa4c6944da5d41f654e96)

I think namedtuple calls eval(), which triggers a race here:

cpython/Python/pythonrun.c

Line 1318 in 0671451

_PyRuntime.signals.unhandled_keyboard_interrupt = 0;

CPython versions tested on:

3.13

Operating systems tested on:

Linux

Linked PRs
@hawkinsp hawkinsp added the type-bug An unexpected behavior, bug, or error label Dec 20, 2024
@picnixz picnixz added extension-modules C modules in the Modules dir 3.13 bugs and security fixes topic-free-threading 3.14 new features, bugs and security fixes labels Dec 20, 2024
@colesbury colesbury changed the title Race when calling collections.namedtuple() concurrently in free-threading mode Race on _PyRuntime.signals.unhandled_keyboard_interrupt when calling eval() concurrently in free-threading mode Dec 21, 2024
@colesbury
Copy link
Contributor

I think there's a race condition here even in the default build where a concurrent eval() may wipe out the unhandled_keyboard_interrupt from the main thread.

@colesbury
Copy link
Contributor

Also seen when running ./python -m test test_exceptions --parallel-threads=10 -v under Tsan in main (3.14). It's in a different function, but still on _PyRuntime.signals.unhandled_keyboard_interrupt.

WARNING: ThreadSanitizer: data race (pid=990837)
  Write of size 4 at 0x555555cd2530 by thread T595:
    #0 _PyErr_Display Python/pythonrun.c:1127 (python+0x43bad3) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #1 thread_excepthook_file Modules/_threadmodule.c:2189 (python+0x52f068) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #2 thread_excepthook Modules/_threadmodule.c:2268 (python+0x52f068)
    #3 cfunction_vectorcall_O Objects/methodobject.c:523 (python+0x1fd56e) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #4 _PyObject_VectorcallTstate Include/internal/pycore_call.h:167 (python+0x149161) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #5 PyObject_Vectorcall Objects/call.c:327 (python+0x149161)
    #6 _PyEval_EvalFrameDefault Python/generated_cases.c.h:1367 (python+0x8a1ff) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #7 _PyEval_EvalFrame Include/internal/pycore_ceval.h:116 (python+0x37d7fc) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #8 _PyEval_Vector Python/ceval.c:1761 (python+0x37d7fc)
    #9 _PyFunction_Vectorcall Objects/call.c:413 (python+0x14957b) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #10 _PyObject_VectorcallTstate Include/internal/pycore_call.h:167 (python+0x14f138) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #11 method_vectorcall Objects/classobject.c:72 (python+0x14f138)
    #12 _PyVectorcall_Call Objects/call.c:273 (python+0x14b9b7) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #13 _PyObject_Call Objects/call.c:348 (python+0x14be80) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #14 PyObject_Call Objects/call.c:373 (python+0x14be80)
    #15 thread_run Modules/_threadmodule.c:354 (python+0x52e235) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #16 pythread_wrapper Python/thread_pthread.h:242 (python+0x466079) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)

  Previous write of size 4 at 0x555555cd2530 by thread T599:
    #0 _PyErr_Display Python/pythonrun.c:1127 (python+0x43bad3) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #1 thread_excepthook_file Modules/_threadmodule.c:2189 (python+0x52f068) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #2 thread_excepthook Modules/_threadmodule.c:2268 (python+0x52f068)
    #3 cfunction_vectorcall_O Objects/methodobject.c:523 (python+0x1fd56e) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #4 _PyObject_VectorcallTstate Include/internal/pycore_call.h:167 (python+0x149161) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #5 PyObject_Vectorcall Objects/call.c:327 (python+0x149161)
    #6 _PyEval_EvalFrameDefault Python/generated_cases.c.h:1367 (python+0x8a1ff) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #7 _PyEval_EvalFrame Include/internal/pycore_ceval.h:116 (python+0x37d7fc) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #8 _PyEval_Vector Python/ceval.c:1761 (python+0x37d7fc)
    #9 _PyFunction_Vectorcall Objects/call.c:413 (python+0x14957b) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #10 _PyObject_VectorcallTstate Include/internal/pycore_call.h:167 (python+0x14f138) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #11 method_vectorcall Objects/classobject.c:72 (python+0x14f138)
    #12 _PyVectorcall_Call Objects/call.c:273 (python+0x14b9b7) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #13 _PyObject_Call Objects/call.c:348 (python+0x14be80) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #14 PyObject_Call Objects/call.c:373 (python+0x14be80)
    #15 thread_run Modules/_threadmodule.c:354 (python+0x52e235) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)
    #16 pythread_wrapper Python/thread_pthread.h:242 (python+0x466079) (BuildId: f460519d72c998d19781985effd47fe68db6f3c2)

@colesbury colesbury mentioned this issue Feb 7, 2025
Open
colesbury added a commit to colesbury/cpython that referenced this issue Feb 10, 2025
@colesbury
pythongh-128130: Fix unhandled keyboard interrupt data race
9bf5214 
Use an atomic operation when setting
`_PyRuntime.signals.unhandled_keyboard_interrupt`. We now only clear the
variable at the start of `_PyRun_Main`, which is the same function where
we check it.

This avoids race conditions where previously another thread might call
`run_eval_code_obj()` and erroneously clear the unhandled keyboard
interrupt.
@bedevere-app bedevere-app bot mentioned this issue Feb 10, 2025
Merged
colesbury added a commit to colesbury/cpython that referenced this issue Feb 10, 2025
@colesbury
pythongh-128130: Fix unhandled keyboard interrupt data race
a5267ca 
Use an atomic operation when setting
`_PyRuntime.signals.unhandled_keyboard_interrupt`. We now only clear the
variable at the start of `_PyRun_Main`, which is the same function where
we check it.

This avoids race conditions where previously another thread might call
`run_eval_code_obj()` and erroneously clear the unhandled keyboard
interrupt.
colesbury added a commit that referenced this issue Feb 13, 2025
@colesbury
gh-128130: Fix unhandled keyboard interrupt data race (gh-129975)
451f291 
Use an atomic operation when setting
`_PyRuntime.signals.unhandled_keyboard_interrupt`. We now only clear the
variable at the start of `_PyRun_Main`, which is the same function where
we check it.

This avoids race conditions where previously another thread might call
`run_eval_code_obj()` and erroneously clear the unhandled keyboard
interrupt.
@nascheme
Copy link
Member

It looks like this bug has been fixed by gh-128130 and can be closed.

While testing with TSAN, I found another warning and made a PR to fix it: gh-131126.

@hawkinsp
Copy link
Contributor Author

Closing, on the assumption the PR linked above fixed this!

@hawkinsp hawkinsp reopened this Mar 24, 2025
@hawkinsp
Copy link
Contributor Author

Not fixed on the 3.13 branch:

WARNING: ThreadSanitizer: data race (pid=113280)
  Write of size 4 at 0x567f47699168 by thread T6 (mutexes: read M0):
    #0 run_eval_code_obj /__w/jax/jax/cpython/Python/pythonrun.c:1362:53 (python3.13+0x4a2ee6) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #1 run_mod /__w/jax/jax/cpython/Python/pythonrun.c:1466:19 (python3.13+0x4a26d5) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #2 _PyRun_StringFlagsWithName /__w/jax/jax/cpython/Python/pythonrun.c:1261:15 (python3.13+0x49fa64) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #3 PyRun_StringFlags /__w/jax/jax/cpython/Python/pythonrun.c:1273:12 (python3.13+0x49fa64)
    #4 builtin_exec_impl /__w/jax/jax/cpython/Python/bltinmodule.c (python3.13+0x3d99c2) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #5 builtin_exec /__w/jax/jax/cpython/Python/clinic/bltinmodule.c.h:556:20 (python3.13+0x3d99c2)
    #6 cfunction_vectorcall_FASTCALL_KEYWORDS /__w/jax/jax/cpython/Objects/methodobject.c:441:24 (python3.13+0x28a5d0) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #7 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1eb0ea) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #8 PyObject_Vectorcall /__w/jax/jax/cpython/Objects/call.c:327:12 (python3.13+0x1eb0ea)
    #9 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e385b) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #10 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3df98a) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #11 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1814:12 (python3.13+0x3df98a)
    #12 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb75f) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #13 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef7e0) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #14 method_vectorcall /__w/jax/jax/cpython/Objects/classobject.c:92:18 (python3.13+0x1ef7e0)
    #15 _PyVectorcall_Call /__w/jax/jax/cpython/Objects/call.c:273:16 (python3.13+0x1eb3d3) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #16 _PyObject_Call /__w/jax/jax/cpython/Objects/call.c:348:16 (python3.13+0x1eb3d3)
    #17 PyObject_Call /__w/jax/jax/cpython/Objects/call.c:373:12 (python3.13+0x1eb455) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #18 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e5a42) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #19 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3df98a) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #20 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1814:12 (python3.13+0x3df98a)
    #21 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb75f) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #22 _PyObject_VectorcallDictTstate /__w/jax/jax/cpython/Objects/call.c:135:15 (python3.13+0x1ea2dd) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #23 _PyObject_Call_Prepend /__w/jax/jax/cpython/Objects/call.c:504:24 (python3.13+0x1ebdd7) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #24 slot_tp_call /__w/jax/jax/cpython/Objects/typeobject.c:9556:15 (python3.13+0x2f9718) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #25 _PyObject_MakeTpCall /__w/jax/jax/cpython/Objects/call.c:242:18 (python3.13+0x1ea54c) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #26 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:166:16 (python3.13+0x1eb1a8) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #27 PyObject_Vectorcall /__w/jax/jax/cpython/Objects/call.c:327:12 (python3.13+0x1eb1a8)
    #28 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e385b) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #29 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3df98a) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #30 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1814:12 (python3.13+0x3df98a)
    #31 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb75f) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #32 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef72f) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #33 method_vectorcall /__w/jax/jax/cpython/Objects/classobject.c:70:20 (python3.13+0x1ef72f)
    #34 _PyVectorcall_Call /__w/jax/jax/cpython/Objects/call.c:273:16 (python3.13+0x1eb3d3) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #35 _PyObject_Call /__w/jax/jax/cpython/Objects/call.c:348:16 (python3.13+0x1eb3d3)
    #36 PyObject_Call /__w/jax/jax/cpython/Objects/call.c:373:12 (python3.13+0x1eb455) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #37 thread_run /__w/jax/jax/cpython/./Modules/_threadmodule.c:337:21 (python3.13+0x568202) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #38 pythread_wrapper /__w/jax/jax/cpython/Python/thread_pthread.h:243:5 (python3.13+0x4c1107) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)

  Previous write of size 4 at 0x567f47699168 by thread T7 (mutexes: read M0):
    #0 run_eval_code_obj /__w/jax/jax/cpython/Python/pythonrun.c:1362:53 (python3.13+0x4a2ee6) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #1 run_mod /__w/jax/jax/cpython/Python/pythonrun.c:1466:19 (python3.13+0x4a26d5) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #2 _PyRun_StringFlagsWithName /__w/jax/jax/cpython/Python/pythonrun.c:1261:15 (python3.13+0x49fa64) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #3 PyRun_StringFlags /__w/jax/jax/cpython/Python/pythonrun.c:1273:12 (python3.13+0x49fa64)
    #4 builtin_exec_impl /__w/jax/jax/cpython/Python/bltinmodule.c (python3.13+0x3d99c2) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #5 builtin_exec /__w/jax/jax/cpython/Python/clinic/bltinmodule.c.h:556:20 (python3.13+0x3d99c2)
    #6 cfunction_vectorcall_FASTCALL_KEYWORDS /__w/jax/jax/cpython/Objects/methodobject.c:441:24 (python3.13+0x28a5d0) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #7 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1eb0ea) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #8 PyObject_Vectorcall /__w/jax/jax/cpython/Objects/call.c:327:12 (python3.13+0x1eb0ea)
    #9 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e385b) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #10 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3df98a) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #11 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1814:12 (python3.13+0x3df98a)
    #12 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb75f) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #13 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef7e0) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #14 method_vectorcall /__w/jax/jax/cpython/Objects/classobject.c:92:18 (python3.13+0x1ef7e0)
    #15 _PyVectorcall_Call /__w/jax/jax/cpython/Objects/call.c:273:16 (python3.13+0x1eb3d3) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #16 _PyObject_Call /__w/jax/jax/cpython/Objects/call.c:348:16 (python3.13+0x1eb3d3)
    #17 PyObject_Call /__w/jax/jax/cpython/Objects/call.c:373:12 (python3.13+0x1eb455) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #18 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:1355:26 (python3.13+0x3e5a42) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #19 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3df98a) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #20 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1814:12 (python3.13+0x3df98a)
    #21 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb75f) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #22 _PyObject_VectorcallDictTstate /__w/jax/jax/cpython/Objects/call.c:135:15 (python3.13+0x1ea2dd) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #23 _PyObject_Call_Prepend /__w/jax/jax/cpython/Objects/call.c:504:24 (python3.13+0x1ebdd7) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #24 slot_tp_call /__w/jax/jax/cpython/Objects/typeobject.c:9556:15 (python3.13+0x2f9718) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #25 _PyObject_MakeTpCall /__w/jax/jax/cpython/Objects/call.c:242:18 (python3.13+0x1ea54c) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #26 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:166:16 (python3.13+0x1eb1a8) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #27 PyObject_Vectorcall /__w/jax/jax/cpython/Objects/call.c:327:12 (python3.13+0x1eb1a8)
    #28 _PyEval_EvalFrameDefault /__w/jax/jax/cpython/Python/generated_cases.c.h:813:23 (python3.13+0x3e385b) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #29 _PyEval_EvalFrame /__w/jax/jax/cpython/./Include/internal/pycore_ceval.h:119:16 (python3.13+0x3df98a) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #30 _PyEval_Vector /__w/jax/jax/cpython/Python/ceval.c:1814:12 (python3.13+0x3df98a)
    #31 _PyFunction_Vectorcall /__w/jax/jax/cpython/Objects/call.c (python3.13+0x1eb75f) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #32 _PyObject_VectorcallTstate /__w/jax/jax/cpython/./Include/internal/pycore_call.h:168:11 (python3.13+0x1ef72f) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #33 method_vectorcall /__w/jax/jax/cpython/Objects/classobject.c:70:20 (python3.13+0x1ef72f)
    #34 _PyVectorcall_Call /__w/jax/jax/cpython/Objects/call.c:273:16 (python3.13+0x1eb3d3) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #35 _PyObject_Call /__w/jax/jax/cpython/Objects/call.c:348:16 (python3.13+0x1eb3d3)
    #36 PyObject_Call /__w/jax/jax/cpython/Objects/call.c:373:12 (python3.13+0x1eb455) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #37 thread_run /__w/jax/jax/cpython/./Modules/_threadmodule.c:337:21 (python3.13+0x568202) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)
    #38 pythread_wrapper /__w/jax/jax/cpython/Python/thread_pthread.h:243:5 (python3.13+0x4c1107) (BuildId: d6d54de3753b7c6777cd2a5c08fbef6663ab66e8)

@colesbury
Copy link
Contributor

The PR was merged into main, but I didn't backport it to 3.13. I've been generally hesitant to backport PRs that primarily fix TSAN warnings as opposed to crashes or other user visible misbehavior.

If it makes testing Jax easier, let me know and we can reconsider backporting the fix.

@hawkinsp
Copy link
Contributor Author

The PR was merged into main, but I didn't backport it to 3.13. I've been generally hesitant to backport PRs that primarily fix TSAN warnings as opposed to crashes or other user visible misbehavior.

If it makes testing Jax easier, let me know and we can reconsider backporting the fix.

It's not critical. It's just an entry in a tsan suppression file.
https://github.com/jax-ml/jax/blob/d3836f160cc6270280ba75c0ae5b12cf1b99535f/.github/workflows/tsan-suppressions.txt#L27
I'm just trying to be aggressive in making sure that these issues get fixed upstream since I want 3.14 to be race free for us.

Actually, we're doing pretty well overall. We're down to only 2 unfixed races in our test suite (#129748, #128050), assuming the things tagged as fixed in 3.14 are indeed fixed.

@colesbury
Copy link
Contributor

Are you able to test with 3.14 as well? If not, what are the blockers?

@hawkinsp
Copy link
Contributor Author

Are you able to test with 3.14 as well? If not, what are the blockers?

No blockers. We just don't have that test. We've done it manually a few times. Let me add that to our CI.

@colesbury colesbury mentioned this issue Mar 24, 2025
Closed
@hawkinsp
Copy link
Contributor Author

hawkinsp commented Apr 7, 2025

Closing, we haven't seen this race under 3.14.

@hawkinsp hawkinsp closed this as completed Apr 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
3.13 bugs and security fixes 3.14 new features, bugs and security fixes extension-modules C modules in the Modules dir topic-free-threading type-bug An unexpected behavior, bug, or error
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@hawkinsp @colesbury @nascheme @picnixz