Deadlock when shutting down ThreadPoolExecutor from inside OS Signal handler #121649
Comments
|
For what its worth, I workaround this like using a var to track if signal has been received: from concurrent.futures import ThreadPoolExecutor
import signal
import threading
def main():
with ThreadPoolExecutor(max_workers=2) as executor:
exit_signal = threading.Event()
def __exit(signal, frame):
exit_signal.set()
signal.signal(signal.SIGTERM, __exit)
while True:
if exit_signal.is_set():
executor.shutdown(wait=True, cancel_futures=True)
break
executor.submit(lambda: 1 + 1)
if __name__ == "__main__":
main() |
|
This is expected: it is not safe to take possibly contended non-recursive locks in signal handling code. Signal handler code "interrupts" the main thread and runs at an arbitrary point, which means it may be holding arbitrary locks. In this case the main thread is in a hot loop running code that holds the thread pool's (non-recursive) lock, so when the handler runs and calls shutdown it immediately deadlocks when it tries to acquire the same lock again. Something along the lines of your work-around is the correct way to implement this sort of functionality. Note that setting the event also takes a lock, but one that your main thread never takes itself. Be careful if that might ever change: e.g. if you had a graceful shutdown path that sets it from the main thread, that would introduce the possibility of a deadlock. Signal handler code needs to be written very carefully! |
Uh oh!
There was an error while loading. Please reload this page.
Bug report
Bug description:
When running the code below and sending a SIGTERM to the process results in what looks like a deadlock on the
ThreadPoolExecutor._shutdown_lock. I guess the issue is that the SIGTERM is being handled in main thread while in thesubmitfunction where the_shutdown_lockis already locked. When I try callingshutdowninside the signal handler it waits forever.TBH I'm not sure if this is a bug or expected behavior but I wanted to file this anyways. Thanks!
AFTER sending SIGTERM to the process, this is what
pystackshows:CPython versions tested on:
3.12
Operating systems tested on:
Linux
The text was updated successfully, but these errors were encountered: