From cfd65b8d85cfd513344ff498c3468fb0a395a462 Mon Sep 17 00:00:00 2001 From: Raymond Hettinger Date: Tue, 22 Jul 2025 18:25:23 -0600 Subject: [PATCH 01/14] Fix docs for Queue.shutdown --- Doc/library/queue.rst | 45 +++++++++++++++++++++++++++++-------------- 1 file changed, 31 insertions(+), 14 deletions(-) diff --git a/Doc/library/queue.rst b/Doc/library/queue.rst index fbbebcf4ed8f92..94fd36310382d1 100644 --- a/Doc/library/queue.rst +++ b/Doc/library/queue.rst @@ -187,9 +187,6 @@ fully processed by daemon consumer threads. processed (meaning that a :meth:`task_done` call was received for every item that had been :meth:`put` into the queue). - ``shutdown(immediate=True)`` calls :meth:`task_done` for each remaining item - in the queue. - Raises a :exc:`ValueError` if called more times than there were items placed in the queue. @@ -233,22 +230,42 @@ Example of how to wait for enqueued tasks to be completed:: Terminating queues ^^^^^^^^^^^^^^^^^^ -:class:`Queue` objects can be made to prevent further interaction by shutting -them down. +When no longer needed, :class:`Queue` objects can be wound down +or terminated immediately. .. method:: Queue.shutdown(immediate=False) - Shut down the queue, making :meth:`~Queue.get` and :meth:`~Queue.put` raise - :exc:`ShutDown`. + Put a :class:`Queue` instance into a shutdown mode. + + The queue can no longer grow. + Future calls to :meth:`~Queue.put` raise :exc:`ShutDown`. + Currently blocked callers of :meth:`~Queue.put` will be unblocked + and will raise :exc:`ShutDown` in the formerly blocked thread. + + If *immediate* is false (the default), the queue can be wound + down normally with calls :meth:`~Queue.get` to extract tasks + that have already been loaded. + + If the shutdown occurs during the brief window where the queue still + has data and callers to :meth:`~Queue.get` are blocked, those callers + will be unblocked. + + And if :meth:`~Queue.task_done` is called for each remaining task, a + pending :meth:`~Queue.join` will be unblocked normally. + + Once the queue is empty, future calls to :meth:`~Queue.get` will + raise :exc:`ShutDown`. - By default, :meth:`~Queue.get` on a shut down queue will only raise once the - queue is empty. Set *immediate* to true to make :meth:`~Queue.get` raise - immediately instead. + If *immediate* is true, the queue is terminated immediately. + The queue is drained to be completely empty. The count of + unfinished tasks is reduced to zero but without calling + :meth:`~Queue.task_done`. That then unblocks all callers of + :meth:`~Queue.join`. In addition, blocked callers of + :meth:`~Queue.get` are unblocked and will raise :exc:`ShutDown`. - All blocked callers of :meth:`~Queue.put` and :meth:`~Queue.get` will be - unblocked. If *immediate* is true, a task will be marked as done for each - remaining item in the queue, which may unblock callers of - :meth:`~Queue.join`. + Use caution when using :meth:`~Queue.join` with *immediate* set + to true. This unblocks the join even when no work has been done + on the tasks, violating the usual invariant for joining a queue. .. versionadded:: 3.13 From dc762d77e74fbece256432da875cc6ed86c3cc81 Mon Sep 17 00:00:00 2001 From: Raymond Hettinger Date: Tue, 22 Jul 2025 18:37:14 -0600 Subject: [PATCH 02/14] Clarify adjustment to unfinished tasks --- Doc/library/queue.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/library/queue.rst b/Doc/library/queue.rst index 94fd36310382d1..04faa1b071d808 100644 --- a/Doc/library/queue.rst +++ b/Doc/library/queue.rst @@ -258,7 +258,7 @@ or terminated immediately. If *immediate* is true, the queue is terminated immediately. The queue is drained to be completely empty. The count of - unfinished tasks is reduced to zero but without calling + unfinished tasks is reduced by number drained but without calling :meth:`~Queue.task_done`. That then unblocks all callers of :meth:`~Queue.join`. In addition, blocked callers of :meth:`~Queue.get` are unblocked and will raise :exc:`ShutDown`. From 4e132c4695245ac55a2f72c61e685c9159ffe045 Mon Sep 17 00:00:00 2001 From: Raymond Hettinger Date: Tue, 22 Jul 2025 18:41:07 -0600 Subject: [PATCH 03/14] Clarify that Queue.join is unblocked unconditionally --- Doc/library/queue.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/Doc/library/queue.rst b/Doc/library/queue.rst index 04faa1b071d808..be6668a41ce958 100644 --- a/Doc/library/queue.rst +++ b/Doc/library/queue.rst @@ -259,8 +259,9 @@ or terminated immediately. If *immediate* is true, the queue is terminated immediately. The queue is drained to be completely empty. The count of unfinished tasks is reduced by number drained but without calling - :meth:`~Queue.task_done`. That then unblocks all callers of - :meth:`~Queue.join`. In addition, blocked callers of + :meth:`~Queue.task_done`. All callers of :meth:`~Queue.join` + are unblocked even if the unfinished tasks is more than zero. + In addition, blocked callers of :meth:`~Queue.get` are unblocked and will raise :exc:`ShutDown`. Use caution when using :meth:`~Queue.join` with *immediate* set From 5a93c53f4be5ce4e903d569f3d1618f46dbf5388 Mon Sep 17 00:00:00 2001 From: Raymond Hettinger Date: Thu, 24 Jul 2025 00:57:23 -0600 Subject: [PATCH 04/14] Clarify "wound down" --- Doc/library/queue.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/queue.rst b/Doc/library/queue.rst index be6668a41ce958..025feee61847f5 100644 --- a/Doc/library/queue.rst +++ b/Doc/library/queue.rst @@ -231,7 +231,7 @@ Terminating queues ^^^^^^^^^^^^^^^^^^ When no longer needed, :class:`Queue` objects can be wound down -or terminated immediately. +until empty or terminated immediately (a hard shutdown). .. method:: Queue.shutdown(immediate=False) @@ -258,7 +258,7 @@ or terminated immediately. If *immediate* is true, the queue is terminated immediately. The queue is drained to be completely empty. The count of - unfinished tasks is reduced by number drained but without calling + unfinished tasks is reduced by the number drained but without calling :meth:`~Queue.task_done`. All callers of :meth:`~Queue.join` are unblocked even if the unfinished tasks is more than zero. In addition, blocked callers of From 478cbab6be5db791187f7217eb773d3513745948 Mon Sep 17 00:00:00 2001 From: Raymond Hettinger Date: Thu, 24 Jul 2025 01:09:01 -0600 Subject: [PATCH 05/14] Focus discussion on effects instead of process --- Doc/library/queue.rst | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/Doc/library/queue.rst b/Doc/library/queue.rst index 025feee61847f5..72e1e7791773a1 100644 --- a/Doc/library/queue.rst +++ b/Doc/library/queue.rst @@ -257,12 +257,11 @@ until empty or terminated immediately (a hard shutdown). raise :exc:`ShutDown`. If *immediate* is true, the queue is terminated immediately. - The queue is drained to be completely empty. The count of - unfinished tasks is reduced by the number drained but without calling - :meth:`~Queue.task_done`. All callers of :meth:`~Queue.join` - are unblocked even if the unfinished tasks is more than zero. - In addition, blocked callers of - :meth:`~Queue.get` are unblocked and will raise :exc:`ShutDown`. + The queue is drained to be completely empty. All callers of + :meth:`~Queue.join` are unblocked regardless of the number + of unfinished tasks. Blocked callers of :meth:`~Queue.get` + are unblocked and will raise :exc:`ShutDown` because the + queue is empty. Use caution when using :meth:`~Queue.join` with *immediate* set to true. This unblocks the join even when no work has been done From 2c20579edb8752d4ffd2ff462c8fdeb08e01e8bc Mon Sep 17 00:00:00 2001 From: Raymond Hettinger Date: Thu, 24 Jul 2025 01:10:42 -0600 Subject: [PATCH 06/14] Remove distraction about unblocking blocked gets for an empty queue --- Doc/library/queue.rst | 4 ---- 1 file changed, 4 deletions(-) diff --git a/Doc/library/queue.rst b/Doc/library/queue.rst index 72e1e7791773a1..597018b426ff73 100644 --- a/Doc/library/queue.rst +++ b/Doc/library/queue.rst @@ -246,10 +246,6 @@ until empty or terminated immediately (a hard shutdown). down normally with calls :meth:`~Queue.get` to extract tasks that have already been loaded. - If the shutdown occurs during the brief window where the queue still - has data and callers to :meth:`~Queue.get` are blocked, those callers - will be unblocked. - And if :meth:`~Queue.task_done` is called for each remaining task, a pending :meth:`~Queue.join` will be unblocked normally. From 4d14799b5bd6e1ba248d2819fadfb66e287171a0 Mon Sep 17 00:00:00 2001 From: Raymond Hettinger Date: Thu, 24 Jul 2025 01:12:18 -0600 Subject: [PATCH 07/14] Update the docstring --- Lib/queue.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/queue.py b/Lib/queue.py index 25beb46e30d6bd..efb595fe03d08c 100644 --- a/Lib/queue.py +++ b/Lib/queue.py @@ -240,8 +240,8 @@ def shutdown(self, immediate=False): 'immediate' to True to make gets raise immediately instead. All blocked callers of put() and get() will be unblocked. If - 'immediate', a task is marked as done for each item remaining in - the queue, which may unblock callers of join(). + 'immediate', callers of join() are unblocked regardless of + the number of unfinished tasks. ''' with self.mutex: self.is_shutdown = True From 6740c3f808d89a49ac0cf51917637545c6fd8b9d Mon Sep 17 00:00:00 2001 From: Raymond Hettinger Date: Thu, 24 Jul 2025 01:18:17 -0600 Subject: [PATCH 08/14] Update asyncio docstring --- Lib/asyncio/queues.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Lib/asyncio/queues.py b/Lib/asyncio/queues.py index 2f3865114a84f9..6b52bbdbc89de1 100644 --- a/Lib/asyncio/queues.py +++ b/Lib/asyncio/queues.py @@ -257,8 +257,8 @@ def shutdown(self, immediate=False): 'immediate' to True to make gets raise immediately instead. All blocked callers of put() and get() will be unblocked. If - 'immediate', a task is marked as done for each item remaining in - the queue, which may unblock callers of join(). + 'immediate', unblock callers of join() regardless of the + number of unfinished tasks. """ self._is_shutdown = True if immediate: From af1744e1956694dd8533909a67f037d5a0e88fbe Mon Sep 17 00:00:00 2001 From: Raymond Hettinger Date: Thu, 24 Jul 2025 01:22:09 -0600 Subject: [PATCH 09/14] Update asynico main docs --- Doc/library/asyncio-queue.rst | 37 +++++++++++++++++++++++------------ 1 file changed, 25 insertions(+), 12 deletions(-) diff --git a/Doc/library/asyncio-queue.rst b/Doc/library/asyncio-queue.rst index d99213aa81d53e..8954b7e655c7cd 100644 --- a/Doc/library/asyncio-queue.rst +++ b/Doc/library/asyncio-queue.rst @@ -102,17 +102,33 @@ Queue .. method:: shutdown(immediate=False) - Shut down the queue, making :meth:`~Queue.get` and :meth:`~Queue.put` - raise :exc:`QueueShutDown`. + Put a :class:`Queue` instance into a shutdown mode. - By default, :meth:`~Queue.get` on a shut down queue will only - raise once the queue is empty. Set *immediate* to true to make - :meth:`~Queue.get` raise immediately instead. + The queue can no longer grow. + Future calls to :meth:`~Queue.put` raise :exc:`ShutDown`. + Currently blocked callers of :meth:`~Queue.put` will be unblocked + and will raise :exc:`ShutDown` in the formerly blocked thread. - All blocked callers of :meth:`~Queue.put` and :meth:`~Queue.get` - will be unblocked. If *immediate* is true, a task will be marked - as done for each remaining item in the queue, which may unblock - callers of :meth:`~Queue.join`. + If *immediate* is false (the default), the queue can be wound + down normally with calls :meth:`~Queue.get` to extract tasks + that have already been loaded. + + And if :meth:`~Queue.task_done` is called for each remaining task, a + pending :meth:`~Queue.join` will be unblocked normally. + + Once the queue is empty, future calls to :meth:`~Queue.get` will + raise :exc:`ShutDown`. + + If *immediate* is true, the queue is terminated immediately. + The queue is drained to be completely empty. All callers of + :meth:`~Queue.join` are unblocked regardless of the number + of unfinished tasks. Blocked callers of :meth:`~Queue.get` + are unblocked and will raise :exc:`ShutDown` because the + queue is empty. + + Use caution when using :meth:`~Queue.join` with *immediate* set + to true. This unblocks the join even when no work has been done + on the tasks, violating the usual invariant for joining a queue. .. versionadded:: 3.13 @@ -129,9 +145,6 @@ Queue call was received for every item that had been :meth:`~Queue.put` into the queue). - ``shutdown(immediate=True)`` calls :meth:`task_done` for each - remaining item in the queue. - Raises :exc:`ValueError` if called more times than there were items placed in the queue. From 12d1ea96edf826373a5a759cc86160a58464e1c0 Mon Sep 17 00:00:00 2001 From: Raymond Hettinger Date: Thu, 24 Jul 2025 03:19:27 -0600 Subject: [PATCH 10/14] Match the exception used by asyncio --- Doc/library/asyncio-queue.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/Doc/library/asyncio-queue.rst b/Doc/library/asyncio-queue.rst index 8954b7e655c7cd..dbeb1ad5778681 100644 --- a/Doc/library/asyncio-queue.rst +++ b/Doc/library/asyncio-queue.rst @@ -105,9 +105,9 @@ Queue Put a :class:`Queue` instance into a shutdown mode. The queue can no longer grow. - Future calls to :meth:`~Queue.put` raise :exc:`ShutDown`. + Future calls to :meth:`~Queue.put` raise :exc:`QueueShutDown`. Currently blocked callers of :meth:`~Queue.put` will be unblocked - and will raise :exc:`ShutDown` in the formerly blocked thread. + and will raise :exc:`QueueShutDown` in the formerly blocked thread. If *immediate* is false (the default), the queue can be wound down normally with calls :meth:`~Queue.get` to extract tasks @@ -117,13 +117,13 @@ Queue pending :meth:`~Queue.join` will be unblocked normally. Once the queue is empty, future calls to :meth:`~Queue.get` will - raise :exc:`ShutDown`. + raise :exc:`QueueShutDown`. If *immediate* is true, the queue is terminated immediately. The queue is drained to be completely empty. All callers of :meth:`~Queue.join` are unblocked regardless of the number of unfinished tasks. Blocked callers of :meth:`~Queue.get` - are unblocked and will raise :exc:`ShutDown` because the + are unblocked and will raise :exc:`QueueShutDown` because the queue is empty. Use caution when using :meth:`~Queue.join` with *immediate* set From 4620deb0e75d0088e87f7772e10bb2186a4c7b09 Mon Sep 17 00:00:00 2001 From: Raymond Hettinger Date: Thu, 24 Jul 2025 03:31:12 -0600 Subject: [PATCH 11/14] Update task_done docstrings --- Lib/asyncio/queues.py | 3 --- Lib/queue.py | 3 --- 2 files changed, 6 deletions(-) diff --git a/Lib/asyncio/queues.py b/Lib/asyncio/queues.py index 6b52bbdbc89de1..e5d6f2e4b61e17 100644 --- a/Lib/asyncio/queues.py +++ b/Lib/asyncio/queues.py @@ -227,9 +227,6 @@ def task_done(self): been processed (meaning that a task_done() call was received for every item that had been put() into the queue). - shutdown(immediate=True) calls task_done() for each remaining item in - the queue. - Raises ValueError if called more times than there were items placed in the queue. """ diff --git a/Lib/queue.py b/Lib/queue.py index efb595fe03d08c..c90de8edc76c34 100644 --- a/Lib/queue.py +++ b/Lib/queue.py @@ -80,9 +80,6 @@ def task_done(self): have been processed (meaning that a task_done() call was received for every item that had been put() into the queue). - shutdown(immediate=True) calls task_done() for each remaining item in - the queue. - Raises a ValueError if called more times than there were items placed in the queue. ''' From 2ab910ed6711df65db68bc7d65fc730b2d1079e7 Mon Sep 17 00:00:00 2001 From: Raymond Hettinger Date: Thu, 24 Jul 2025 10:17:02 -0600 Subject: [PATCH 12/14] Fix word order --- Doc/library/asyncio-queue.rst | 2 +- Doc/library/queue.rst | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/library/asyncio-queue.rst b/Doc/library/asyncio-queue.rst index dbeb1ad5778681..963bc1fb82c12f 100644 --- a/Doc/library/asyncio-queue.rst +++ b/Doc/library/asyncio-queue.rst @@ -110,7 +110,7 @@ Queue and will raise :exc:`QueueShutDown` in the formerly blocked thread. If *immediate* is false (the default), the queue can be wound - down normally with calls :meth:`~Queue.get` to extract tasks + down normally with :meth:`~Queue.get` calls to extract tasks that have already been loaded. And if :meth:`~Queue.task_done` is called for each remaining task, a diff --git a/Doc/library/queue.rst b/Doc/library/queue.rst index 597018b426ff73..9c32d95b44d07b 100644 --- a/Doc/library/queue.rst +++ b/Doc/library/queue.rst @@ -243,7 +243,7 @@ until empty or terminated immediately (a hard shutdown). and will raise :exc:`ShutDown` in the formerly blocked thread. If *immediate* is false (the default), the queue can be wound - down normally with calls :meth:`~Queue.get` to extract tasks + down normally with :meth:`~Queue.get` calls to extract tasks that have already been loaded. And if :meth:`~Queue.task_done` is called for each remaining task, a From 7ad150b7c244ffed322a0983382cfd9add327db0 Mon Sep 17 00:00:00 2001 From: Raymond Hettinger Date: Thu, 24 Jul 2025 10:48:48 -0600 Subject: [PATCH 13/14] Add missing subsection header --- Doc/library/queue.rst | 3 +++ 1 file changed, 3 insertions(+) diff --git a/Doc/library/queue.rst b/Doc/library/queue.rst index 9c32d95b44d07b..8202abdd3c1351 100644 --- a/Doc/library/queue.rst +++ b/Doc/library/queue.rst @@ -201,6 +201,9 @@ fully processed by daemon consumer threads. count of unfinished tasks drops to zero, :meth:`join` unblocks. +Waiting for task completion +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + Example of how to wait for enqueued tasks to be completed:: import threading From 7de4f73196c29581a9efc0181cf163c0a485d51b Mon Sep 17 00:00:00 2001 From: Raymond Hettinger Date: Thu, 24 Jul 2025 10:53:10 -0600 Subject: [PATCH 14/14] Replace parenthetical with a prepositional phrase. --- Doc/library/queue.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/library/queue.rst b/Doc/library/queue.rst index 8202abdd3c1351..6dcf06aab00295 100644 --- a/Doc/library/queue.rst +++ b/Doc/library/queue.rst @@ -234,7 +234,7 @@ Terminating queues ^^^^^^^^^^^^^^^^^^ When no longer needed, :class:`Queue` objects can be wound down -until empty or terminated immediately (a hard shutdown). +until empty or terminated immediately with a hard shutdown. .. method:: Queue.shutdown(immediate=False)