Skip to content

[3.7] bpo-33649: More improvements (GH-9439) #9454

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Sep 20, 2018
Merged

[3.7] bpo-33649: More improvements (GH-9439) #9454

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 45 additions & 4 deletions Doc/library/asyncio-eventloop.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -749,7 +749,7 @@ Watching file descriptors
invoke *callback* with the specified arguments once *fd* is available for
writing.

Use :func:`functools.partial` :ref:`to pass keywords
Use :func:`functools.partial` :ref:`to pass keyword arguments
<asyncio-pass-keywords>` to *func*.

.. method:: loop.remove_writer(fd)
Expand Down Expand Up @@ -963,7 +963,7 @@ Unix signals
Raise :exc:`ValueError` if the signal number is invalid or uncatchable.
Raise :exc:`RuntimeError` if there is a problem setting up the handler.

Use :func:`functools.partial` :ref:`to pass keywords
Use :func:`functools.partial` :ref:`to pass keyword arguments
<asyncio-pass-keywords>` to *func*.

.. method:: loop.remove_signal_handler(sig)
Expand All @@ -990,11 +990,52 @@ Executing code in thread or process pools
The *executor* argument should be an :class:`concurrent.futures.Executor`
instance. The default executor is used if *executor* is ``None``.

Use :func:`functools.partial` :ref:`to pass keywords
<asyncio-pass-keywords>` to *func*.
Example::

import asyncio
import concurrent.futures

def blocking_io():
# File operations (such as logging) can block the
# event loop: run them in a thread pool.
with open('/dev/urandom', 'rb') as f:
return f.read(100)

def cpu_bound():
# CPU-bound operations will block the event loop:
# in general it is preferable to run them in a
# process pool.
return sum(i * i for i in range(10 ** 7))

async def main():
loop = asyncio.get_running_loop()

## Options:

# 1. Run in the default loop's executor:
result = await loop.run_in_executor(
None, blocking_io)
print('default thread pool', result)

# 2. Run in a custom thread pool:
with concurrent.futures.ThreadPoolExecutor() as pool:
result = await loop.run_in_executor(
pool, blocking_io)
print('custom thread pool', result)

# 3. Run in a custom process pool:
with concurrent.futures.ProcessPoolExecutor() as pool:
result = await loop.run_in_executor(
pool, cpu_bound)
print('custom process pool', result)

asyncio.run(main())

This method returns a :class:`asyncio.Future` object.

Use :func:`functools.partial` :ref:`to pass keyword arguments
<asyncio-pass-keywords>` to *func*.

.. versionchanged:: 3.5.3
:meth:`loop.run_in_executor` no longer configures the
``max_workers`` of the thread pool executor it creates, instead
Expand Down
174 changes: 131 additions & 43 deletions Doc/library/asyncio-task.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,50 +109,89 @@ To actually run a coroutine asyncio provides three main mechanisms:
Awaitables
==========

We say that an object is an *awaitable* object if it can be used
in an :keyword:`await` expression.
We say that an object is an **awaitable** object if it can be used
in an :keyword:`await` expression. Many asyncio APIs are designed to
accept awaitables.

There are three main types of *awaitable* objects:
**coroutines**, **Tasks**, and **Futures**.

.. rubric:: Coroutines and Tasks

Python coroutines are *awaitables*::
.. rubric:: Coroutines

Python coroutines are *awaitables* and therefore can be awaited from
other coroutines::

import asyncio

async def nested():
return 42

async def main():
# Will print "42":
print(await nested())
# Nothing happens if we just call "nested()".
# (a coroutine object is created but not awaited)
nested()

# Let's do it differently now and await it:
print(await nested()) # will print "42".

asyncio.run(main())

.. important::

In this documentation the term "coroutine" can be used for
two closely related concepts:

* a *coroutine function*: an :keyword:`async def` function;

* a *coroutine object*: an object returned by calling a
*coroutine function*.

asyncio also supports legacy :ref:`generator-based
<asyncio_generator_based_coro>` coroutines.


.. rubric:: Tasks

*Tasks* are used to schedule coroutines *concurrently*.
See the previous :ref:`section <coroutine>` for an introduction
to coroutines and tasks.

Note that in this documentation the term "coroutine" can be used for
two closely related concepts:
When a coroutine is wrapped into a *Task* with functions like
:func:`asyncio.create_task` the coroutine is automatically
scheduled to run soon::

import asyncio

async def nested():
return 42

* a *coroutine function*: an :keyword:`async def` function;
async def main():
# Schedule nested() to run soon concurrently
# with "main()".
task = asyncio.create_task(nested())

* a *coroutine object*: object returned by calling a
*coroutine function*.
# "task" can now be used to cancel "nested()", or
# can simply be awaited to wait until it is complete:
await task

asyncio.run(main())


.. rubric:: Futures

There is a dedicated section about the :ref:`asyncio Future object
<asyncio-futures>`, but the concept is fundamental to asyncio so
it needs a brief introduction in this section.
A :class:`Future` is a special **low-level** awaitable object that
represents an **eventual result** of an asynchronous operation.

When a Future object is *awaited* it means that the coroutine will
wait until the Future is resolved in some other place.

A Future is a special **low-level** awaitable object that represents
an **eventual result** of an asynchronous operation.
Future objects in asyncio are needed to allow callback-based code
to be used with async/await.

Normally, **there is no need** to create Future objects at the
Normally **there is no need** to create Future objects at the
application level code.

Future objects, sometimes exposed by libraries and some asyncio
APIs, should be awaited::
APIs, can be awaited::

async def main():
await function_that_returns_a_future_object()
Expand All @@ -163,6 +202,9 @@ APIs, should be awaited::
some_python_coroutine()
)

A good example of a low-level function that returns a Future object
is :meth:`loop.run_in_executor`.


Running an asyncio Program
==========================
Expand Down Expand Up @@ -192,8 +234,8 @@ Creating Tasks

.. function:: create_task(coro)

Wrap the *coro* :ref:`coroutine <coroutine>` into a Task and
schedule its execution. Return the Task object.
Wrap the *coro* :ref:`coroutine <coroutine>` into a :class:`Task`
and schedule its execution. Return the Task object.

The task is executed in the loop returned by :func:`get_running_loop`,
:exc:`RuntimeError` is raised if there is no running loop in
Expand Down Expand Up @@ -253,17 +295,17 @@ Sleeping
Running Tasks Concurrently
==========================

.. awaitablefunction:: gather(\*fs, loop=None, return_exceptions=False)
.. awaitablefunction:: gather(\*aws, loop=None, return_exceptions=False)

Run :ref:`awaitable objects <asyncio-awaitables>` in the *fs*
Run :ref:`awaitable objects <asyncio-awaitables>` in the *aws*
sequence *concurrently*.

If any awaitable in *fs* is a coroutine, it is automatically
If any awaitable in *aws* is a coroutine, it is automatically
scheduled as a Task.

If all awaitables are completed successfully, the result is an
aggregate list of returned values. The order of result values
corresponds to the order of awaitables in *fs*.
corresponds to the order of awaitables in *aws*.

If *return_exceptions* is ``True``, exceptions are treated the
same as successful results, and aggregated in the result list.
Expand All @@ -273,7 +315,7 @@ Running Tasks Concurrently
If ``gather`` is *cancelled*, all submitted awaitables
(that have not completed yet) are also *cancelled*.

If any Task or Future from the *fs* sequence is *cancelled*, it is
If any Task or Future from the *aws* sequence is *cancelled*, it is
treated as if it raised :exc:`CancelledError` -- the ``gather()``
call is **not** cancelled in this case. This is to prevent the
cancellation of one submitted Task/Future to cause other
Expand Down Expand Up @@ -323,13 +365,13 @@ Running Tasks Concurrently
Shielding Tasks From Cancellation
=================================

.. awaitablefunction:: shield(fut, \*, loop=None)
.. awaitablefunction:: shield(aw, \*, loop=None)

Protect an :ref:`awaitable object <asyncio-awaitables>`
from being :meth:`cancelled <Task.cancel>`.

*fut* can be a coroutine, a Task, or a Future-like object. If
*fut* is a coroutine it is automatically scheduled as a Task.
*aw* can be a coroutine, a Task, or a Future-like object. If
*aw* is a coroutine it is automatically scheduled as a Task.

The statement::

Expand Down Expand Up @@ -361,12 +403,12 @@ Shielding Tasks From Cancellation
Timeouts
========

.. coroutinefunction:: wait_for(fut, timeout, \*, loop=None)
.. coroutinefunction:: wait_for(aw, timeout, \*, loop=None)

Wait for the *fut* :ref:`awaitable <asyncio-awaitables>`
Wait for the *aw* :ref:`awaitable <asyncio-awaitables>`
to complete with a timeout.

If *fut* is a coroutine it is automatically scheduled as a Task.
If *aw* is a coroutine it is automatically scheduled as a Task.

*timeout* can either be ``None`` or a float or int number of seconds
to wait for. If *timeout* is ``None``, block until the future
Expand All @@ -381,7 +423,7 @@ Timeouts
The function will wait until the future is actually cancelled,
so the total wait time may exceed the *timeout*.

If the wait is cancelled, the future *fut* is also cancelled.
If the wait is cancelled, the future *aw* is also cancelled.

The *loop* argument is deprecated and scheduled for removal
in Python 4.0.
Expand Down Expand Up @@ -409,22 +451,22 @@ Timeouts
# timeout!

.. versionchanged:: 3.7
When *fut* is cancelled due to a timeout, ``wait_for`` waits
for *fut* to be cancelled. Previously, it raised
When *aw* is cancelled due to a timeout, ``wait_for`` waits
for *aw* to be cancelled. Previously, it raised
:exc:`asyncio.TimeoutError` immediately.


Waiting Primitives
==================

.. coroutinefunction:: wait(fs, \*, loop=None, timeout=None,\
.. coroutinefunction:: wait(aws, \*, loop=None, timeout=None,\
return_when=ALL_COMPLETED)

Run :ref:`awaitable objects <asyncio-awaitables>` in the *fs*
Run :ref:`awaitable objects <asyncio-awaitables>` in the *aws*
sequence concurrently and block until the condition specified
by *return_when*.

If any awaitable in *fs* is a coroutine, it is automatically
If any awaitable in *aws* is a coroutine, it is automatically
scheduled as a Task.

Returns two sets of Tasks/Futures: ``(done, pending)``.
Expand Down Expand Up @@ -465,12 +507,12 @@ Waiting Primitives

Usage::

done, pending = await asyncio.wait(fs)
done, pending = await asyncio.wait(aws)


.. function:: as_completed(fs, \*, loop=None, timeout=None)
.. function:: as_completed(aws, \*, loop=None, timeout=None)

Run :ref:`awaitable objects <asyncio-awaitables>` in the *fs*
Run :ref:`awaitable objects <asyncio-awaitables>` in the *aws*
set concurrently. Return an iterator of :class:`Future` objects.
Each Future object returned represents the earliest result
from the set of the remaining awaitables.
Expand All @@ -480,7 +522,7 @@ Waiting Primitives

Example::

for f in as_completed(fs):
for f in as_completed(aws):
earliest_result = await f
# ...

Expand Down Expand Up @@ -670,6 +712,52 @@ Task Object
A Task is *done* when the wrapped coroutine either returned
a value, raised an exception, or the Task was cancelled.

.. method:: result()

Return the result of the Task.

If the Task is *done*, the result of the wrapped coroutine
is returned (or if the coroutine raised an exception, that
exception is re-raised.)

If the Task has been *cancelled*, this method raises
a :exc:`CancelledError` exception.

If the Task's result isn't yet available, this method raises
a :exc:`InvalidStateError` exception.

.. method:: exception()

Return the exception of the Task.

If the wrapped coroutine raised an exception that exception
is returned. If the wrapped coroutine returned normally
this method returns ``None``.

If the Task has been *cancelled*, this method raises a
:exc:`CancelledError` exception.

If the Task isn't *done* yet, this method raises an
:exc:`InvalidStateError` exception.

.. method:: add_done_callback(callback, *, context=None)

Add a callback to be run when the Task is *done*.

This method should only be used in low-level callback-based code.

See the documentation of :meth:`Future.add_done_callback`
for more details.

.. method:: remove_done_callback(callback)

Remove *callback* from the callbacks list.

This method should only be used in low-level callback-based code.

See the documentation of :meth:`Future.remove_done_callback`
for more details.

.. method:: get_stack(\*, limit=None)

Return the list of stack frames for this Task.
Expand Down