Skip to content

bpo-34270: added the ability to name asyncio tasks #8547

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 22 commits into from
Aug 8, 2018
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
19204e1
bpo-34270: added the "name" attribute to asyncio tasks
agronholm Jul 29, 2018
25939b9
Made changes requested by the reviewers
agronholm Jul 30, 2018
369eba9
Added whatsnew entry
agronholm Jul 30, 2018
ce8e014
Changed unsigned long long to uint64_t
agronholm Jul 30, 2018
11258ed
Added explanation for the use of itertools.count() in asyncio.tasks
agronholm Jul 30, 2018
1f3dda6
Reverted task factory API changes and added Task.set_name()
agronholm Aug 1, 2018
de125ce
Updated the changelog entries to match the latest changes
agronholm Aug 1, 2018
98ebde1
Fixed PEP 8 issue
agronholm Aug 1, 2018
5aadb36
Fixed C function name for Task.get_name()
agronholm Aug 1, 2018
ad5c572
Added back the "name" parameter to create_task() and asyncio.Task()
agronholm Aug 4, 2018
5fba009
Added back the "name" parameter to create_task()
agronholm Aug 4, 2018
fc02fd4
Updated the news items
agronholm Aug 4, 2018
77f2147
Added the "name" argument back to AbstractEventLoop.create_task()
agronholm Aug 4, 2018
645b638
Updated the news items
agronholm Aug 4, 2018
b352983
More documentation updates
agronholm Aug 4, 2018
56e2883
Refactored the task name setting logic into a function
agronholm Aug 4, 2018
d655f24
Documentation tweaks
agronholm Aug 4, 2018
dd4d4a5
Made the changes requested by the reviewer
agronholm Aug 8, 2018
860d67a
Fixed segfault if "name" is already a string
agronholm Aug 8, 2018
2cca5e7
Fixed the new test to use a custom factory
agronholm Aug 8, 2018
a1be2e0
Fixed last two nits
agronholm Aug 8, 2018
2ee3972
Fixed incompatible return value from set_name()
agronholm Aug 8, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion Doc/library/asyncio-eventloop.rst
Original file line number Diff line number Diff line change
Expand Up @@ -250,7 +250,7 @@ Futures
Tasks
-----

.. method:: AbstractEventLoop.create_task(coro)
.. method:: AbstractEventLoop.create_task(coro, \*, name=None)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We also need to update the documentation of set_task_factory

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point.


Schedule the execution of a :ref:`coroutine object <coroutine>`: wrap it in
a future. Return a :class:`Task` object.
Expand All @@ -259,8 +259,14 @@ Tasks
interoperability. In this case, the result type is a subclass of
:class:`Task`.

If the *name* argument is provided and not ``None``, it is set as the name
of the task using :meth:`Task.set_name`.

.. versionadded:: 3.4.2

.. versionchanged:: 3.8
Added the ``name`` parameter.

.. method:: AbstractEventLoop.set_task_factory(factory)

Set a task factory that will be used by
Expand Down
36 changes: 33 additions & 3 deletions Doc/library/asyncio-task.rst
Original file line number Diff line number Diff line change
Expand Up @@ -387,18 +387,24 @@ with the result.
Task
----

.. function:: create_task(coro)
.. function:: create_task(coro, \*, name=None)

Wrap a :ref:`coroutine <coroutine>` *coro* into a task and schedule
its execution. Return the task object.
its execution. Return the task object.

If *name* is not ``None``, it is set as the name of the task using
:meth:`Task.set_name`.

The task is executed in :func:`get_running_loop` context,
:exc:`RuntimeError` is raised if there is no running loop in
current thread.

.. versionadded:: 3.7

.. class:: Task(coro, \*, loop=None)
.. versionchanged:: 3.8
Added the ``name`` parameter.

.. class:: Task(coro, \*, loop=None, name=None)

A unit for concurrent running of :ref:`coroutines <coroutine>`,
subclass of :class:`Future`.
Expand Down Expand Up @@ -438,6 +444,9 @@ Task
.. versionchanged:: 3.7
Added support for the :mod:`contextvars` module.

.. versionchanged:: 3.8
Added the ``name`` parameter.

.. classmethod:: all_tasks(loop=None)

Return a set of all tasks for an event loop.
Expand Down Expand Up @@ -504,6 +513,27 @@ Task
get_stack(). The file argument is an I/O stream to which the output
is written; by default output is written to sys.stderr.

.. method:: get_name()

Return the name of the task.

If no name has been explicitly assigned to the task, the default
``Task`` implementation generates a default name during instantiation.

.. versionadded:: 3.8

.. method:: set_name(value)

Set the name of the task.

The *value* argument can be any object, which is then converted to a
string.

In the default ``Task`` implementation, the name will be visible in the
:func:`repr` output of a task object.

.. versionadded:: 3.8


Example: Parallel execution of tasks
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Expand Down
7 changes: 7 additions & 0 deletions Doc/whatsnew/3.8.rst
Original file line number Diff line number Diff line change
Expand Up @@ -243,6 +243,13 @@ Changes in the Python API
* ``PyGC_Head`` struct is changed completely. All code touched the
struct member should be rewritten. (See :issue:`33597`)

* Asyncio tasks can now be named, either by passing the ``name`` keyword
argument to :func:`asyncio.create_task` or
the :meth:`~asyncio.AbstractEventLoop.create_task` event loop method, or by
calling the :meth:`~asyncio.Task.set_name` method on the task object. The
task name is visible in the ``repr()`` output of :class:`asyncio.Task` and
can also be retrieved using the :meth:`~asyncio.Task.get_name` method.


CPython bytecode changes
------------------------
Expand Down
6 changes: 4 additions & 2 deletions Lib/asyncio/base_events.py
Original file line number Diff line number Diff line change
Expand Up @@ -381,18 +381,20 @@ def create_future(self):
"""Create a Future object attached to the loop."""
return futures.Future(loop=self)

def create_task(self, coro):
def create_task(self, coro, *, name=None):
"""Schedule a coroutine object.

Return a task object.
"""
self._check_closed()
if self._task_factory is None:
task = tasks.Task(coro, loop=self)
task = tasks.Task(coro, loop=self, name=name)
if task._source_traceback:
del task._source_traceback[-1]
else:
task = self._task_factory(self, coro)
tasks._set_task_name(task, name)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line should be unindented or you should pass the name to task.Task(..) few lines above. Please add a test that Tasks get custom names when created via loop.create_task(name=..) with a standard factory.


return task

def set_task_factory(self, factory):
Expand Down
6 changes: 4 additions & 2 deletions Lib/asyncio/base_tasks.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,13 @@ def _task_repr_info(task):
# replace status
info[0] = 'cancelling'

info.insert(1, 'name=%r' % task.get_name())

coro = coroutines._format_coroutine(task._coro)
info.insert(1, f'coro=<{coro}>')
info.insert(2, f'coro=<{coro}>')

if task._fut_waiter is not None:
info.insert(2, f'wait_for={task._fut_waiter!r}')
info.insert(3, f'wait_for={task._fut_waiter!r}')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, feel free to make another pull request to refactor this code. I don't like us modifying some list object with insert() calls.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How would you prefer that we display the name of the task in repr() then?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be great to refactor the code that formats things (tasks, callbacks, reprs) etc somehow. Right now it feels hacky and it's one of the areas of asyncio code that I'd like to be rewritten to be clearer. Discussing that is slightly out of the scope of this PR though, but if you have any ideas how to avoid mutating lists and having a non-linear formatting logic I'd be happy to accept a new PR. For this PR, your current code is totally fine!

return info


Expand Down
2 changes: 1 addition & 1 deletion Lib/asyncio/events.py
Original file line number Diff line number Diff line change
Expand Up @@ -277,7 +277,7 @@ def create_future(self):

# Method scheduling a coroutine object: create a task.

def create_task(self, coro):
def create_task(self, coro, *, name=None):
raise NotImplementedError

# Methods for interacting with threads.
Expand Down
35 changes: 32 additions & 3 deletions Lib/asyncio/tasks.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
import contextvars
import functools
import inspect
import itertools
import types
import warnings
import weakref
Expand All @@ -23,6 +24,11 @@
from . import futures
from .coroutines import coroutine

# Helper to generate new task names
# This uses itertools.count() instead of a "+= 1" operation because the latter
# is not thread safe. See bpo-11866 for a longer explanation.
_task_name_counter = itertools.count(1).__next__


def current_task(loop=None):
"""Return a currently executed task."""
Expand All @@ -48,6 +54,16 @@ def _all_tasks_compat(loop=None):
return {t for t in _all_tasks if futures._get_loop(t) is loop}


def _set_task_name(task, name):
if name is not None:
try:
set_name = task.set_name
except AttributeError:
pass
else:
set_name(name)


class Task(futures._PyFuture): # Inherit Python Task implementation
# from a Python Future implementation.

Expand Down Expand Up @@ -94,7 +110,7 @@ def all_tasks(cls, loop=None):
stacklevel=2)
return _all_tasks_compat(loop)

def __init__(self, coro, *, loop=None):
def __init__(self, coro, *, loop=None, name=None):
super().__init__(loop=loop)
if self._source_traceback:
del self._source_traceback[-1]
Expand All @@ -104,6 +120,11 @@ def __init__(self, coro, *, loop=None):
self._log_destroy_pending = False
raise TypeError(f"a coroutine was expected, got {coro!r}")

if name is None:
self._name = f'Task-{_task_name_counter()}'
else:
self._name = str(name)

self._must_cancel = False
self._fut_waiter = None
self._coro = coro
Expand All @@ -126,6 +147,12 @@ def __del__(self):
def _repr_info(self):
return base_tasks._task_repr_info(self)

def get_name(self):
return self._name

def set_name(self, value):
self._name = str(value)

def set_result(self, result):
raise RuntimeError('Task does not support set_result operation')

Expand Down Expand Up @@ -312,13 +339,15 @@ def __wakeup(self, future):
Task = _CTask = _asyncio.Task


def create_task(coro):
def create_task(coro, *, name=None):
"""Schedule the execution of a coroutine object in a spawn task.

Return a Task object.
"""
loop = events.get_running_loop()
return loop.create_task(coro)
task = loop.create_task(coro)
_set_task_name(task, name)
return task


# wait() and as_completed() similar to those in PEP 3148.
Expand Down
28 changes: 28 additions & 0 deletions Lib/test/test_asyncio/test_base_events.py
Original file line number Diff line number Diff line change
Expand Up @@ -813,6 +813,34 @@ def create_task(self, coro):
task._log_destroy_pending = False
coro.close()

def test_create_named_task_with_default_factory(self):
async def test():
pass

loop = asyncio.new_event_loop()
task = loop.create_task(test(), name='test_task')
try:
self.assertEqual(task.get_name(), 'test_task')
finally:
loop.run_until_complete(task)
loop.close()

def test_create_named_task_with_custom_factory(self):
def task_factory(loop, coro):
return asyncio.Task(coro, loop=loop)

async def test():
pass

loop = asyncio.new_event_loop()
loop.set_task_factory(task_factory)
task = loop.create_task(test(), name='test_task')
try:
self.assertEqual(task.get_name(), 'test_task')
finally:
loop.run_until_complete(task)
loop.close()

def test_run_forever_keyboard_interrupt(self):
# Python issue #22601: ensure that the temporary task created by
# run_forever() consumes the KeyboardInterrupt and so don't log
Expand Down
55 changes: 48 additions & 7 deletions Lib/test/test_asyncio/test_tasks.py
Original file line number Diff line number Diff line change
Expand Up @@ -87,8 +87,8 @@ class BaseTaskTests:
Task = None
Future = None

def new_task(self, loop, coro):
return self.__class__.Task(coro, loop=loop)
def new_task(self, loop, coro, name='TestTask'):
return self.__class__.Task(coro, loop=loop, name=name)

def new_future(self, loop):
return self.__class__.Future(loop=loop)
Expand Down Expand Up @@ -295,28 +295,57 @@ def notmuch():
coro = format_coroutine(coro_qualname, 'running', src,
t._source_traceback, generator=True)
self.assertEqual(repr(t),
'<Task pending %s cb=[<Dummy>()]>' % coro)
"<Task pending name='TestTask' %s cb=[<Dummy>()]>" % coro)

# test cancelling Task
t.cancel() # Does not take immediate effect!
self.assertEqual(repr(t),
'<Task cancelling %s cb=[<Dummy>()]>' % coro)
"<Task cancelling name='TestTask' %s cb=[<Dummy>()]>" % coro)

# test cancelled Task
self.assertRaises(asyncio.CancelledError,
self.loop.run_until_complete, t)
coro = format_coroutine(coro_qualname, 'done', src,
t._source_traceback)
self.assertEqual(repr(t),
'<Task cancelled %s>' % coro)
"<Task cancelled name='TestTask' %s>" % coro)

# test finished Task
t = self.new_task(self.loop, notmuch())
self.loop.run_until_complete(t)
coro = format_coroutine(coro_qualname, 'done', src,
t._source_traceback)
self.assertEqual(repr(t),
"<Task finished %s result='abc'>" % coro)
"<Task finished name='TestTask' %s result='abc'>" % coro)

def test_task_repr_autogenerated(self):
@asyncio.coroutine
def notmuch():
return 123

t1 = self.new_task(self.loop, notmuch(), None)
t2 = self.new_task(self.loop, notmuch(), None)
self.assertNotEqual(repr(t1), repr(t2))

match1 = re.match("^<Task pending name='Task-(\d+)'", repr(t1))
self.assertIsNotNone(match1)
match2 = re.match("^<Task pending name='Task-(\d+)'", repr(t2))
self.assertIsNotNone(match2)

# Autogenerated task names should have monotonically increasing numbers
self.assertLess(int(match1.group(1)), int(match2.group(1)))
self.loop.run_until_complete(t1)
self.loop.run_until_complete(t2)

def test_task_repr_name_not_str(self):
@asyncio.coroutine
def notmuch():
return 123

t = self.new_task(self.loop, notmuch())
t.set_name({6})
self.assertEqual(t.get_name(), '{6}')
self.loop.run_until_complete(t)

def test_task_repr_coro_decorator(self):
self.loop.set_debug(False)
Expand Down Expand Up @@ -376,7 +405,7 @@ def notmuch():
t._source_traceback,
generator=not coroutines._DEBUG)
self.assertEqual(repr(t),
'<Task pending %s cb=[<Dummy>()]>' % coro)
"<Task pending name='TestTask' %s cb=[<Dummy>()]>" % coro)
self.loop.run_until_complete(t)

def test_task_repr_wait_for(self):
Expand Down Expand Up @@ -2260,6 +2289,18 @@ async def coro():

self.loop.run_until_complete(coro())

def test_bare_create_named_task(self):

async def coro_noop():
pass

async def coro():
task = asyncio.create_task(coro_noop(), name='No-op')
self.assertEqual(task.get_name(), 'No-op')
await task

self.loop.run_until_complete(coro())

def test_context_1(self):
cvar = contextvars.ContextVar('cvar', default='nope')

Expand Down
2 changes: 2 additions & 0 deletions Misc/ACKS
Original file line number Diff line number Diff line change
Expand Up @@ -573,6 +573,7 @@ Elliot Gorokhovsky
Hans de Graaff
Tim Graham
Kim Gräsman
Alex Grönholm
Nathaniel Gray
Eddy De Greef
Duane Griffin
Expand All @@ -594,6 +595,7 @@ Michael Guravage
Lars Gustäbel
Thomas Güttler
Jonas H.
Antti Haapala
Joseph Hackman
Barry Haddow
Philipp Hagemeister
Expand Down
Loading