extmod/asyncio: Add `Task` methods from CPython #13000

imnotjames · 2023-11-17T06:08:35Z

This updates the way that Task is defined in the C module for asyncio which implements tasks & task queues. The goal is to bring asyncio in MicroPython slightly closer to CPython.

This adds the following methods to Task:

get_coro() - how CPython exposes the coroutine backing the task (CPython Docs)
result() - a helper method from CPython for returning the successful response (CPython Docs)
exception() - a helper method from CPython for returning the raised values (CPython Docs)
cancelled() - helper for true / false if the task has been cancelled (CPython Docs)
add_done_callback() - adds a callback to the task for completion or if already complete executes it (CPython docs)
remove_done_callback() - removes a specific "done" callback from the task (CPython docs)

~~Adds a hash unary operation so Tasks can be added to sets like in CPython.~~

~~Also adds set_result and set_exception for consistency but neither of these do anything - they raise a RuntimeError because they aren't supported on Tasks. 🤷~~

github-actions · 2023-11-17T06:14:15Z

Code size report:

   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64: +1760 +0.220% standard[incl +224(data)]
      stm32:  +620 +0.158% PYBV10
     mimxrt:  +672 +0.185% TEENSY40
        rp2:  +688 +0.210% RPI_PICO
       samd:  +664 +0.253% ADAFRUIT_ITSYBITSY_M4_EXPRESS

codecov · 2023-11-17T07:05:46Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (a968888) 98.43% compared to head (8d260d8) 98.43%.
Report is 13 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master   #13000   +/-   ##
=======================================
  Coverage   98.43%   98.43%           
=======================================
  Files         158      158           
  Lines       20978    21027   +49     
=======================================
+ Hits        20649    20698   +49     
  Misses        329      329

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

extmod/modasyncio.c

Adds methods that are in CPython, such as `exception`, `result`, `get_coro`, `cancelled`, `add_done_callback`, and `remove_done_callback`. Also adds support for the unary hash so tasks may be collected in a python set. Signed-off-by: James Ward <james@notjam.es>

Signed-off-by: James Ward <james@notjam.es>

andrewleech · 2023-11-19T20:47:48Z

I personally wouldn't worry about adding:

Also adds set_result and set_exception for consistency but neither of these do anything - they raise a RuntimeError because they aren't supported on Tasks.

Adding even function names for these costs space, and not adding them will just raise an exception anyway, it's really not much different!

imnotjames · 2023-11-19T22:04:50Z

I personally wouldn't worry about adding:

Also adds set_result and set_exception for consistency but neither of these do anything - they raise a RuntimeError because they aren't supported on Tasks.

Adding even function names for these costs space, and not adding them will just raise an exception anyway, it's really not much different!

I guess so. My concern was that the exception would be a different kind of error but that's probably ok.

Signed-off-by: James Ward <james@notjam.es>

imnotjames · 2023-11-20T01:13:59Z

Okay, dropped those functions and fixed the commit messages.

dpgeorge · 2023-11-22T05:39:31Z

Thanks for the contribution.

MicroPython aims to be compact so it can run on devices without many resources. As such, these kinds of methods/functions were left out of the asyncio implementation on purpose (among many others).

That said, we can add them if they are important and useful.

What motivated you to add these particular features? Do they add anything that you can't already do?

I don't think the done-callback functions are really needed. MicroPython doesn't support any kind of callback in asyncio. Instead you can wait on the task to complete then run the required code.

dpgeorge · 2023-11-22T05:40:44Z

extmod/asyncio/task.py

+
+    def add_done_callback(self, callback):
+        if not self.state:
+            callback(self, self.data)


This signature of the callback does not match CPython. In CPython the callback takes only one argument.

That's right - I'm hoping to follow up to bring this in line with CPython in cases where only one argument is accepted in the function in a follow up PR.

However, changing the callback signature here would mean it's different from where it gets called elsewhere.

I could add a comment to clarify that if it helps?

dpgeorge · 2023-11-22T05:42:19Z

extmod/modasyncio.c

@@ -255,6 +387,15 @@ STATIC void task_attr(mp_obj_t self_in, qstr attr, mp_obj_t *dest) {
    }
 }

+STATIC mp_obj_t task_unary_op(mp_unary_op_t op, mp_obj_t o_in) {


Hashing of Task should actually work, since commit eaccaa3. So I think this bit of code can be removed.

I'll confirm it without this, it's very likely that's the case!

Yup, seems fine with that patch, awesome. Removed that change from this PR.

peterhinch · 2023-11-22T10:08:23Z

MicroPython doesn't support any kind of callback in asyncio. Instead you can wait on the task to complete then run the required code.

Definitely +1.

imnotjames · 2023-11-22T19:24:03Z

Thanks for the contribution.

Of course! :)

MicroPython aims to be compact so it can run on devices without many resources. As such, these kinds of methods/functions were left out of the asyncio implementation on purpose (among many others).

Compact is a great goal, and I do recognize that a lot of effort was put in to make asyncio as small as possible -- and also a lot of effort to make the compatibility with the CPython asyncio implementation as high as possible within that constraint.

That said, we can add them if they are important and useful.

What motivated you to add these particular features? Do they add anything that you can't already do?

All of these features expose existing "private" APIs (plumbing?) via the CPython public API (porcelain?). So everything can technically be done today - but in a way that's more brittle.

In particular, this PR heavily simplifies understanding if a task is cancelled or not, and if it's "done" what the result of it running was. It's not documented anywhere how the state or data properties works and I think that's intentional - it's internal aspects of the implementation that work well but shouldn't be touched by other parties.

The use case I had for this was to implement the wait and other waiting primitives in CPython as well as Task groups via these kinds of features.

The example CPython gives for tasks groups is as such:

async with asyncio.TaskGroup() as tg:
  task1 = tg.create_task(some_coro(...))
  task2 = tg.create_task(another_coro(...))
print(f"Both tasks have completed now: {task1.result()}, {task2.result()}")

Where the result is retrieved via task.result()

I don't think the done-callback functions are really needed. MicroPython doesn't support any kind of callback in asyncio.

I had added them because micropython kind of supports them via state (if you squint hard enough) but it's not clear without looking at the internals that they work that way, nor the rules you would need to follow so as not to break the standard async/await functionality. This encodes some of those rules into the add_done_callback and remove_done_callback methods.

Instead you can wait on the task to complete then run the required code.

If you await the task you'll get them serially rather than the result in parallel. The example case I have is supporting the waiting primitives on multiple tasks with the FIRST_COMPLETED or FIRST_EXCEPTION option.

Signed-off-by: James Ward <james@notjam.es>

peterhinch · 2023-11-23T16:11:16Z

For task groups see #8791. A very useful enhancement.

projectgus · 2024-03-07T23:48:58Z

This is an automated heads-up that we've just merged a Pull Request
that removes the STATIC macro from MicroPython's C API.

See #13763

A search suggests this PR might apply the STATIC macro to some C code. If it
does, then next time you rebase the PR (or merge from master) then you should
please replace all the STATIC keywords with static.

Although this is an automated message, feel free to @-reply to me directly if
you have any questions about this.

imnotjames force-pushed the feat/upython/async-tasks branch 2 times, most recently from 964a9a9 to f029578 Compare November 17, 2023 06:24

imnotjames changed the title ~~feat: add Task methods to make _asyncio more similar to cpython~~ extmod/asyncio: add Task methods to make _asyncio more similar to cpython Nov 17, 2023

imnotjames changed the title ~~extmod/asyncio: add Task methods to make _asyncio more similar to cpython~~ extmod/asyncio: Add Task methods from CPython Nov 17, 2023

imnotjames force-pushed the feat/upython/async-tasks branch 7 times, most recently from da05a15 to fb170d9 Compare November 18, 2023 18:30

imnotjames commented Nov 18, 2023

View reviewed changes

extmod/modasyncio.c Outdated Show resolved Hide resolved

imnotjames marked this pull request as ready for review November 18, 2023 18:33

imnotjames force-pushed the feat/upython/async-tasks branch 2 times, most recently from 921984b to ae0905c Compare November 18, 2023 19:14

imnotjames mentioned this pull request Nov 19, 2023

feat: add Task methods to make _asyncio more similar to cpython adafruit/circuitpython#8576

Closed

2 tasks

imnotjames added 3 commits November 19, 2023 14:41

extmod/asyncio: Add cpython task methods.

df39e51

Signed-off-by: James Ward <james@notjam.es>

tests/extmod: Add asyncio tests for new task features.

e1ef6ed

Signed-off-by: James Ward <james@notjam.es>

imnotjames force-pushed the feat/upython/async-tasks branch from ae0905c to e1ef6ed Compare November 19, 2023 19:41

imnotjames force-pushed the feat/upython/async-tasks branch 2 times, most recently from d65d495 to a97ef0d Compare November 20, 2023 00:11

imnotjames added 2 commits November 19, 2023 19:14

tests/extmod: Drop set_result and set_exception tests.

5ecb859

Signed-off-by: James Ward <james@notjam.es>

extmod/modasyncio.c: Drop set_result and set_exception.

92092f0

Signed-off-by: James Ward <james@notjam.es>

imnotjames force-pushed the feat/upython/async-tasks branch 2 times, most recently from 0d98d3b to db9db03 Compare November 20, 2023 00:48

extmod/asyncio: Drop set_exception and set_result.

d23edc3

Signed-off-by: James Ward <james@notjam.es>

imnotjames force-pushed the feat/upython/async-tasks branch from db9db03 to d23edc3 Compare November 20, 2023 00:52

dpgeorge added the extmod Relates to extmod/ directory in source label Nov 21, 2023

imnotjames mentioned this pull request Nov 22, 2023

feat: add Task methods to make _asyncio more similar to cpython adafruit/circuitpython#8647

Draft

dpgeorge reviewed Nov 22, 2023

View reviewed changes

extmod/modasyncio.c: Rely on the generic hash during runtime.

8d260d8

Signed-off-by: James Ward <james@notjam.es>

imnotjames force-pushed the feat/upython/async-tasks branch from d8459c3 to 8d260d8 Compare November 23, 2023 04:45

dpgeorge mentioned this pull request Jan 22, 2024

extmod/asyncio: Support gather of tasks that finish early. #13474

Merged

Gadgetoid mentioned this pull request Feb 29, 2024

global: Remove the STATIC macro. #13763

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

extmod/asyncio: Add `Task` methods from CPython #13000

extmod/asyncio: Add `Task` methods from CPython #13000

imnotjames commented Nov 17, 2023 •

edited

Loading

github-actions bot commented Nov 17, 2023 •

edited

Loading

codecov bot commented Nov 17, 2023 •

edited

Loading

andrewleech commented Nov 19, 2023

imnotjames commented Nov 19, 2023

imnotjames commented Nov 20, 2023

dpgeorge commented Nov 22, 2023

dpgeorge Nov 22, 2023

imnotjames Nov 22, 2023

imnotjames Nov 22, 2023

dpgeorge Nov 22, 2023

imnotjames Nov 22, 2023

imnotjames Nov 23, 2023 •

edited

Loading

peterhinch commented Nov 22, 2023

imnotjames commented Nov 22, 2023 •

edited

Loading

peterhinch commented Nov 23, 2023

projectgus commented Mar 7, 2024

extmod/asyncio: Add Task methods from CPython #13000

Are you sure you want to change the base?

extmod/asyncio: Add Task methods from CPython #13000

Conversation

imnotjames commented Nov 17, 2023 • edited Loading

github-actions bot commented Nov 17, 2023 • edited Loading

codecov bot commented Nov 17, 2023 • edited Loading

Codecov Report

andrewleech commented Nov 19, 2023

imnotjames commented Nov 19, 2023

imnotjames commented Nov 20, 2023

dpgeorge commented Nov 22, 2023

dpgeorge Nov 22, 2023

Choose a reason for hiding this comment

imnotjames Nov 22, 2023

Choose a reason for hiding this comment

imnotjames Nov 22, 2023

Choose a reason for hiding this comment

dpgeorge Nov 22, 2023

Choose a reason for hiding this comment

imnotjames Nov 22, 2023

Choose a reason for hiding this comment

imnotjames Nov 23, 2023 • edited Loading

Choose a reason for hiding this comment

peterhinch commented Nov 22, 2023

imnotjames commented Nov 22, 2023 • edited Loading

peterhinch commented Nov 23, 2023

projectgus commented Mar 7, 2024

extmod/asyncio: Add `Task` methods from CPython #13000

extmod/asyncio: Add `Task` methods from CPython #13000

imnotjames commented Nov 17, 2023 •

edited

Loading

github-actions bot commented Nov 17, 2023 •

edited

Loading

codecov bot commented Nov 17, 2023 •

edited

Loading

imnotjames Nov 23, 2023 •

edited

Loading

imnotjames commented Nov 22, 2023 •

edited

Loading