Skip to content

Conversation

kevinkk525
Copy link
Contributor

This PR adds asyncio.shield() to uasyncio which prevents the shielded task from being cancelled if shield() gets cancelled.
It includes testcases too.
It does however not modify _uasyncio/tasks which is not written in python.

Also I don't know how happy you are with the test-case format I chose or with the way shield is integrated. Consider it a "proof-of-concept".

Additionally there is one test "test_cancel_wait_for_shield()" which gives a different order than CPython due to the differences in wait_for.

@kevinkk525
Copy link
Contributor Author

I just realized that my implementation is actually wrong because shield() needs to return a Task. Therefore it fails in some scenarios. Will update as soon as I found a fix.

@@ -128,6 +128,7 @@ def __init__(self, coro, globals=None):
self.ph_child_last = None # Paring heap
self.ph_next = None # Paring heap
self.ph_rightmost_parent = None # Paring heap
self.shield = False
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding this here means increasing the size of all tasks, even those that don't use shield, and would ideally be avoided. In the C implementation of Task you can see that the size of the task is 8 words, so increasing it by one will mean it now needs 3 GC blocks rather than 2 (there are 4 words per GC block). Would be better to find a way to not need this variable....

Copy link
Contributor Author

@kevinkk525 kevinkk525 Apr 16, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Absolutely. It's meant as a proof-of-concept. You'll know how to optmizie it better than I do.
But currently I am having trouble getting it to work correctly with shield() returning a Task object. An oversight from the start, so this code (except for self.shield=False) is actually not how it will work.. You can ignore it for now.

@kevinkk525
Copy link
Contributor Author

Now it should work correctly! Took me a while to figure it out but now it actually looks like a fairly small change to "task.py".
By subclassing Task in shield.py I can prevent the "shield" attribute from taking additional resources unless you use shield().

Only CPython difference in the tests is the cancellation order of wait_for.

@kevinkk525
Copy link
Contributor Author

I put shield() back into funcs.py and changed the way Task.shield is assigned to the same way Task.waiting is done.
Will have to be implemented in _uasyncio Task too

Testcases updated and extended. The result is now the same as Cpython even in wait_for cases if the fix in #5931 is applied.

Little bugfix setting shield correctly after task finished (actually not even needed because that tasks are being destroyed afterwards anyways).
@dpgeorge
Copy link
Member

dpgeorge commented Dec 1, 2020

Thanks for updating this. It is quite clean although it still requires an additional entry in the Task object when shielding is used. This is not a show-stopper but does mean the C implementation (see moduasyncio.c) will be much more complex because it has to manage allocation of extra memory to hold this shield entry when it is assigned to.

It would be good to look for an alternative implementation of shield that did not need any modifications to the existing core code (but I'm not asking you to do that!).

With #6667, wait-for can be fixed without shield, but IMO shield is still useful in its own right.

@dpgeorge dpgeorge added the extmod Relates to extmod/ directory in source label Dec 1, 2020
@kevinkk525
Copy link
Contributor Author

Thanks for taking a look at this and wait_for!
I thought it'd be a small and non-complex addition to moduasyncio.c, similar to Task.waiting, which gets lazily initialized too:

STATIC mp_obj_t task_make_new(const mp_obj_type_t *type, size_t n_args, size_t n_kw, const mp_obj_t *args) {
[...]
    self->waiting = mp_const_none;

@dpgeorge
Copy link
Member

dpgeorge commented Dec 1, 2020

I thought it'd be a small and non-complex addition to moduasyncio.c, similar to Task.waiting, which gets lazily initialized too:

Except that adding a new entry to mp_obj_task_t would make it 9 words big, which now takes up 3 GC blocks instead of 2 (there are 4 words per GC block). So the lazy initialisation would need to do something like make task->waiting into a tuple of 2 elements, on for the original waiting value and another for the shield value (I did something similar to this in a very early version but it was not needed in the end).

@dpgeorge
Copy link
Member

This needs a different solution that doesn't require adding a member to the Task object.

@dpgeorge dpgeorge closed this Jun 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
extmod Relates to extmod/ directory in source
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants