-
-
Notifications
You must be signed in to change notification settings - Fork 8.2k
RFC: Cooperative concurrency model of choice for MicroPython #242
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
In my brief evaluation of this substantial info. It looks to me like: Of course moost of these discussions center around webservers (as they want concurrent socket based ops right now) rather than the focus on async processes which we would like in an embedded approach. |
I saw Protothreads and the description sounded good, but it may be too simplistic to get uPy functions running asynchronously. |
@dpgeorge: Well, protothreads are inferior analogue of python generators (which are both kinds of coroutines). We don't need them in uPy, they're needed when you have 1K of flash and 128b of RAM ;-). What you already implemented of generators is more advanced and flexible than protothreads. Btw, I thought you had generators destined as concurrency model yourself, because I'd personally implemented generator support the last thing otherwise ;-). |
FWIW, async generators have been picking up momentum in recent years in more mature mainstream frameworks - for example the relatively recent introduction of it into the core c# language, which is now the recommended approach on that platform. My point being that developers are more accepting of the approach today than they would have been several years ago. Personally, I like them. |
Hah, well, I just wanted to get as Python-conformant as I could, and generators didn't need much extra work, so they are there. I'll have to learn more about how to use them as async concurrency. Do you think it's enough to have concurrency at the Python high-level, and not at the C level? It's much easier at the Python level (since it's basically done). To get it working, is there more to implement in the core of uPy, or is it just a pure Python library that needs to be ported? |
It's probably worth getting terminology straight, because there's duality between synchronous/asynchronous stuff in multitasking framework. With (typical implementation of) cooperative multitasking we want to handle async events/processes with sync event loop. And with preemptive multitasking, we want to have synchronous processes scheduled and running asynchronously. So, the talk here about cooperative multitasking model, I updated the title. This can be implemented in pure Python. I don't even raise question of preemptive multitasking. First of all, I don't think that's useful for MCU usage. And that surely would require RTOS, enough memory for thread stacks, etc. Thread-based multitasking would be more useful for unix port, even though best practice would be to implement thread pool to adapt blocking sync services into cooperative async ones. But even that would require either pseudo-threadsafety with locks (oh no, we had enough of GIL!) or doing true "lockless" threadsafety by supporting multiple VMs (Squirrel does that, each and every API function takes VM pointer, how do you like that?)
So, I submitted this informal ticket essentially as a prelude to #243. Trivial "embedded" coop concurrency trivially doable with py2.3 generators:
To add seamless async i/o (which looks like "system service", though actually implemented by the similar coroutine as any other task), it takes 2.5 .send() method, implemented by me recently. But to make it real-world usable, exception handling needs to be implemented, which is concern of #243. |
I recommend David Beazley's tutorial (# 4 above), it in particular clearly addresses @Neon22's concern that "they center around webservers rather than the focus on async processes which we would like in an embedded approach." - no, there's no difference, they center around arbitrary async events/processes and there's elegant way to plug anything and everything. |
Ok, Python 3.4 was released, featuring asyncio module: http://docs.python.org/3.4/library/asyncio.html . Googling around, it's getting its critical acclaim. So, it's no longer a question, it's the fact: cooperative concurrency model of choice for Python is coroutines via generators. But asyncio is based on "yield from" construct, which itself appeared only in py3.3. uPy doesn't have it implemented so far. So, what is "yield from" and why it was selected as the base for async implementation? Explanation directly from Guido van Rossum (Python author): https://groups.google.com/forum/#!msg/python-tulip/bmphRrryuFk/aB45sEJUomYJ |
Nice read from Guido about yield from. Now we just need to implement it... :) |
Yeah, opened #366 for that, because it's not clear at all how it should be implemented ;-). |
Current state:
Closing this ticket thus. |
This questions was raised yet in comments to Kickstarter campaign, and I'd like to set up this ticket for informal background discussion, to allow interested parties to further discuss/share ideas on this matter, and serve as guide for further implementation directions.
Summary of my proposal: Choose asynchronous concurrency via generators, as native, Pythonic, high-level way to do concurrency in Python.
I guess almost everyone heard that Python generators can be used to implement coroutines, and possibly few actually use them and up to date with their status. Myself neither, so I was taking last couple of weeks to survey state of the art in Python async concurrency in general, and generators in particular. There appear to be gradual advances since 2.5, and process continues, with upcoming 3.4 offering standardized async coroutine framework.
Formal references:
Tutorials:
4. http://www.dabeaz.com/coroutines/index.html "A Curious Course on Coroutines and Concurrency"
5. http://lgiordani.github.io/blog/2013/03/29/python-generators-from-iterators-to-cooperative-multitasking-3/ "Python Generators - From Iterators to Cooperative Multitasking"
Reviews and comparisons:
6. http://nichol.as/asynchronous-servers-in-python "Asynchronous Servers in Python"
The text was updated successfully, but these errors were encountered: