Hello everyone.

My goal is to be able to dynamically select Python files from the filesystem and run them 'concurrently' - also choosing when to stop one and begin another. In an ideal world you could provide the equivalent to a 'yield()' call in those python files that would allow others to execute, but running them end to start could work as well. I am raising this issue to a) seek advice from those who are more familiar with the complexities here and b) see if anyone is interested in such an ability / would like to collaborate to make it happen.

I have considered a few possible ways to make this happen, but I don't know which fits best in the ideology of MicroPython or which is the easiest to implement.

(My sights are focused on the ESP32 port but if this could be a MicroPython wide addition then I'm game)

1. It seems like the Python 'subprocess' module could at least run one file and then another from a Python script, and I could imagine setting up a method for users to start/stop execution of certain files with some kind of user interface. But from my very limited knowledge of the module it seems like there would be no way to yield execution part-way though.
   • If this is the right way to go I'd need to read up on how to port a module. With a little time, guidance, or combination I could make it happen.
   • This might be a nice way to avoid having to worry about memory and storing states of new interpreters?
2. Since the interpreter is run underneath a FreeRTOS task, and the 'pyexec_file()' function exists I'd be very happy to be able to create another task, and allow users to call the FreeRTOS 'yield()' function from a python module. I think this is my preferred solution but it goes against 'anything that can be written in Python should be.'
   • If going this way my main concern would be making sure there are no conflicts between states of the main and spawned interpreters.
   • I've begun looking through the code to try to identify global state variables that would need to be isolated, but I feel a little overwhelmed. Advice from someone who is familiar with the state requirements of the interpreter would be so greatly appreciated.
   • One last concern here is how to decide how much memory each process should be allotted.
3. I briefly considered the '_thread' module but I don't think it will work to start a whole file... Am I wrong? Maybe users could write module files with a common 'run' function and then import / del the modules as needed - and somehow call the run function for the right module as a thread in the main process?
   • The way I see it this is the least favorable solution, though if it could work it would require little to no new development.

I'd really appreciate advice / thoughts on this topic. If it can be a worthy addition to MicroPython then I'll be eager to help out! If it's not right then I'd still appreciate advice on how to proceed for my own extension that I can offer.

I guess that about wraps it up for now. Looking forward to talking about this. Thanks!