RFC: 'true' dynamic loading for unix port #6767
Conversation
|
Do you plan on some kind of version checking to ensure that the binary module is compatible with the runtime that is trying to load it (e.g. module was compiled against MicroPython 1.11 but trying to load in 1.12)? |
|
Never considered that, also because a different version doesn't necessarily mean it won't run, but shouldn't be too hard. |
|
What are the use-cases? I ask naively because I always think of the unix port being just for testing, debugging, proof of concept, or demos. |
|
Well we use the windows port (which is essentially the unix one) here for large-scale applications already from way before there were user C modules etc. So for me that's the primary use-case: extend MicroPython without modifying the build process, without being restricted to certain directory structures, without being limited to a subset of the API etc. Another important aspect of having separate modules (as opposed to user C modules which still result in 1 executable) is ability to dsitribute all of them and having dependencies handled client-side: if a machine doesn't have a certain dependency installed, it can just test that and still run MicroPython using other modules. If you have one executable though, all depenencies must always be satisfied so if we'd have to do that, we'd either need to install everything we ever use on all machines (complete no-go) or make all dependencies in turn dynamically loadable (i.e. instead of just linking against the library - which is a lot of boring work and not convenient at all). Another aspect but less important is build time: complete application from scratch takes like 10 minutes. But if you need just one module you can also build just that one module which is way faster. For personal usage and small projects though, most of this doesn't matter (though build time and not being invasive can still be nice) |
|
This is really interesting... thanks @stinos! Would you be able to provide an example of a more comprehensive module implemented this way... (i.e. maybe a simple class and some methods?) I think I'm mainly a bit unsure how messy the registration code of the module dict / locals dict etc is going to be? Especially without QSTR handling in the build process. I'm guessing that using the existing native .mpy feature is out of the question for your use case (for the same reason as on, say, ESP32) because you're likely going to want the full system linker/loader (i.e. to access other libraries and syscalls). (Hence the "true" in the PR title :) ) And FFI isn't really an alternative, because I'm guessing you want to provide a Python-level API from your dynamic library, not just have the ability for Python code to call your library. So now we have:
What should this feature be called? From the code it looks like you're suggesting maybe "Dynamic Libraries" (i.e DYNLIB) or "Dynamic Modules" (i.e. "dynmod"). Naming is hard!
Yeah. However, this is exactly the sort of thing that's being discussed in #5711 which is basically how much of a dynamic loader/linker should MicroPython implement... |
It's possible to add that but the only way I found is to first preprocess the complete source (i.e. MicroPython's plus the module source) which makes things a bit complicated, plus loses the benefit of 'standalone' builds since if you have multiple modules they all need to be preprocessed in one go so they need to know about each other. I'll update the example so it's more complete.
Basically yes, and also because it maybe implements half of the MicroPython API I'm using at the moment. Also see comments starting at #5711 (comment); in theory it would be possible to have a hybrid solution and use the native .mpy bit just for linking against MicroPython but then you still need the system's loader to get all other dependencies so seems needlessly complicated to do that combination. On the other hand it would have one benefit on windows: last time I checked the only way to make objects (as opposed to functions) dynamically loadable is to decorate them with
Indeed, way too much extra work and code layers.
Good question. I indeed chose 'dynamic module' (after what |
Introduce CPython-style dynamic module loading: upon import look for .so or .pyd files on the module search path and load it using the dynamic loader, lookup the symbol 'init_<modulename>' and call it as a function which returns the actual uPy module.
|
Stinos, you're not going to like this comment, and I don't want to be mean, but I'm honestly wondering whether this is something that should be added to the main MP repo. It seems to enable something that is somewhat outside of MP's focus area. My fear is that you will be the only user and that it adds overall confusion with yet another dynamic loading mechanism, and thereby adds more baggage to carry around. I think it's great that you have opened this issue, where others can find it, and if there are a bunch more people chiming in with "oh, this is just what I needed" then by all means go forward! But without this happening I'm on the thumbs-down side, especially given the immense backlog of pending PRs that are squarely in MP's focus area. I hope you're not taking this personally against you, I very much appreciate and respect all the work you're doing on MP! |
|
@tve no worries, no offense taken, the whole point of this PR is the 'RFC' bit. As I mentioned, I'm well aware clear the scope of this feature is rather limited and for a lot of usecases out there the alternative exists to just use CPython. I think the only reasons this could be part of the mainline are that it's a relatively small change, and that it makes MicroPython more widely usable on unix and potentially other more restricted platforms where full CPython is just too much. I assume that area is now mainly taken by Lua for instance. Which really isn't as nice of a language as Python imo :) |
|
I'm rather interested in this, in order to be able to make use of a port of the cpython wrapper to BearLibTerminal.so, an OpenGL text mode user interface for roguelike games and other stacked-tile-entity purposes. The dynamic library is cross platform for linux, windows, and macos. One of my side projects has been to make a nice 'advanced' serial terminal link to various microcontrollers running micropython -- ESP32, STM32, and the coming RP2040. As a "user story", I'll submit some sample code that may or may not be of use in testing dynamic shared module loading. I'd like to publish this game to Valve's Steam game service, along with micropython executable builds for linux, osx, and windows. If I got far enough along to be able to charge $3.99 for a slightly enhanced version of the game, I could figure out some way of remunerating the micropython team some of the proceeds to keep the continuing maintenance, development, and unit testing of features like these. It is a very basic roguelike engine built on top of BearLibTerminal's pip package. No binary code is included other than an AtariST8x16SystemFont.ttf representing the artistic feel for the game, which is optional. Beyond that engine demo, I'd want to establish some kind of packet-based serial interchange format allowing a microcontroller UART to push content to a memory filesystem which BearLibTerminal could access, allowing a MCU to push a font to the host, or several of the supported bitmap formats as graphic tiles, and command their movement around a cellgrid without having to understand or output OpenGL themselves. LittleVGL is out there, but it's... Not really suitable for an entertainment use-case in the way this PR could be, at least for myself. Hopefully that's enough of a "hey, someone wants this pretty badly" to consider pushing forward the development of this PR. Cheers! |
Not sure if this is clear, but the feature in this PR isn't strictly required to do what you want: you can still write a MicroPython wrapper around any library then build it into MicroPython using http://docs.micropython.org/en/latest/develop/cmodules.html |
BearLibTerminal is published as a binary shared object in PyPi, with wheels for windows .dll, macos .dylib and linux .so files. The pypi package also includes bindings that exposes the dlopen'd library as python functions and does minor abstractions to choose which shared object style to load based on the platform. There are also bindings for other language runtimes. The source is under the MIT license, available at https://github.com/cfyzium/bearlibterminal The existing ctypes binding is here: https://github.com/cfyzium/bearlibterminal/blob/master/Terminal/Include/Python/bearlibterminal/terminal.py I am aware I could fork/vendor bearlibterminal and compile it into the final micropython executable, but, I am more interested in making this work in a general sense with existing art already in the field. That said; if this sort of support lands in micropython proper, I would probably submit a pull request to the BLT repo with middleware bindings to micropython so all the work is upstream, and I am merely just another .py file floating in the breeze. This is just a very explicit "heads up" that there is interest in where this PR is headed, and a code gift of a commented example that has worked for five years. As well as expressing an interest in using micropython for widespread binary deployment over, say embedding lua and porting the code over. Or trying to figure out how to package cpython with zips or other opaque solutions to deployment. Micropython is simple, elegant, and only a few hundred kilobytes. It's literally more useful than trying to rely on bash being around on windows. As it is, I already fool around with a windows build of micropython.exe to automate some file deployment actions in a single executable. Were I to deploy the game as I envision it, the only thing sitting next to that executable would be the existing precompiled BearLibTerminal.dll file and the appropriate micropython specific version of terminal.py, and some derivative of the game code I previously threw out into the world. As the above example of micropython in the wild, being currently deployed alongside an existing game on Steam. Or did I misunderstand your earlier comment, "It is of limited usage, i.e. for PC ports only, since AFAIK no mirocontrollers implement dlopen and the likes." ? And ctypes.CDLL sort of behavior is not what is being proposed here? (with the windows port being a child of the unix port?) |
|
Yes, but that doesn't change anything. This PR is about being able to dynamically load a MicroPython module, i.e. What you are talking about is loading BearLibTerminal.so using CPython's ctypes.CDLL, for which there is no MicroPython equivalent (unix port does have an ffi module though which on second thought might be your best option here). Otherwise you'd write a wrapper in C around BearLibTerminal.so be it by loading it at runtime using dynload etc or at build time by compiling against BearLibTerminal.h and linking against its import library. |
|
@kamilion thanks for the elaborate explanation, but note that if you edit comments well after you've created the first version, things get really hard to follow. So, to avoid confusion: my previous comment was a reply to this comment:
or in other words, the answer to the question you added afterwards
is: indeed, that is not what is being proposed here. |
|
Thanks @stinos for posting this. I've been wanting to have a detailed look but didn't get a chance yet. |
…-report-fix shared-module/usb_hid: Fix behavior of Device.get_last_received_report()
Introduce CPython-style dynamic module loading: upon import look
for .so or .pyd files on the module search path and load it using
the dynamic loader, lookup the symbol
init_<modulename>andcall it as a function which returns the actual uPy module.
I've been using this patch for years but for some reason never made it into a PR so this is a question for feedback before I get this into a proper shape (less invasive test running, comments, documentation perhaps, more elaborate example, adding windows couterpart): is there any interest in this? It is of limited usage, i.e. for PC ports only, since AFAIK no mirocontrollers implement
dlopenand the likes. However it has none of the disadvantages/fiddling of user C modules/native modules and allows complete out of source building of modules.