As usual, naming things is hard. There is now the MICROPY_PERSISTENT_NATIVE config variable to enable this dynamic loadable C module feature, along with some structures and functions named with "persistent_native"

Ok, why not call it "loadable native" after all?

Otherwise, I looked thru code. In my current state, I can't understand it well ;-). Well, I see that it reuses a lot of machinery from persistent bytecode and native codegens, which causes you to select "persistent native" term and ".mpy" extension. I'm not sure I agree with either. Regardless of underlying implementation, cached (byte)code and code purposely written in C are 2 rather different things, and I don't think they should be mixed up on terminology level and external user interface levels. I'd suggest ".mpd" extension for compiled modules (following ".pyd" for CPython). Unless you have arguments why it should be .mpy. For example, I'd +0.5 argument that it saves a file look up. But we'll definitely end up confusing users (for example, "cached" .mpy can always be deleted and will be regenerated from source, not so for "implemented in C" .mpy).

Some random thoughts: nice way to get forward with this would be building some real-world code with this. Like sqlite ;-).

And no, examples/modx/modx.c here in the patch doesn't give warm fuzzy feelings - it's very different from how built-in modules are coded. I'd set that as one of main goals to be able to code modules so they were able to be used both statically and dynamically (efficiently!). That will certainly require resurrecting my idea of having custom preprocessor for stuff like QSTR(foo) (for dynamic module that would resolve to something like __qstr_arr[QSTR_foo] and code to init __qstr_arr).

And no, examples/modx/modx.c here in the patch doesn't give warm fuzzy feelings - it's very different from how built-in modules are coded. I'd set that as one of main goals to be able to code modules so they were able to be used both statically and dynamically (efficiently!).

I went to a lot of effort with this new version to provide this feature. See these macros:
https://github.com/micropython/micropython/pull/1627/files#diff-f66815f6580dbaa8ad2614d01ad1dad3R34

QSTR(foo) (for dynamic module that would resolve to something like __qstr_arr[QSTR_foo]

That's how it works at the moment.

For dynamic modules everything must go through a table: qstrs, constants, and calls to the runtime (unless you want to provide a proper linker in uPy to link external symbols, but that's going to require a lot of code, and specific code for each arch).

This was rebased on top of current master and force pushed. @aykevl you may want to try it out.

Thanks! It doesn't seem to work out of the box (probably due to the age), I'm now fixing a few things. Will share when I have something working.

Thank you for the Python 2 fix. I use Debian which uses Python 2 by default. It now works on my side.

For an MCU with no filesystem and if qstrings are not used, is it possible to load new modules or native code by modifying the table static const mp_rom_map_elem_t mp_builtin_module_table[] and the other tables that you'd normally modify when creating a new module?

If it's not possible now, could this become possible with some linker file modification or is this fundamentally impossible for micropython? Or fundamentally impossible for any executable?

I can imagine that with qstrings, this would be impossible or very hard because the qstring table would not be populated with the new module.

@dpgeorge @pfalcon @aykevl

For an MCU with no filesystem and if qstrings are not used

So what would be the use case for dynamically loadable modules, then? Why not just integrate them into the ROM?
I'm curious, because I've seen such mentions before and I'm not sure in which way dynamically loadable modules could be useful when there is no filesystem.

It might be possible if you make mp_builtin_module_table[] not const and leave some room at the end for newly loaded modules.

So what would be the use case for dynamically loadable modules, then? Why not just integrate them into the ROM?

We want to give our customers the ability to extend the micropython installation but only allow them to write to certain areas of flash (so they can't brick the module).

The micropython installation is very integrated into our product so to flash everything as part of a monolithic .elf, we'd have to give out our .objs for them to link to their .objs . . . which we don't want to do.

I'm not saying it's a compelling enough feature for the community here to spend oodles of time on it; I just want to know whether it's possible to do this assuming you don't use qstrings.

Though, if all it takes is adding a #define MAKE_MODULES_EXTENSIBLE along with adding some number of blank entries in some tables . . . it seems like it'd pass the ROI smell test.

We want to give our customers the ability to extend the micropython installation but only allow them to write to certain areas of flash (so they can't brick the module).

Ah, that's certainly a use case, though I wonder if such a feature could be useful for open source projects. But I'm not a maintainer so can't say anything about whether it will be done.

I think the harder problem is disabling qstrs. I suspect they're integrated so deeply they can't simply be disabled - e.g. they're used for fast comparisons between any two strings.

So, I'm looking into this again. And for the life of me I cannot understand what were the ideas and requirements which went into this "persistent native" stuff. The only reason I may imagine is desire to prototype something for saving @micropython.native/@micropython.viper code into .mpy. Because there's no other explanation why would all this over-engineering, all these parallel hierarchies of "persistent native" functions and types be required to implement just "dynamically loadable modules". (Heck, it's not clear why they would be needed even for persisting - there's already types for native and viper functions).

Perhaps I'm just stupid.

But real fun just begins. The included modx.c doesn't have any (global) variables. That's not realistic, any more or less non-trivial code will have variables, data structures, etc. Trying to add those, I see [rip+xx] in the .o file. But within the produced .elf, there're direct addressing/immediate values instead! Stupid modern compiler smartasses! Ok, adding ld -fPIC. It complains that it's possible only with -shared. Ok, building that, and trying to comprehend the resulting code. Remembering that a few things in the world are as overengineered and bloated as ELF shared libraries, and the only way to not get nausea with them is -Bsymbolic. Ok, but the code in .mpy doesn't disassemble well. After enough peering into it, becomes clear that objcopy -O binary manages to corrupt the section content when copying it out of shlib. That's the modern compiler infrastructure again - automagically "relaxing" PIC code into non-PIC without anybody asking, and being unable to copy a hundred of bytes verbatim without corruption.

Looking at the generated code again, the persistence worthy of Sisyphus becomes apparent:

 226:   48 8d 2d f3 00 00 00    lea    rbp,[rip+0xf3]        # 320 <_ctx>
 22d:   53                      push   rbx
 22e:   48 8b 45 00             mov    rax,QWORD PTR [rbp+0x0]
 232:   ff 50 08                call   QWORD PTR [rax+0x8]
 235:   48 89 c3                mov    rbx,rax
 238:   48 8b 45 00             mov    rax,QWORD PTR [rbp+0x0]
 23c:   4c 89 e7                mov    rdi,r12
 23f:   ff 50 08                call   QWORD PTR [rax+0x8]
 242:   48 8d 3c 03             lea    rdi,[rbx+rax*1]
 246:   48 8b 45 00             mov    rax,QWORD PTR [rbp+0x0]
 24a:   5b                      pop    rbx
 24b:   5d                      pop    rbp
 24c:   41 5c                   pop    r12
 24e:   48 8b 00                mov    rax,QWORD PTR [rax]

Look how carefully it loads pointer to the function table again and again, again and again - instead of just caching it in a register! After jerking back and forth, this one becomes apparent too - wonderful semantics of the C language, where every function is suspected of being able to modify a global. Wait, this should, work, right:

restrict const mod_ctx_t *_ctx;

?

No! It's 2019, but in C, it's possible to only declare var as ever-changing (volatile) or fully constant-down-to-being-immediate-value. Ok, after some thinking (which included abusing .got table to do the needful), solution was found of declaring it extern const mod_ctx_t * const _ctx; in one compilation unit, so there it was treated as cachable constant, and in another - as const mod_ctx_t *_ctx;, so it could be actually initialized at runtime, not compile-time.

Summing up: the idea to use -fPIC seemed bright, but actually is brittle like hell. And I'm looking at the most popular arch. Something like Xtensa will just crumble down. Anyway, I'm proceeding ;-).

After enough peering into it, becomes clear that objcopy -O binary manages to corrupt the section content when copying it out of shlib.

Heh, that was done by elftompy.py (of course, I'm changing module format). Poor binutils slandered by me!

Some more notes on the design of dynaloaded modules format: #4535 (comment)

This PR is well and truly superseded by #5083