Skip to content

RFC: Do we need (better) support for runtime initialization of C modules #5654

Open
@stinos

Description

@stinos

All (?) modules in the MicroPython source code and external C modules are created by defining a const mp_obj_module_t object which basically gets populated at compile time by specifying the globals table. This was done so because it allows putting everything in ROM, I assume. The module object then somehow has to make it into mp_builtin_module_table so it can be found by the import logic. However another way to make modules available within MicroPython is creating an mp_obj_module_t at runtime and register it. Essentially something like

const qstr module_name = qstr_from_str("mymodule");
mp_obj_module_t *module = (mp_obj_module_t*) mp_obj_new_module(module_name);
//Populate the module with variables/functions/classes/....
mp_obj_dict_store(module->globals, "global_var", mp_const_true);
...
//Register module so 'import mymodule' in Python now works
mp_module_register(module_name, module);

this offers some flexibility for making certain things available or not based on runtime conditions for instance. Or if you're short on ROM it allows creating your module in RAM instead (well, just a small part i.e. the globals table not sure if that really helps). It's similar to what gets done in the mpy_init function of native modules, it's also something which is standard in CPython (C modules have a PyInit_xxx which gets called upon first import) and actually there's an example of something similar in unix/main.c (although that just registers a type globally not in a specific module, but the principle is the same).

However there isn't really (see below) any build support which allows doing this: there's no way to say 'here's an initialization function for a module X, call it to create the module X when encountering import X. Is that something we need? I know I could use it because literally all my modules work like that, just not sure if other people ever felt any need for it. Or even thought about it. Implementation could be relatively simple I think: a table similar to how mp_builtin_module_table works but with key/value pairs of module name and a pointer to their init function which gets called to initialize the module once, and a cache for the loaded ones.

Addendum: currently there are some ways to have these 'runtime initialized modules':

  • you can just patch e.g. main.c and add whatever code you need to create a module
  • I found a way to leverage the external C module stuff to do it but it's a bit hacky and needs some boilerplate for each module. Idea is that you enable MICROPY_MODULE_BUILTIN_INIT, create a non-const mp_obj_module_t where the globals just contain one __init__ entry, pointing to a function which then replaces the module globals with a RAM dict populated with whatever the module needs. See https://github.com/stinos/micropython-wrap/blob/master/tests/cmodule.c for example

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions