Some further thoughts...

Creating simple stubs to allow esp-idf functions to be called from native modules means that there is still python->C glue code that someone has to write somewhere. If the native code is complex and calls some esp-idf functions in the middle then there's no real way around having to hand-code the python->C glue. However, if drivers can be written in 99% python code plus 1% of more or less direct calls into esp-idf then it's a pain to have to hand write the glue code.

As simple example: the machine.Counter module proposal defines a pause() and a resume() method. Each of these methods would ideally be written in python using something like:

def pause(self):
    if self.active:
        pcnt_counter_pause(self.unit) # call ESP-IDF C function with int arg
    else:
        raise("Unit inactive")

The current proposal makes it easy to generate the stubs 'cause one doesn't have to provide type information, but what if one could? Then we could generate stubs that are directly callable from python! E.g., suppose pcnt_counter_pause was defined using something like:

[ 'pcnt_counter_pause', 'I', 'I']

where the tuple specifies the function name, the input types (in struct.pack notation) and the return type. The stubs could all be placed into a machine.native module such that the above code could be written as

import machine.native
def pause(self):
    if self.active:
        res = native.pcnt_counter_pause(self.unit) # call ESP-IDF C function with int arg
        if res:
            raise("Pause failed")
    else:
        raise("Unit inactive")```

There are a lot of things that could be discussed in this context, but let me start with just a few.

The goal is to make it easy to write native modules that call ESP-IDF functions by just adding the names of the needed functions to a list

It should in principal be possible to do this by just adding the new functions to mp_fun_table and also to the table in mpy_ld.py. But I guess that doesn't work in practice due to the need for stubs, because the linker can't relocate function calls that are directly made to entries in mp_fun_table (code must instead use mpy_fun_table.foo(...)).

A completely different approach that I did not investigate in detail would be to enhance the dynamic linker in the firmware so it can fix up the targets of the CALL instructions, i.e., perform proper linking instead of the whole stub stuff.

IMO this is the "right" way to do it. But instead of changing the firmware's loader/linker (which would get quite complex to support new linking mechanisms) I'd suggest to improve mpy_ld.py so that it can handle such cases. For some archs the "CALL" instructions emitted by the C compiler are not big enough to reach all of the (4gb) address space, so stubs are necessary in these cases. The idea would then be for mpy_ld.py itself to generate the stubs automatically when needed (and insert them into the .mpy file it generates). For archs like esp32, where "CALL" already has enough room for a 32-bit label, it might be possible to avoid a stub and just add a relocation entry to the .mpy that fixes this label up during import.

The current proposal makes it easy to generate the stubs 'cause one doesn't have to provide type information, but what if one could? Then we could generate stubs that are directly callable from python! E.g., suppose pcnt_counter_pause was defined using something like:

This is very similar to ffi (see unix ffi module) and what ctypes does. But for the IDF it'd be static. So really what it could be is just an (automatically generated) espidf module that would expose all IDF functions (and constant). This might be a good idea, to allow arbitrary access to esp32 functionality, and not be limited by what's implemented in the machine and esp32 modules. [On MCUs like stm32 the corresponding API is essentially the peripheral register set, and this is already exposed to Python code via direct memory reads/writes and the stm module for convenient constants.]

There are a lot of things that could be discussed in this context, but let me start with just a few.

I hope that's a positive statement ;-)

The goal is to make it easy to write native modules that call ESP-IDF functions by just adding the names of the needed functions to a list

It should in principal be possible to do this by just adding the new functions to mp_fun_table and also to the table in mpy_ld.py. But I guess that doesn't work in practice due to the need for stubs, because the linker can't relocate function calls that are directly made to entries in mp_fun_table (code must instead use mpy_fun_table.foo(...)).

It doesn't work in practice because py/nativeglue.[hc] are machine independent files. In order to get the right functions in there a pile of esp32-specific -I options would be needed, plus a pile of esp32-specific #includes, plus appropriate #ifdefs around all that. Not appealing IMHO...

A completely different approach that I did not investigate in detail would be to enhance the dynamic linker in the firmware so it can fix up the targets of the CALL instructions, i.e., perform proper linking instead of the whole stub stuff.

IMO this is the "right" way to do it. But instead of changing the firmware's loader/linker (which would get quite complex to support new linking mechanisms) I'd suggest to improve mpy_ld.py so that it can handle such cases.

👍

For archs like esp32, where "CALL" already has enough room for a 32-bit label, it might be possible to avoid a stub and just add a relocation entry to the .mpy that fixes this label up during import.

Do you have an example for any arch that produces such relocation entries ("a relocation entry to the .mpy that fixes this label up during import"). Or can you point me at a couple of relevant lines of code in mpy_ld or the dyn linker? Just a starting point for me to understand...

The current proposal makes it easy to generate the stubs 'cause one doesn't have to provide type information, but what if one could? Then we could generate stubs that are directly callable from python! E.g., suppose pcnt_counter_pause was defined using something like:

This is very similar to ffi (see unix ffi module) and what ctypes does. But for the IDF it'd be static.

I'll take a look at the ffi stuff, but generating an espidf module doesn't sound too complicated for a manually provided list of entry points. I'm not sure how I would extract a list of "all" functions given how everything is spread across include directories...

Do you have an example for any arch that produces such relocation entries ("a relocation entry to the .mpy that fixes this label up during import")

In mpy_ld.py the MPYOutput.write_reloc() method writes a general relocation entry to the mpy file. This then gets parsed by mp_native_relocate() in persistentcode.c. As long as you can express the relocation that you need in this form then it will work. Eg if a CALL points to somewhere to find the destination, you need to relocate that "somewhere".

@dpgeorge I looked at the code some more and need a decision from you.
The relocation entries in the mpy table start with an op byte which encodes the destination of the relocation. The LSB is used to encode whether there is a fixup offset, that leaves 127 op values and other than values 3..5 they're all used. In particular, values 7..127 are used to encode entries in mp_fun_table, of which 80 are currently used (if I counted correctly). How would you like me to add the platform-specific entries to this encoding?
The proposal I have is to use op==5 to encode "entry in plat_relo_tab" and to add a 16-bit field for the entry number after the offset field (and there would be no count field just like for the mp_fun_table elements).

The proposal I have is to use op==5 to encode "entry in plat_relo_tab" and to add a 16-bit field for the entry number after the offset field (and there would be no count field just like for the mp_fun_table elements).

op==5 is already used. Maybe op==126?

The proposal I have is to use op==5 to encode "entry in plat_relo_tab" and to add a 16-bit field for the entry number after the offset field (and there would be no count field just like for the mp_fun_table elements).

op==5 is already used. Maybe op==126?

Hmmmm, given https://github.com/micropython/micropython/blob/master/tools/mpy_ld.py#L771-L775 can you tell me how op codes 3..5 can be "already used"?

can you tell me how op codes 3..5 can be "already used"?

dest=2, so 2*2+1=5?

OK, I implemented the relocation using op=126. Seems to work nicely :-)
If the machinery passes muster I'll clean-up the PR (remove debug stuff, add #ifdef, single copy of plat_relo_tab symbol names, etc).

I added a partial implementation of generating an espidf module to call the esp-idf functions from python and then implemented examples/natmod/esp-counter fully in python using the espidf module. This code generation is very, very primitive compared to the littlelvgl generator; it has its pros and cons...

Closing in favor of #5711