@andrewleech Maybe it will interest you.

Good work, I started idf 5.1 compatibility here too, maybe you'd like to compare: #11869

I thought ADC also needed updating, or has that already been updated?
Bluetooth was also broken for me, though maybe that's just on the C6

The CI setup is configured here:

I think there's a desire to not build multiple versions in CI, better to pick one compatible version and just focus on that, that would be my preference at least.

I thought ADC also needed updating, or has that already been updated? Bluetooth was also broken for me, though maybe that's just on the C6

I've been using the current ESP-IDF 5.2 preview on the ESP32-S3. No issues so far with BLE. So it's probably related to the C6.

I think there's a desire to not build multiple versions in CI, better to pick one compatible version and just focus on that, that would be my preference at least.

I also agree that it's better to pick one compatible version and focus on it.

The deprecated warning on the pm_config struct caused by ESP-IDF commit 8f415a7f448d0438091f2456c173b5e32a829dd1 can be fixed easily:

diff --git a/ports/esp32/modmachine.c b/ports/esp32/modmachine.c
index 1e83935a8..35b1ad0fe 100644
--- a/ports/esp32/modmachine.c
+++ b/ports/esp32/modmachine.c
@@ -78,15 +80,7 @@ STATIC mp_obj_t machine_freq(size_t n_args, const mp_obj_t *args) {
             mp_raise_ValueError(MP_ERROR_TEXT("frequency must be 20MHz, 40MHz, 80Mhz, 160MHz or 240MHz"));
             #endif
         }
-        #if CONFIG_IDF_TARGET_ESP32
-        esp_pm_config_esp32_t pm;
-        #elif CONFIG_IDF_TARGET_ESP32C3
-        esp_pm_config_esp32c3_t pm;
-        #elif CONFIG_IDF_TARGET_ESP32S2
-        esp_pm_config_esp32s2_t pm;
-        #elif CONFIG_IDF_TARGET_ESP32S3
-        esp_pm_config_esp32s3_t pm;
-        #endif
+        esp_pm_config_t pm;
         pm.max_freq_mhz = freq;
         pm.min_freq_mhz = freq;
         pm.light_sleep_enable = false;

There's another change in coming in 5.3 where support for the ESP32-C6 LP UART changes some of the API for all uarts. Like in 57b3f6b, but there's another I've already found too.

This PR is for 5.1, but maybe 5.2 will be out before it ever gets merged.

Maybe it should be possible to add support for ESP-IDF versions sooner? The further behind one gets the harder it is to catch up. Thankfully, when using 5.3-dev there were only a few things to fix to build, so I fixed them. If it had been a miles long list that filled the console scrollback, I'd probably just give up because I don't have time to do all that now, and not fix anything.

There's another change in coming in 5.3 where support for the ESP32-C6 LP UART changes some of the API for all uarts.

EDITED:
ESP-IDF v5.3 and v5.2 look unstable unfortunatly.

It's not released. Look at the master branch. They've already made a v5.3-dev tag.

Thanks @IhorNehrutsa for working on this, it's a good step forward.

However, building with this new IDF version the firmware size has increased by about 60k-70k! So now the D2WD (and probably also OTA) board variants no longer fit in their flash partition.

Furthermore, it looks like the available RAM has decreased as well.

EDIT: the following is wrong! See next comment for updated and correct RAM numbers.

For a standard ESP32, on current firmware running we get:

MicroPython v1.22.0-preview.167.ga800ed5ae on 2023-11-22; Generic ESP32 module with ESP32
Type "help()" for more information.
>>> import micropython
>>> micropython.mem_info()
stack: 704 out of 15360
GC: total: 64000, used: 5008, free: 58992, max new split: 110592
 No. of 1-blocks: 28, 2-blocks: 6, max blk sz: 264, max free sz: 3674
>>> import esp32
>>> esp32.idf_heap_info(esp32.HEAP_DATA)
[(240, 4, 0, 4), (7288, 244, 224, 52), (16648, 3344, 2688, 3184), (87144, 4344, 4096, 4344), (15072, 14656, 14336, 14656), (113840, 112968, 110592, 112968)]
>>> 

With this PR:

>>> micropython.mem_info()
stack: 704 out of 15360
GC: total: 64000, used: 5152, free: 58848, max new split: 49152
 No. of 1-blocks: 35, 2-blocks: 7, max blk sz: 264, max free sz: 3665

>>> esp32.idf_heap_info(esp32.HEAP_DATA)
[(15448, 14804, 14336, 4), (240, 4, 0, 4), (7288, 4, 0, 4), (16648, 44, 36, 4), (86184, 4, 0, 4), (15072, 4, 0, 4), (113840, 84932, 49152, 45392)]

Comparing the heap regions:

• region of 240 bytes size is unchanged
• region of 7288 bytes goes from 224 bytes available to 0 bytes available
• region of 16648 bytes goes from 3344 bytes available to 36 bytes available
• region of 87144 bytes is now a region of 86184 bytes, and goes from 4344 available to 0 available
• region of 15072 bytes goes from 14656 available to 0 available
• region of 113840 bytes goes from 112968 available (110592 max contiguous) to 84932 available (49152 max contiguous)
• there's a new region in IDF 5.1.2 that is size 15448 and has 14804 available

So in total we previously had 135536 total available (max 110592 contiguous), but now we have only 99772 (49152 max contiguous).

That's a loss of 35764 bytes of heap, which is a huge amount considering the standard ESP32 doesn't have that much to begin with.

I'm happy to merge this PR so that we can support newer versions of the IDF, but we should stick to building with v5.0.2 (or move to v5.0.4) until we can reclaim some of that flash and RAM.

The above analysis of free RAM is wrong, I had already started WiFi on the v5.1.2 version.

Here is a correct analysis.

Current master on ESP32_GENERIC:

MicroPython v1.22.0-preview.167.ga800ed5ae on 2023-11-22; Generic ESP32 module with ESP32
Type "help()" for more information.
>>> import platform
>>> platform.platform()
'MicroPython-1.22.0-preview-xtensa-IDFv5.0.2-with-newlib4.1.0'
>>> import micropython
>>> micropython.mem_info()
stack: 704 out of 15360
GC: total: 64000, used: 5344, free: 58656, max new split: 110592
 No. of 1-blocks: 43, 2-blocks: 9, max blk sz: 264, max free sz: 3653
>>> import esp32
>>> esp32.idf_heap_info(esp32.HEAP_DATA)
[(240, 4, 0, 4), (7288, 212, 192, 24), (16648, 3148, 2432, 3052), (87144, 4344, 4096, 4344), (15072, 14656, 14336, 14656), (113840, 112968, 110592, 112968)]

With this PR:

MicroPython v1.22.0-preview.180.g94e13d056 on 2023-11-23; Generic ESP32 module with ESP32
Type "help()" for more information.
>>> import platform
>>> platform.platform()
'MicroPython-1.22.0-preview-xtensa-IDFv5.1.2-with-newlib4.1.0'
>>> import micropython
>>> micropython.mem_info()
stack: 704 out of 15360
GC: total: 64000, used: 1920, free: 62080, max new split: 110592
 No. of 1-blocks: 44, 2-blocks: 10, max blk sz: 18, max free sz: 3867
>>> import esp32
>>> esp32.idf_heap_info(esp32.HEAP_DATA)
[(240, 4, 0, 4), (7288, 4, 0, 4), (16648, 3284, 2560, 3116), (86184, 3384, 3328, 3384), (15072, 14656, 14336, 14656), (113840, 112968, 110592, 112968)]

That's after a fresh machine.reset().

With this PR there's about 1k less due to the region 87144 shrinking down to 86184. But that's about it. The big region of 113840 bytes is unchanged.

When enabling WiFi and connecting to an access point the memory use is:

Current master:

>>> esp32.idf_heap_info(esp32.HEAP_DATA)
[(240, 4, 0, 4), (7288, 4, 0, 4), (16648, 4, 0, 4), (87144, 4, 0, 4), (15072, 4, 0, 4), (113840, 91404, 90112, 88904)]

This PR:

>>> esp32.idf_heap_info(esp32.HEAP_DATA)
[(240, 4, 0, 4), (7288, 4, 0, 4), (16648, 4, 0, 4), (86184, 4, 0, 4), (15072, 4, 0, 4), (113840, 85792, 81920, 83760)]

So in this case this PR uses about 5k extra IDF heap.

After connecting to WiFi and running a few requests, both HTTP and HTTPS, the memory use is:

Current master:

>>> esp32.idf_heap_info(esp32.HEAP_DATA)
[(240, 4, 0, 4), (7288, 4, 0, 4), (16648, 4, 0, 4), (87144, 4, 0, 4), (15072, 4, 0, 4), (113840, 90620, 55296, 53804)]

This PR:

>>> esp32.idf_heap_info(esp32.HEAP_DATA)
[(240, 4, 0, 4), (7288, 4, 0, 4), (16648, 4, 0, 4), (86184, 4, 0, 4), (15072, 4, 0, 4), (113840, 84940, 49152, 49152)]

Again, this PR uses about 5k extra IDF heap (which is not due to the requests, just WiFi connection).

This PR with IDFv5.0.4

MicroPython v1.22.0-preview.182.g8fa053fe0 on 2023-11-23; Generic ESP32 module with ESP32
Type "help()" for more information.
>>> import platform
>>> platform.platform()
'MicroPython-1.22.0-preview-xtensa-IDFv5.0.4-with-newlib4.1.0'
>>> import micropython
>>> micropython.mem_info()
stack: 704 out of 15360
GC: total: 64000, used: 1552, free: 62448, max new split: 110592
 No. of 1-blocks: 37, 2-blocks: 10, max blk sz: 18, max free sz: 3890
>>> import esp32
>>> esp32.idf_heap_info(esp32.HEAP_DATA)
[(240, 4, 0, 4), (7288, 36, 28, 4), (16648, 6372, 4096, 2272), (87576, 1956, 1920, 1956), (15072, 14656, 14336, 14656), (113840, 112968, 110592, 112968)]

@dpgeorge I suggest switching MicroPython master to the IDF v5.0.4

I suggest switching MicroPython master to the IDF v5.0.4

Yes I think that's a good idea. Please can you also update the tools/ci.sh, the IDF_VER variable to use this version?

Code size report:

   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +0 +0.000% RPI_PICO
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (2c828a8) 98.36% compared to head (d0f9499) 98.36%.
Report is 2 commits behind head on master.

@@           Coverage Diff           @@
##           master   #12952   +/-   ##
=======================================
  Coverage   98.36%   98.36%           
=======================================
  Files         159      159           
  Lines       20989    20989           
=======================================
  Hits        20646    20646           
  Misses        343      343           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Thanks for updating, this is now rebased and merged in 88778be through 4365edb (I squashed a few of the related commits together).

I also had to add two additional commits to reduce code size for ESP32-SPIRAM and D2WD builds: b4b77c1 and d0758d8

Has anyone tried building for the C6 with these changes?

I'll do so when I get a chance but it may be a day or two away...

I get an error incompatible .mpy sub-version when add my xxxx.py to ~/micropython/port/esp32/modules

~/micropython/ports/esp32/build-ESP32_GENERIC/frozen_mpy/angles.mpy: incompatible .mpy sub-version

~/micropython/ports/esp32$ python ~/.espressif/python_env/idf5.0_py3.11_env/bin/mpy-cross --version

MicroPython v1.21.0 on 2023-10-07; mpy-cross emitting mpy v6.1

micropython/tools/mpy-tool.py

class Config:
    MPY_VERSION = 6
    MPY_SUB_VERSION = 2

mpyfiles.rst

=================== ============
MicroPython release .mpy version
=================== ============
v1.22.0 and up      6.2
v1.20 - v1.21.0     6.1
v1.19.x             6

6967ff3

I get an error incompatible .mpy sub-version when add my xxxx.py to ~/micropython/port/esp32/modules

If you're using the latest master version of MicroPython then you need to build and use mpy-cross from the repository, not from PyPI (or any other source).

@dpgeorge Thanks, solved.