WIP: esp32: Add support to build using IDF cmake, with MicroPython as a cmake component #6473

dpgeorge · 2020-09-23T06:04:03Z

This PR adds a set of cmake files to allow building the esp32 port as a cmake component in the ESP-IDF. The existing Makefile is left untouched and can still be used.

The PR also updates the code to work with IDF v4.1, and the latest esp32 toolchain.

To build and flash to a device:

<set up environment, virtualenv, path, etc, eg source $IDF_PATH/export.sh>

cd ports/esp32
mkdir build
cd build
cmake ..
make
ESPBAUD=921600 make flash

TODO:

existing make is broken due to changes to support IDF v4.1
work out a way to select the board to build

dpgeorge · 2020-09-23T06:19:11Z

@tve any thoughts/comments on this approach?

UnexpectedMaker · 2020-09-23T06:58:24Z

Wowza @dpgeorge - DID NOT SEE THIS COMING!

dpgeorge · 2020-09-23T07:02:46Z

This should allow building for esp32s2, but I didn't try that yet (did get it running on a standard TinyPICO tho).

UnexpectedMaker · 2020-09-23T07:13:46Z

So there is no more "must be on IDF XXX commit" anymore? I can just grab latest and follow those steps ?
What about sdkconfig? Do I need to run menuconfig and setup stuff?

I just branched and added this PR - want to give it a whirl.

pumelo

Awesome. Really like this.
Tested to build micropython with the docker image espressif/idf:release-v4.1 despite the mentioned change this works.

pumelo · 2020-09-23T09:52:41Z

ports/esp32/main/CMakeLists.txt

+
+# Include core source components.
+include(${MPY_DIR}/py/micropy_py.cmake)
+include(${MPY_DIR}/py/micropy_extmod.cmake)


I think this should be:
include(${MPY_DIR}/extmod/micropy_extmod.cmake)

Yes! now fixed.

pumelo · 2020-09-23T11:07:44Z

I just pushed an approach to specify the board with something like that:
cmake .. -D MPY_BOARD=GENERIC_OTA
here: https://github.com/pumelo/micropython/tree/esp32-cmake

https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-guides/build-system.html#custom-sdkconfig-defaults
esp-idf expects the sdkconfig files to be separated by a semicolon thus the additional file sdkconfig.cmake in each board dir.

dpgeorge · 2020-09-23T12:08:00Z

I just pushed an approach to specify the board with something like that:

Ok, interesting. I was thinking to parse the existing mpconfigboard.mk, so there was only one "source of truth", but it is probably better to have a dedicated mpconfigboard.cmake file per board, with any required settings (like SDKCONFIG_DEAFULT).

tve

This seems an unavoidable change in the long run. I'm not a fan of the esp-idf config/make system, but it's what everyone using the esp32 kind'a expects. The less MP reinvents the better in general...

Comments from doing a test run with this PR branch:

The fact that one gets 59% through the compilation and then it complains about mpy-cross isn't very user-friendly:

[ 59%] Generating ../../genhdr/qstrdefs.generated.h
[ 59%] Generating ../../frozen_content.c
mpy-cross not found at /home/src/esp32/mpy-esp-idf/micropython/mpy-cross/mpy-cross, please build it
first
make[2]: *** [esp-idf/main/CMakeFiles/__idf_main.dir/build.make:89: frozen_content.c] Error 1

The compilation seems to spit out a lot more warnings than when using the old makefile. I guess that's good, but only really if they're going to be addressed?

After a make running ESPBAUD=921600 make flash is very slow because it re-generates qstr headers, and it's awfully verbose (I just wanted to flash what's already built and not see the full list of what didn't get rebuilt...).

With the old makefile, if I run touch main.c; make it takes just shy of 12 seconds on my machine. With the new stuff touch main.c; make takes 26 seconds. Not great.

I suppose to really make MP a component of esp-idf the config stuff needs to be addressed too, right?

tve · 2020-09-25T05:10:55Z

ports/esp32/machine_uart.c

@@ -68,28 +70,28 @@ STATIC void machine_uart_print(const mp_print_t *print, mp_obj_t self_in, mp_pri
    if (self->invert) {
        mp_printf(print, ", invert=");
        uint32_t invert_mask = self->invert;
-        if (invert_mask & UART_INVERSE_TXD) {
+        if (invert_mask & UART_SIGNAL_TXD_INV) {


Does this work with esp-idf pre-4.1? Or is the idea to deprecate the use of older esp-idf versions?

I didn't get that far, I just wanted to get something working and get feedback (will add WIP to the PR title).

We can continue to support older IDF versions if it's not difficult, but I'd rather just support the latest (easier to test, etc).

NP, I just wanted to flag it.

tve · 2020-09-25T05:11:49Z

ports/esp32/main/CMakeLists.txt

@@ -0,0 +1,114 @@
+# Set location of base MicroPython directory, and this port's directory.


Does this file have to be in a subdirectory? (I know nothing about cmake)

Yes. Or at least I couldn't work out how to put it in ports/esp32/ directly...

tve · 2020-09-25T05:14:31Z

ports/esp32/modnetwork.c

@@ -491,7 +490,7 @@ STATIC mp_obj_t esp_ifconfig(size_t n_args, const mp_obj_t *args) {
    tcpip_adapter_ip_info_t info;
    tcpip_adapter_dns_info_t dns_info;
    tcpip_adapter_get_ip_info(self->if_id, &info);
-    tcpip_adapter_get_dns_info(self->if_id, TCPIP_ADAPTER_DNS_MAIN, &dns_info);
+    tcpip_adapter_get_dns_info(self->if_id, ESP_NETIF_DNS_MAIN, &dns_info);


Compat with older esp-idf?

Not sure, but easy to make it compat.

dpgeorge · 2020-09-25T06:24:28Z

This seems an unavoidable change in the long run. I'm not a fan of the esp-idf config/make system, but it's what everyone using the esp32 kind'a expects. The less MP reinvents the better in general...

I feel the same, I'd rather pure make but I think we need to move to cmake. I tried updating the IDF to v4.1 with the existing Makefile approach but could not get it working after about 2 hours (eventually ended up with a link error for libc/libm), and then thought time would be better spent looking into cmake, hence this PR.

For esp32s2 support (and other esp32 series chips) I think the cmake approach is necessary (just too complex otherwise, time gets wasted on duplicating effort already done by the IDF).

The fact that one gets 59% through the compilation and then it complains about mpy-cross isn't very user-friendly:

Can be fixed by a Makefile wrapper that checks for mpy-cross (or a cmake rule...).

The compilation seems to spit out a lot more warnings than when using the old makefile. I guess that's good, but only really if they're going to be addressed?

Not sure why they are popping up but, yes, they'd need to be dealt with (silenced or fixed).

After a make running ESPBAUD=921600 make flash is very slow because it re-generates qstr headers, and it's awfully verbose (I just wanted to flash what's already built and not see the full list of what didn't get rebuilt...).

Yes... the verbosity is part of cmake I guess and cannot be changed. The qstr regeration is a pain and I could not work out how to fix it, it might not be possible to fix because cmake has a more restricted concept of dependencies than make.

With the old makefile, if I run touch main.c; make it takes just shy of 12 seconds on my machine. With the new stuff touch main.c; make takes 26 seconds. Not great.

That's mostly down to qstr regeneration, again.

I suppose to really make MP a component of esp-idf the config stuff needs to be addressed too, right?

The build system doesn't need to use menuconfig, it can continue to have sdkconfig fragments, it just needs to be set up that way.

In the end it's a choice between:

existing make approach, with a burden (and time sink) when IDF needs to be updated
cmake approach as per this PR, which is much cleaner, can more easily support esp32s2 etc and IDF updates, but is slower to build

We could support both make and cmake.... and maybe that's a solution for now, until cmake has proven itself (and users wanting to use cmake/esp32s2 can). From my point of view, I have a preference to switch to cmkae completely and remove make support (although have a simple Makefile to wrap cmake so existing make BOARD=xyz works).

tve · 2020-09-25T06:34:07Z

It would be really good to fix the qstr issue.. Ugh.

I agree with your overall comments. I would switch completely and only bring back the Makefile if there are real issues uncovered. Just because of less maintenance...

dpgeorge · 2020-09-25T07:04:32Z

It would be really good to fix the qstr issue.. Ugh.

Agreed... I already spent way too long trying to fix this, but my cmake skills are low so I could be missing something simple.

I agree with your overall comments. I would switch completely and only bring back the Makefile if there are real issues uncovered. Just because of less maintenance...

Yes, I'd rather spend time working on real issues (eg uevent, ssl, features, etc) than fixing and maintaining the build.

So, I think that's a plan then, to move fully to cmake... (if anyone else has an opinion, please raise it).

Signed-off-by: Damien George <damien@micropython.org>

dpgeorge · 2020-10-06T07:16:37Z

This is updated to: 1) fix qstr generation so it's faster, now very similar to how it works with normal make; 2) remove build warnings.

Signed-off-by: Damien George <damien@micropython.org>

MaureenHelm · 2020-10-07T01:56:53Z

So, I think that's a plan then, to move fully to cmake... (if anyone else has an opinion, please raise it).

Do you mean just for the esp32 port or for all ports? Zephyr already uses cmake and I've been working on a refactoring in #6392

dpgeorge · 2020-10-07T02:19:16Z

Do you mean just for the esp32 port or for all ports?

Aah, sorry, I meant just for the esp32 port. But of course other ports like Zephyr that have an SDK using cmake could also switch to use cmake for everything, if that makes sense.

IhorNehrutsa · 2020-10-30T08:55:36Z

Is it possible to use CMake under Windows with Espressif Standard Setup of Toolchain for Windows?
ESP-IDF Windows Tools Installer
That would be great for me.
Thanks.

p1ng0o · 2020-12-07T09:07:45Z

Tested successfully on esp-idf 4.2.

Also, I added support for ubluetooth module.

dpgeorge#14

tve · 2020-12-24T00:36:52Z

I pulled in pumelo's https://github.com/pumelo/micropython/tree/esp32-cmake and change MPY_ to MICROPY_ for the prefixes to match the rest. Works pretty well for me so far...

I also pulled in and hacked on dpgeorge#14 to get BLE to work. For me this ends up overflowing iram0_0_seg by 3808 bytes, which is a significant amount. This using the esp-idf v4.1 tag. I'm not sure what to do about this. I increased the iram segment size by 4KB, but that's not exactly satisfying. But it allowed me to do a make size-components...

I tried the release/v4.2 branch and that results in a pile of errors saying that esp_system.h is not found, plus a bunch more header files that must have moved. Dunno what @p1ng0o did that made it work...

Is there any hope to get this stuff finalized anytime soon?

Edit:

I needed to add a dependency on the esp_system and esp_timer components
in order to get iram0_0_seg not to overflow I had to add:

CONFIG_WIFI_IRAM_OPT=n
CONFIG_WIFI_RX_IRAM_OPT=n
CONFIG_FREERTOS_UNICORE=y
CONFIG_COMPILER_OPTIMIZATION_DEFAULT=n
CONFIG_COMPILER_OPTIMIZATION_SIZE=y
CONFIG_COMPILER_OPTIMIZATION_ASSERTIONS_ENABLE=y
CONFIG_COMPILER_OPTIMIZATION_ASSERTIONS_SILENT=n
CONFIG_COMPILER_OPTIMIZATION_ASSERTIONS_DISABLE=n

Some more options suggested by igrr: https://www.esp32.com/viewtopic.php?f=13&t=18683&p=69415#p69415

tve · 2020-12-24T20:21:33Z

FYI, I took a stab at CI: tve@c42bff4
No PR 'cause you don't have any of the git workflow stuff in your branch...

dpgeorge · 2021-02-09T03:21:17Z

I was able to build this PR using IDF v4.0.2 and basic smoke tests pass, even connecting to the WiFi.

dpgeorge · 2021-02-15T06:02:55Z

Superseded by #6892, which was merged in 66098c0 through e017f27

Wi-Fi autoconnect and title bar status

dpgeorge added the port-esp32 label Sep 23, 2020

dpgeorge mentioned this pull request Sep 23, 2020

[WIP] esp32/Makefile: Move to v4.1 #6413

Closed

pumelo reviewed Sep 23, 2020

View reviewed changes

tve reviewed Sep 25, 2020

View reviewed changes

dpgeorge changed the title ~~esp32: Add support to build using IDF cmake, with MicroPython as a cmake component~~ WIP: esp32: Add support to build using IDF cmake, with MicroPython as a cmake component Sep 25, 2020

dpgeorge added 4 commits October 6, 2020 17:16

py,extmod: Add core cmake rule files.

31dd25e

Signed-off-by: Damien George <damien@micropython.org>

esp32: Update to support IDF v4.1.

0ddce69

Signed-off-by: Damien George <damien@micropython.org>

esp32: Add support to build using IDF with cmake.

35f6c3d

Signed-off-by: Damien George <damien@micropython.org>

esp32/esp32_rmt: Don't do unnecessary check for unsigned less than zero.

6f5e026

dpgeorge force-pushed the esp32-cmake branch from 8582817 to 6f5e026 Compare October 6, 2020 07:09

travis: Update esp32 CI to build with cmake.

5ea9433

Signed-off-by: Damien George <damien@micropython.org>

dpgeorge added 5 commits October 6, 2020 18:40

travis: Reorder esp32 install rules.

9d61245

Signed-off-by: Damien George <damien@micropython.org>

travis: Checkout all submodules of esp-idf.

c38f627

Signed-off-by: Damien George <damien@micropython.org>

travis: Install setuptools for idf.

bf2ee6c

Signed-off-by: Damien George <damien@micropython.org>

travis: Print cmake version and don't use -S or -B options.

b6b68a3

Signed-off-by: Damien George <damien@micropython.org>

travis: Explicitly use PYTHON=python3 for cmake build.

cf4a64b

Signed-off-by: Damien George <damien@micropython.org>

dpgeorge mentioned this pull request Oct 7, 2020

zephyr: Use cmake ExternalProject to build libmicropython.a. #6392

Closed

MaureenHelm mentioned this pull request Oct 11, 2020

zephyr: Build MicroPython as a cmake target. #6542

Closed

dpgeorge mentioned this pull request Oct 22, 2020

Enable extra warnings for mpy-cross and unix port #6510

Merged

tve mentioned this pull request Oct 24, 2020

esp32: remove unix socket error munging #6569

Closed

tve mentioned this pull request Dec 25, 2020

esp32: switch esp-idf to v4.2, add CI, support MICROPY_BOARD #6731

Closed

amirgon mentioned this pull request Jan 14, 2021

Added draft with submodules and CMakeLists [untested] lvgl/lv_lib_rlottie#1

Merged

3 tasks

dpgeorge mentioned this pull request Feb 13, 2021

esp32: Convert to build using cmake as an IDF component (IDF v4.0.2) #6892

Merged

2 tasks

dpgeorge closed this Feb 15, 2021

dpgeorge deleted the esp32-cmake branch April 14, 2021 03:02

tannewt pushed a commit to tannewt/circuitpython that referenced this pull request Jun 15, 2022

Merge pull request micropython#6473 from tannewt/auto_wifi

5c941ca

Wi-Fi autoconnect and title bar status

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: esp32: Add support to build using IDF cmake, with MicroPython as a cmake component #6473

WIP: esp32: Add support to build using IDF cmake, with MicroPython as a cmake component #6473

dpgeorge commented Sep 23, 2020

dpgeorge commented Sep 23, 2020

UnexpectedMaker commented Sep 23, 2020

dpgeorge commented Sep 23, 2020

UnexpectedMaker commented Sep 23, 2020

pumelo left a comment

pumelo Sep 23, 2020

dpgeorge Sep 23, 2020

pumelo commented Sep 23, 2020

dpgeorge commented Sep 23, 2020

tve left a comment

tve Sep 25, 2020

dpgeorge Sep 25, 2020

tve Sep 25, 2020

tve Sep 25, 2020

dpgeorge Sep 25, 2020

tve Sep 25, 2020

dpgeorge Sep 25, 2020

dpgeorge commented Sep 25, 2020

tve commented Sep 25, 2020

dpgeorge commented Sep 25, 2020

dpgeorge commented Oct 6, 2020

MaureenHelm commented Oct 7, 2020

dpgeorge commented Oct 7, 2020

IhorNehrutsa commented Oct 30, 2020

p1ng0o commented Dec 7, 2020

tve commented Dec 24, 2020 •

edited

Loading

tve commented Dec 24, 2020

dpgeorge commented Feb 9, 2021

dpgeorge commented Feb 15, 2021

		@@ -0,0 +1,114 @@
		# Set location of base MicroPython directory, and this port's directory.

WIP: esp32: Add support to build using IDF cmake, with MicroPython as a cmake component #6473

WIP: esp32: Add support to build using IDF cmake, with MicroPython as a cmake component #6473

Conversation

dpgeorge commented Sep 23, 2020

dpgeorge commented Sep 23, 2020

UnexpectedMaker commented Sep 23, 2020

dpgeorge commented Sep 23, 2020

UnexpectedMaker commented Sep 23, 2020

pumelo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pumelo commented Sep 23, 2020

dpgeorge commented Sep 23, 2020

tve left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dpgeorge commented Sep 25, 2020

tve commented Sep 25, 2020

dpgeorge commented Sep 25, 2020

dpgeorge commented Oct 6, 2020

MaureenHelm commented Oct 7, 2020

dpgeorge commented Oct 7, 2020

IhorNehrutsa commented Oct 30, 2020

p1ng0o commented Dec 7, 2020

tve commented Dec 24, 2020 • edited Loading

tve commented Dec 24, 2020

dpgeorge commented Feb 9, 2021

dpgeorge commented Feb 15, 2021

tve commented Dec 24, 2020 •

edited

Loading