Skip to content

Super enhancement for ESP8266: software-initiated MicroPython base firmware update from image file on LFS2 file-system and dynamic frozen modules #12429

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 69 commits into
base: master
Choose a base branch
from

Conversation

xuancong84
Copy link
Contributor

@xuancong84 xuancong84 commented Sep 12, 2023

This PR added support for software-initiated OTA/non-OTA firmware update from a firmware image file stored on the internal LFS2 file-system, and managing dynamic frozen modules.

  1. Full device firmware update (DFU) (from a file)

This provides a more useful and practical way of full MicroPython base firmware update. Typically, one can create an ESP8266 web server (using e.g., microWebSrv, microdot, etc.) and upload the new firmware image file firmware.bin onto the LFS2 file-system; after that, instruct the backend server to call inisetup.fwupdate('firmware.bin') to update to the new MicroPython firmware.

This implementation of DFU works by first reading through the entire firmware image file so as to dig out all sectors and offsets using a hook in LFS2. After that, invoke the IRAM function esp.DFU() to erase needed sectors, read the firmware image and write it to the flash sector by sector. This function and all the functions it calls must not lie in the ROM (that is mapped to the 1st 1MB of flash) so that the entire 1st 1MB can be flashed without the need to ping-ponging between 2 usable sections of the firmware image (as in yaota8266's OTA solution). Thus, this resolution allows MicroPython base code size to grow to the full 1MB (including the bootloader). Moreover, you can also use this DFU function to flash a non-MicroPython ESP8266 firmware image to switch into non-MicroPython ESP8266 firmware.

Usage:

from inisetup import fwupdate
fwupdate('firmware-image-file.bin', erase_all=False, safe_check=True, verbose=True)

By default, the file-system will not be touched unless erase_all=True is set.

  1. Dynamic frozen modules (DFM)

You can now add/remove/list dynamic frozen modules (named as compared to static frozen modules that cannot be changed without re-flashing MicroPython firmware). Dynamic frozen modules are stored on sectors immediately after the main firmware on the 1st 1MB ROM-mapped flash. Importing DFM takes slightly (about 10%) more RAM than static frozen module, but it takes about 40-45% less RAM than importing .mpy files, which in turn, takes less than half of the memory than importing .py files (because compiling .py files consumes huge amount of memory (5-10 times of the .mpy file size)).

Functions added to the esp module for managing DFMs:

  • esp.add_frozen(file_name, module_name) : add a DFM (with <file_name>) as a module (of name <module_name>) to ROM from an .mpy file on the file-system; if already exists, overwrite.
  • esp.del_frozen(module_name): delete a DFM from ROM
  • esp.ls_frozen(): list DFMs in the ROM
  • esp.reset_frozen(): delete all DFMs in the ROM

Take note: when a module's .mpy file exists both on the filesystem and in the ROM, the version on the filesystem will be loaded, so more memory will be consumed. To avoid that, delete the .mpy file from the filesystem or move it to a different folder or rename it.

@github-actions
Copy link

github-actions bot commented Sep 12, 2023

Code size report:


@xuancong84 xuancong84 changed the title Major enhancement for ESP8266: MicroPython firmware update from image file on internal file-system (LFS2) Major enhancement for ESP8266: software-initiated MicroPython base firmware update from image file on LFS2 file-system Sep 12, 2023
@xuancong84 xuancong84 force-pushed the master branch 2 times, most recently from 006b99e to be9ead9 Compare September 12, 2023 16:31
@codecov
Copy link

codecov bot commented Sep 12, 2023

Codecov Report

Attention: Patch coverage is 81.48148% with 5 lines in your changes missing coverage. Please review.

Project coverage is 98.40%. Comparing base (f1bdac3) to head (f231bf8).
Report is 6 commits behind head on master.

Files with missing lines Patch % Lines
py/persistentcode.c 77.27% 5 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master   #12429      +/-   ##
==========================================
- Coverage   98.43%   98.40%   -0.03%     
==========================================
  Files         163      163              
  Lines       21294    21306      +12     
==========================================
+ Hits        20960    20967       +7     
- Misses        334      339       +5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@xuancong84 xuancong84 force-pushed the master branch 2 times, most recently from 488fedd to d192877 Compare September 12, 2023 17:38
@jimmo
Copy link
Member

jimmo commented Sep 13, 2023

Thanks @xuancong84 - This is definitely a much more straightforward solution than e.g. yaota8266. However a number of issues need to be addressed:

  1. The branch needs to be cleanly rebased, no merge commits, etc and just a couple commits corresponding to the independent changes in the PR.
  2. There are a lot of unrelated changes here (formatting, other changes, etc), these need to be removed.
  3. We don't want to modify the littlefs driver, as this tracks the upstream version (it would ideally be a submodule if it weren't for the prefix/renaming). I understand why you have added this change to get the blocks and offsets, but I don't think this is something we're likely to want to merge. Can we explore other options -- for example one option would be to have a dedicated region of flash to write the firmware image to (you could even make a simple filesystem for this region that ensures the file is contiguous, which means that it's still very easy to write to from e.g. your web server). Have a look at Add VfsMap filesystem, mpremote deploy-mapfs, and ability to import .mpy files from ROM (WIP) #8381 which requires solving the same problem.
  4. On stm32 we have a custom bootloader (mboot) which supports loading firmware images from the filesystem. It also supports verifying the image against a signing key, which may be a necessary feature for people doing OTA? (But I imagine for many users this might not actually be that important...?)

@xuancong84
Copy link
Contributor Author

I have force committed many times so as to get pass as many checks as possible, but I am really not sure how to rebase cleanly as so to pass all checks, maybe you can help me.

Regarding the LittleFS driver issue, I had considered the choice by copying all relevant functions in LFS driver code into the DFU module. However, if there is some new changes in the LFS driver that changes the way file data blocks are located, the copied-over code will break. I am wondering whether it is possible to pull LFS driver code and auto merge instead of cloning the entire driver as it is?

@jimmo
Copy link
Member

jimmo commented Sep 13, 2023

I have force committed many times so as to get pass as many checks as possible, but I am really not sure how to rebase cleanly as so to pass all checks, maybe you can help me.

git fetch origin  # replace origin with whatever remote you have that points to micropython/micropython
git reset --soft origin/master
git commit -a

This will:

  • fetch the latest upstream code (use git remote -v to find out what to use for origin)
  • reset your branch to be the same as master, with just the differences from your branch (this is what --soft does)
  • make a new commit with just your changes

@jimmo
Copy link
Member

jimmo commented Sep 13, 2023

(And git push -f at the end once you've re-made your commit. Despite what git and many guides tell you, force push is a totally normal and expected thing to do)

@xuancong84 xuancong84 force-pushed the master branch 2 times, most recently from ed0ed15 to c45833a Compare September 13, 2023 08:38
@xuancong84
Copy link
Contributor Author

xuancong84 commented Sep 13, 2023

Thanks @jimmo . Now only 1 check failed. I really do not understand why code format check fails. It says

Error: Process completed with exit code 1.

What does it mean?

@robert-hh
Copy link
Contributor

If you look at the details of the failed test, you see the places where it failed. Usually that can be cured by running uncrustify on the file with the configuration for Micropython. The configuration is at micropython/tools/uncrustify,cfg. The command line I use is:

uncrustify -c uncrustify.cfg <filesname>

I run uncrustify version Uncrustify_d-0.72.0

@jimmo
Copy link
Member

jimmo commented Sep 13, 2023

Error: Process completed with exit code 1.
What does it mean?

The rest of the output above that line is the diff reported by the code formatting tool.

You can also run this locally as per the code conventions documentation: https://github.com/micropython/micropython/blob/master/CODECONVENTIONS.md#code-auto-formatting

@jimmo
Copy link
Member

jimmo commented Sep 13, 2023

The configuration is at micropython/tools/uncrustify,cfg. The command line I use is:

You can't run uncrustify directly, as there are additional rules applied (that cannot be expressed in uncrustify config). You need to use tools/codeformat.py

@xuancong84 xuancong84 force-pushed the master branch 2 times, most recently from a8da95b to bdccaa3 Compare September 13, 2023 15:55
@xuancong84
Copy link
Contributor Author

xuancong84 commented Sep 13, 2023

Thanks to @jimmo and @robert-hh , all checks passed. Now comes back to the LFS hook issue that I discussed previously:

Regarding the LittleFS driver issue, I had considered the choice by copying all relevant functions in LFS driver code into the DFU module. However, if there is some new changes in the LFS driver that changes the way file data blocks are located, the copied-over code will break. I am wondering whether it is possible to pull LFS driver code and auto merge instead of cloning the entire driver as it is?

@jimm
Copy link

jimm commented Sep 13, 2023

Correction: I think you mean @jimmo not me (@jimm). :-)

@xuancong84
Copy link
Contributor Author

Correction: I think you mean @jimmo not me (@jimm). :-)

Sorry for the incorrect "autocomplete". I have corrected. Thanks!

@xuancong84
Copy link
Contributor Author

xuancong84 commented Sep 19, 2023

d6e431c Sorry, I did not know that variables defined in boards/ESP8266_GENERIC/mpconfigboard.mk only affects ports/esp8266, but does not affect lib/littlefs/lfs2.c. Now fixed!

@xuancong84 xuancong84 force-pushed the master branch 2 times, most recently from 6cf679a to 3d967ba Compare September 22, 2023 05:23
@xuancong84 xuancong84 changed the title Major enhancement for ESP8266: software-initiated MicroPython base firmware update from image file on LFS2 file-system Super enhancement for ESP8266: software-initiated MicroPython base firmware update from image file on LFS2 file-system and dynamic frozen modules Sep 22, 2023
pi-anl and others added 23 commits November 18, 2024 23:27
This commit fixes PWM configuration across C3, C6, S2 and S3 chips, which
was broken by 6d79937.  Without this fix
the PWM frequency is limited to a maximum of 2446Hz (on S2 at least).

Signed-off-by: Andrew Leech <andrew@alelec.net>
The PIC16 port didn't catch up with the other ports, so it required a bit
of work to make it build with the latest version of XC16.

Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
In `deque_subscr()`, if `index_val` equals `self->alloc`, the index
correction `index_val -= self->alloc` does not execute, leading to an
out-of-bounds access in `self->items[index_val]`.

The fix in this commit ensures that the index correction is applied
whenever `index_val >= self->alloc`, preventing access beyond the allocated
buffer size.

Signed-off-by: Jan Sturm <jansturm92@googlemail.com>
This was missed in 628abf8.  The the bug
was that, when IPv6 is enabled, the `sizeof(ip_addr_t)` is much larger than
IPv4 size, which is what's needed for IGMP addressing.

Fixes issue micropython#16100.

Signed-off-by: Damien George <damien@micropython.org>
The cleanup in 548babf relies on some functions not available in older
ESP-IDF. Temporarily restore them, until we drop support for ESP-IDF <5.2.

PWM functionality should end up the same regardless of ESP-IDF version, and
also no different from MicroPython V1.23.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
Seemingly ESP-IDF incorrectly marks RTC FAST memory region
as MALLOC_CAP_EXEC on ESP32-S2 when it isn't. This memory is
the lowest priority, so it only is returned if D/IRAM is exhausted.

Apply this workaround to treat the allocation as failed if it gives us
non-executable RAM back, rather than crashing.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
mpremote error messages now go to stderr, so make sure stdout is flushed
before printing them.

Also update the test runner to capture error messages.

Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
This fixes a regression in db59e55: prior
to that commit `mpremote` supported trailing slashes on the destination of
a normal (non-recursive) copy.

Add back support for that, with the semantics that a trailing slash
requires the destination to be an existing directory.

Also add a test for this.

Signed-off-by: Damien George <damien@micropython.org>
Because the `ai_canonname` field is subsequently used.

ESP32_GENERIC_S3 (at least) crashes with IDF 5.2.3 without this set.

Signed-off-by: Damien George <damien@micropython.org>
Commit f4ab9d9 inadvertently broke some
Python block devices, for example esp32 and stm32 SDCard classes.  Those
classes return a bool from their `readblocks` and `writeblocks` methods
instead of an integer errno code.  With that change, both `False` and
`True` return values are now be interpreted as non-zero and hence the block
device call fails.

The fix in this commit is to allow a bool and explicitly convert `True` to
0 and `False` to `-MP_EIO`.

Signed-off-by: Damien George <damien@micropython.org>
This function is documented to return True if any stations are connected to
the AP. Without this fix it returns True whenever the driver has brought
the AP interface up.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
The `num_stas` was uninitialised and if it happened to take the value 0
then no results were returned.  It now has the correct maximum value.

Signed-off-by: Damien George <damien@micropython.org>
Configuring the AP for cyw43 writes to some buffers that are only sent to
the modem when the interface is brought up. This means you can't configure
the AP after calling active(True), the new settings seem to be accepted but
the radio doesn't change.

This is different to the WLAN behaviour on other ports. The esp8266 port
requires calling active(True) on the AP before configuring, even.

Fix this by bouncing the AP interface after a config change, if it's
active. Configuring with active(False) still works the same as before.

Adds a static variable to track interface active state, rather than relying
on the LWIP interface state. This is because the interface state is updated
by a driver callback and there's a race: if code calls active(True) and
then config(a=b) then the driver doesn't know it's active yet and the
changes aren't correctly applied.

It is possible this pattern will cause the AP to come up briefly with the
default "PICOabcd" SSID before being reconfigured, however (due to the
aforementioned race condition) it seems like this may not happen at all
before the new config is applied.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
- Previously the call to esp_wifi_set_channel() would be immediately
  overridden by calling esp_wifi_config(...) with the previous channel set.

- AP interface doesn't seem to need more than esp_wifi_config(...) to work.
  It will automatically configure 40MHz bandwidth and place the secondary
  channel using similar logic to what was being explicitly calculated here.

- However, calling esp_wifi_set_channel() on the STA interface is necessary
  if using this interface with ESP-NOW (without connecting to an AP). So
  the esp_wifi_set_channel() call is kept in for this purpose. Without
  this, tests/multi_espnow/70_channel.py fails.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
ESP32 has hardware V1 and S2/S3 has V2, and future chips
may have different versions.

This should still compile to the same binary before and after.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
Closes micropython#13178.

TouchPad confirmed working on both chips, and fixes the the ESP32-S3
reading constant max value. Was unable to reproduce the bug on ESP32-S2 but
this may be due to my test setup, and it still works with the fix.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
This fixes a bug in FrameBuffer.ellipse where it goes into an infinite loop
if both radii are 0.

This fixes the bug with a simple pre-check to see if both radii are 0, and
in that case sets a single pixel at the center. This is consistent with the
behaviour of the method when called with just one of the radii set to 0,
where it will draw a horizontal or vertical line of 1 pixel width.

The pixel is set with setpixel_checked so it should handle out-of-bounds
drawing correctly.

This fix also includes three new tests: one for the default behaviour, one
for drawing out-of-bounds, and one for when the sector mask is 0.

Fixes issue micropython#16053.

Signed-off-by: Corran Webster <cwebster@unital.dev>
The micro:bit board (and probably other boards using the music or display
module) locked up on soft reboot.  Reason was a buffer overflow caused by
an index counter, which was not reset on soft_reboot.

That's fixed in this commit.  Tested with a micro:bit board, performing a
series of soft reboots.

Signed-off-by: robert-hh <robert@hammelrath.com>
Recent MSVC versions have changed the definition of NAN to a non-constant
expression!  This is a bug, C standard says it should be a constant.

Good explanation and workaround at: https://stackoverflow.com/a/79199887

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
Signed-off-by: Damien George <damien@micropython.org>
Copy link
Contributor

@smurfix smurfix Mar 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I almost hate to ask, but what are tests for empty ellipses doing in a "firmeware update from LFS" PR?

@@ -2246,12 +2246,12 @@ static mp_obj_t mp_obj_new_str_type_from_vstr(const mp_obj_type_t *type, vstr_t
}

byte *data;
if (vstr->len + 1 == vstr->alloc) {
if (vstr->alloc == vstr->len + 1) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason for this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could it be due to that some strings might be null-terminated? So for example, the string "banana" is of length 6, but you typically need to allocate 7 bytes (last character to be null) to be safe for functions like strlen(), strcmp(), or strcpy() because they look for the null character.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Umm, no? This is just two integers, both of type size_t.

…ard interrupt on ESP32*

This is useful for ESP32* as it allows STDIO ports to be used as UART ports for interacting with external devices.
mp_hal_set_interrupt_char(int) is supposed to work but does not work because every time when
parse_compile_execute() is called, the intr_char is reset to CHAR_CTRL_C depending on exec_flags.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.