-
-
Notifications
You must be signed in to change notification settings - Fork 8.2k
Super enhancement for ESP8266: software-initiated MicroPython base firmware update from image file on LFS2 file-system and dynamic frozen modules #12429
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
Code size report:
|
006b99e
to
be9ead9
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #12429 +/- ##
==========================================
- Coverage 98.43% 98.40% -0.03%
==========================================
Files 163 163
Lines 21294 21306 +12
==========================================
+ Hits 20960 20967 +7
- Misses 334 339 +5 ☔ View full report in Codecov by Sentry. |
488fedd
to
d192877
Compare
Thanks @xuancong84 - This is definitely a much more straightforward solution than e.g. yaota8266. However a number of issues need to be addressed:
|
I have force committed many times so as to get pass as many checks as possible, but I am really not sure how to rebase cleanly as so to pass all checks, maybe you can help me. Regarding the LittleFS driver issue, I had considered the choice by copying all relevant functions in LFS driver code into the DFU module. However, if there is some new changes in the LFS driver that changes the way file data blocks are located, the copied-over code will break. I am wondering whether it is possible to pull LFS driver code and auto merge instead of cloning the entire driver as it is? |
This will:
|
(And |
ed0ed15
to
c45833a
Compare
Thanks @jimmo . Now only 1 check failed. I really do not understand why code format check fails. It says
What does it mean? |
If you look at the details of the failed test, you see the places where it failed. Usually that can be cured by running uncrustify on the file with the configuration for Micropython. The configuration is at micropython/tools/uncrustify,cfg. The command line I use is:
I run uncrustify version |
The rest of the output above that line is the diff reported by the code formatting tool. You can also run this locally as per the code conventions documentation: https://github.com/micropython/micropython/blob/master/CODECONVENTIONS.md#code-auto-formatting |
You can't run uncrustify directly, as there are additional rules applied (that cannot be expressed in uncrustify config). You need to use tools/codeformat.py |
a8da95b
to
bdccaa3
Compare
Thanks to @jimmo and @robert-hh , all checks passed. Now comes back to the LFS hook issue that I discussed previously:
|
0ec341e
to
d6e431c
Compare
d6e431c Sorry, I did not know that variables defined in |
6cf679a
to
3d967ba
Compare
This commit fixes PWM configuration across C3, C6, S2 and S3 chips, which was broken by 6d79937. Without this fix the PWM frequency is limited to a maximum of 2446Hz (on S2 at least). Signed-off-by: Andrew Leech <andrew@alelec.net>
The PIC16 port didn't catch up with the other ports, so it required a bit of work to make it build with the latest version of XC16. Signed-off-by: Alessandro Gatti <a.gatti@frob.it>
In `deque_subscr()`, if `index_val` equals `self->alloc`, the index correction `index_val -= self->alloc` does not execute, leading to an out-of-bounds access in `self->items[index_val]`. The fix in this commit ensures that the index correction is applied whenever `index_val >= self->alloc`, preventing access beyond the allocated buffer size. Signed-off-by: Jan Sturm <jansturm92@googlemail.com>
This was missed in 628abf8. The the bug was that, when IPv6 is enabled, the `sizeof(ip_addr_t)` is much larger than IPv4 size, which is what's needed for IGMP addressing. Fixes issue micropython#16100. Signed-off-by: Damien George <damien@micropython.org>
The cleanup in 548babf relies on some functions not available in older ESP-IDF. Temporarily restore them, until we drop support for ESP-IDF <5.2. PWM functionality should end up the same regardless of ESP-IDF version, and also no different from MicroPython V1.23. Signed-off-by: Angus Gratton <angus@redyak.com.au>
Seemingly ESP-IDF incorrectly marks RTC FAST memory region as MALLOC_CAP_EXEC on ESP32-S2 when it isn't. This memory is the lowest priority, so it only is returned if D/IRAM is exhausted. Apply this workaround to treat the allocation as failed if it gives us non-executable RAM back, rather than crashing. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <angus@redyak.com.au>
mpremote error messages now go to stderr, so make sure stdout is flushed before printing them. Also update the test runner to capture error messages. Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
This fixes a regression in db59e55: prior to that commit `mpremote` supported trailing slashes on the destination of a normal (non-recursive) copy. Add back support for that, with the semantics that a trailing slash requires the destination to be an existing directory. Also add a test for this. Signed-off-by: Damien George <damien@micropython.org>
Because the `ai_canonname` field is subsequently used. ESP32_GENERIC_S3 (at least) crashes with IDF 5.2.3 without this set. Signed-off-by: Damien George <damien@micropython.org>
Commit f4ab9d9 inadvertently broke some Python block devices, for example esp32 and stm32 SDCard classes. Those classes return a bool from their `readblocks` and `writeblocks` methods instead of an integer errno code. With that change, both `False` and `True` return values are now be interpreted as non-zero and hence the block device call fails. The fix in this commit is to allow a bool and explicitly convert `True` to 0 and `False` to `-MP_EIO`. Signed-off-by: Damien George <damien@micropython.org>
This function is documented to return True if any stations are connected to the AP. Without this fix it returns True whenever the driver has brought the AP interface up. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <angus@redyak.com.au>
The `num_stas` was uninitialised and if it happened to take the value 0 then no results were returned. It now has the correct maximum value. Signed-off-by: Damien George <damien@micropython.org>
Configuring the AP for cyw43 writes to some buffers that are only sent to the modem when the interface is brought up. This means you can't configure the AP after calling active(True), the new settings seem to be accepted but the radio doesn't change. This is different to the WLAN behaviour on other ports. The esp8266 port requires calling active(True) on the AP before configuring, even. Fix this by bouncing the AP interface after a config change, if it's active. Configuring with active(False) still works the same as before. Adds a static variable to track interface active state, rather than relying on the LWIP interface state. This is because the interface state is updated by a driver callback and there's a race: if code calls active(True) and then config(a=b) then the driver doesn't know it's active yet and the changes aren't correctly applied. It is possible this pattern will cause the AP to come up briefly with the default "PICOabcd" SSID before being reconfigured, however (due to the aforementioned race condition) it seems like this may not happen at all before the new config is applied. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <angus@redyak.com.au>
- Previously the call to esp_wifi_set_channel() would be immediately overridden by calling esp_wifi_config(...) with the previous channel set. - AP interface doesn't seem to need more than esp_wifi_config(...) to work. It will automatically configure 40MHz bandwidth and place the secondary channel using similar logic to what was being explicitly calculated here. - However, calling esp_wifi_set_channel() on the STA interface is necessary if using this interface with ESP-NOW (without connecting to an AP). So the esp_wifi_set_channel() call is kept in for this purpose. Without this, tests/multi_espnow/70_channel.py fails. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <angus@redyak.com.au>
ESP32 has hardware V1 and S2/S3 has V2, and future chips may have different versions. This should still compile to the same binary before and after. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <angus@redyak.com.au>
Closes micropython#13178. TouchPad confirmed working on both chips, and fixes the the ESP32-S3 reading constant max value. Was unable to reproduce the bug on ESP32-S2 but this may be due to my test setup, and it still works with the fix. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <angus@redyak.com.au>
This fixes a bug in FrameBuffer.ellipse where it goes into an infinite loop if both radii are 0. This fixes the bug with a simple pre-check to see if both radii are 0, and in that case sets a single pixel at the center. This is consistent with the behaviour of the method when called with just one of the radii set to 0, where it will draw a horizontal or vertical line of 1 pixel width. The pixel is set with setpixel_checked so it should handle out-of-bounds drawing correctly. This fix also includes three new tests: one for the default behaviour, one for drawing out-of-bounds, and one for when the sector mask is 0. Fixes issue micropython#16053. Signed-off-by: Corran Webster <cwebster@unital.dev>
The micro:bit board (and probably other boards using the music or display module) locked up on soft reboot. Reason was a buffer overflow caused by an index counter, which was not reset on soft_reboot. That's fixed in this commit. Tested with a micro:bit board, performing a series of soft reboots. Signed-off-by: robert-hh <robert@hammelrath.com>
Recent MSVC versions have changed the definition of NAN to a non-constant expression! This is a bug, C standard says it should be a constant. Good explanation and workaround at: https://stackoverflow.com/a/79199887 This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <angus@redyak.com.au>
Signed-off-by: Damien George <damien@micropython.org>
tests/extmod/framebuf_ellipse.py
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I almost hate to ask, but what are tests for empty ellipses doing in a "firmeware update from LFS" PR?
@@ -2246,12 +2246,12 @@ static mp_obj_t mp_obj_new_str_type_from_vstr(const mp_obj_type_t *type, vstr_t | |||
} | |||
|
|||
byte *data; | |||
if (vstr->len + 1 == vstr->alloc) { | |||
if (vstr->alloc == vstr->len + 1) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a reason for this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could it be due to that some strings might be null-terminated? So for example, the string "banana" is of length 6, but you typically need to allocate 7 bytes (last character to be null) to be safe for functions like strlen()
, strcmp()
, or strcpy()
because they look for the null character.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Umm, no? This is just two integers, both of type size_t
.
…ard interrupt on ESP32* This is useful for ESP32* as it allows STDIO ports to be used as UART ports for interacting with external devices. mp_hal_set_interrupt_char(int) is supposed to work but does not work because every time when parse_compile_execute() is called, the intr_char is reset to CHAR_CTRL_C depending on exec_flags.
This PR added support for software-initiated OTA/non-OTA firmware update from a firmware image file stored on the internal LFS2 file-system, and managing dynamic frozen modules.
This provides a more useful and practical way of full MicroPython base firmware update. Typically, one can create an ESP8266 web server (using e.g., microWebSrv, microdot, etc.) and upload the new firmware image file
firmware.bin
onto the LFS2 file-system; after that, instruct the backend server to callinisetup.fwupdate('firmware.bin')
to update to the new MicroPython firmware.This implementation of DFU works by first reading through the entire firmware image file so as to dig out all sectors and offsets using a hook in LFS2. After that, invoke the IRAM function
esp.DFU()
to erase needed sectors, read the firmware image and write it to the flash sector by sector. This function and all the functions it calls must not lie in the ROM (that is mapped to the 1st 1MB of flash) so that the entire 1st 1MB can be flashed without the need to ping-ponging between 2 usable sections of the firmware image (as in yaota8266's OTA solution). Thus, this resolution allows MicroPython base code size to grow to the full 1MB (including the bootloader). Moreover, you can also use this DFU function to flash a non-MicroPython ESP8266 firmware image to switch into non-MicroPython ESP8266 firmware.Usage:
By default, the file-system will not be touched unless
erase_all=True
is set.You can now add/remove/list dynamic frozen modules (named as compared to static frozen modules that cannot be changed without re-flashing MicroPython firmware). Dynamic frozen modules are stored on sectors immediately after the main firmware on the 1st 1MB ROM-mapped flash. Importing DFM takes slightly (about 10%) more RAM than static frozen module, but it takes about 40-45% less RAM than importing
.mpy
files, which in turn, takes less than half of the memory than importing.py
files (because compiling.py
files consumes huge amount of memory (5-10 times of the.mpy
file size)).Functions added to the
esp
module for managing DFMs:.mpy
file on the file-system; if already exists, overwrite.Take note: when a module's
.mpy
file exists both on the filesystem and in the ROM, the version on the filesystem will be loaded, so more memory will be consumed. To avoid that, delete the.mpy
file from the filesystem or move it to a different folder or rename it.