-
-
Notifications
You must be signed in to change notification settings - Fork 8.2k
ports/esp32: Add basic espressif IDF v5.3 compatibility. #15733
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Code size report:
|
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #15733 +/- ##
=======================================
Coverage 98.43% 98.43%
=======================================
Files 163 163
Lines 21295 21295
=======================================
Hits 20961 20961
Misses 334 334 ☔ View full report in Codecov by Sentry. |
ports/esp32/usb_serial_jtag.c
Outdated
@@ -95,6 +95,10 @@ void usb_serial_jtag_poll_rx(void) { | |||
} | |||
} | |||
|
|||
#ifndef USB_SERIAL_JTAG_PACKET_SZ_BYTES | |||
#define USB_SERIAL_JTAG_PACKET_SZ_BYTES USB_SERIAL_JTAG_RDWR_BYTE_S |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The value of USB_SERIAL_JTAG_RDWR_BYTE_S
is 0 which is why you don't get any bytes on stdout any more.
Try defining to 64 for now, that was the value before the macro was removed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for that!
I did scan the 5.3 migration guide and didn't see anything about that but I guess USB_SERIAL_JTAG_RDWR_BYTE_S
wasn't necessarily ever intended as an end user/application variable.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, talking directly to the usb serial peripheral (rather than via stdio) is a private "low level" interface so no guarantees, I think!
I updated to the release/v5.2 branch tip (which is how I came across the usb jtag constant thing) and also needed this patch for sockets to work. It seems now diff --git i/ports/esp32/modsocket.c w/ports/esp32/modsocket.c
index 916eb79bd9..641bcd1170 100644
--- i/ports/esp32/modsocket.c
+++ w/ports/esp32/modsocket.c
@@ -253,7 +253,7 @@ static void _getaddrinfo_inner(const mp_obj_t host, const mp_obj_t portx,
// Somehow LwIP returns a resolution of 0.0.0.0 for failed lookups, traced it as far back
// as netconn_gethostbyname_addrtype returning OK instead of error.
if (*res == NULL ||
- (strcmp(res[0]->ai_canonname, "0.0.0.0") == 0 && strcmp(host_str, "0.0.0.0") != 0)) {
+ (res[0]->ai_canonname && strcmp(res[0]->ai_canonname, "0.0.0.0") == 0 && strcmp(host_str, "0.0.0.0") != 0)) {
lwip_freeaddrinfo(*res);
mp_raise_OSError(-2); // name or service not known
}
@@ -261,7 +261,7 @@ static void _getaddrinfo_inner(const mp_obj_t host, const mp_obj_t portx,
assert(retval == 0 && *res != NULL);
}
-static void _socket_getaddrinfo(const mp_obj_t addrtuple, struct addrinfo **resp) {
+static void _socket_getaddrinfo const mp_obj_t addrtuple, struct addrinfo **resp) {
mp_obj_t *elem;
mp_obj_get_array_fixed_n(addrtuple, 2, &elem);
_getaddrinfo_inner(elem[0], elem[1], NULL, resp);
@@ -951,7 +951,7 @@ static mp_obj_t esp_socket_getaddrinfo(size_t n_args, const mp_obj_t *args) {
mp_obj_new_int(resi->ai_family),
mp_obj_new_int(resi->ai_socktype),
mp_obj_new_int(resi->ai_protocol),
- mp_obj_new_str_from_cstr(resi->ai_canonname),
+ resi->ai_canonname ? mp_obj_new_str_from_cstr(resi->ai_canonname) : mp_const_empty_bytes,
mp_const_none
};
|
Oh weird, I saw that you have v5.3 working maybe without the sockets change! Guess not... |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me! Can confirm seems to at least start up and have basic working functionality with the ESP32, ESP32-C3 and ESP32-S3 boards I just now flashed with these changes.
Changed are almost identical to my own (uncommited) changes to support 5.3 from a month ago. Can confirm that indeed the 64 bytes JTAG serial size is no longer available (removed here), for my changes I did go through the IDF internals a bit to check and the size seems always be 64 for every chip variant, so hardcoding it on our side should be fine.
Two differences that might be worth considering: I did also change the JTAG serial RX buffer to use the packet size instead of its own constant, since I didn't see a reason it should use a different value. And, after looking through NimBLE and seeing ediv_random
was fully removed there, I figured we can probably remove all references to it altogether. And, should we also be updating the list of constants in network_wlan.c
?
(My own uncommited changes)
diff --git a/extmod/nimble/modbluetooth_nimble.c b/extmod/nimble/modbluetooth_nimble.c
index f582661ff..9cb12ee77 100644
--- a/extmod/nimble/modbluetooth_nimble.c
+++ b/extmod/nimble/modbluetooth_nimble.c
@@ -1911,24 +1911,21 @@ static int ble_secret_store_read(int obj_type, const union ble_store_key *key, u
// <type=peer,addr,*> (single)
// Find the entry for this specific peer.
assert(key->sec.idx == 0);
- // assert(!key->sec.ediv_rand_present);
key_data = (const uint8_t *)&key->sec.peer_addr;
key_data_len = sizeof(ble_addr_t);
} else {
// <type=peer,*> (with index)
// Iterate all known peers.
- // assert(!key->sec.ediv_rand_present);
key_data = NULL;
key_data_len = 0;
}
break;
}
case BLE_STORE_OBJ_TYPE_OUR_SEC: {
- // <type=our,addr,ediv_rand>
- // Find our secret for this remote device, matching this ediv/rand key.
+ // <type=our,addr,*>
+ // Find our secret for this remote device.
assert(ble_addr_cmp(&key->sec.peer_addr, BLE_ADDR_ANY)); // Must have address.
assert(key->sec.idx == 0);
- // assert(key->sec.ediv_rand_present);
key_data = (const uint8_t *)&key->sec.peer_addr;
key_data_len = sizeof(ble_addr_t);
break;
@@ -1958,10 +1955,6 @@ static int ble_secret_store_read(int obj_type, const union ble_store_key *key, u
DEBUG_printf("ble_secret_store_read: found secret\n");
- if (obj_type == BLE_STORE_OBJ_TYPE_OUR_SEC) {
- // TODO: Verify ediv_rand matches.
- }
-
return 0;
}
@@ -1970,14 +1963,13 @@ static int ble_secret_store_write(int obj_type, const union ble_store_value *val
switch (obj_type) {
case BLE_STORE_OBJ_TYPE_PEER_SEC:
case BLE_STORE_OBJ_TYPE_OUR_SEC: {
- // <type=peer,addr,edivrand>
+ // <type=peer,addr,*>
struct ble_store_key_sec key_sec;
const struct ble_store_value_sec *value_sec = &val->sec;
ble_store_key_from_value_sec(&key_sec, value_sec);
assert(ble_addr_cmp(&key_sec.peer_addr, BLE_ADDR_ANY)); // Must have address.
- // assert(key_sec.ediv_rand_present);
if (!mp_bluetooth_gap_on_set_secret(obj_type, (const uint8_t *)&key_sec.peer_addr, sizeof(ble_addr_t), (const uint8_t *)value_sec, sizeof(struct ble_store_value_sec))) {
DEBUG_printf("Failed to write key: type=%d\n", obj_type);
@@ -2007,7 +1999,6 @@ static int ble_secret_store_delete(int obj_type, const union ble_store_key *key)
// <type=peer,addr,*>
assert(ble_addr_cmp(&key->sec.peer_addr, BLE_ADDR_ANY)); // Must have address.
- // ediv_rand is optional (will not be present for delete).
if (!mp_bluetooth_gap_on_set_secret(obj_type, (const uint8_t *)&key->sec.peer_addr, sizeof(ble_addr_t), NULL, 0)) {
DEBUG_printf("Failed to delete key: type=%d\n", obj_type);
diff --git a/ports/esp32/usb_serial_jtag.c b/ports/esp32/usb_serial_jtag.c
index 23a73d053..709b93e85 100644
--- a/ports/esp32/usb_serial_jtag.c
+++ b/ports/esp32/usb_serial_jtag.c
@@ -35,10 +35,12 @@
#include "soc/periph_defs.h"
#include "freertos/portmacro.h"
-#define USB_SERIAL_JTAG_BUF_SIZE (64)
+#if ESP_IDF_VERSION >= ESP_IDF_VERSION_VAL(5, 3, 0)
+#define USB_SERIAL_JTAG_PACKET_SZ_BYTES 64
+#endif
static DRAM_ATTR portMUX_TYPE rx_mux = portMUX_INITIALIZER_UNLOCKED;
-static uint8_t rx_buf[USB_SERIAL_JTAG_BUF_SIZE];
+static uint8_t rx_buf[USB_SERIAL_JTAG_PACKET_SZ_BYTES];
static volatile bool terminal_connected = false;
static void usb_serial_jtag_handle_rx(void) {
@@ -48,8 +50,8 @@ static void usb_serial_jtag_handle_rx(void) {
portENTER_CRITICAL(&rx_mux);
}
size_t req_len = ringbuf_free(&stdin_ringbuf);
- if (req_len > USB_SERIAL_JTAG_BUF_SIZE) {
- req_len = USB_SERIAL_JTAG_BUF_SIZE;
+ if (req_len > USB_SERIAL_JTAG_PACKET_SZ_BYTES) {
+ req_len = USB_SERIAL_JTAG_PACKET_SZ_BYTES;
}
size_t len = usb_serial_jtag_ll_read_rxfifo(rx_buf, req_len);
for (size_t i = 0; i < len; ++i) {
@@ -95,10 +97,6 @@ void usb_serial_jtag_poll_rx(void) {
}
}
-#ifndef USB_SERIAL_JTAG_PACKET_SZ_BYTES
-#define USB_SERIAL_JTAG_PACKET_SZ_BYTES USB_SERIAL_JTAG_RDWR_BYTE_S
-#endif
-
void usb_serial_jtag_tx_strn(const char *str, size_t len) {
while (len) {
size_t l = len;
Thanks for sharing your changes, I'll compare a bit later when I get a chance
I wondered that too, it like like they weren't updated for the last couple of versions either. Reflecting these as qstr's does take up flash space so I guess it depends on whether these are things end users are likely to need to reference? |
Yeah I guess sockets worked for the mqtt comms, that was only trialled on C3 though, and not for long. Perhaps there's still other situations a null would need to be checked for / handled |
Since I did the v5.2 PR (#13775) I looked into why I didn't update them there, and the reason is they didn't exist yet, they were added later in #14015 (or to be more precise, these PR's happened in parallel, so I added the new 5.2 constants but didn't add them to If I'm reading it correctly, the decision made in the discussion on that PR was that these constants should live as part of the If that's the case, then we should probably:
|
That is correct.
Yes, I agree with this list of TODOs (see comment immediately above).
Yes, |
Dear All, Is it possible to sponsor a specific effort? Sincerely, |
I'm in the middle of other USB work which is taking up my limited personal time for this kind of work, as such I don't when I'd have time to do more with this PR. It seems like there probably isn't much more work needed though? I'd be quite happy if anyone else wanted to continue working on it or even just testing what's missing here? |
Hi All, Unfortunately, I am not familiar with the PR process at all....though I would be happy to test if someone told me how. Sincerely, |
I've now done this, see #16119. |
I removed the |
I've rebased this on latest master, cleaned up a few things, and force pushed. Tested on ESP32_GENERIC. Everything seems to work except Tested on ESP32_GENERIC_S2, that also locks up trying to run some native code (although I think that's a problem on master...). |
I rebased this on master to get the above-mentioned patches. This PR is now quite a bit simpler. |
Just updated my tree with these commits, building against the latest IDF v5.3.1, and can confirm it seems to be running well so far. |
Note: #16169 has broken building against 5.3 due to espressif/esp-idf@f35ec64 having renamed the |
A small change that seems to fix that:
|
Thanks @DvdGiessen , I have applied your patch. I also rebased this on latest master. @projectgus I think we can merge this, what do you think? We won't official move to 5.3 yet so don't need to exhaustively test this with that IDF version. And the changes should not affect IDF<5.3. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agree. The only thing I (just) noticed missing is adding v5.3 to the list of supported versions in the README.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The only thing I (just) noticed missing is adding v5.3 to the list of supported versions in the README.
@dpgeorge I went ahead and rebased again to add this, so I think it's good to go!
Note as we're adding a version that there is also an open PR to drop older versions - #16128 (Will rebase after this is merged.) |
FWIW I just built with these changes against the newly released IDF v5.3.2 and that also runs fine. :) |
I also did a quick test with a project using an esp32-s3 and v5.3.2 with this branch and everything seemed to work as expected. |
Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au> Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Merged. Thanks everyone who worked on this. |
Yes thank you all! I'm an avid user of the esp32 + micropython and unglamorous tasks like this really help me out. I appreciate you! |
Indeed, I am in the same camp. Excellent progress!
Sincerely,
Duane Kaufman
…----- On Dec 9, 2024, at 10:56 PM, Keenan Johnson ***@***.***> wrote:
Yes thank you all! I'm an avid user of the esp32 + micropython and unglamorous
tasks like this really help me out. I appreciate you!
—
Reply to this email directly, [
#15733 (comment) |
view it on GitHub ] , or [
https://github.com/notifications/unsubscribe-auth/ABK2PQAKDTZ7YMN24PC7VXL2EZX6FAVCNFSM6AAAAABNHM3ZNGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMZQGM4DSMZWGQ
| unsubscribe ] .
You are receiving this because you are subscribed to this thread. Message ID: <
***@***.*** >
|
Summary
Espressif have recently release IDF v5.3, this PR is working towards basic compatibility with it.
Testing
I've initially tried running this on a C3 device which only has the builtin USB exposed (no uart->usb).
I do not get repl on the USB connection however, nothing is seen over the USB port after the initial reboot / startup text:
Trade-offs and Alternatives
Unknown at this point.