extmod: Add generic network.PPP to work with lwIP
#14461
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #14461 +/- ##
=======================================
Coverage 98.43% 98.43%
=======================================
Files 163 163
Lines 21295 21295
=======================================
Hits 20961 20961
Misses 334 334 ☔ View full report in Codecov by Sentry. |
|
Code size report: |
|
That PR looks good, supporting the use of e.g. LTE modems or AT-Type WiFi interfaces. |
|
Thanks @robert-hh for the review.
Yes, thank you for that. That PR will definitely be needed! |
|
I've been trying to get this working with a SimCom A7683E on a Pico W. I seem to be able to bring it up to the point where I it's "connected" and I can see the IP address. I can use Setting up the same chip using Linux ppp on a Raspberry Pi 5 will get us ~11k (saturated 115200 baud rate), and the module also works with some custom Python to use AT commands to make web requests. Any ideas what could be amiss? I guess if this code works on ESP32 and the A76783 works otherwise, it could be related to the RP2's implementation of To elaborate, And attempting to request a file: (note I seem to be getting |
The ESP32 uses a completely different implementation of PPP (provided by the ESP-IDF) so that's not the best comparison except to prove the cellular module works. |
The rp2 port doesn't yet support user-side UART IRQs, so this PR will only work if you either merge #14041 as well as this PR, or call the Apart from that, you probably need to set a timeout on the UART, eg |
|
I merged both this and #14041 though it's done via CI so I'll double check we're actually getting the UART IRQ stuff as expected. I figured if Tried For anyone else who wants to poke this on Pico. The build is here: pimoroni/pimoroni-pico#947 |
dad02e5 to
9a94ca6
Compare
Hello @victorallume, do you have more details about how the PPP implementation in the ESP32 uses the UART? I’m working on a cmux library to multiplex 4 virtual UARTs over the physical UART of a SIMCom modem. The cmux logic is working well but I need to emulate the UART with a stream object to pass it to network.ppp, something is missing because the REPL connection is crashing when I try to connect the PPP with my virtual UART. Please take a look here for more details: https://github.com/orgs/micropython/discussions/9263 |
Not sure; all I know is that the ESP32 implementation of PPP comes from the ESP-IDF (which I think uses LWIP internally) |
|
Still no luck with the SimCom A7683E on RP2, though I've noticed the UART code has been updated since when I last cut a build (yesterday) and now so I'll try bumping that too. Edit: Still no dice. |
|
Okay I have finally got a response out of PPP, and despite merging the UART IRQ pull-request and seeing the tmr = Timer()
tmr.init(mode=Timer.PERIODIC, period=20, callback=ppp.poll)I've just seen the UART IRQ stuff had recent updates, though, so I guess I should try that again. Failing that I will try a minimal repro, since it's clear the IRQ stuff is not working for me (or perhaps I am missing some means of enabling it properly?) Update: To answer my own question, no. The UART IRQ stuff works and the If I hook def poll(u):
if u.irq().flags() & u.IRQ_RXIDLE:
print("poll!")
ppp.poll()
uart.irq(poll, uart.IRQ_RXIDLE)But this does not get me working network requests. Eventually it stalls and... nada. If I use the above code to "manually" poll every 20ms, then I get working network responses instantly. (Edit: Snipped off my nonsense confusing freq with period in the Timer class) |
I think I may have found the problem: upon UART IRQ, I've now fixed this by making sure I also added PPP debug printing as a temporary measure. That will be disabled once things work. |
|
Fantastic! I’ll do my merge dance and give it a try this morning! |
|
Somehow the latest code is running into a hardfault. There doesn't seem to be any pattern as to where. Here are some runs of Once instance even failed before getting any input.... This is a pretty glued together MicroPython including RP2350 fixes (though these tests are running on RP2040), the UART IRQ stuff and this PR so it's possible any number of things have gone awry. Though... this is new! Edit:
|
18ca76a to
654711c
Compare
This could simply be due to the debug output, which will attempt to write to the USB CDC within an lwIP callback, which is probably not safe (it's safe enough for debug purposes but not in general, probably). I've updated this PR to disable the debug printing. Please try it again! |
|
Tried again and was hitting the same stalling issue (with and without polling), so I bumped both Continuing to poke some other settings to see if it'll make any difference, but it's not looking promising and the failure is repeatable. With polling I can see the IRQ fire a couple of times after making a request, and then nothing. Without polling i fires ~12 times and also hangs. Same class of problem, I guess, but it can't be papered over with polling anymore. (Ha, if I do the maths (256 bytes * 8 bits / 115200 bps) and plug in a 10ms poll interval it works again.) I am still having trouble constructing a UART instance after a soft reset, perhaps related to the IRQ stuff. |
I made a small change in the RP2 software that might fix this problem. Not sure since I do not know the state of your code whwn it happens. |
That makes two of us! But thanks for trying- I'll test ASAP. Edit: I think it's made an improvement. No stalls on creating the No further luck getting the cursed thing to work without polling! I'm now doing: POLL_TIME = int(256 * 8 / UART_BAUD * 1000)
uart.irq(lambda x: None, uart.IRQ_RXIDLE)
tmr = Timer()
tmr.init(mode=Timer.PERIODIC, period=POLL_TIME, callback=ppp.poll)Which works great 😆 I've also noticed that if I set |
Okay this was a bug in #14041 and it has now been fixed. Plot twist, now if I set my rxbuf to 1k+ PPP seems to be working 🤔 Maybe the IRQ being garbage collected was my problem all along. 512 bytes does not seem to work, only a couple of requests will be successful (eventual buffer overrun?). I've tested a 1k bytes rxbuf up to 100 successive requests, albeit the payload is quite small. This is all at 115200 baud and agonisingly slow so I'll do some poking at faster speeds to see how those fare too. With the UART fixes if my code doesn't crash and I I think I also tested with a hardware IRQ, replacing the default one by using: Anyway, TLDR: with an
|
That sounds good. Thank you for testing and confirmation, and for revealing the bugs. |
|
Hello guys, I apologize for insisting, but maybe you can help on something that would be very useful for PPP connections with modems. I have a custom library in MicroPython to implement cmux between an ESP32 and a SIMCom modem. It would allow to use 4 virtual UARTs over a physical UART to send AT commands and to have a PPP connection on the same physical UART. It is working well but only failing at the very last step when I pass a virtual UART to network.ppp. For some reason the ppp service is not able to handle the virtual UART even when I’m adding to it all the methods that it should have to handle read and write. It is just crashing the Thonny console when executing ppp.connect(). The same data flow is correctly setting the ppp connection when I pass it to a physical UART of a second microcontroller, which means the cmux handshaking is doing well. Here is my library (https://github.com/MicropythonShare/cmux). Please take a look, I have not been able to find what is missing or what the ppp module is expecting from the virtual UART. This would be useful not only for ESP32 but for any other microcontroller running MicroPython. I’d really appreciate any help on this 🙂 |
|
There's more detail / context on the efforts by @nestorld here: https://github.com/orgs/micropython/discussions/9263 |
|
@nestorld AIUI and at a bleary glance through misty eyes this morning, you’re missing the “irq” method. Called here: micropython/extmod/network_ppp_lwip.c Line 232 in 654711c Your virtual UART could presumably just create a timer and call the “poll” method it’s handed using my timings above. (basically a function of buffer size and baud rate) |
Thanks for helping @Gadgetoid. That irq method seems not to be used in ESP32's UARTs. I have tried creating a fake method like this (like the one shown in the UART documentation), just to confirm if it is invoked at any moment: ...and I never see that print in the console. I have done the same for all the methods in my virtual UART and none of them are invoked between ppp.connect() and the crash. Micropython UART documentation says that the irq method is only available for WiPy, which is not my case. In fact, if I see all the properties and methods for a physical UART in Thonny, this is what I get (no irq method): I don't have big skills with the C code, but here is the last point where I was able to get some printf in the micropython/ports/esp32/network_ppp.c file before the crash (inserting a printf before the return line): mp_stream_rw function is in micropython/py/streams.c file. I suspect about line 50 there:
Where "stream" is the UART, or the virtual UART in this case. |
It will not be available until PR #14041 is merged. |
But is it needed for a virtual UART to emulate what a physical UART does? I would guess no |
|
Based on my (mostly positive) experience working with PPP on ESP32, I'd like to raise a concern regarding DNS in multi-interface configurations. ESP32 docs say:
For example, if you have Ethernet, WiFi, and PPP interfaces, the peer DNS of PPP will typically override the system DNS server settings (because PPP often takes the longest time to initialize). This will almost certainly render WiFi and Ethernet interfaces unusable for DNS. I understand this may be beyond the scope of this PR, but I wanted to highlight it so that appropriate action can be taken. |
654711c to
9defa9c
Compare
|
Documentation for This is now ready to merged. |
This updates lwIP from STABLE-2_1_3_RELEASE, which was released in November 2021. The latest STABLE-2_2_0_RELEASE was released in September 2023. Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
This commit adds a new `network.PPP` interface which works on any port that
has bare-metal lwIP, eg rp2, stm32, mimxrt.
It has been tested on stm32. A board needs to enable
`MICROPY_PY_NETWORK_PPP_LWIP` and then it can use it as follows:
import network
ppp = network.PPP(uart)
ppp.connect()
while not ppp.isconnected():
pass
# use `socket` module as usual, etc
ppp.disconnect()
Usually the application must first configure the cellular/etc UART link to
get it connected and in to PPP mode first (eg ATD*99#), before handing over
control to `network.PPP`.
The PPP interface automatically configures the UART IRQ callback to call
PPP.poll() on incoming data.
Signed-off-by: Damien George <damien@micropython.org>
Can be enabled by a board by enabling `MICROPY_PY_NETWORK_PPP_LWIP`. Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
The errcode should be cleared so the caller sees a successful write, even if it's a short write. Signed-off-by: Damien George <damien@micropython.org>
When timeout=0 (non-blocking mode) the UART should still wait for each character to go out. Otherwise non-blocking mode with CTS enabled is useless because it can only write one character at a time. Signed-off-by: Damien George <damien@micropython.org>
This allows enabling lwIP debugging output. For example, to enable PPP
debugging add the following to `mpconfigboard.h`:
#define LWIP_DEBUG 1
#define PPP_DEBUG LWIP_DBG_ON
Signed-off-by: Damien George <damien@micropython.org>
Can be enabled by a board by enabling `MICROPY_PY_NETWORK_PPP_LWIP`. Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
9defa9c to
35b6a66
Compare
|
Please check this issue: #15900 |
|
Hello friends, in our project that we developed with MP v1.24.0-preview on Mimixrt1020, we access the internet with PPP via GSM module. While there is no problem when sending 300-500 bytes of data with socket, the application freezes when data is larger (for example: 1KB, 2KB), sometimes it acts as if the data is gone before it is completely gone. We tried the following tests that we saw in the comments on Discord and GitHub, but we still could not get any results: We have tried each of the above separately, and also all combinations, but still no results. A problem that has been blocking our way for about two months, we are open to any suggestions. I am waiting for your ideas that may help. |
|
Hello. |
@eskereli @dpgeorge I found a bug in MIMXRT UART, causing the UART buffer size to be reset to 256B on every call to uart.init() unless the size is deliberately set in this call. Maybe that is you issue as well in your attempts. The same applies to RP2 and SAMD. Will send fixes. |
Can confirm. Had been working around this without recognising it as a bug 🤦 |
This PR adds a new
network.PPPinterface which works on any port that has bare-metal lwIP, eg rp2, stm32, mimxrt.It has been tested on stm32. A board needs to enable
MICROPY_PY_NETWORK_PPP_LWIPand then it can use it as follows:Notes:
ATD*99#), before handing over control tonetwork.PPP.PPP.poll()on incoming data.