Add new raw REPL paste mode that has flow control, and compiles as it receives data #6527
Conversation
a199493 to
33d7d92
Compare
|
We use the raw repl via USB a lot, so increasing transfer speed by 10x sounds... great! Will test this asap when back at work. And possibly it would also benefit other ports/boards where paste mode was not always reliable. eg on the pca10059 dev board, usb paste mode is almost unusable due to lost characters even for small scripts. |
Thanks, would be good to get more testing on this.
Note that this PR won't help friendly REPL paste mode (ctrl-E, === mode) because that just accepts characters as you paste them (eg via ctrl-V) from your OS. To improve the situation there with pca10059 you can increase |
33d7d92 to
a5a209c
Compare
|
This looks great! When the code finishes, does the REPL go to normal prompt, just like with current paste mode? What about errors -- is it possible to distinguish between normal stdout and stacktrace like in current raw mode or is all the output given together like in current paste mode? Will all this work over WebREPL as well? |
It goes back to raw REPL, just like normal raw REPL after executing a command (ie it does not go back to friendly REPL, it needs a ctrl-A after executing for that).
It's the same as current raw mode: there are ctrl-D characters delineating normal output and a stacktrace. It's very similar to normal raw mode (you can see the changes in this PR to pyboard.py are localised), just the way the code is sent to the device is different.
Good question. I have not tested it but it should work, it should work with anything that plugs in to the dupterm interface. |
Testing with a command-line WebREPL client (not the browser WebREPL client), this new raw mode does work with it. |
a5a209c to
1841437
Compare
| STATIC void mp_reader_new_stdin(mp_reader_t *reader, mp_reader_stdin_t *reader_stdin, size_t buf_max) { | ||
| // Make flow-control window half the buffer size, and indicate to the host that 2x windows are free. | ||
| size_t window = buf_max / 2; | ||
| char reply[3] = { window & 0xff, window >> 8, 0x01 }; |
There was a problem hiding this comment.
I'm sorry if this is a stupid question, but given that window is transmitted as a 16-bit value, should this function be checking that buf_max is less than 0x20000 ?
There was a problem hiding this comment.
Thanks, yes, that should be constrained. I've now changed the type of the arg to a uint16_t so that the compiler enforces the size to a maximum of 65535 (which should be plenty).
1841437 to
83a4d74
Compare
|
I changed the title of this PR to remove the "RFC". It should be ready to go in now. |
|
Would it be appropriate for this PR to also update https://github.com/micropython/micropython/blob/master/docs/reference/repl.rst with a description of this new mode? (and possibly an overview of how the different modes are entered / exited?) |
Background: the friendly/normal REPL is intended for human use whereas the
raw REPL is for computer use/automation. Raw REPL is used for things like
pyboard.py script_to_run.py. The normal REPL has built-in flow control
because it echos back the characters. That's not so with raw REPL and flow
control is just implemented by rate limiting the amount of data that goes
in. Currently it's fixed at 256 byte chunks every 10ms. This is sometimes
too fast for slow MCUs or systems with small stdin buffers. It's also too
slow for a lot of higher-end MCUs, ie it could be a lot faster.
This commit adds a new raw REPL mode which includes flow control: the
device will echo back a character after a certain number of bytes are sent
to the host, and the host can use this to regulate the data going out to
the device. The amount of characters is controlled by the device and sent
to the host before communication starts. This flow control allows getting
the maximum speed out of a serial link, regardless of the link or the
device at the other end.
Also, this new raw REPL mode parses and compiles the incoming data as it
comes in. It does this by creating a "stdin reader" object which is then
passed to the lexer. The lexer requests bytes from this "stdin reader"
which retrieves bytes from the host, and does flow control. What this
means is that no memory is used to store the script (in the existing raw
REPL mode the device needs a big buffer to read in the script before it can
pass it on to the lexer/parser/compiler). The only memory needed on the
device is enough to parse and compile.
Finally, it would be possible to extend this new raw REPL to allow bytecode
(.mpy files) to be sent as well as text mode scripts (but that's not done
in this commit).
Some results follow. The test was to send a large 33k script that contains
mostly comments and then prints out the heap, run via pyboard.py large.py.
On PYBD-SF6, prior to this PR:
$ ./pyboard.py large.py
stack: 524 out of 23552
GC: total: 392192, used: 34464, free: 357728
No. of 1-blocks: 12, 2-blocks: 2, max blk sz: 2075, max free sz: 22345
GC memory layout; from 2001a3f0:
00000: h=hhhh=======================================hhBShShh==h=======h
00400: =====hh=B........h==h===========================================
00800: ================================================================
00c00: ================================================================
01000: ================================================================
01400: ================================================================
01800: ================================================================
01c00: ================================================================
02000: ================================================================
02400: ================================================================
02800: ================================================================
02c00: ================================================================
03000: ================================================================
03400: ================================================================
03800: ================================================================
03c00: ================================================================
04000: ================================================================
04400: ================================================================
04800: ================================================================
04c00: ================================================================
05000: ================================================================
05400: ================================================================
05800: ================================================================
05c00: ================================================================
06000: ================================================================
06400: ================================================================
06800: ================================================================
06c00: ================================================================
07000: ================================================================
07400: ================================================================
07800: ================================================================
07c00: ================================================================
08000: ================================================================
08400: ===============================================.....h==.........
(349 lines all free)
(the big blob of used memory is the large script).
Same but with this PR:
$ ./pyboard.py large.py
stack: 524 out of 23552
GC: total: 392192, used: 1296, free: 390896
No. of 1-blocks: 12, 2-blocks: 3, max blk sz: 40, max free sz: 24420
GC memory layout; from 2001a3f0:
00000: h=hhhh=======================================hhBShShh==h=======h
00400: =====hh=h=B......h==.....h==....................................
(381 lines all free)
The only thing in RAM is the compiled script (and some other unrelated
items).
Time to download before this PR: 1438ms, data rate: 230,799 bits/sec.
Time to download with this PR: 119ms, data rate: 2,788,991 bits/sec.
So it's more than 10 times faster, and uses significantly less RAM.
Results are similar on other boards. On an stm32 board that connects via
UART only at 115200 baud, the data rate goes from 80kbit/sec to
113kbit/sec, so gets close to saturating the UART link without loss of
data.
The new raw REPL mode also supports a single ctrl-C to break out of this
flow-control mode, so that a ctrl-C can always get back to a known state.
It's also backwards compatible with the original raw REPL mode, which is
still supported with the same sequence of commands. The new raw REPL
mode is activated by ctrl-E, which gives an error on devices that do not
support the new mode.
Signed-off-by: Damien George <damien@micropython.org>
528f4bf to
d9d5693
Compare
Yes, good idea. Now done. |
docs/reference/repl.rst
Outdated
| b'\r\nraw REPL; CTRL-B to exit\r\n>R\x01\x80\x00\x01\x04123\r\n\x04\x04>' | ||
|
|
||
| In this case the flow-control window size is 128 and there is a new window | ||
| immediately available at the start (ie up to 256 bytes can be written to begin |
There was a problem hiding this comment.
Why does a window size of 128 mean that 256 bytes can be written?
There was a problem hiding this comment.
because 2 windows worth are available immediately, the implicit one and the explicitly indicated one (this is mentioned in the protocol text above in the dot points... but maybe not very clear)
There was a problem hiding this comment.
updated to be more explicit
There was a problem hiding this comment.
Yup, documentation is much clearer now, thanks.
docs/reference/repl.rst
Outdated
|
|
||
| * While there are bytes to send, write up to the remaining window size. | ||
|
|
||
| * If the window size is 0, or there is a byte waiting to read, read 1 byte. |
There was a problem hiding this comment.
Was this supposed to be "if the remaining window size is 0" ?
| def raw_paste_write(self, command_bytes): | ||
| # Read initial header, with window size. | ||
| data = self.serial.read(2) | ||
| window_size = data[0] | data[1] << 8 |
There was a problem hiding this comment.
Would it be "more pythonic" to use something like struct.unpack here? 😉
There was a problem hiding this comment.
I try not to depend on entire modules if it's just for a very simple thing like this... too much micro, not enough Python :)
d9d5693 to
6793108
Compare
| # Device supports raw-paste mode, write out the command using this mode. | ||
| return self.raw_paste_write(command_bytes) | ||
| else: | ||
| # Device doesn't support raw-paste, fall back to normal raw REPL. |
There was a problem hiding this comment.
Should this then set self.use_raw_paste = False ?
There was a problem hiding this comment.
Yeah that's probably a good idea, so it doesn't keep trying the failing raw-paste each time. And it's not like the device can suddenly support it, you'd have to at least disconnect/reconnect to get support for raw-paste, which means resetting this variable.
6793108 to
1de811e
Compare
tools/pyboard.py
Outdated
| print(data) | ||
| raise PyboardError("could not enter raw repl") | ||
| # Don't try to use raw-paste mode again for this connection. | ||
| self.use_raw_paste = True |
There was a problem hiding this comment.
Aaah! Thanks for reviewing! Should be fixed now.
This commit adds support to pyboard.py for the new raw REPL paste mode. Note that this new pyboard.py is fully backwards compatible with old devices (it detects if the device supports the new raw REPL paste mode). Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
1de811e to
a14ca31
Compare
Connects up read, write and ctrl_transfer to TinyUSB. USB Host support is available on iMX RT and RP2040. Fixes micropython#6527 (imx) and fixes micropython#5986 (rp2).
Connects up read, write and ctrl_transfer to TinyUSB. USB Host support is available on iMX RT and RP2040. Fixes micropython#6527 (imx) and fixes micropython#5986 (rp2).
Background: the friendly/normal REPL is intended for human use whereas the raw REPL is for computer use/automation. Raw REPL is used for things like
pyboard.py script_to_run.py. The normal REPL has built-in flow control because it echos back the characters. That's not so with raw REPL and flow control is just implemented by rate limiting the amount of data that goes in. Currently it's fixed at 256 byte chunks every 10ms. This is sometimes too fast for slow MCUs or systems with small stdin buffers. It's also too slow for a lot of higher-end MCUs, ie it could be a lot faster.This PR adds a new raw REPL mode which includes flow control: the device will echo back a character after a certain number of bytes are sent to the host, and the host can use this to regulate the data going out to the device. The amount of characters is controlled by the device and sent to the host before communication starts. This flow control allows getting the maximum speed out of a serial link, regardless of the link or the device at the other end.
Also, this new raw REPL mode parses and compiles the incoming data as it comes in. It does this by creating a "stdin reader" object which is then passed to the lexer. The lexer requests bytes from this "stdin reader" which retrieves bytes from the host, and does flow control. What this means is that no memory is used to store the script (in the existing raw REPL mode the device needs a big buffer to read in the script before it can pass it on to the lexer/parser/compiler). The only memory needed on the device is enough to parse and compile.
Finally, it would be possible to extend this new raw REPL to allow bytecode (.mpy files) to be sent as well as text mode scripts (but that's not done in this PR).
Some results follow. The test was to send a large 33k script that contains mostly comments and then prints out the heap, run via
pyboard.py large.py.On PYBD-SF6, prior to this PR:
(the big blob of used memory is the large script).
Same but with this PR:
The only thing in RAM is the compiled script (and some other unrelated items).
Time to download before this PR: 1438ms, data rate: 230,799 bits/sec.
Time to download with this PR: 119ms, data rate: 2,788,991 bits/sec.
So it's more than 10 times faster, and uses significantly less RAM.
Results are similar on other boards. On an stm32 board that connects via UART only at 115200 baud, the data rate goes from 80kbit/sec to 113kbit/sec, so gets close to saturating the UART link without loss of data.
Note that the new mode is fully backwards compatible with old
pyboard.py, and newpyboard.pyis fully backwards compatible with old devices (it detects if the device supports the new raw REPL mode).The new raw REPL mode also supports a single ctrl-C to break out of this flow-control mode, so that a ctrl-C can always get back to a known state.
To summarise: this PR introduces a new raw REPL mode which 1) has flow control to make it faster, 2) compiles as data comes in to use less RAM, and 3) can be extended to support reading in .mpy files.
There are other ways to achieve these items (and this is my attempt number 4...) and comments are welcome.