Codecov Report

Merging #8231 (10f2db7) into master (0a21762) will increase coverage by 0.00%.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #8231   +/-   ##
=======================================
  Coverage   98.39%   98.40%           
=======================================
  Files         153      153           
  Lines       20187    20188    +1     
=======================================
+ Hits        19864    19865    +1     
  Misses        323      323           

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0a21762...10f2db7. Read the comment docs.

mpremote is also being packaged to wheel in github-actions in this PR, can be tested with

Go to actions page for the task, download artifact link at bottom: https://github.com/micropython/micropython/actions/runs/1775611454

https://github.com/micropython/micropython/suites/5129239273/artifacts/154344973

unzip mpremote.zip
pip install mpremote*.whl

I still want to do some more work here to speed up the sync process, make it as efficient as I can.

With a remote littlefs partition on esp32 even just doing recursive file listing can be painfully slow ( ~ 12 seconds for 128 files across 28 folders ), not sure how much could be done to improve this.

One of the main things I'd like to fix is more related to fs cp -r than the manifest.
When copying a folder from computer over an existing remote folder, existing files will be overwritten however remote files/folders contained within that have since been deleted on computer will be left on the device. This way you can have stale content causing unexpected results.
On linux, a cp -r will entirely replace an existing folder unless you add the -u flag.

I want to look into general file transfer speed using mpremote. It should be possible to improve the speed for files going in to the device.

On linux, a cp -r will entirely replace an existing folder unless you add the -u flag.

+1

I haven't looked at the actual file transfer speed over the serial port, however I've got re-copying folders working much faster with either a sha or (size + mtime) check on remote vs local filesystems.

The best speed so far is by scanning the remote folder tree into a dict and passing that back to mpremote to check against local folder; this can be used to delete extra remote files and not transfer ones that aren't needed.

Most of the time for this operation on a mostly unchanged folder is just in a recursive walk in micropython, even with just one os.stat per file it can be quite slow. Might not be able to do much about that though, no way to know if there are extra files to delete without doing a os.listdir and then os.stat on each one to know if it's a dir or a file...

no way to know if there are extra files to delete without doing a os.listdir and then os.stat on each one to know if it's a dir or a file...

try uos.ilistdir()

Yes of course, I did forget about the difference between the two. I'll try that later and see how much improvement it makes

Is there any progress on #7731?

Is there any progress on #7731?

I haven't run into this issue myself so hadn't seen that ticket. At least it looks reasonably easy to reproduce, though there could be a few issues in that mix too

For now I've trimmed this PR back down to just the manifest command, no sync. This command combined with mount is enough on many devices (ones with native usb at least) to allow fast development of projects which use a manifest file to organise the files / libraries.

I'll continue to work on sync / deploy command/s, along with any file transfer speed improvements & fixes in a follow up PR.