micropython/mip: Add a new `mip` library for on-device installation. #542

jimmo · 2022-09-28T06:17:15Z

This PR adds:

A library (mip) that can be frozen (or manually installed) on network-capable boards for on-device installation of packages from micropython-lib (or other similar package indices).
A tool (tools/build.py) to deploy micropython-lib to a static web server suitable for access from mip (or mpremote when similar functionality is added).

The structure of the static web server output is explained in the comments at the top of deploy.py.

mip.install() supports the following use cases:

mip.install("foo") -- this will download the latest version of the foo package (and dependencies) from micropython-lib.
mip.install("foo", version="x.y") -- install foo version x.y, including dependencies at the version when foo x.y was published
mip.install("foo", target="path/to/lib") -- install foo and dependencies in specified path (otherwise defaults to the first "lib" directory in sys.path)
mip.install("foo", index="https://example.com/") -- install foo from example.com's index
mip.install("http://example.com/x/y/foo.py") -- download foo.py directly from the URL
mip.install("http://example.com/x/y/foo.json") -- download a package, including dependencies, described by foo.json (see below)
mip.install("http://example.com/x/y") -- implicitly http://example.com/x/y/package.json
mip.install("github:org/user/path/foo.py") -- shortcut for github-hosted content (fetches user/path/foo.py from the default branch)
mip.install("github:org/user/path/mypackage.json") -- as above
mip.install("github:org/user") -- this will hopefully be the common case for "self-hosted" packages.
mip.install("github:org/user", version="devel") -- as above, but using the "devel" branch
`mip.install("foo", mpy=False) -- fetch foo but as .py files (i.e. for debugging).

Major differences in mip compared to upip:

Packages installed from the index are bytecode compiled to .mpy by default. The request includes the device's supported version.
pypi is not supported as a source.
It uses urequests rather than having its own HTTP client. (Most boards that freeze mip also freeze urequests anyway)

The package.json looks like

{
  "hashes": [
    ["aioble/server.mpy", "e39dbf64"],
    ...
  ],
  "urls": [
    ["target/path.py", "http://url/foo/bar/path.py"],
    ...
  ],
  "deps": [
    ["name", "version"],
    ...
  ]
  "version": "0.1"
}

The "hashes" are for referencing .mpy files directly from micropython-lib. This is used by micropython-lib packages, but shouldn't be used for self-hosted ones. The "urls" allows arbritrary URLs to be fetched, and "deps" allows recursive dependencies, e.g. a self-hosted package might want to provide some urls to their own files, and then depend on some micropython-lib packages (or a package json in another github repo).

Note: deploy.py requires micropython/micropython#9437

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>

jimmo · 2022-09-28T06:34:26Z

I've written a small demo of what writing a "self-hosted" (on GitHub) package will look like, including how to specify dependencies: https://github.com/jimmo/micropython-mlx90640

This can be installed by

>>> mip.install("jimmo/micropython-mlx90640") # installs from default branch (main)
>>> mip.install("jimmo/micropython-mlx90640", version="devel") # use the devel branch (equivalently, tag)

Of course I'd prefer for a driver like this one to end up in micropython-lib and therefore get all the benefits of being bytecode compiled etc, but this third-party publishing (to github or other web server) is still very much a use case we want to support.

jimmo · 2022-09-28T12:01:22Z

Another artifact of the build is a top-level index.json (which will be https://micropython.org/pi/v2/index.json) which contains the full list of available packages from micropython-lib. mip doesn't consume this, but other tools that want to discover available packages can. What's included in this for now is pretty minimal, but unlike the package metadata I don't imagine this being consumed by low-memory devices so this could have more info added (possibly version history etc). (Edit: added full version history for each bytecode version, as well as description, author, license)

tools/deploy.py

dpgeorge · 2022-09-29T02:14:02Z

I checked the frozen version of this new upip, compared to the old one (compiled with mpy-cross v6 using -O0, ie no optimisations). This is to see how the new one compares to the old one in terms of flash size. Note that the new upip relies on urequests, so the total size of the new upip needs to include the size of urequests.

Old upip size:

esp8266: 6056 bytes frozen
rp2: 5832 bytes frozen

New upip+urequests combined size:

esp8266: 5536 bytes frozen
rp2: 5360 bytes frozen

So the new upip combined with urequests is about 500 bytes smaller (around 10% smaller) than the old upip. Even bigger savings are made when urequests already exists for other reasons. In that case upip on its own is just 2464 bytes frozen (on rp2).

Furthermore the dependencies on built-in C modules are reduced with the new upip. The dependencies are:

Old upip: sys, gc, os, errno, json, zlib, uctypes, ssl, socket
New upip: sys, os, binascii (optional), hashlib (optional), and then json, ssl, socket via urequests.

So gc, errno, zlib and uctypes are no longer required by the new upip, compared to the old one.

jimmo · 2022-09-29T02:25:07Z

I've pushed an updated version that adds two features requested via private feedback:

Ability to install "source only" versions of packages. Alongside the supported bytecode versions, there's now a "py" version. This can be requested using mip.install(..., mpy=False).
Added version history to the index.json. This allows you to discover the available versions for each bytecode version.

micropython/upip/upip.py

dpgeorge · 2022-09-29T02:37:33Z

What about the ability to do, on the unix port, micropython -m upip install foo? That could be done with the old upip. Do we need to support that?

Edit: maybe we can have a separate upip-cli that supports command-line for unix?

jimmo · 2022-09-29T02:49:48Z

Addressed comments above.

Edit: maybe we can have a separate upip-cli that supports command-line for unix?

Yep, I want to do this like the change we made for unittest where if you install unittest-discover then you also get micropython -m unittest. (Separate PR)

This populates https://micropython.org/pi/v2 with compiled packages, suitable for use by `mip` and `mpremote`. This work was funded through GitHub Sponsors. Signed-off-by: Jim Mussared <jim.mussared@gmail.com>

Riffing on "pip", "mip installs packages". This is a replacement for the previous `upip` tool for on-device installation of packages. This work was funded through GitHub Sponsors. Signed-off-by: Jim Mussared <jim.mussared@gmail.com>

jimmo · 2022-09-29T13:10:42Z

As discussed, renamed upip to mip, and deploy.py to build.py.

dpgeorge · 2022-09-29T13:38:35Z

Thank you, this is awesome!

peterhinch · 2022-09-30T08:02:22Z

Are there any ideas on how to support platforms like Pico that aren't network-enabled?

jimmo · 2022-09-30T10:36:35Z

Are there any ideas on how to support platforms like Pico that aren't network-enabled?

@peterhinch
Yep! micropython/micropython#9467

jimmo · 2022-09-30T10:41:52Z

And micropython/micropython#9463 included a significant overhaul of https://docs.micropython.org/en/latest/reference/packages.html (which hasn't gone live yet, but will hopefully update soon).

uu: Fix dependency on os-path.

122b689

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>

jimmo added a commit to jimmo/micropython-mlx90640 that referenced this pull request Sep 28, 2022

Demo for micropython/micropython-lib#542

14cebc1

jimmo force-pushed the upip-deploy branch from b086eaf to f4e1522 Compare September 28, 2022 13:29

jimmo added a commit to jimmo/micropython-mlx90640 that referenced this pull request Sep 28, 2022

Demo for micropython/micropython-lib#542

5d9e84c

jimmo force-pushed the upip-deploy branch from f4e1522 to 00a9506 Compare September 28, 2022 14:33

dpgeorge reviewed Sep 29, 2022

View reviewed changes

tools/deploy.py Outdated Show resolved Hide resolved

jimmo force-pushed the upip-deploy branch from 00a9506 to c01b2c2 Compare September 29, 2022 02:18

jimmo force-pushed the upip-deploy branch from c01b2c2 to 8e8cc73 Compare September 29, 2022 02:28

dpgeorge reviewed Sep 29, 2022

View reviewed changes

jimmo force-pushed the upip-deploy branch from 8e8cc73 to b395c44 Compare September 29, 2022 02:37

jimmo force-pushed the upip-deploy branch 2 times, most recently from 5ae5fdd to 1094a03 Compare September 29, 2022 02:48

jimmo added 2 commits September 29, 2022 22:54

tools/build.py: Add script for deploying to a static web server.

58a93f3

This populates https://micropython.org/pi/v2 with compiled packages, suitable for use by `mip` and `mpremote`. This work was funded through GitHub Sponsors. Signed-off-by: Jim Mussared <jim.mussared@gmail.com>

jimmo force-pushed the upip-deploy branch from 1094a03 to 5e7bac1 Compare September 29, 2022 13:02

jimmo mentioned this pull request Sep 29, 2022

top: Replace upip with mip everywhere. micropython/micropython#9463

Merged

jimmo changed the title ~~micropython/upip: Add a new upip library for on-device installation.~~ micropython/upip: Add a new mip library for on-device installation. Sep 29, 2022

jimmo changed the title ~~micropython/upip: Add a new mip library for on-device installation.~~ micropython/mip: Add a new mip library for on-device installation. Sep 29, 2022

dpgeorge merged commit 5e7bac1 into micropython:master Sep 29, 2022

jimmo mentioned this pull request Sep 29, 2022

tools/mpremote: Add mpremote mip install to install packages. micropython/micropython#9467

Merged

brainelectronics mentioned this pull request Oct 23, 2022

Feature: Allow upip to install from test.pypi.org pfalcon/pycopy-lib#72

Closed

beriberikix mentioned this pull request Nov 6, 2022

Micropython Example chrysn/aiocoap#287

Open

brainelectronics mentioned this pull request Mar 21, 2023

upip: Add PyPI support #632

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

micropython/mip: Add a new `mip` library for on-device installation. #542

micropython/mip: Add a new `mip` library for on-device installation. #542

jimmo commented Sep 28, 2022 •

edited

Loading

jimmo commented Sep 28, 2022 •

edited

Loading

jimmo commented Sep 28, 2022 •

edited

Loading

dpgeorge commented Sep 29, 2022

jimmo commented Sep 29, 2022 •

edited

Loading

dpgeorge commented Sep 29, 2022 •

edited

Loading

jimmo commented Sep 29, 2022

jimmo commented Sep 29, 2022

dpgeorge commented Sep 29, 2022

peterhinch commented Sep 30, 2022

jimmo commented Sep 30, 2022

jimmo commented Sep 30, 2022

micropython/mip: Add a new mip library for on-device installation. #542

micropython/mip: Add a new mip library for on-device installation. #542

Conversation

jimmo commented Sep 28, 2022 • edited Loading

jimmo commented Sep 28, 2022 • edited Loading

jimmo commented Sep 28, 2022 • edited Loading

dpgeorge commented Sep 29, 2022

jimmo commented Sep 29, 2022 • edited Loading

dpgeorge commented Sep 29, 2022 • edited Loading

jimmo commented Sep 29, 2022

jimmo commented Sep 29, 2022

dpgeorge commented Sep 29, 2022

peterhinch commented Sep 30, 2022

jimmo commented Sep 30, 2022

jimmo commented Sep 30, 2022

micropython/mip: Add a new `mip` library for on-device installation. #542

micropython/mip: Add a new `mip` library for on-device installation. #542

jimmo commented Sep 28, 2022 •

edited

Loading

jimmo commented Sep 28, 2022 •

edited

Loading

jimmo commented Sep 28, 2022 •

edited

Loading

jimmo commented Sep 29, 2022 •

edited

Loading

dpgeorge commented Sep 29, 2022 •

edited

Loading