Add ability to have frozen bytecode #1811

dpgeorge · 2016-01-31T22:35:05Z

This PR brings proper frozen bytecode. Bytecode is compiled using a "MicroPython cross compiler", and then frozen using tools/mpytool.py. The output file is then a .c file that's compiled into your uPy binary. Then "import frozenmod" works as expected. The resulting frozen module requires zero RAM to compile (but does require about 1 GC block to hold the function object, although that could eventually be optimised away).

The main changes in the PR are:

support in core for frozen modules (including import and adding a new qstr pool for the frozen qstrs)
addition of top-level dir upy-compiler/ which contains an example cross compiler (that's real-world usable, just needs editing of mpconfigport.h to suit a given target)
minor modifications to minimal port to include an example of frozen bytecode (.mpy file is included so you don't need the cross-compiler to build it)

One thing to note is that MICROPY_MODULE_FROZEN has changed meaning, and all old uses of this macro should (and are in this PR) changed to MICROPY_MODULE_FROZEN_STR. There is now also MICROPY_MODULE_FROZEN_MPY to support both original frozen strings, and new frozen bytecode/mpy files.

Question: should upy-compiler be included? I think so. What should the dir be called? What should the executable be called (I just made it "micropython").

dpgeorge · 2016-01-31T22:38:51Z

To test:

$ cd minimal
$ make
$ make run
>>> import frozentest

dpgeorge · 2016-02-01T18:09:53Z

Regarding the cross compiler (upy-compiler/ in this PR): currently it needs to be rebuilt to suit a given target. This is because certain compile-time options make it produce different bytecode (eg whether bytecode has dict caching for certain opcodes). This is a bit of a pain, because you'll need separate binaries per target.

I think it's worth considering making the relevant configuration options that do modify the bytecode, dynamically selectable. Then you could specify them on the command line of the cross compiler, and only need one binary for all targets. Eg something like: $ micropy_cross --small-int-bits 31 --dict-cache 0 inputfile.py.

Also, as per @dhylands' suggestion, we'd also want to make constants (eg stm module) dynamically selectable.

dpgeorge · 2016-02-02T17:33:19Z

I took a look at the code and it would be quite straightforward to make the upy-cross-compiler dynamically configurable (via command line) so it could produce mpy files for all targets.

dpgeorge · 2016-02-25T12:25:47Z

This is now updated to use the new mpy-cross compiler. It also includes ability to freeze bytecode into the stmhal build.

@peterhinch this may interest you :) Try the following (start from root dir):

$ make -C mpy-cross
$ cd stmhal
$ mkdir scripts
(put your .py scripts to freeze in scripts/ dir, eg font.py)
$ make FROZEN_MPY_DIR=scripts

Then just import your script as usual, eg "import font". I tested it with a large bytes object and indeed it stays in flash. Please let me know if/how it works for you.

peterhinch · 2016-02-26T09:13:37Z

I hope I'm not doing something daft but I'm failing at the first hurdle. I cloned the source to ~/temp/micropython, cd'd to the micropython root, and did the following:

[adminpete@axolotl]: ~/temp/micropython
$ make -C mpy-cross
Use make V=1 or set BUILD_VERBOSE in your environment to increase build verbosity.
make: Entering directory `/home/adminpete/temp/micropython/mpy-cross'
mkdir -p build/genhdr
GEN build/genhdr/qstrdefs.generated.h
Generating build/genhdr/mpversion.h
mkdir -p build/py
mkdir -p build/py/../extmod
CC ../py/mpstate.c
CC ../py/nlrx86.S
... [many lines omitted]
CC ../py/../extmod/vfs_fat_file.c
CC ../py/../extmod/moduos_dupterm.c
In file included from /usr/include/bits/errno.h:24:0,
                 from /usr/include/errno.h:35,
                 from ../py/../extmod/moduos_dupterm.c:27:
/usr/include/linux/errno.h:1:23: fatal error: asm/errno.h: No such file or directory
 #include <asm/errno.h>
                       ^
compilation terminated.
make: *** [build/py/../extmod/moduos_dupterm.o] Error 1
make: Leaving directory `/home/adminpete/temp/micropython/mpy-cross'
[adminpete@axolotl]: ~/temp/micropython
$

chuckbook · 2016-02-26T09:48:05Z

Thanks Damien!
This works great. Now we can do all the eye candy by utilizing some virgin flash cells.
Unnecessary to mention that the 1 MByte was flooded immediately :-)

chuckbook · 2016-02-26T09:51:10Z

@peterhinch: Make sure to have 32-bit code generation enabled on your host machine.

dpgeorge · 2016-02-26T10:29:22Z

Thanks for testing @chuckbook! You must have some large scripts to fill 1mb... out of interest what is the number of lines of Python code that got to 1mb?

@peterhinch it builds 32-bit by default to reduce the size of the binary. You can comment out the first line of the Makefile and try building again. It should work in 64-bit mode as well.

chuckbook · 2016-02-26T13:17:13Z

@dpgeorge ~16000, however just a frozen database, after that it ran into a emit->code_info_offset assertion.

dpgeorge · 2016-02-26T14:05:17Z

@chuckbook a failed assertion is bad. If you can reproduce it then I can try to fix it.

chuckbook · 2016-02-26T22:01:28Z

@dpgeorge I raise to 49107 lines of code. The assertion results from an 'if else chain' in a function with more than 15000 lines of python code. Most of the code is random generated. I would just drive this thing to the top. Flash is now plugged like a Christmas goose!

peterhinch · 2016-02-27T06:48:28Z

Doubtless I'm doing something dumb here, but I can't get this to work. Everything appears OK but I end up with a build identical to the standard one and lacking my modules. Is there a cross-compilation step I'm missing? Here is my session, with comments and some lines omitted. It's notable that the names of my modules never appear in any of the build output listings.

[delete old build]

[adminpete@axolotl]: ~/temp
$ sudo rm -r micropython/
[sudo] password for adminpete: 

[start from scratch with new clone]

[adminpete@axolotl]: ~/temp
$ git clone https://github.com/micropython/micropython.git
Cloning into 'micropython'...
remote: Counting objects: 37622, done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 37622 (delta 0), reused 0 (delta 0), pack-reused 37618
Receiving objects: 100% (37622/37622), 23.70 MiB | 2.34 MiB/s, done.
Resolving deltas: 100% (27068/27068), done.
Checking connectivity... done.
[adminpete@axolotl]: ~/temp


[edit makefile commenting out 1st line]

$ cd micropython/mpy-cross/
[adminpete@axolotl]: ~/temp/micropython/mpy-cross
$ nano Makefile
[adminpete@axolotl]: ~/temp/micropython/mpy-cross


[build cross]

$ cd ..
[adminpete@axolotl]: ~/temp/micropython
$ make -C mpy-cross
Use make V=1 or set BUILD_VERBOSE in your environment to increase build verbosity.
make: Entering directory `/home/adminpete/temp/micropython/mpy-cross'
mkdir -p build/genhdr
GEN build/genhdr/qstrdefs.generated.h
Generating build/genhdr/mpversion.h
mkdir -p build/py
mkdir -p build/py/../extmod
CC ../py/mpstate.c

[lines omitted]

CC ../py/../extmod/moduos_dupterm.c
CC main.c
CC gccollect.c
LINK mpy-cross
   text    data     bss     dec     hex filename
 135594     808     856  137258   2182a mpy-cross
make: Leaving directory `/home/adminpete/temp/micropython/mpy-cross'

[put scripts into tree]

[adminpete@axolotl]: ~/temp/micropython
$ cd stmhal/
[adminpete@axolotl]: ~/temp/micropython/stmhal
$ mkdir scripts
[adminpete@axolotl]: ~/temp/micropython/stmhal
$ cp /mnt/qnap2/data/Projects/MicroPython/projects/barometer/frozen/* scripts/
[adminpete@axolotl]: ~/temp/micropython/stmhal
$ ls -l scripts/
total 80
-rw-r--r-- 1 adminpete adminpete 15442 Feb 27 06:23 epaper.py
-rw-r--r-- 1 adminpete adminpete 17947 Feb 27 06:23 epd.py
-rw-r--r-- 1 adminpete adminpete 11825 Feb 27 06:23 flash.py
-rw-r--r-- 1 adminpete adminpete  2606 Feb 27 06:23 micropower.py
-rw-r--r-- 1 adminpete adminpete  1271 Feb 27 06:23 panel.py
-rw-r--r-- 1 adminpete adminpete  7524 Feb 27 06:23 sdcard.py
-rw-r--r-- 1 adminpete adminpete 13709 Feb 27 06:23 upower.py


[build firmware]

[adminpete@axolotl]: ~/temp/micropython/stmhal
$ make FROZEN_MPY_DIR=scripts
Use make V=1 or set BUILD_VERBOSE in your environment to increase build verbosity.
mkdir -p build-PYBV10/genhdr
Create build-PYBV10/genhdr/pins.h
Create stmconst build-PYBV10/modstm_qstr.h
GEN build-PYBV10/genhdr/qstrdefs.generated.h
Generating build-PYBV10/genhdr/mpversion.h
mkdir -p build-PYBV10/hal/f4/src
mkdir -p build-PYBV10/lib/fatfs
mkdir -p build-PYBV10/lib/fatfs/option
mkdir -p build-PYBV10/lib/libc
mkdir -p build-PYBV10/lib/libm
mkdir -p build-PYBV10/lib/mp-readline
mkdir -p build-PYBV10/lib/netutils
mkdir -p build-PYBV10/lib/timeutils
mkdir -p build-PYBV10/lib/utils
mkdir -p build-PYBV10/py
mkdir -p build-PYBV10/py/../extmod
mkdir -p build-PYBV10/usbdev/class/src
mkdir -p build-PYBV10/usbdev/core/src
CC ../py/mpstate.c
CC ../py/nlrx86.S

[lines omitted]

CC ../lib/utils/printf.c
Create build-PYBV10/genhdr/pybcdc.inf
Create build-PYBV10/genhdr/pybcdc_inf.h
CC main.c
CC system_stm32.c
CC stm32_it.c

[lines omitted]

CC usbdev/class/src/usbd_msc_data.c
CC build-PYBV10/pins_PYBV10.c
LINK build-PYBV10/firmware.elf
   text    data     bss     dec     hex filename
 287480     336   27948  315764   4d174 build-PYBV10/firmware.elf
Create build-PYBV10/firmware.dfu
Create build-PYBV10/firmware.hex
[adminpete@axolotl]: ~/temp/micropython/stmhal


[check size of build]

$ cd build-PYBV10/
[adminpete@axolotl]: ~/temp/micropython/stmhal/build-PYBV10
$ ls -l firmware.dfu 
-rw-r--r-- 1 adminpete adminpete 288133 Feb 27 06:24 firmware.dfu
[adminpete@axolotl]: ~/temp/micropython/stmhal/build-PYBV10
$

dpgeorge · 2016-02-27T07:24:18Z

You will need to clone the branch from the PR, not the master branch!

dhylands · 2016-02-27T08:17:21Z

Try to use FROZEN_DIR instead of FROZEN_MPY_DIR

It looks like importing frozen mpy files isn't yet supported.

Both of my comments applied to the master branch. I see @dpgeorge clarified that these changes are on the PR branch.

peterhinch · 2016-02-27T08:24:12Z

Sorry I'm being dumb but I'm still a n00b when it comes to git. I tried

git fetch origin pull/1811/head

Is that correct? I'm still getting no joy. I can explicitly compile individual modules to mpy files in my scripts directory (is this step necessary?) but the build file remains the same size.

chuckbook · 2016-02-27T08:27:19Z

@peterhinch, try:
git pull origin pull/1811/head

dhylands · 2016-02-27T08:35:44Z

I was going to say to do:

git fetch origin pull/1811/head:new-branch-name
git checkout new-branch-name

replace new-branch-name with what you'd like.

On Sat, Feb 27, 2016 at 12:27 AM, chuckbook notifications@github.com
wrote:

@peterhinch https://github.com/peterhinch, try:

git pull origin pull/1811/head

—
Reply to this email directly or view it on GitHub
#1811 (comment)
.

Dave Hylands
Shuswap, BC, Canada
http://www.davehylands.com

dhylands · 2016-02-27T08:40:23Z

Just out of curiosity, I tried the git pull mentioned by @chuckbook but it wants to do a merge commit.

The git fetch/checkout doesn't require any merge commit.

chuckbook · 2016-02-27T08:52:35Z

As a novice, also out of curiosity, if one starts with a fresh clone, isn't the pull thing adequate?

peterhinch · 2016-02-27T09:49:14Z

Right, got it, thanks.
What an awesome feature! One minor wrinkle, I had to edit mkenv.mk

PYTHON = python3

otherwise mpy-tool.py fell over. The usual issue of defaulting to Python 2.x.

dhylands · 2016-02-27T18:34:33Z

As a novice, also out of curiosity, if one starts with a fresh clone, isn't the pull thing adequate?

You get the files, but the history doesn't look right because it does a merge commit. So if I do:

git clone https://github.com/micropython/micropython mp
cd mp
git checkout new-branch-2
git pull origin pull/1811/head
git log -2
commit 9959f14c13f7f3eb41793fbe7b1f50a36f0e01b5
Merge: 8a18084 42bc25b
Author: Dave Hylands <dhylands@gmail.com>
Date:   Sat Feb 27 10:14:20 2016 -0800

    Merge commit 'refs/pull/1811/head' of https://github.com/micropython/micropython into new-branch-1

commit 8a180845711d881d31876a43259092eaf03268f7
Author: danicampora <daniel@wipy.io>
Date:   Sat Feb 27 00:12:21 2016 +0100

    cc3200: Update WiPy software version to 1.2.0

whereas if I do:

git clone https://github.com/micropython/micropython mp
cd mp
git fetch origin pull/1811/head:new-branch
git checkout new-branch
git log -3
commit 42bc25b66fd5e3272928b29d50ab82fd03189af6
Author: Damien George <damien.p.george@gmail.com>
Date:   Thu Feb 25 12:19:40 2016 +0000

    stmhal: Add Makefile option FROZEN_MPY_DIR to support frozen bytecode.

commit 42b6bbe2298ff4502b8af1c444d8237deea50189
Author: Damien George <damien.p.george@gmail.com>
Date:   Sun Jan 31 22:16:41 2016 +0000

    minimal: Add example of frozen persistent bytecode (.mpy file).

    frozentest.py is frozen into the binary as frozen bytecode.  The .mpy
    file is included so that there is no dependency on the cross compiler.

commit 9b4a6dfaf4b321c361a56302bb749cac7f37d996
Author: Damien George <damien.p.george@gmail.com>
Date:   Sun Jan 31 22:24:16 2016 +0000

    py: Add ability to have frozen persistent bytecode from .mpy files.

    The config variable MICROPY_MODULE_FROZEN is now made of two separate
    parts: MICROPY_MODULE_FROZEN_STR and MICROPY_MODULE_FROZEN_MPY.  This
    allows to have none, either or both of frozen strings and frozen mpy
    files (aka frozen bytecode).

then I get to see the commit history exactly like I was in @dpgeorge's repository.

chuckbook · 2016-02-27T18:46:39Z

@dhylands , thanks a lot for the explanation!

peterhinch · 2016-02-28T08:36:32Z

@dpgeorge I'm deeply impressed by this and can see plenty of applications :-) Do you plan to extend it to handle decorators emitting native code (including asm)?

peterhinch · 2016-03-01T09:55:59Z

I'm hitting an error trying to freeze a file upower.py. I can compile the file to an mpy file. The error occurs when trying to produce a firmware build - in the absence of any line number information I'm not clear what it's objecting to. The file can be found here

[adminpete@axolotl]: ~/temp/micropython/stmhal
$ make FROZEN_MPY_DIR=scripts
Use make V=1 or set BUILD_VERBOSE in your environment to increase build verbosity.
MPY scripts/micropower.py
MPY scripts/upower.py
Creating build-PYBV10/scripts/frozen_mpy.c
error while freezing scripts/upower.py: freezing of object 1.21 is not implemented
make: *** [build-PYBV10/scripts/frozen_mpy.c] Error 1
make: *** Deleting file `build-PYBV10/scripts/frozen_mpy.c'
[adminpete@axolotl]: ~/temp/micropython/stmhal

@dpgeorge If it's any help I could do a binary chop of the file to try to narrow down the cause, but perhaps the object number is clue enough?

dpgeorge · 2016-03-01T10:12:01Z

error while freezing scripts/upower.py: freezing of object 1.21 is not implemented

Freezing of floats is not yet supported.

There are plent of directions frozen bytecode can go, for example:

support freezing of more constant data like floats, tuples, etc
support freezing of natively generated code (native, viper, asm)
support package imports
dynamic freezing (freezing a set of .py or .mpy files at runtime)

The problem is time, just not enough of it :)

dpgeorge · 2016-03-02T15:54:16Z

As the chars to replace are those used in regexp, I assume there is a (small) problem within
mpy-tool.py.

There is no regex in mpy-tool... I'll have to look into it.

chuckbook · 2016-03-02T16:39:49Z

Not regex per se but string encoding/substitution in general.

zumpchke · 2016-03-14T12:51:46Z

When will this be merged to master?

dpgeorge · 2016-03-14T13:13:35Z

When will this be merged to master?

Soon :) There are a lot of other things pending that need to be done first.

It's needed by frozen bytecode.

Currently it can freeze .mpy files.

The config variable MICROPY_MODULE_FROZEN is now made of two separate parts: MICROPY_MODULE_FROZEN_STR and MICROPY_MODULE_FROZEN_MPY. This allows to have none, either or both of frozen strings and frozen mpy files (aka frozen bytecode).

frozentest.py is frozen into the binary as frozen bytecode. The .mpy file is included so that there is no dependency on the cross compiler.

dpgeorge · 2016-04-13T15:21:59Z

Merged, starting at commit 6d24dc2, ending 733db52.

chuckbook · 2016-04-13T17:53:59Z

@dpgeorge this does not work as a frozen bytecode:
def f(): return ';'
whereas
def f(): return ':'
works fine.
Probably an error in mpy-tool.py.

dpgeorge · 2016-04-13T21:15:01Z

@chuckbook thanks, should now be fixed.

chuckbook · 2016-04-13T21:43:38Z

nearly :-)
chr(0x5c) handling still needs some attention...

def f(): return '$;=?@'+chr(0x5c)+'^\'|~'

works
def f(): return '$;=?@\^\'|~'
doesn't

dpgeorge · 2016-04-14T13:23:52Z

@chuckbook should be fixed by 49bb04e.

chuckbook · 2016-04-15T17:45:36Z

@dpgeorge thanks a lot!

Tweak RTD to install svg2pdfconverter

dpgeorge force-pushed the mpytool branch from f7533bb to 8d1e0d5 Compare January 31, 2016 22:37

This was referenced Jan 31, 2016

Prepare for v1.6 #1803

Closed

RFC: Add zipfile support #1797

Closed

dpgeorge mentioned this pull request Feb 11, 2016

Add MicroPython cross compiler: mpycross #1839

Merged

dpgeorge force-pushed the mpytool branch from 8d1e0d5 to 42bc25b Compare February 25, 2016 12:22

dpgeorge mentioned this pull request Mar 7, 2016

Access Frozen Module from SD Card #1858

Closed

zumpchke mentioned this pull request Mar 11, 2016

Standalone binary? #1892

Closed

dpgeorge force-pushed the mpytool branch from 42bc25b to 102f2d7 Compare March 13, 2016 22:13

dpgeorge added 7 commits March 15, 2016 21:31

py/emitglue: Move typedef of mp_raw_code_t from .c to .h file.

e57b7ae

It's needed by frozen bytecode.

py/emitglue: Make mp_raw_code_t* arguments constant pointers.

92db286

py/makeqstrdata: Factor out some code to functions that can be reused.

0cc4fbf

tools: Add mpy-tool.py, to work with .mpy files.

42d4867

Currently it can freeze .mpy files.

minimal: Add example of frozen persistent bytecode (.mpy file).

2412f82

frozentest.py is frozen into the binary as frozen bytecode. The .mpy file is included so that there is no dependency on the cross compiler.

stmhal: Add Makefile option FROZEN_MPY_DIR to support frozen bytecode.

828a8a8

dpgeorge force-pushed the mpytool branch from 102f2d7 to 828a8a8 Compare March 15, 2016 21:31

pfalcon force-pushed the master branch 6 times, most recently from 9167980 to 1cc81ed Compare April 10, 2016 22:16

dpgeorge closed this Apr 13, 2016

dpgeorge deleted the mpytool branch April 13, 2016 15:22

tannewt added a commit to tannewt/circuitpython that referenced this pull request Apr 18, 2019

Merge pull request micropython#1811 from tannewt/doc_build_tweak

70dd616

Tweak RTD to install svg2pdfconverter

Add ability to have frozen bytecode #1811

Add ability to have frozen bytecode #1811

Conversation

dpgeorge commented Jan 31, 2016

dpgeorge commented Jan 31, 2016

dpgeorge commented Feb 1, 2016

dpgeorge commented Feb 2, 2016

dpgeorge commented Feb 25, 2016

peterhinch commented Feb 26, 2016

chuckbook commented Feb 26, 2016

chuckbook commented Feb 26, 2016

dpgeorge commented Feb 26, 2016

chuckbook commented Feb 26, 2016

dpgeorge commented Feb 26, 2016

chuckbook commented Feb 26, 2016

peterhinch commented Feb 27, 2016

dpgeorge commented Feb 27, 2016 via email

dhylands commented Feb 27, 2016

peterhinch commented Feb 27, 2016

chuckbook commented Feb 27, 2016

dhylands commented Feb 27, 2016

dhylands commented Feb 27, 2016

chuckbook commented Feb 27, 2016

peterhinch commented Feb 27, 2016

dhylands commented Feb 27, 2016

chuckbook commented Feb 27, 2016

peterhinch commented Feb 28, 2016

peterhinch commented Mar 1, 2016

dpgeorge commented Mar 1, 2016

dpgeorge commented Mar 2, 2016

chuckbook commented Mar 2, 2016

zumpchke commented Mar 14, 2016

dpgeorge commented Mar 14, 2016

dpgeorge commented Apr 13, 2016

chuckbook commented Apr 13, 2016

dpgeorge commented Apr 13, 2016

chuckbook commented Apr 13, 2016

dpgeorge commented Apr 14, 2016

chuckbook commented Apr 15, 2016