tools/manifestfile: Add --unix-ffi option. #13655

projectgus · 2024-02-13T06:19:45Z

Allows adding unix-ffi library from the command line, to fix micropython-lib builds.

Follow up to 35dd959 from #13620

This doesn't expose the full add_library() function to the command line, as currently the only use case is unix-ffi libraries. The only "external" user of this tool is currently micropython-lib.

Before:

 python ../../tools/manifestfile.py --lib . --compile unix-ffi/email.message/manifest.py
Error in manifest file: /home/gus/ry/george/micropython/lib/micropython-lib/unix-ffi/email.message/manifest.py: Package 're' not found in any known library.

After:

❯ python ../../tools/manifestfile.py --unix-ffi --lib . --compile unix-ffi/email.message/manifest.py
version=0.5.3 description=None license=None author=None pypi=None pypi_publish=None
ManifestOutput(file_type=1, full_path='/home/gus/ry/george/micropython/lib/micropython-lib/unix-ffi/ffilib/ffilib.py', target_path='ffilib.py', timestamp=1676863157.6312485, kind=6, metadata=<__main__.ManifestPackageMetadata object at 0x7a8cd86d3a10>, opt=None)
ManifestOutput(file_type=1, full_path='/home/gus/ry/george/micropython/lib/micropython-lib/unix-ffi/re/re.py', target_path='re.py', timestamp=1707800376.548602, kind=6, metadata=<__main__.ManifestPackageMetadata object at 0x7a8cd86d3190>, opt=None)
ManifestOutput(file_type=1, full_path='/home/gus/ry/george/micropython/lib/micropython-lib/python-stdlib/binascii/binascii.py', target_path='binascii.py', timestamp=1676863157.6112483, kind=6, metadata=<__main__.ManifestPackageMetadata object at 0x7a8cd86dcc90>, opt=None)
ManifestOutput(file_type=1, full_path='/home/gus/ry/george/micropython/lib/micropython-lib/python-stdlib/os/os/__init__.py', target_path='os/__init__.py', timestamp=1677138094.1277487, kind=6, metadata=<__main__.ManifestPackageMetadata object at 0x7a8cd86de150>, opt=None)
ManifestOutput(file_type=1, full_path='/home/gus/ry/george/micropython/lib/micropython-lib/python-stdlib/os-path/os/path.py', target_path='os/path.py', timestamp=1707800376.548602, kind=6, metadata=<__main__.ManifestPackageMetadata object at 0x7a8cd86dc9d0>, opt=None)
ManifestOutput(file_type=1, full_path='/home/gus/ry/george/micropython/lib/micropython-lib/python-stdlib/uu/uu.py', target_path='uu.py', timestamp=1676863157.6245818, kind=6, metadata=<__main__.ManifestPackageMetadata object at 0x7a8cd86d3610>, opt=None)
ManifestOutput(file_type=1, full_path='/home/gus/ry/george/micropython/lib/micropython-lib/python-stdlib/struct/struct.py', target_path='struct.py', timestamp=1676863157.617915, kind=6, metadata=<__main__.ManifestPackageMetadata object at 0x7a8cd86dc910>, opt=None)
ManifestOutput(file_type=1, full_path='/home/gus/ry/george/micropython/lib/micropython-lib/python-stdlib/base64/base64.py', target_path='base64.py', timestamp=1707800376.545269, kind=6, metadata=<__main__.ManifestPackageMetadata object at 0x7a8cd86d3490>, opt=None)
ManifestOutput(file_type=1, full_path='/home/gus/ry/george/micropython/lib/micropython-lib/python-stdlib/random/random.py', target_path='random.py', timestamp=1676863157.617915, kind=6, metadata=<__main__.ManifestPackageMetadata object at 0x7a8cd86dc510>, opt=None)
ManifestOutput(file_type=1, full_path='/home/gus/ry/george/micropython/lib/micropython-lib/python-stdlib/datetime/datetime.py', target_path='datetime.py', timestamp=1700610061.7551172, kind=6, metadata=<__main__.ManifestPackageMetadata object at 0x7a8cd86dcbd0>, opt=None)
ManifestOutput(file_type=1, full_path='/home/gus/ry/george/micropython/lib/micropython-lib/python-stdlib/collections/collections/__init__.py', target_path='collections/__init__.py', timestamp=1677138094.1244152, kind=6, metadata=<__main__.ManifestPackageMetadata object at 0x7a8cd86dd7d0>, opt=None)
ManifestOutput(file_type=1, full_path='/home/gus/ry/george/micropython/lib/micropython-lib/python-stdlib/collections-defaultdict/collections/defaultdict.py', target_path='collections/defaultdict.py', timestamp=1677138094.1244152, kind=6, metadata=<__main__.ManifestPackageMetadata object at 0x7a8cd86dda50>, opt=None)
ManifestOutput(file_type=1, full_path='/home/gus/ry/george/micropython/lib/micropython-lib/unix-ffi/urllib.parse/urllib/parse.py', target_path='urllib/parse.py', timestamp=1680739294.753335, kind=6, metadata=<__main__.ManifestPackageMetadata object at 0x7a8cd86dc090>, opt=None)
ManifestOutput(file_type=1, full_path='/home/gus/ry/george/micropython/lib/micropython-lib/python-stdlib/warnings/warnings.py', target_path='warnings.py', timestamp=1676863157.6245818, kind=6, metadata=<__main__.ManifestPackageMetadata object at 0x7a8cd86dc950>, opt=None)
ManifestOutput(file_type=1, full_path='/home/gus/ry/george/micropython/lib/micropython-lib/python-stdlib/quopri/quopri.py', target_path='quopri.py', timestamp=1676863157.617915, kind=6, metadata=<__main__.ManifestPackageMetadata object at 0x7a8cd86dd290>, opt=None)
ManifestOutput(file_type=1, full_path='/home/gus/ry/george/micropython/lib/micropython-lib/python-stdlib/functools/functools.py', target_path='functools.py', timestamp=1676863157.6112483, kind=6, metadata=<__main__.ManifestPackageMetadata object at 0x7a8cd86dd590>, opt=None)
ManifestOutput(file_type=1, full_path='/home/gus/ry/george/micropython/lib/micropython-lib/python-stdlib/string/string.py', target_path='string.py', timestamp=1676863157.617915, kind=6, metadata=<__main__.ManifestPackageMetadata object at 0x7a8cd86df0d0>, opt=None)
ManifestOutput(file_type=1, full_path='/home/gus/ry/george/micropython/lib/micropython-lib/unix-ffi/email.encoders/email/quoprimime.py', target_path='email/quoprimime.py', timestamp=1677138094.1344154, kind=6, metadata=<__main__.ManifestPackageMetadata object at 0x7a8cd86dd150>, opt=None)
ManifestOutput(file_type=1, full_path='/home/gus/ry/george/micropython/lib/micropython-lib/unix-ffi/email.encoders/email/base64mime.py', target_path='email/base64mime.py', timestamp=1677138094.1344154, kind=6, metadata=<__main__.ManifestPackageMetadata object at 0x7a8cd86dd150>, opt=None)
ManifestOutput(file_type=1, full_path='/home/gus/ry/george/micropython/lib/micropython-lib/unix-ffi/email.encoders/email/encoders.py', target_path='email/encoders.py', timestamp=1677138094.1344154, kind=6, metadata=<__main__.ManifestPackageMetadata object at 0x7a8cd86dd150>, opt=None)
ManifestOutput(file_type=1, full_path='/home/gus/ry/george/micropython/lib/micropython-lib/unix-ffi/email.errors/email/errors.py', target_path='email/errors.py', timestamp=1677138094.1344154, kind=6, metadata=<__main__.ManifestPackageMetadata object at 0x7a8cd86dc3d0>, opt=None)
ManifestOutput(file_type=1, full_path='/home/gus/ry/george/micropython/lib/micropython-lib/unix-ffi/email.charset/email/charset.py', target_path='email/charset.py', timestamp=1677138094.1344154, kind=6, metadata=<__main__.ManifestPackageMetadata object at 0x7a8cd86dd8d0>, opt=None)
ManifestOutput(file_type=1, full_path='/home/gus/ry/george/micropython/lib/micropython-lib/unix-ffi/email.utils/email/utils.py', target_path='email/utils.py', timestamp=1680739294.753335, kind=6, metadata=<__main__.ManifestPackageMetadata object at 0x7a8cd86d28d0>, opt=None)
ManifestOutput(file_type=1, full_path='/home/gus/ry/george/micropython/lib/micropython-lib/unix-ffi/email.message/email/message.py', target_path='email/message.py', timestamp=1677138094.1377487, kind=6, metadata=<__main__.ManifestPackageMetadata object at 0x7a8cd86d1bd0>, opt=None)
ManifestOutput(file_type=1, full_path='/home/gus/ry/george/micropython/lib/micropython-lib/unix-ffi/email.message/email/iterators.py', target_path='email/iterators.py', timestamp=1677138094.1377487, kind=6, metadata=<__main__.ManifestPackageMetadata object at 0x7a8cd86d1bd0>, opt=None)

This work was funded through GitHub Sponsors.

tools/manifestfile.py

codecov · 2024-02-13T06:26:28Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (def6ad4) 98.36% compared to head (1ef2944) 98.36%.

Additional details and impacted files

@@           Coverage Diff           @@
##           master   #13655   +/-   ##
=======================================
  Coverage   98.36%   98.36%           
=======================================
  Files         160      160           
  Lines       21075    21075           
=======================================
  Hits        20730    20730           
  Misses        345      345

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

github-actions · 2024-02-13T06:28:28Z

Code size report:

   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +0 +0.000% RPI_PICO
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS

tools/manifestfile.py

dpgeorge · 2024-02-15T05:00:58Z

I think it's fine to make this a special option called --unix-ffi that just does the right thing.

This tool running as a command-line tool is anyway an internal tool to MicroPython (only used by micropython-lib CI as far as I can tell), so we don't need to worry too much about changing/adding/removing these command-line options.

projectgus · 2024-02-16T02:53:56Z

@dpgeorge @jimmo I've updated this to add --unix-ffi only, with a short additional rationale in the description.

tools/manifestfile.py

jimmo · 2024-02-17T02:20:54Z

Yep happy to make this just --unix-ffi. Thanks @projectgus

jimmo · 2024-02-17T02:22:09Z

(Although it might be useful for CI in a different non-micropython-lib repo, which was the reason we added support for add_library initially. Anyway we can deal with that in a future PR, let's just get the CI working for micropython-lib)

projectgus · 2024-02-19T01:38:22Z

(Although it might be useful for CI in a different non-micropython-lib repo, which was the reason we added support for add_library initially. Anyway we can deal with that in a future PR, let's just get the CI working for micropython-lib)

I think it would be really good to have a good story on how to build and publish micropython packages outside of micropython-lib. Maybe I'm just missing the relevant knowledge, but it seems to me like actually building that out might result in a slightly different high-level design to what's currently here anyhow.

Follow up to 35dd959, allows explicitly adding the unix-ffi library path from the command line. This option is needed when building unix-ffi manifests in micropython-lib CI. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <angus@redyak.com.au>

dpgeorge · 2024-02-19T02:49:01Z

Thanks for updating. Now merged.

projectgus added the tools Relates to tools/ directory in source, or other tooling label Feb 13, 2024

dpgeorge reviewed Feb 13, 2024

View reviewed changes

tools/manifestfile.py Outdated Show resolved Hide resolved

projectgus force-pushed the bugfix/manifestfile/add_library branch from b385cfd to 57b7913 Compare February 13, 2024 06:29

projectgus mentioned this pull request Feb 13, 2024

ci: Add unix-ffi library when building unix-ffi subdirectory. micropython/micropython-lib#805

Merged

projectgus force-pushed the bugfix/manifestfile/add_library branch from 57b7913 to 4cb1d5e Compare February 13, 2024 06:38

jimmo reviewed Feb 14, 2024

View reviewed changes

tools/manifestfile.py Show resolved Hide resolved

projectgus force-pushed the bugfix/manifestfile/add_library branch from 4cb1d5e to 61ba293 Compare February 16, 2024 02:52

projectgus changed the title ~~tools/manifestfile: Add --add-library option.~~ tools/manifestfile: Add --unix-ffi option. Feb 16, 2024

dpgeorge reviewed Feb 16, 2024

View reviewed changes

tools/manifestfile.py Outdated Show resolved Hide resolved

dpgeorge reviewed Feb 16, 2024

View reviewed changes

tools/manifestfile.py Outdated Show resolved Hide resolved

projectgus force-pushed the bugfix/manifestfile/add_library branch from 61ba293 to 5c70b15 Compare February 19, 2024 01:36

dpgeorge force-pushed the bugfix/manifestfile/add_library branch from 5c70b15 to 1ef2944 Compare February 19, 2024 02:34

dpgeorge merged commit 1ef2944 into micropython:master Feb 19, 2024

projectgus deleted the bugfix/manifestfile/add_library branch February 19, 2024 23:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

tools/manifestfile: Add --unix-ffi option. #13655

tools/manifestfile: Add --unix-ffi option. #13655

Uh oh!

projectgus commented Feb 13, 2024 •

edited

Loading

Uh oh!

Uh oh!

codecov bot commented Feb 13, 2024 •

edited

Loading

Uh oh!

github-actions bot commented Feb 13, 2024

Uh oh!

Uh oh!

dpgeorge commented Feb 15, 2024

Uh oh!

projectgus commented Feb 16, 2024

Uh oh!

Uh oh!

Uh oh!

jimmo commented Feb 17, 2024

Uh oh!

jimmo commented Feb 17, 2024 •

edited

Loading

Uh oh!

projectgus commented Feb 19, 2024

Uh oh!

dpgeorge commented Feb 19, 2024

Uh oh!

Uh oh!

Uh oh!

tools/manifestfile: Add --unix-ffi option. #13655

tools/manifestfile: Add --unix-ffi option. #13655

Uh oh!

Conversation

projectgus commented Feb 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Feb 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Feb 13, 2024

Uh oh!

Uh oh!

dpgeorge commented Feb 15, 2024

Uh oh!

projectgus commented Feb 16, 2024

Uh oh!

Uh oh!

Uh oh!

jimmo commented Feb 17, 2024

Uh oh!

jimmo commented Feb 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

projectgus commented Feb 19, 2024

Uh oh!

dpgeorge commented Feb 19, 2024

Uh oh!

Uh oh!

projectgus commented Feb 13, 2024 •

edited

Loading

codecov bot commented Feb 13, 2024 •

edited

Loading

jimmo commented Feb 17, 2024 •

edited

Loading