Skip to content

tools/boardgen: Ensure locals_dict is generated with consistent order. #17433

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jun 10, 2025
Merged

tools/boardgen: Ensure locals_dict is generated with consistent order. #17433

merged 1 commit into from
Jun 10, 2025
+6 −2

Conversation

andrewleech
Copy link
Contributor

Summary

tools/boardgen.py is used by the make-pins.py scripts in many ports to generate the pin definitions for the machine module.

In #17391 (comment) I found that this is currently generating the C structs for pin definitions with inconsistent ordering which makes it sometimes impossible to get a consistent built binary file even for no change in source files.

Testing

Reviewed the generated files from running to see the content is the same as previously, however now with different / repeatable order.

cd ports/stm32; 
python boards/make-pins.py --board-csv boards/PYBD_SF6/pins.csv --af-csv boards/stm32f767_af.csv --prefix boards/stm32f4xx_prefix.c          │
│   --output-source build-PYBD_SF6/pins_PYBD_SF6.c --output-header build-PYBD_SF6/genhdr/pins.h --output-af-const build-PYBD_SF6/genhdr/pins_af_const.h --output-af-defs build-PYBD_SF6/genhdr/pins_af_defs.h

@codecov Codecov
Copy link

codecov bot commented Jun 5, 2025 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.54%. Comparing base (b8e56a1) to head (95203ab).
Report is 3 commits behind head on master.

Additional details and impacted files 
@@           Coverage Diff           @@
##           master   #17433   +/-   ##
=======================================
  Coverage   98.54%   98.54%           
=======================================
  Files         169      169           
  Lines       21943    21943           
=======================================
  Hits        21623    21623           
  Misses        320      320

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions GitHub Actions
Copy link

github-actions bot commented Jun 5, 2025

Code size report:

   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +0 +0.000% RPI_PICO_W
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:    +0 +0.000% VIRT_RV32

dpgeorge
dpgeorge reviewed Jun 5, 2025
View reviewed changes
@andrewleech andrewleech force-pushed the boardgen_pin_order branch from ecbabf0 to 54cfd75 Compare June 5, 2025 02:16
@andrewleech
Copy link
Contributor Author

Updated to just order the board name iterating (over a set)

Also created some unit tests for this module, I don't believe there was any automated testing on it previously?

@dpgeorge
Copy link
Member

dpgeorge commented Jun 5, 2025

Also created some unit tests for this module, I don't believe there was any automated testing on it previously?

Correct, there's no existing test for it. Except of course that boards build without error.

I can see it's AI generated, and not actually run as part of the CI (yet). It's quite a large test and would take time to review. Maybe put that in a separate PR, so this one (sorting pin names) is easier and quicker to get merged?

@andrewleech andrewleech force-pushed the boardgen_pin_order branch from 54cfd75 to 1143c9d Compare June 5, 2025 02:56
@andrewleech andrewleech mentioned this pull request Jun 5, 2025
Open
@andrewleech
Copy link
Contributor Author

andrewleech commented Jun 5, 2025 •
edited
Loading

I can see it's AI generated, and not actually run as part of the CI (yet). It's quite a large test and would take time to review. Maybe put that in a separate PR, so this one (sorting pin names) is easier and quicker to get merged?

Yes it was generated, I used it to check the results of the change here and gave a quick sanity check over the other tests thinking they looked better than nothing. I didn't think a build / no build of ports is quite enough coverage as doens't show if things are missed, nor the ordering as found today!

It does warrant separate discussion though so good idea, I've split it out into its own PR here #17434

@dpgeorge dpgeorge added the tools Relates to tools/ directory in source, or other tooling label Jun 5, 2025
@andrewleech andrewleech mentioned this pull request Jun 5, 2025
Merged
dpgeorge
dpgeorge approved these changes Jun 10, 2025
View reviewed changes
Copy link
Member

@dpgeorge dpgeorge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested on PYBD_SF2. The board pins are now in a sorted order (sorting is first done on CPU name, then boards name).

@pi-anl @dpgeorge
tools/boardgen.py: Ensure board pin locals_dict has consistent order.
95203ab 
`tools/boardgen.py` is used by the `make-pins.py` scripts in many ports to
generate the pin definitions for the machine module.

In micropython#17391 it was found that this is currently generating the C structs for
board pin definitions with inconsistent ordering (across different build
runs), which makes it sometimes impossible to get a consistent binary file
even for no change in source files.

This commit fixes that by sorting the board pin names alphabetically.

Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
@dpgeorge dpgeorge force-pushed the boardgen_pin_order branch from 1143c9d to 95203ab Compare June 10, 2025 01:10
@dpgeorge dpgeorge merged commit 95203ab into micropython:master Jun 10, 2025
66 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dpgeorge dpgeorge dpgeorge approved these changes

Assignees
No one assigned
Labels
tools Relates to tools/ directory in source, or other tooling
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@andrewleech @dpgeorge @pi-anl