Skip to content

py/mpconfig: Add make show-config to parse and view overall config. #8437

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

py/mpconfig: Add make show-config to parse and view overall config. #8437

wants to merge 1 commit into from

Conversation

andrewleech
Copy link
Contributor

This PR provides a new make show-config target which will search for all MICROPY_* defines used in current build and write them with their computed values for a file in the current build folder, eg.

anl@STEP:~/mpy_iot/src/micropython/ports/nrf$ make show-config
Use make V=1 or set BUILD_VERBOSE in your environment to increase build verbosity.
GEN build-pca10040/genhdr/mpconfig.cfg
Build configuration recorded in: build-pca10040/genhdr/mpconfig.cfg

Which produces a mpconfig.cfg file like

...
MICROPY_FLOAT_IMPL = 1
MICROPY_FLOAT_IMPL_DOUBLE = 2
MICROPY_FLOAT_IMPL_FLOAT = 1
MICROPY_FLOAT_IMPL_NONE = 0
# MICROPY_FLOAT_ZERO is not defined
MICROPY_FULL_CHECKS = 1
MICROPY_GC_ALLOC_THRESHOLD = 1
MICROPY_GC_CONSERVATIVE_CLEAR = 1
MICROPY_GC_HOOK_LOOP = ""  # is defined
MICROPY_GC_STACK_ENTRY_TYPE = "size_t"
# MICROPY_GIT_TAG is not defined
# MICROPY_HAL_HAS_VT100 is not defined
MICROPY_HAS_FILE_READER = 1
MICROPY_HELPER_LEXER_UNIX = 0
MICROPY_HELPER_REPL = 1
MICROPY_HW_BOARD_NAME = "PCA10040"
MICROPY_HW_ENABLE_INTERNAL_FLASH_STORAGE = 1
MICROPY_HW_ENABLE_RNG = 1
# MICROPY_HW_ENABLE_USB is not defined
MICROPY_HW_HAS_LED = 1
# MICROPY_HW_HAS_SDCARD is not defined
MICROPY_HW_LED1 = 17
# MICROPY_HW_LED1_PULLUP is not defined
MICROPY_HW_LED2 = 18
# MICROPY_HW_LED2_PULLUP is not defined
...

This is generated by:

  • Use the same list of *.c files used for QSTR generation
  • searching for all MICROPY_* patterns in the C files
  • list these in a temporary C file along with #include <py/mpconfig.h>
  • preprocess this temporary C file
  • parse the proprocessor output into format described above

By building on top of the proven QSTR make infrastructure all other standard mechanisms to specify build settings should work with this target, eg. make show-config BOARD=microbit to show the configuration used by that board definition etc.

background

On a number of occasions I've wanted a way to easily view the overall configuration for a particular build, both to see what settings are available, and to see what settings are resolved down to.

In particular, I'm trying to rebuild #8317 to correctly start with a commit that includes the ROM level definition, but then override the individual settings to have an overall null-change initial commit. I wanted a tool like this which could be used to show before and after that no change to settings has been made.

By parsing all the C files that will go into a build, all settings that are actually used in the code should found and documented, ignoring any that are set in board definitions but not actually used.

The corner case here is if a setting is defined and used in header files only, not ever being referenced directly in a C file, it will be missed. I'm not sure how often this scenareo is likely to happen?
Perhaps all C files should be pre-processed with -dD to capture and keep all defines in the .i output, then scan those outputs rather than the c files directly. If this flag is added to the existing pp stage in $(HEADER_BUILD)/qstr.i.last generation then this generated file could be re-used here, however I'm not sure if adding this flag would adversely affect the speed / compatibility of the existing qstr generation tasks.
Doing all this however would mean you lose visibility of which settings are used in code however, it would include everything defined, regardless of whether they're actively used or not.

@andrewleech andrewleech force-pushed the mpconfig_show_config branch 2 times, most recently from ecbf333 to ec0b69e Compare March 23, 2022 09:30
@dlech
Copy link
Contributor

dlech commented Mar 23, 2022

I'm still in favor of using Kconfig (rather than reinventing the wheel, so to speak). It has already solved this problem and more.

@andrewleech
Copy link
Contributor Author

Yep I'm interested in moving toward a config management system too, and I agree the Linux kernel one seems like it would be good. I don't know though about how the kconfig infrastructure fits together to know how to deal with multiple levels of settings (to match to the hierarchical arrangement of port/boards) but I presume it's supported.

Regardless of that, this PR is currently about visualising the output of a config system, rather than managing the inputs.

I am hoping however that this could be a first step to help gather together all the settings that are in use, could be a parser to use once-off to help create the definitions for use with kconfig?

@jej
Copy link

jej commented Mar 23, 2022 •
edited
Loading

Something like OpenWrt config? It manages many target boards and use the linux kernel config tool.

https://github.com/openwrt/openwrt

@andrewleech
Copy link
Contributor Author

Something like OpenWrt config? It manages many target boards and use the linux kernel config tool.

Yes! I had thought of them as an example showing menuconfig (kconfig) it can be used to manage a similar arrangement of targets - I'm not sure how easy it's going to be to learn how all this fits toghether though, still assuming it's going to be a lot of work to bring such a system in here.
I also don't want to lose the convenience of being able to just cd ports/stm; make BOARD=PYBD_SF6 so the system will need to be flexible enough to have board-level defaults that can just be visiualised and modified in a local-user fashion.

Again though, that's definitely a larger future task I'm not quite willing to take on until I get some of my other in-progress stuff finished!

@andrewleech andrewleech changed the title py/mpconfig: Add make show-config to parse and record overall config. py/mpconfig: Add make show-config to parse and view overall config. Mar 23, 2022
@dlech
Copy link
Contributor

dlech commented Mar 24, 2022

I am hoping however that this could be a first step to help gather together all the settings that are in use, could be a parser to use once-off to help create the definitions for use with kconfig?

Brilliant. 😄

@andrewleech
Copy link
Contributor Author

I've been thinking more on what I really want this tool to be able to do, will add some more features.

  • also generate a mpconfig.out.h file that can be pointed to in IDE's to improve introspection of what's enabled/disabled
  • also parse header files to ensure I'm not missing some configs
  • add commenting into the generated file to describe where a define is confugured, and where it's used. This will highlight any defines that are created but never consumed, and simplify navigating to the usages of each define to provide better context.

As suggested earlier, some of this will require pre-processing all the C files similar to qstr.i.last with the -dD setting so that #defines are kept in the .i file - this will allow me to find where everything is defined. As a trial I added this setting to the existing qstr.i.last make target but did not like the result. qstr.i.last went from being a ~700k lines long file to over 2.1M lines long. More importantly, time make build-pca10040/genhdr/qstrdefs.generated.h went from real 0m0.912s to real 0m1.679s on my really beefy desktop, that's not a performance hit I'm ok with on every normal build. So I'll just replicate the preprocessing commands needed from qstr.i.last into my script with that extra setting, so this slower processing is only done on demand for make show-config

@andrewleech andrewleech force-pushed the mpconfig_show_config branch 4 times, most recently from a9621f1 to 70e80d9 Compare April 3, 2022 00:31
@andrewleech andrewleech mentioned this pull request Apr 3, 2022
Closed
@dpgeorge dpgeorge added the py-core Relates to py/ directory in source label Apr 29, 2022
@andrewleech andrewleech force-pushed the mpconfig_show_config branch from 70e80d9 to 71dab52 Compare May 29, 2022 10:57
@andrewleech andrewleech force-pushed the mpconfig_show_config branch from 71dab52 to 285793a Compare August 9, 2022 01:29
tannewt added a commit to tannewt/circuitpython that referenced this pull request Jan 30, 2024
@tannewt
Re-enable paralleldisplaybus on ESP
3b15931 
Add lilygo t-display s3 because it has a parallel display.

Fixes micropython#8437
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
py-core Relates to py/ directory in source
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@andrewleech @dlech @jej @dpgeorge @pi-anl