Skip to content

unix: Add additional testing targets. #17921

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

jepler
Copy link
Contributor

@jepler jepler commented Aug 14, 2025

Summary

These are convenience targets for running specific tests as a developer. They are more useful that invoking run-tests directly as they take account of the VARIANT= specified on the make command-line.

For instance, you can run all tests matching the regular expression "int" with make VARIANT=... test//int. (the new targets are all documented in README.md)

Testing

I used each of the added targets locally and verified that they seemed to work as intended.

Trade-offs and Alternatives

This doesn't do anything that isn't possible to day by properly setting MICROPY_MICROPYTHON and invoking run-tests.py directly but I find it's a good QOL enhancement. CircuitPython carries something similar but not quite the same. (note to @dhalbert in the future: if this is accepted, you should take this version and drop the one in circuitpython)

These are convenience targets for running specific tests as a
developer. They are more useful that invoking run-tests
directly as they take account of the VARIANT= specified on the
make command-line.

For instance, you can run all tests matching the regular expression
"int" with `make VARIANT=... test//int`. (the new targets are
all documented in README.md)

Signed-off-by: Jeff Epler <jepler@gmail.com>
Copy link

Code size report:

   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +0 +0.000% RPI_PICO_W
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:    +0 +0.000% VIRT_RV32

@jepler
Copy link
Contributor Author

jepler commented Aug 14, 2025

There are of course different naming conventions that could be used for the "run a test subset" rules test/% and test//%. Here's the reasoning that led to the current forms:

  • I started with test-a-% and test-d-% to mirror the commandline flags, but for sub directories to be accepted by the directory test pattern, the % had to follow a /, so it would have to be test/% test-d/% or similar. I went with the shortest test/%.
  • similarly the regular expression based search would need to be of a similar form if patterns with an embedded / are important. Such patterns might occur, like if you want to test everything that matches the pattern basics/.*int for example. It turns out that with //% and /% patterns, make picks the correct (longest prefix?) one, at least in my testing. If one "/" introduces "here's what to test" (and has to for technical reasons of GNU make), the vi user in me finds "//" as "ditto, but it's a search" to be natural.

@Josverl
Copy link
Contributor

Josverl commented Aug 14, 2025

And would that pattern work with WindowsPaths \ as well ?

@jepler
Copy link
Contributor Author

jepler commented Aug 14, 2025

That's a good question! I don't have any Windows machines available to test.

Since most path like stuff is handled just fine by make even though Windows uses a different directory separator, I think the part where it gets the test selection into $* will be the same.

The run-tests script is careful to replace windows-style directory separators with forward slash ones before applying filters (see around line 884), so I think there's a good chance that make test//basics/.*int would work in a cross-platform way.

It would be great if someone can actually test on Windows and advise how to update the readme about any caveats that may exist. If someone does I'll update it.

Copy link

codecov bot commented Aug 15, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.38%. Comparing base (744270a) to head (0a119b8).
⚠️ Report is 17 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master   #17921   +/-   ##
=======================================
  Coverage   98.38%   98.38%           
=======================================
  Files         171      171           
  Lines       22295    22298    +3     
=======================================
+ Hits        21936    21939    +3     
  Misses        359      359           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@Josverl
Copy link
Contributor

Josverl commented Aug 15, 2025

I found some time.

under MSYS2 + the same changes to the windows makefile - the test selection also works fine.
while make itself runs on plain powershell - the rest of the toolchain not.

So while it works, I don't think its worth the effort to add to the windows makefile.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants