tests/run-tests.py: Change --target/--device options to --test-instance. #16111

dpgeorge · 2024-10-30T02:02:14Z

Summary

Previously to this PR, running the test suite on a bare-metal board required specifying the target (really platform) and device, eg:

$ ./run-tests.py --target pyboard --device /dev/ttyACM1

That's quite a lot to type, and you also need to know what the target platform is, when a lot of the time you either don't care or it doesn't matter.

This commit makes it easier to run the tests by replacing both of these options with a single --test-instance (-t for short) option. That option specifies the executable/port/device to test. Then the target platform is automatically detected.

The --test-instance can be passed:

"micropython" (the default) to use the unix version of MicroPython (this is the same as the previous default behaviour)
"webassembly" to test the webassembly port
anything else is considered a port/device to pass to Pyboard

There are also some shortcuts to specify a port/device, following mpremote:

a is short for /dev/ttyACM
u is short for /dev/ttyUSB
c is short for COM

For example:

$ ./run-tests.py -t a1

As part of this change, the platform (and it's native architecture if it supports importing native .mpy files) is show at the start of the test run.

Testing

Ran the tests on a Pico and ESP8266 to verify it still works as expected. Also CI for this PR will test a lot of things (unix, webassembly, qemu, zephyr).

Trade-offs and Alternatives

One can no longer specify the platform (eg unix, pyboard, rp2, etc), it's now always automatically detected. I don't think this is an issue.

The new argument is --test-instance with -t for short. It could be --target or --dut or something else, but I think --test-instance captures the intent pretty well.

Eventually this scheme would be adopted by the other test runner scripts like run-multitests.py and run-perfbench.py.

dpgeorge · 2024-10-30T02:02:58Z

@stinos FYI

github-actions · 2024-10-30T02:10:29Z

Code size report:

   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +0 +0.000% RPI_PICO_W
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:    +0 +0.000% VIRT_RV32

codecov · 2024-10-30T02:15:19Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.57%. Comparing base (c88a9d6) to head (85053ad).
Report is 3 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master   #16111   +/-   ##
=======================================
  Coverage   98.57%   98.57%           
=======================================
  Files         164      164           
  Lines       21345    21345           
=======================================
  Hits        21041    21041           
  Misses        304      304

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

andrewleech · 2024-10-30T02:44:02Z

This looks amazing @dpgeorge I'm a huge supporter of automating testing like this in hardware!

It should make it much easier for myself and other developers to run these tests, which lowers the barriers to extending them and creating new ones to improve test coverage!

@mattytrentini @Josverl @hmaerki this could integrate nicely into a chain with mpbuild, mpflash and octoprobe

dpgeorge · 2024-10-30T02:50:50Z

@andrewleech This is not the full auto test runner yet 😄 Just the first small piece.

dpgeorge · 2024-10-30T02:53:37Z

One can no longer specify the platform (eg unix, pyboard, rp2, etc), it's now always automatically detected. I don't think this is an issue.

Actually, this could possibly be an issue in some corner cases, eg when you want to use the "minimal" target profile on a non-minimal port, say zephyr (that's what the CI used to do). But I'd rather solve that problem by doing better auto-detection of features that the port has, rather than needing to specify it on the command line.

So I still stick with this PR and the fact you can no longer specify the platform, it's always taken from sys.platform (with a fallback to "minimal" if sys.platform doesn't exist).

dpgeorge · 2024-10-30T03:26:04Z

It should make it much easier for myself and other developers to run these tests

See #16112 for that.

hmaerki · 2024-10-30T10:50:17Z

@andrewleech This is not the full auto test runner yet 😄 Just the first small piece.

Octoprobe will provide everything to make it fully automated, even as self-hosted github runner - let me first finish the documentation.

stinos · 2024-10-30T16:42:44Z

tests/run-tests.py

@@ -115,6 +200,53 @@ def convert_regex_escapes(line):
    return bytes("".join(cs), "utf8")


+def get_test_instance(test_instance, baudrate, user, password):
+    if test_instance.startswith("port:"):


It's not really clear to me where/how this can be specified, should this be in the help (-h) output?

Ah, I missed documenting this. Now added to the help text.

stinos · 2024-10-30T16:43:53Z

This looks good. Since for the default (running with unix port and/or using MICROPY_MICROPYTHON) nothing changes, it might be clearer if the commit mentions that explicitly.

andrewleech · 2024-10-30T21:18:15Z

@andrewleech This is not the full auto test runner yet 😄 Just the first small piece.

Octoprobe will provide everything to make it fully automated, even as self-hosted github runner - let me first finish the documentation.

@hmaerki I mixed up this PR with another one, take a look at: #16112

dpgeorge · 2024-10-30T23:45:35Z

Since for the default (running with unix port and/or using MICROPY_MICROPYTHON) nothing changes, it might be clearer if the commit mentions that explicitly.

Commit message now updated to mention that. Also I embellished the help text further to explain more about the MICROPY_MICROPYTHON env variable.

stinos · 2024-10-31T07:51:15Z

Ok, nice.

projectgus

One small comment, but this looks good to me. Yay for improved ergonomics!

projectgus · 2024-10-31T05:57:02Z

tests/run-tests.py

 If a test fails, run-tests.py produces a pair of <test>.out and <test>.exp files in the result
 directory with the MicroPython output and the expectations, respectively.
 """,
        epilog="""\
+The -t option accepts the following for the test instance:
+- micropython - use the unix port of MicroPython, specified by the


This is a bit nitpicky, but how about naming this option unix?

(In a follow-up, I'd be interested to add support for unix:<VARIANT> to cut down on environment variable wrangling when testing minimal or coverage variants.)

This was called "micropython" to match run-multitests.py (and eg to distinguish from "cpython" possibility in the multitests).

But, happy to change it to "unix" here. That makes more sense, especially if you can also select "webassembly" as an option.

Eventually we can update run-multitests.py to use the same conventions (and even provide "micropython" as a backwards-compatible alias to "unix" there).

Oh, and the current (on master) target for the unix port is actually "unix"! And is actually the default value for --target. So makes sense to keep it as "unix".

Changed "micropython" to "unix".

Signed-off-by: Damien George <damien@micropython.org>

Previously to this commit, running the test suite on a bare-metal board required specifying the target (really platform) and device, eg: $ ./run-tests.py --target pyboard --device /dev/ttyACM1 That's quite a lot to type, and you also need to know what the target platform is, when a lot of the time you either don't care or it doesn't matter. This commit makes it easier to run the tests by replacing both of these options with a single `--test-instance` (`-t` for short) option. That option specifies the executable/port/device to test. Then the target platform is automatically detected. The `--test-instance` can be passed: - "unix" (the default) to use the unix version of MicroPython - "webassembly" to test the webassembly port - anything else is considered a port/device to pass to Pyboard There are also some shortcuts to specify a port/device, following `mpremote`: - a<n> is short for /dev/ttyACM<n> - u<n> is short for /dev/ttyUSB<n> - c<n> is short for COM<n> For example: $ ./run-tests.py -t a1 Note that the default test instance is "unix" and so this commit does not change the standard way to run tests on the unix port, by just doing `./run-tests.py`. As part of this change, the platform (and it's native architecture if it supports importing native .mpy files) is show at the start of the test run. Signed-off-by: Damien George <damien@micropython.org>

Signed-off-by: Damien George <damien@micropython.org>

dpgeorge added the tests Relates to tests/ directory in source label Oct 30, 2024

dpgeorge requested review from projectgus and andrewleech October 30, 2024 02:02

dpgeorge force-pushed the tests-simplify-target-selection branch from 5979b7d to 4f6e371 Compare October 30, 2024 02:12

dpgeorge force-pushed the tests-simplify-target-selection branch from 4f6e371 to bd7a336 Compare October 30, 2024 02:51

dpgeorge force-pushed the tests-simplify-target-selection branch from bd7a336 to 1424f23 Compare October 30, 2024 03:03

dpgeorge mentioned this pull request Oct 30, 2024

tests: add full, automated, top-level test runner for hardware #16112

Draft

stinos reviewed Oct 30, 2024

View reviewed changes

dpgeorge force-pushed the tests-simplify-target-selection branch from 1424f23 to 600162a Compare October 30, 2024 23:44

projectgus approved these changes Nov 1, 2024

View reviewed changes

dpgeorge added 3 commits November 4, 2024 12:02

tests/run-tests.py: Simplify the way target-specific tests are given.

f2ac471

Signed-off-by: Damien George <damien@micropython.org>

tests/run-tests.py: Add mimxrt and samd platforms.

85053ad

Signed-off-by: Damien George <damien@micropython.org>

dpgeorge force-pushed the tests-simplify-target-selection branch from 698f5fb to 85053ad Compare November 4, 2024 01:50

dpgeorge merged commit 85053ad into micropython:master Nov 4, 2024
66 checks passed

dpgeorge deleted the tests-simplify-target-selection branch November 4, 2024 02:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tests/run-tests.py: Change --target/--device options to --test-instance. #16111

tests/run-tests.py: Change --target/--device options to --test-instance. #16111

dpgeorge commented Oct 30, 2024

dpgeorge commented Oct 30, 2024

github-actions bot commented Oct 30, 2024

codecov bot commented Oct 30, 2024 •

edited

Loading

andrewleech commented Oct 30, 2024

dpgeorge commented Oct 30, 2024

dpgeorge commented Oct 30, 2024

dpgeorge commented Oct 30, 2024

hmaerki commented Oct 30, 2024

stinos Oct 30, 2024

dpgeorge Oct 30, 2024

stinos commented Oct 30, 2024

andrewleech commented Oct 30, 2024

dpgeorge commented Oct 30, 2024

stinos commented Oct 31, 2024

projectgus left a comment

projectgus Oct 31, 2024

dpgeorge Nov 1, 2024

dpgeorge Nov 1, 2024

dpgeorge Nov 1, 2024

tests/run-tests.py: Change --target/--device options to --test-instance. #16111

tests/run-tests.py: Change --target/--device options to --test-instance. #16111

Conversation

dpgeorge commented Oct 30, 2024

Summary

Testing

Trade-offs and Alternatives

dpgeorge commented Oct 30, 2024

github-actions bot commented Oct 30, 2024

codecov bot commented Oct 30, 2024 • edited Loading

Codecov Report

andrewleech commented Oct 30, 2024

dpgeorge commented Oct 30, 2024

dpgeorge commented Oct 30, 2024

dpgeorge commented Oct 30, 2024

hmaerki commented Oct 30, 2024

stinos Oct 30, 2024

Choose a reason for hiding this comment

dpgeorge Oct 30, 2024

Choose a reason for hiding this comment

stinos commented Oct 30, 2024

andrewleech commented Oct 30, 2024

dpgeorge commented Oct 30, 2024

stinos commented Oct 31, 2024

projectgus left a comment

Choose a reason for hiding this comment

projectgus Oct 31, 2024

Choose a reason for hiding this comment

dpgeorge Nov 1, 2024

Choose a reason for hiding this comment

dpgeorge Nov 1, 2024

Choose a reason for hiding this comment

dpgeorge Nov 1, 2024

Choose a reason for hiding this comment

codecov bot commented Oct 30, 2024 •

edited

Loading