Skip to content

Commit 5979b7d

Browse files
dpgeorgedpgeorge
committed
tests/run-tests.py: Change --target/--device options to --test-instance.
Previously to this commit, running the test suite on a bare-metal board required specifying the target (really platform) and device, eg: $ ./run-tests.py --target pyboard --device /dev/ttyACM1 That's quite a lot to type, and you also need to know what the target platform is, when a lot of the time you either don't care or it doesn't matter. This commit makes it easier to run the tests by replacing both of these options with a single `--test-instance` (`-t` for short) option. That option specifies the executable/port/device to test. Then the target platform is automatically detected. The `--test-instance` can be passed: - "micropython" (the default) to use the unix version of MicroPython - "webassembly" to test the webassembly port - anything else is considered a port/device to pass to Pyboard There are also some shortcuts to specify a port/device, following `mpremote`: - a<n> is short for /dev/ttyACM<n> - u<n> is short for /dev/ttyUSB<n> - c<n> is short for COM<n> For example: $ ./run-tests.py -t a1 As part of this change, the platform (and it's native architecture if it supports importing native .mpy files) is show at the start of the test run. Signed-off-by: Damien George <damien@micropython.org>
1 parent 995343d commit 5979b7d

File tree

10 files changed

+105
-66
lines changed

10 files changed

+105
-66
lines changed

‎docs/develop/gettingstarted.rst

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -278,7 +278,7 @@ To run a selection of tests on a board/device connected over USB use:
278278
.. code-block:: bash
279279
280280
$ cd tests
281-
$ ./run-tests.py --target minimal --device /dev/ttyACM0
281+
$ ./run-tests.py -t /dev/ttyACM0
282282
283283
See also :ref:`writingtests`.
284284

‎docs/develop/writingtests.rst

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -60,7 +60,7 @@ Then to run on a board:
6060

6161
.. code-block:: bash
6262
63-
$ ./run-tests.py --target minimal --device /dev/ttyACM0
63+
$ ./run-tests.py -t /dev/ttyACM0
6464
6565
And to run only a certain set of tests (eg a directory):
6666

‎ports/cc3200/tools/smoke.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
"""
77
Execute it like this:
88
9-
python3 run-tests.py --target wipy --device 192.168.1.1 ../cc3200/tools/smoke.py
9+
python3 run-tests.py -t 192.168.1.1 ../cc3200/tools/smoke.py
1010
"""
1111

1212
pin_map = [23, 24, 11, 12, 13, 14, 15, 16, 17, 22, 28, 10, 9, 8, 7, 6, 30, 31, 3, 0, 4, 5]

‎ports/qemu/Makefile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -166,7 +166,7 @@ run: $(BUILD)/firmware.elf
166166
.PHONY: test
167167
test: $(BUILD)/firmware.elf
168168
$(eval DIRNAME=ports/$(notdir $(CURDIR)))
169-
cd $(TOP)/tests && ./run-tests.py --target qemu --device execpty:"$(QEMU_SYSTEM) $(QEMU_ARGS) -serial pty -kernel ../$(DIRNAME)/$<" $(RUN_TESTS_ARGS) $(RUN_TESTS_EXTRA)
169+
cd $(TOP)/tests && ./run-tests.py -t execpty:"$(QEMU_SYSTEM) $(QEMU_ARGS) -serial pty -kernel ../$(DIRNAME)/$<" $(RUN_TESTS_ARGS) $(RUN_TESTS_EXTRA)
170170

171171
$(BUILD)/firmware.elf: $(LDSCRIPT) $(OBJ)
172172
$(Q)$(CC) $(LDFLAGS) -o $@ $(OBJ) $(LIBS)

‎ports/qemu/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -95,7 +95,7 @@ Or manually by first starting the emulation with `make run` and then running the
9595
tests against the serial device, for example:
9696

9797
$ cd ../../tests
98-
$ ./run-tests.py --target qemu --device /dev/pts/1
98+
$ ./run-tests.py -t /dev/pts/1
9999

100100
Extra make options
101101
------------------

‎ports/webassembly/Makefile

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -148,10 +148,10 @@ repl: $(BUILD)/micropython.mjs
148148
min: $(BUILD)/micropython.min.mjs
149149

150150
test: $(BUILD)/micropython.mjs $(TOP)/tests/run-tests.py
151-
cd $(TOP)/tests && MICROPY_MICROPYTHON_MJS=../ports/webassembly/$< ./run-tests.py --target webassembly
151+
cd $(TOP)/tests && MICROPY_MICROPYTHON_MJS=../ports/webassembly/$< ./run-tests.py -t webassembly
152152

153153
test_min: $(BUILD)/micropython.min.mjs $(TOP)/tests/run-tests.py
154-
cd $(TOP)/tests && MICROPY_MICROPYTHON_MJS=../ports/webassembly/$< ./run-tests.py --target webassembly
154+
cd $(TOP)/tests && MICROPY_MICROPYTHON_MJS=../ports/webassembly/$< ./run-tests.py -t webassembly
155155

156156
################################################################################
157157
# Remaining make rules.

‎tests/README.md

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,19 @@
11
# MicroPython Test Suite
22

3-
This directory contains tests for various functionality areas of MicroPython.
4-
To run all stable tests, run "run-tests.py" script in this directory.
3+
This directory contains tests for most parts of MicroPython.
4+
5+
To run all stable tests, run the "run-tests.py" script in this directory. By default
6+
that will run the test suite against the unix port of MicroPython.
7+
8+
To run the test suite against a bare-metal target (a board running MicroPython firmware)
9+
use the `-t` option to specify the serial port. This will automatically detect the
10+
target platform and run the appropriate set of tests for that platform. For example:
11+
12+
$ ./run-tests.py -t /dev/ttyACM0
13+
14+
That will run tests on the `/dev/ttyACM0` serial port. You can also use shortcut
15+
device names like `a<n>` for `/dev/ttyACM<n>` and `c<n>` for `COM<n>`. Use
16+
`./run-tests.py --help` to see all of the device possibilites, and other options.
517

618
Tests of capabilities not supported on all platforms should be written
719
to check for the capability being present. If it is not, the test

‎tests/feature_check/target_info.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,4 +19,4 @@
1919
"xtensawin",
2020
"rv32imc",
2121
][sys_mpy >> 10]
22-
print(arch)
22+
print(sys.platform, arch)

‎tests/run-tests.py

Lines changed: 82 additions & 55 deletions
Original file line numberDiff line numberDiff line change
@@ -91,7 +91,7 @@ def open(self, path, mode):
9191

9292
# Tests to skip on specific targets.
9393
# These are tests that are difficult to detect that they should not be run on the given target.
94-
target_tests_to_skip = {
94+
platform_tests_to_skip = {
9595
"esp8266": (
9696
"micropython/viper_args.py", # too large
9797
"micropython/viper_binop_arith.py", # too large
@@ -197,6 +197,53 @@ def convert_regex_escapes(line):
197197
return bytes("".join(cs), "utf8")
198198

199199

200+
def get_test_instance(test_instance, baudrate, user, password):
201+
if test_instance.startswith("port:"):
202+
_, port = test_instance.split(":", 1)
203+
elif test_instance == "micropython":
204+
return None
205+
elif test_instance == "webassembly":
206+
return PyboardNodeRunner()
207+
elif test_instance.startswith("a") and test_instance[1:].isdigit():
208+
port = "/dev/ttyACM" + test_instance[1:]
209+
elif test_instance.startswith("u") and test_instance[1:].isdigit():
210+
port = "/dev/ttyUSB" + test_instance[1:]
211+
elif test_instance.startswith("c") and test_instance[1:].isdigit():
212+
port = "COM" + test_instance[1:]
213+
else:
214+
# Assume it's a device path.
215+
port = test_instance
216+
217+
global pyboard
218+
sys.path.append(base_path("../tools"))
219+
import pyboard
220+
221+
pyb = pyboard.Pyboard(port, baudrate, user, password)
222+
pyboard.Pyboard.run_script_on_remote_target = run_script_on_remote_target
223+
pyb.enter_raw_repl()
224+
return pyb
225+
226+
227+
def detect_test_platform(pyb, args):
228+
# Run a script to detect various bits of information about the target test instance.
229+
output = run_feature_check(pyb, args, "target_info.py")
230+
if output.endswith(b"CRASH"):
231+
raise ValueError("cannot detect platform: {}".format(output))
232+
platform, arch = str(output, "ascii").strip().split()
233+
if arch == "None":
234+
arch = None
235+
236+
args.platform = platform
237+
args.arch = arch
238+
if arch and not args.mpy_cross_flags:
239+
args.mpy_cross_flags = "-march=" + arch
240+
241+
print("platform={}".format(platform), end="")
242+
if arch:
243+
print(" arch={}".format(arch), end="")
244+
print()
245+
246+
200247
def prepare_script_for_target(args, *, script_filename=None, script_text=None, force_plain=False):
201248
if force_plain or (not args.via_mpy and args.emit == "bytecode"):
202249
if script_filename is not None:
@@ -706,18 +753,18 @@ def run_tests(pyb, tests, args, result_dir, num_threads=1):
706753
skip_tests.add("extmod/ssl_poll.py")
707754

708755
# Skip thread mutation tests on targets that don't have the GIL.
709-
if args.target in ("rp2", "unix"):
756+
if args.platform in ("darwin", "rp2", "unix", "win32"):
710757
for t in tests:
711758
if t.startswith("thread/mutate_"):
712759
skip_tests.add(t)
713760

714761
# Some tests shouldn't be run on pyboard
715-
if args.target != "unix":
762+
if args.platform not in ("darwin", "unix", "win32"):
716763
skip_tests.add("basics/exception_chain.py") # warning is not printed
717764
skip_tests.add("micropython/meminfo.py") # output is very different to PC output
718765

719-
# Skip target-specific tests.
720-
skip_tests.update(target_tests_to_skip.get(args.target, ()))
766+
# Skip platform-specific tests.
767+
skip_tests.update(platform_tests_to_skip.get(args.platform, ()))
721768

722769
# Some tests are known to fail on 64-bit machines
723770
if pyb is None and platform.architecture()[0] == "64bit":
@@ -936,13 +983,28 @@ def main():
936983
specified test files. If test files nor directories are specified, the script
937984
expects to be ran in the tests directory (where this file is located) and the
938985
builtin tests suitable for the target platform are ran.
986+
939987
When running tests, run-tests.py compares the MicroPython output of the test with the output
940988
produced by running the test through CPython unless a <test>.exp file is found, in which
941989
case it is used as comparison.
990+
942991
If a test fails, run-tests.py produces a pair of <test>.out and <test>.exp files in the result
943992
directory with the MicroPython output and the expectations, respectively.
944993
""",
945994
epilog="""\
995+
The -t option accepts the following for the test instance:
996+
- micropython - use the unix port of MicroPython, specified by the
997+
MICROPY_MICROPYTHON environment variable
998+
- webassembly - use the webassembly port of MicroPython, specified by the
999+
MICROPY_MICROPYTHON_MJS environment variable
1000+
- a<n> - use /dev/ttyACM<n>
1001+
- u<n> - use /dev/ttyUSB<n>
1002+
- c<n> - use COM<n>
1003+
- exec:<command> - execute a command and attach to its stdin/stdout
1004+
- execpty:<command> - execute a command and attach to the printed /dev/pts/<n> device
1005+
- <a>.<b>.<c>.<d> - connect to the given IPv4 address
1006+
- anything else specifies a serial port
1007+
9461008
Options -i and -e can be multiple and processed in the order given. Regex
9471009
"search" (vs "match") operation is used. An action (include/exclude) of
9481010
the last matching regex is used:
@@ -951,11 +1013,8 @@ def main():
9511013
run-tests.py -e async -i async_foo - include all, exclude async, yet still include async_foo
9521014
""",
9531015
)
954-
cmd_parser.add_argument("--target", default="unix", help="the target platform")
9551016
cmd_parser.add_argument(
956-
"--device",
957-
default="/dev/ttyACM0",
958-
help="the serial device or the IP address of the pyboard",
1017+
"-t", "--test-instance", default="micropython", help="the MicroPython instance to test"
9591018
)
9601019
cmd_parser.add_argument(
9611020
"-b", "--baudrate", default=115200, help="the baud rate of the serial device"
@@ -1039,43 +1098,11 @@ def main():
10391098

10401099
sys.exit(0)
10411100

1042-
LOCAL_TARGETS = (
1043-
"unix",
1044-
"webassembly",
1045-
)
1046-
EXTERNAL_TARGETS = (
1047-
"pyboard",
1048-
"wipy",
1049-
"esp8266",
1050-
"esp32",
1051-
"minimal",
1052-
"nrf",
1053-
"qemu",
1054-
"renesas-ra",
1055-
"rp2",
1056-
"zephyr",
1057-
)
1058-
if args.target in LOCAL_TARGETS:
1059-
pyb = None
1060-
if args.target == "webassembly":
1061-
pyb = PyboardNodeRunner()
1062-
elif args.target in EXTERNAL_TARGETS:
1063-
global pyboard
1064-
sys.path.append(base_path("../tools"))
1065-
import pyboard
1066-
1067-
pyb = pyboard.Pyboard(args.device, args.baudrate, args.user, args.password)
1068-
pyboard.Pyboard.run_script_on_remote_target = run_script_on_remote_target
1069-
pyb.enter_raw_repl()
1070-
else:
1071-
raise ValueError("target must be one of %s" % ", ".join(LOCAL_TARGETS + EXTERNAL_TARGETS))
1101+
# Get the test instance to run on.
1102+
pyb = get_test_instance(args.test_instance, args.baudrate, args.user, args.password)
10721103

1073-
# Automatically detect the native architecture for mpy-cross if not given.
1074-
if not args.mpy_cross_flags:
1075-
output = run_feature_check(pyb, args, "target_info.py")
1076-
arch = str(output, "ascii").strip()
1077-
if arch != "None":
1078-
args.mpy_cross_flags = "-march=" + arch
1104+
# Automatically detect the platform.
1105+
detect_test_platform(pyb, args)
10791106

10801107
if args.run_failures and (any(args.files) or args.test_dirs is not None):
10811108
raise ValueError(
@@ -1091,7 +1118,7 @@ def main():
10911118
tests = []
10921119
elif len(args.files) == 0:
10931120
test_extensions = ("*.py",)
1094-
if args.target == "webassembly":
1121+
if args.platform == "webassembly":
10951122
test_extensions += ("*.js", "*.mjs")
10961123

10971124
if args.test_dirs is None:
@@ -1101,23 +1128,23 @@ def main():
11011128
"misc",
11021129
"extmod",
11031130
)
1104-
if args.target == "pyboard":
1131+
if args.platform == "pyboard":
11051132
# run pyboard tests
11061133
test_dirs += ("float", "stress", "inlineasm", "ports/stm32")
1107-
elif args.target in ("renesas-ra"):
1134+
elif args.platform in ("renesas-ra"):
11081135
test_dirs += ("float", "inlineasm", "ports/renesas-ra")
1109-
elif args.target == "rp2":
1136+
elif args.platform == "rp2":
11101137
test_dirs += ("float", "stress", "thread", "ports/rp2")
11111138
if "arm" in args.mpy_cross_flags:
11121139
test_dirs += ("inlineasm",)
1113-
elif args.target == "esp32":
1140+
elif args.platform == "esp32":
11141141
test_dirs += ("float", "stress", "thread")
1115-
elif args.target in ("esp8266", "minimal", "nrf"):
1142+
elif args.platform in ("esp8266", "minimal", "nrf"):
11161143
test_dirs += ("float",)
1117-
elif args.target == "wipy":
1144+
elif args.platform == "wipy":
11181145
# run WiPy tests
11191146
test_dirs += ("ports/cc3200",)
1120-
elif args.target == "unix":
1147+
elif args.platform == "unix":
11211148
# run PC tests
11221149
test_dirs += (
11231150
"float",
@@ -1128,13 +1155,13 @@ def main():
11281155
"cmdline",
11291156
"ports/unix",
11301157
)
1131-
elif args.target == "qemu":
1158+
elif args.platform == "qemu":
11321159
test_dirs += (
11331160
"float",
11341161
"inlineasm",
11351162
"ports/qemu",
11361163
)
1137-
elif args.target == "webassembly":
1164+
elif args.platform == "webassembly":
11381165
test_dirs += ("float", "ports/webassembly")
11391166
else:
11401167
# run tests from these directories

‎tools/ci.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -767,5 +767,5 @@ function ci_zephyr_run_tests {
767767
docker exec zephyr-ci west build -p auto -b qemu_cortex_m3 -- -DCONF_FILE=prj_minimal.conf
768768
# Issues with zephyr tests:
769769
# - inf_nan_arith fails pow(-1, nan) test
770-
(cd tests && ./run-tests.py --target minimal --device execpty:"qemu-system-arm -cpu cortex-m3 -machine lm3s6965evb -nographic -monitor null -serial pty -kernel ../ports/zephyr/build/zephyr/zephyr.elf" -d basics float --exclude inf_nan_arith)
770+
(cd tests && ./run-tests.py -t execpty:"qemu-system-arm -cpu cortex-m3 -machine lm3s6965evb -nographic -monitor null -serial pty -kernel ../ports/zephyr/build/zephyr/zephyr.elf" -d basics float --exclude inf_nan_arith)
771771
}

0 commit comments

Comments
 (0)