tests: add full, automated, top-level test runner for hardware #16112

dpgeorge · 2024-10-30T03:22:26Z

Summary

With the aim to speed up testing for a release, I put together a script to automatically run tests on devices running MicroPython. We have a lot of different tests these days, with different ways to run them. The idea with this top-level test runner is:

detect all serial devices connected
detect what platform is running on each device and what its capabilities are (eg WLAN, BLE, ability to import .mpy files)
run as many tests as possible given the connected devices (eg if there are two WLAN boards connected, then run the network multitests against them, among other things)

Basically all you need to do is connect boards to your PC via USB and run:

$ cd tests
$ ./hwtest.py

Then it does the rest. It takes between 10 and 15 minutes per board, so if you have many connected it can take a few hours. You can also restrict to given devices by specifying them on the command line, eg:

$ ./hwtest.py a0 a1 u0

(Probably the name of the script hwtest.py can change to something better.)

Testing

This was used to test 16 distinct boards for the recent v1.24.0 release.

Trade-offs and Alternatives

This is a bit of a hacky script at the moment but can be improved over time.

Currently there are no boards that pass all tests (!!) and so over time we need to fix things and improve the tests so more of them can pass.

dpgeorge · 2024-10-30T03:24:24Z

PR #16111 will help simplify this test runner a bit.

PR #15909 is included in this PR.

codecov · 2024-10-30T03:24:50Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.38%. Comparing base (b3cd1a3) to head (d668cbe).
⚠️ Report is 7 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master   #16112   +/-   ##
=======================================
  Coverage   98.38%   98.38%           
=======================================
  Files         171      171           
  Lines       22297    22297           
=======================================
  Hits        21938    21938           
  Misses        359      359

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

hmaerki · 2024-10-30T10:46:16Z

Octoprobe is ready so far to run these tests fully automated, including build of all firmware variants and to be triggered from github workflows.

We have to agree on triggers (github action on PR, github manual action). I will start the discussion when I am back from vacation.

github-actions · 2025-04-15T04:09:09Z

Code size report:

   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +0 +0.000% RPI_PICO_W
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:    +0 +0.000% VIRT_RV32