Skip to content

tests: add full, automated, top-level test runner for hardware #16112

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 13 commits into
base: master
Choose a base branch
from
Draft

tests: add full, automated, top-level test runner for hardware #16112

wants to merge 13 commits into from

Conversation

dpgeorge
Copy link
Member

Summary

With the aim to speed up testing for a release, I put together a script to automatically run tests on devices running MicroPython. We have a lot of different tests these days, with different ways to run them. The idea with this top-level test runner is:

  • detect all serial devices connected
  • detect what platform is running on each device and what its capabilities are (eg WLAN, BLE, ability to import .mpy files)
  • run as many tests as possible given the connected devices (eg if there are two WLAN boards connected, then run the network multitests against them, among other things)

Basically all you need to do is connect boards to your PC via USB and run:

$ cd tests
$ ./hwtest.py

Then it does the rest. It takes between 10 and 15 minutes per board, so if you have many connected it can take a few hours. You can also restrict to given devices by specifying them on the command line, eg:

$ ./hwtest.py a0 a1 u0

(Probably the name of the script hwtest.py can change to something better.)

Testing

This was used to test 16 distinct boards for the recent v1.24.0 release.

Trade-offs and Alternatives

This is a bit of a hacky script at the moment but can be improved over time.

Currently there are no boards that pass all tests (!!) and so over time we need to fix things and improve the tests so more of them can pass.

@dpgeorge dpgeorge added the tests Relates to tests/ directory in source label Oct 30, 2024
@dpgeorge dpgeorge marked this pull request as draft October 30, 2024 03:22
@dpgeorge dpgeorge changed the title tests: add full, automated test running for hardware tests: add full, automated, top-level test runner for hardware Oct 30, 2024
@dpgeorge
Copy link
Member Author

PR #16111 will help simplify this test runner a bit.

PR #15909 is included in this PR.

@codecov Codecov
Copy link

codecov bot commented Oct 30, 2024 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.54%. Comparing base (9f30627) to head (1aed9f9).

Additional details and impacted files 
@@           Coverage Diff           @@
##           master   #16112   +/-   ##
=======================================
  Coverage   98.54%   98.54%           
=======================================
  Files         169      169           
  Lines       21890    21890           
=======================================
  Hits        21571    21571           
  Misses        319      319

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@dpgeorge dpgeorge mentioned this pull request Oct 30, 2024
Merged
@hmaerki
Copy link
Contributor

hmaerki commented Oct 30, 2024

Octoprobe is ready so far to run these tests fully automated, including build of all firmware variants and to be triggered from github workflows.

We have to agree on triggers (github action on PR, github manual action). I will start the discussion when I am back from vacation.

@dpgeorge dpgeorge force-pushed the tests-full-test-runner branch 2 times, most recently from b8ce03c to 2047b05 Compare November 4, 2024 05:56
@dpgeorge dpgeorge force-pushed the tests-full-test-runner branch from 2047b05 to 39adb22 Compare November 24, 2024 02:26
@dpgeorge dpgeorge force-pushed the tests-full-test-runner branch from 39adb22 to 31a289e Compare January 15, 2025 00:58
dpgeorge added 12 commits April 15, 2025 13:56
@dpgeorge
tools/mpremote: Check hash to skip files for mip install.
5f95a56 
Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge
tests/micropython/extreme_exc.py: Skip on ESP32 with PSRAM.
c78e3e4 
Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge
tests/run-tests.py: Don't print info for tests that pass.
f215ec8 
Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge
tests/run-multitests.py: Suppress diff.
8384792 
TODO: make it a CLI option

Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge
tests/run-tests.py: Capture KeyboardInterrupt for nicer exit.
ea024e0 
Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge
tests/vcprate.py: Add a test for serial data rate and integrity.
2dea3d0 
Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge
tests/hwtest.py: Add a full, automatic test runner.
682be29 
Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge
tests/hwtest.py: Make sure unittest is installed.
dc9a106 
Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge
tests/hwtest.py: Add command to build all natmods.
2a26f46 
Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge
tests/ports/rp2: Wait for IRQ to pass test with native emitter.
e7d85b0 
Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge
tests/run-tests.py: Don't run thread tests on esp32 in native mode.
9251d7b 
Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge
tests/hwtest.py: Improve CLI options.
6ef6bd8 
Signed-off-by: Damien George <damien@micropython.org>
@dpgeorge dpgeorge force-pushed the tests-full-test-runner branch from 31a289e to 6ef6bd8 Compare April 15, 2025 03:56
@dpgeorge
tests: Increase buffer to 249 bytes.
1aed9f9 
Signed-off-by: Damien George <damien@micropython.org>
@github-actions GitHub Actions
Copy link

Code size report:

   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +0 +0.000% RPI_PICO_W
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:    +0 +0.000% VIRT_RV32

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
tests Relates to tests/ directory in source
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@dpgeorge @hmaerki