tests: add full, automated, top-level test runner for hardware #16112
+658
−9
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #16112 +/- ##
=======================================
Coverage 98.54% 98.54%
=======================================
Files 169 169
Lines 21890 21890
=======================================
Hits 21571 21571
Misses 319 319 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
Octoprobe is ready so far to run these tests fully automated, including build of all firmware variants and to be triggered from github workflows. We have to agree on triggers (github action on PR, github manual action). I will start the discussion when I am back from vacation. |
b8ce03c to
2047b05
Compare
2047b05 to
39adb22
Compare
39adb22 to
31a289e
Compare
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
TODO: make it a CLI option Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
Signed-off-by: Damien George <damien@micropython.org>
31a289e to
6ef6bd8
Compare
Signed-off-by: Damien George <damien@micropython.org>
|
Code size report: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
With the aim to speed up testing for a release, I put together a script to automatically run tests on devices running MicroPython. We have a lot of different tests these days, with different ways to run them. The idea with this top-level test runner is:
Basically all you need to do is connect boards to your PC via USB and run:
Then it does the rest. It takes between 10 and 15 minutes per board, so if you have many connected it can take a few hours. You can also restrict to given devices by specifying them on the command line, eg:
(Probably the name of the script
hwtest.pycan change to something better.)Testing
This was used to test 16 distinct boards for the recent v1.24.0 release.
Trade-offs and Alternatives
This is a bit of a hacky script at the moment but can be improved over time.
Currently there are no boards that pass all tests (!!) and so over time we need to fix things and improve the tests so more of them can pass.