tests: Require SSL certificate file to be available for test to run. #17897
Conversation
|
@projectgus I think you had an opinion on this the last time it came up? |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #17897 +/- ##
=======================================
Coverage 98.38% 98.38%
=======================================
Files 171 171
Lines 22295 22295
=======================================
Hits 21936 21936
Misses 359 359 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
I think this a good improvement, as it doesn't mask the failures.
IMO the best thing we could do would be to have the test runner check this (maybe by running on the host some well-formed multi_test_setup.py found in the directory before running any tests from that directory, and therefore avoid the whole problem of false-positive failures due to preconditions.
But this is still a good PR, I think. :)
I guess we could add a To actually copy across certificates, there is #16112 which does that.
We can still make all those other improvements (eventually) on top of this PR. |
Previously, any test needing an SSL certificate file would automatically skip if the file could not be found. But that makes it too easy to accidentally skip tests. Instead, change it so that the test fails if the certificate file doesn't exist. That matches, for example, the fact that the test fails if networking (LAN, WiFi) is not active. Signed-off-by: Damien George <damien@micropython.org>
5b75bf7 to
2bba507
Compare
Summary
Previously, any test needing an SSL certificate file would automatically skip if the file could not be found. But that makes it too easy to accidentally skip tests.
Instead, change it so that the test fails if the certificate file doesn't exist. That matches, for example, the fact that the test fails if networking (LAN, WiFi) is not active.
Testing
Tested on the unix port (files always available, passes), and PYBD_SF6 with and without the files on the local filesystem. It fails without them, passes with them.
Trade-offs and Alternatives
IMO this is a good improvement, it simplifies the test and makes it impossible to accidentally skip it. But it does require users to copy the files across first (as mentioned in the test README).