Skip to content

Commit c8af42b

Browse files
pi-anlpi-anl
committed
docs: Improve documentation for writing and running tests
Reviewed the run-tests.py and run-multitests.py scripts and used the insights to significantly enhance the documentation in docs/develop/writingtests.rst. The updated documentation provides: - More detailed explanation of the test directory structure and key runner scripts. - Clearer instructions on writing simple tests, including the use of print() and .exp files. - Expanded coverage of run-tests.py options for targeting specific devices, filtering tests, and handling failures. - An overview of how run-tests.py works internally (feature detection, skipping, execution, comparison). - Guidance on writing advanced tests (skipping, unittest, special handling, float/endian considerations). - Introduction to multi-instance testing with run-multitests.py, including how to write tests with instanceN() functions and use the `multitest` helper for coordination. - Examples of running multi-instance tests with different instance types. 🤖 Generated with gemini-2.5-pro-exp-03-25 Co-Authored-By: gemini-2.5-pro-exp-03-25 <gemini-2.5-pro-exp-03-25@alelec.net> Signed-off-by: Andrew Leech <andrew.leech@planetinnovation.com.au>
1 parent ef8282c commit c8af42b

File tree

2 files changed

+367
-235
lines changed

2 files changed

+367
-235
lines changed

‎docs/develop/writingtests.rst

Lines changed: 201 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -1,70 +1,230 @@
11
.. _writingtests:
22

3-
Writing tests
4-
=============
3+
Test Suite
4+
==========
55

6-
Tests in MicroPython are located at the path ``tests/``. The following is a listing of
7-
key directories and the run-tests.py runner script:
6+
The MicroPython test suite resides in the ``tests/`` directory and consists of multiple components:
87

9-
.. code-block:: bash
8+
1. **Standard Tests**: The majority of tests that run on a single instance and verify functionality against CPython or expected outputs.
9+
2. **Multi-instance Tests**: Tests that require multiple MicroPython instances to interact with each other (e.g., networking, Bluetooth).
10+
3. **Native Module Tests**: Tests for dynamic native modules compiled to ``.mpy`` files.
11+
4. **Performance Benchmarks**: Python-level benchmarks for measuring execution speed.
12+
5. **Internal Benchmarks**: Low-level benchmarks for core VM operations and C code.
13+
14+
This document primarily focuses on the standard tests and multi-instance tests.
15+
16+
Directory Structure
17+
-------------------
18+
19+
The ``tests/`` directory is organized into subfolders by functionality:
20+
21+
* ``basics/``: Core Python language features
22+
* ``extmod/``: Extended modules (like ``ujson``, ``ure``)
23+
* ``float/``: Floating-point arithmetic specifics
24+
* ``micropython/``: MicroPython-specific features and behaviors
25+
* ``import/``: Import mechanisms
26+
* ``io/``: Input/Output operations
27+
* ``stress/``: Tests designed to push resource limits (memory, recursion)
28+
* ``thread/``: Threading module tests
29+
* ``cmdline/``: Tests for the command-line interface and REPL
30+
* ``ports/<port_name>/``: Tests specific to a particular port
31+
* ``feature_check/``: Scripts used to detect target capabilities
32+
* ``multi_bluetooth/``, ``multi_network/``: Tests involving multiple instances
33+
* ``perf_bench/``: Performance benchmarking scripts
34+
* ``internal_bench/``: Low-level internal benchmarks
35+
36+
Key Test Runner Scripts:
1037

11-
.
12-
├── basics
13-
├── extmod
14-
├── float
15-
├── micropython
16-
├── run-tests.py
17-
...
38+
* ``run-tests.py``: Main script for standard tests
39+
* ``run-multitests.py``: For tests requiring multiple interacting instances
40+
* ``run-natmodtests.py``: For testing dynamic native modules
41+
* ``run-perfbench.py``: For performance benchmarks
42+
* ``run-internalbench.py``: For low-level internal benchmarks
1843

19-
There are subfolders maintained to categorize the tests. Add a test by creating a new file in one of the
20-
existing folders or in a new folder. It's also possible to make custom tests outside this tests folder,
21-
which would be recommended for a custom port.
44+
Writing Standard Tests
45+
----------------------
2246

23-
For example, add the following code in a file ``print.py`` in the ``tests/unix/`` subdirectory:
47+
Standard tests are the most common type. They verify functionality by comparing output against expected results.
48+
49+
1. **Choose a Directory:** Select the appropriate subfolder for your test based on the functionality it tests.
50+
2. **Create a ``.py`` file:** Add your test code to a new Python file.
51+
3. **Use ``print()``:** The test harness works by comparing standard output. Use ``print()`` statements to output results.
52+
4. **Expected Output:**
53+
* **CPython Comparison:** By default, tests are run on both CPython and MicroPython with outputs compared.
54+
* **``.exp`` Files:** For MicroPython-specific features, create a ``<testname>.py.exp`` file with the expected output.
55+
56+
Example:
57+
58+
Create ``tests/basics/simple_print.py``:
2459

2560
.. code-block:: python
2661
27-
def print_one():
28-
print(1)
62+
# tests/basics/simple_print.py
63+
print("Hello")
64+
a = 1 + 2
65+
print(a)
66+
67+
When run, MicroPython's output will be compared to CPython's output (or ``simple_print.py.exp`` if it exists).
68+
69+
Types of Tests
70+
--------------
71+
72+
The test system supports three main testing approaches:
2973

30-
print_one()
74+
1. **CPython comparison tests**:
75+
* Used for behavior that should match CPython
76+
* Tests are run on both CPython and MicroPython
77+
* Passes if outputs match exactly
3178

32-
If you run your tests, this test should appear in the test output:
79+
2. **Expected output (``.exp``) tests**:
80+
* Used for MicroPython-specific features or when CPython behavior differs
81+
* MicroPython output is compared against contents of a ``.py.exp`` file
82+
* Passes if outputs match exactly
83+
84+
3. **``unittest``-based tests**:
85+
* Requires ``unittest`` module to be available on the target
86+
* Used for hardware testing or other MicroPython-specific behavior
87+
* Passes if unittest summary shows "OK"
88+
89+
Running Tests (``run-tests.py``)
90+
--------------------------------
91+
92+
The primary script is ``run-tests.py``, executed from the ``tests/`` directory.
93+
94+
**Basic Usage:**
3395

3496
.. code-block:: bash
3597
36-
$ cd ports/unix
37-
$ make tests
38-
skip unix/extra_coverage.py
39-
pass unix/ffi_callback.py
40-
pass unix/ffi_float.py
41-
pass unix/ffi_float2.py
42-
pass unix/print.py
43-
pass unix/time.py
44-
pass unix/time2.py
98+
# Run default tests for the Unix port
99+
cd tests/
100+
./run-tests.py
101+
102+
**Running on Hardware Targets:**
103+
104+
.. code-block:: bash
45105
46-
Tests are run by comparing the output from the test target against the output from CPython.
47-
So any test should use print statements to indicate test results.
106+
# Run on a specific serial port
107+
./run-tests.py -t /dev/ttyACM0
108+
./run-tests.py -t COM3
48109
49-
For tests that can't be compared to CPython (i.e. micropython-specific functionality),
50-
you can provide a ``.py.exp`` file which will be used as the truth for comparison.
110+
# Use shortcuts for common serial ports
111+
./run-tests.py -t a0 # Maps to /dev/ttyACM0
112+
./run-tests.py -t u1 # Maps to /dev/ttyUSB1
113+
./run-tests.py -t c3 # Maps to COM3
51114
52-
The other way to run tests, which is useful when running on targets other than the Unix port, is:
115+
**Filtering Tests:**
53116

54117
.. code-block:: bash
55118
56-
$ cd tests
57-
$ ./run-tests.py
119+
# Run all tests in specific directories
120+
./run-tests.py -d basics
121+
./run-tests.py -d extmod -d float
122+
123+
# Run specific files
124+
./run-tests.py basics/int*.py
125+
./run-tests.py float/float_parse.py
126+
127+
# Filter by regex
128+
./run-tests.py -e viper # Exclude tests matching 'viper'
129+
./run-tests.py -i asyncio # Include only tests matching 'asyncio'
130+
131+
**Other Useful Options:**
132+
133+
* ``--emit {bytecode | native | viper}``: Run tests using a specific code emitter
134+
* ``--via-mpy``: Compile tests to ``.mpy`` files first before running
135+
* ``-j N`` or ``--jobs N``: Run N tests in parallel (for PC targets)
136+
* ``--print-failures``: Show the diff for tests that failed in the last run
137+
* ``--run-failures``: Re-run only the tests that failed in the last run
138+
* ``--clean-failures``: Remove the ``.exp`` and ``.out`` files from failures
139+
140+
How The Test System Works
141+
-------------------------
58142

59-
Then to run on a board:
143+
1. **Target Detection:** Determines the platform and architecture.
144+
2. **Feature Detection:** Identifies capabilities of the target MicroPython build.
145+
3. **Test Selection:** Gathers test files based on command-line arguments or defaults.
146+
4. **Skipping Tests:** Excludes tests based on:
147+
* Command-line filters
148+
* Missing features
149+
* Platform-specific skip lists
150+
* Emitter type restrictions
151+
* Tests explicitly printing ``SKIP``
152+
5. **Execution:**
153+
* PC targets: Runs MicroPython as a subprocess
154+
* Board targets: Uses ``pyboard.py`` to connect via serial/network
155+
6. **Output Comparison:** Compares target's output with expected output
156+
7. **Reporting:** Shows results and summarizes pass/fail statistics
157+
158+
Writing Advanced Tests
159+
----------------------
160+
161+
**Skipping Tests Conditionally**
162+
163+
If a test needs to skip itself conditionally:
164+
165+
.. code-block:: python
166+
167+
import sys
168+
if not hasattr(sys, 'feature_needed'):
169+
print('SKIP')
170+
sys.exit()
171+
172+
**Special Test Handling**
173+
174+
Some tests need special options:
175+
176+
* Add comments in the test file: ``# cmdline: -X heapsize=...``
177+
* Tests needing redirection are listed in ``special_tests`` in ``run-tests.py``
178+
179+
**Platform Considerations**
180+
181+
* **Floating Point:** Be mindful of precision differences. Use ``math.isclose()`` with appropriate tolerances.
182+
* **Endianness:** Check ``sys.byteorder`` for byte order sensitive operations.
183+
* **Memory/Stack:** Avoid large allocations or deep recursion on constrained devices.
184+
185+
Multi-instance Tests
186+
--------------------
187+
188+
For testing interactions like networking or Bluetooth, ``run-multitests.py`` orchestrates multiple instances.
189+
190+
**Writing Multi-instance Tests:**
191+
192+
1. Define functions named ``instance0()``, ``instance1()``, etc. for each instance's code.
193+
2. Use the ``multitest`` helper for coordination:
194+
* ``multitest.next()``: Complete a stage and wait for other instances
195+
* ``multitest.broadcast(msg)``: Send a message to all other instances
196+
* ``multitest.wait(msg)``: Wait for a specific broadcast message
197+
* ``multitest.skip()``: Indicate the test should be skipped
198+
* ``multitest.globals(var=value, ...)``: Set global variables
199+
* ``multitest.get_network_ip()``: Get the IP address
200+
* ``multitest.expect_reboot(resume_func_name, delay_ms)``: Handle device reboots
201+
202+
**Running Multi-instance Tests:**
60203

61204
.. code-block:: bash
62205
63-
$ ./run-tests.py -t /dev/ttyACM0
206+
# Run a network test using two unix instances
207+
./run-multitests.py -i micropython -i micropython multi_network/tcp_connect.py
208+
209+
# Run a bluetooth test using two pyboards
210+
./run-multitests.py -i pyb:/dev/ttyACM0 -i pyb:/dev/ttyACM1 multi_bluetooth/ble_connect.py
211+
212+
Provide as many ``-i`` arguments as instances required by the tests.
64213

65-
And to run only a certain set of tests (eg a directory):
214+
Test Certificates for SSL/TLS Tests
215+
-----------------------------------
216+
217+
Network tests using SSL/TLS require test certificates to be available on the device. These certificates are included in the repository in ``multi_network/`` and ``net_inet/`` directories.
218+
219+
**Preparing a Device for SSL/TLS Tests:**
220+
221+
Use ``mpremote`` to set the device's RTC and copy the certificate files:
66222

67223
.. code-block:: bash
68224
69-
$ ./run-tests.py -d basics
70-
$ ./run-tests.py float/builtin*.py
225+
# From the micropython/tests/ directory:
226+
mpremote rtc --set cp multi_net/*.der net_inet/*.der :/
227+
228+
This sets the device's Real Time Clock (using the host's time) and copies all ``.der`` files to the root directory of the device's filesystem.
229+
230+
These certificates are self-signed and generated for testing purposes only.

0 commit comments

Comments
 (0)