We make heavy usage of pytest. Here is a quick guide and collection of useful options:

- To run all tests in the current directory and subdirectories: pytest

- To run tests in a specific directory or file: pytest path/to/dir/test_foo.py

- -s: disables output capturing

- --pdb: in case of exception, enter a (Pdb) prompt so that you can inspect what went wrong.

- -v: verbose mode

- -x: stop the execution as soon as one test fails

- -k foo: run only the tests whose full name contains foo

- -k 'foo and bar'

- -k 'foo and not bar'

make test is useful to run all the tests, but during the development is useful to have more control on how tests are run. The following guide assumes that you are in the directory pyscriptjs/tests/integration/.

$ pytest -xv

...

test_00_support.py::TestSupport::test_basic[chromium] PASSED [ 0%]

test_00_support.py::TestSupport::test_console[chromium] PASSED [ 1%]

test_00_support.py::TestSupport::test_check_js_errors_simple[chromium] PASSED [ 2%]

test_00_support.py::TestSupport::test_check_js_errors_expected[chromium] PASSED [ 3%]

test_00_support.py::TestSupport::test_check_js_errors_expected_but_didnt_raise[chromium] PASSED [ 4%]

test_00_support.py::TestSupport::test_check_js_errors_multiple[chromium] PASSED [ 5%]

...

-x means "stop at the first failure". -v means "verbose", so that you can see all the test names one by one. We try to keep tests in a reasonable order, from most basic to most complex. This way, if you introduced some bug in very basic things, you will notice immediately.

If you have the pytest-xdist plugin installed, you can run all the integration tests on 4 cores in parallel:

$ pytest -n 4

$ pytest test_01_basic.py -k test_pyscript_hello -s

...

[ 0.00 page.goto ] pyscript_hello.html

[ 0.01 request ] 200 - fake_server - http://fake_server/pyscript_hello.html

...

[ 0.17 console.info ] [py-loader] Downloading pyodide-x.y.z...

[ 0.18 request ] 200 - CACHED - https://cdn.jsdelivr.net/pyodide/vx.y.z/full/pyodide.js

...

[ 3.59 console.info ] [pyscript/main] PyScript page fully initialized

[ 3.60 console.log ] hello pyscript

-k selects tests by pattern matching as described above. -s instructs pytest to show the output to the terminal instead of capturing it. In the output you can see various useful things, including network requests and JS console messages.

$ pytest test_01_basic.py -k test_pyscript_hello -s --headed

...

Same as above, but with --headed the browser is shown in a window, and you can interact with it. The browser uses a fake server, which means that HTTP requests are cached.

Unfortunately, in this mode source maps does not seem to work, and you cannot debug the original typescript source code. This seems to be a bug in playwright, for which we have a workaround:

$ pytest test_01_basic.py -k test_pyscript_hello -s --headed --no-fake-server

...

As the name implies, -no-fake-server disables the fake server: HTTP requests are not cached, but source-level debugging works.

Finally:

$ pytest test_01_basic.py -k test_pyscript_hello -s --dev

...

--dev implies --headed --no-fake-server. In addition, it also automatically open chrome dev tools.

By default, we run each test twice: one with execution_thread = "main" and one with execution_thread = "worker". If you want to run only half of them, you can use -m:

$ pytest -m main # run only the tests in the main thread

$ pytest -m worker # ron only the tests in the web worker

By default, our test machinery uses a playwright router which intercepts and cache HTTP requests, so that for example you don't have to download pyodide again and again. This also enables the possibility of running tests in parallel on multiple cores.

The cache is stored using the pytest-cache plugin, which means that it survives across sessions.

If you want to temporarily disable the cache, the easiest thing is to use --no-fake-server, which bypasses it completely.

If you want to clear the cache, you can use the special option --clear-http-cache:

NOTE: this works only if you are inside tests/integration, or if you explicitly specify tests/integration from the command line. This is due to how pytest decides to search for and load the various conftest.py.