Optionally run Python regression tests in parallel #127933
Labels
Comments
colesbury
added a commit
to colesbury/cpython
that referenced
this issue
Dec 16, 2024
This adds a new command line argument, `--parallel-threads` to the regression test runner to allow it to run individual tests in multiple threads in parallel in order to find multithreading bugs.
colesbury
added a commit
to colesbury/cpython
that referenced
this issue
Dec 16, 2024
This adds a new command line argument, `--parallel-threads` to the regression test runner to allow it to run individual tests in multiple threads in parallel in order to find multithreading bugs.
colesbury
added a commit
to colesbury/cpython
that referenced
this issue
Feb 4, 2025
colesbury
added a commit
that referenced
this issue
Feb 4, 2025
This adds a new command line argument, `--parallel-threads` to the regression test runner to allow it to run individual tests in multiple threads in parallel in order to find multithreading bugs. Some tests pass when run with `--parallel-threads`, but there's still more work before the entire suite passes.
srinivasreddy
pushed a commit
to srinivasreddy/cpython
that referenced
this issue
Feb 7, 2025
…ongh-128003) This adds a new command line argument, `--parallel-threads` to the regression test runner to allow it to run individual tests in multiple threads in parallel in order to find multithreading bugs. Some tests pass when run with `--parallel-threads`, but there's still more work before the entire suite passes.
cmaloney
pushed a commit
to cmaloney/cpython
that referenced
this issue
Feb 8, 2025
…ongh-128003) This adds a new command line argument, `--parallel-threads` to the regression test runner to allow it to run individual tests in multiple threads in parallel in order to find multithreading bugs. Some tests pass when run with `--parallel-threads`, but there's still more work before the entire suite passes.
|
We can close this ? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Feature or enhancement
This proposes adding an option to the regression test runner to run individual tests multiple times in parallel with the goal of uncovering multithreading bugs, especially in the free threading build.
Note that this is different from
-j, --multiprocess, which uses multiple processes to run different tests files in parallel. The motivation of-jis to speed up the time it takes to run tests and the use of processes improves isolation. Here the goal is to detect thread-safety bugs (at the cost of extra time).This is motivated by the experience of using https://github.com/Quansight-Labs/pytest-run-parallel, which is a pytest plugin written by @lysnikolaou and @andfoy. We've used that effectively to find thread-safety bugs in C API extensions while working on free threading compatibility.
The proposed changes are limited to the Python-internal
test.regrtestandtest.supportpackage and not the publicunittestpackage.New command line arguments
These are chosen to match https://github.com/Quansight-Labs/pytest-run-parallel. I think using the same names across the Python ecosystem makes things a bit easier to remember if you work on multiple projects.
--parallel-threads=N [default=0], runs each test in N threads--iterations=N [default=1], number of times to run each testOther support
Some of our test cases are not thread-safe (even with the GIL) because they modify global state or use non thread-safe unsafe modules (like
warnings):@support.thread_unsafe- marks a test case as not safe to run in multiple threads. The test will be run once in the main thread (like normal) even when--parallel-threads=Nis specified.Similar projects
There are a few existing projects that do essentially this for pytest and unittest, but they're not directly usable for Python's own test suite.
Linked PRs
The text was updated successfully, but these errors were encountered: