Skip to content

Added helper to test.support for parameterizing tests #135120

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
serhiy-storchaka opened this issue Jun 4, 2025 · 1 comment
Open

Added helper to test.support for parameterizing tests #135120

serhiy-storchaka opened this issue Jun 4, 2025 · 1 comment
Labels
3.13 bugs and security fixes 3.14 bugs and security fixes 3.15 new features, bugs and security fixes tests Tests in the Lib/test dir

Comments

@serhiy-storchaka
Copy link
Member

serhiy-storchaka commented Jun 4, 2025

This is a simple decorator modeled after pytest.mark.parametrize. It allows to "parametrize" tests, each parametrized test is run as a subtest.

The decorator specifies the name(s) of parameters (as an iterable or a comma separated string) and their values (an iterable of values for parameters):

@subTests('a,b', [('foo', 2), ('bar', 5)])
def test_foo(self, a, b):
    ...

is equivalent to

def test_foo(self):
    for a, b in [('foo', 2), ('bar', 5)]:
        with self.subTest(a=a, b=b):
            ...

It allows to save 2 levels of indentation.

Multiple decorators can be used, this generates a Descartes production of parameter values:

@subTests('a,b', [('foo', 2), ('bar', 5)])
@subTests('c', range(5))
def test_foo(self, a, b, c):
    ...

is equivalent to

def test_foo(self):
    for a, b in [('foo', 2), ('bar', 5)]:
        with self.subTest(a=a, b=b):
            for c in range(5):
                with self.subTest(c=c):
                    ...

It allows to save even more 2 levels of indentation.

It has also optional keyword-only parameter _do_cleanups. _do_cleanups=True tells to call doCleanups() after each subtest. This is temporary feature, only for use in test_ntpath. In general, it is wrong to do this because doCleanups() will also calls callbacks added in setUp. It will be replaced either by addSubTestCleanup() (see https://discuss.python.org/t/unittest-add-addcleanup-to-subtest/91827 and #134079) or by other methods (for example by adding fences for cleanup). Also, it will not be needed for real test parameterization.

There are differences from "real" parameterization:

  • setUp and tearDown are only run once, before and after all subtests.
  • It is impossible to select or filter out tests with specific parameters.
  • It can only parametrize test methods, not classes.

I plan to implement real parameterization, this is why I only propose to add that decorator in test.support, not in unittest. In any case it would be backported in test.support. We may finally add both decorators in unittest if this does not create confusion.

The name subTests is intentionally chosen to not confuse with parametrize which is reserved for real test parameterization.

The PR contains also several examples of using that decorator.

Linked PRs

@serhiy-storchaka serhiy-storchaka added the tests Tests in the Lib/test dir label Jun 4, 2025
@serhiy-storchaka serhiy-storchaka added 3.13 bugs and security fixes 3.14 bugs and security fixes 3.15 new features, bugs and security fixes labels Jun 4, 2025
serhiy-storchaka added a commit to serhiy-storchaka/cpython that referenced this issue Jun 4, 2025
serhiy-storchaka added a commit to serhiy-storchaka/cpython that referenced this issue Jun 4, 2025
serhiy-storchaka added a commit to serhiy-storchaka/cpython that referenced this issue Jun 4, 2025
@ZeroIntensity
Copy link
Member

@picnixz I think you suggested this exact thing a little while ago!

serhiy-storchaka added a commit to serhiy-storchaka/cpython that referenced this issue Jun 6, 2025
(cherry picked from commit 6ef06fa)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
serhiy-storchaka added a commit to serhiy-storchaka/cpython that referenced this issue Jun 6, 2025
(cherry picked from commit 6ef06fa)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
serhiy-storchaka added a commit that referenced this issue Jun 6, 2025
serhiy-storchaka added a commit that referenced this issue Jun 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3.13 bugs and security fixes 3.14 bugs and security fixes 3.15 new features, bugs and security fixes tests Tests in the Lib/test dir
Projects
None yet
Development

No branches or pull requests

2 participants