Skip to content

Better integration of doctest with unittest and libregrtest #108885

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
serhiy-storchaka opened this issue Sep 4, 2023 · 1 comment
Open

Better integration of doctest with unittest and libregrtest #108885

serhiy-storchaka opened this issue Sep 4, 2023 · 1 comment
Assignees
Labels
stdlib Python modules in the Lib dir tests Tests in the Lib/test dir type-feature A feature request or enhancement

Comments

@serhiy-storchaka
Copy link
Member

serhiy-storchaka commented Sep 4, 2023

Feature or enhancement

A side effect of migration to the unittest test runner (a long process finished in #89392) was the change in the output in verbose mode. In older version, when test.support.run_doctest() was used to run doctests, it outputted all doctest details when regrtests are run with -v. Now doctests are wrapped in doctest.DocTestClass and only test name is shown when regrtests are run with -v.

The former behavior was too verbose, the current behavior is too terse. I want the following behavior:

  • When regrtests are run without -v, do not output anything.
  • When regrtests are run with single -v, output only doctest name, as for other test cases.
  • When regrtests are run with -vv, output detailed doctest output.
  • Every example count as a subtest.
  • When some example fails, count it as failing subtest, and output detailed reports for every failed example. It should not include details of successful examples (except when run with -vv).
  • It would be nice to integrate it also with the unittest CLI, but taking into account that it needs two -vs to show test case names and will need three -vs to show doctest details.

It means that we should found a way to pass verbosity level to doctests and run every example within TestCase.subTest().

Linked PRs

@serhiy-storchaka serhiy-storchaka added the type-feature A feature request or enhancement label Sep 4, 2023
@serhiy-storchaka serhiy-storchaka added tests Tests in the Lib/test dir stdlib Python modules in the Lib dir labels Sep 4, 2023
@serhiy-storchaka serhiy-storchaka self-assigned this Sep 4, 2023
@serhiy-storchaka serhiy-storchaka moved this from Todo to In Progress in Unittest issues Sep 9, 2023
serhiy-storchaka added a commit to serhiy-storchaka/cpython that referenced this issue May 28, 2025
serhiy-storchaka added a commit to serhiy-storchaka/cpython that referenced this issue May 28, 2025
serhiy-storchaka added a commit that referenced this issue May 28, 2025
Test the error and failure report in more detail.
miss-islington pushed a commit to miss-islington/cpython that referenced this issue May 28, 2025
Test the error and failure report in more detail.
(cherry picked from commit f6324bc)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
miss-islington pushed a commit to miss-islington/cpython that referenced this issue May 28, 2025
Test the error and failure report in more detail.
(cherry picked from commit f6324bc)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
serhiy-storchaka added a commit that referenced this issue May 28, 2025
Test the error and failure report in more detail.
(cherry picked from commit f6324bc)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
serhiy-storchaka added a commit that referenced this issue May 28, 2025
Test the error and failure report in more detail.
(cherry picked from commit f6324bc)

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
serhiy-storchaka added a commit to serhiy-storchaka/cpython that referenced this issue May 29, 2025
Run each example as a subtest in unit tests synthesized by
doctest.DocFileSuite() and doctest.DocTestSuite().
serhiy-storchaka added a commit to serhiy-storchaka/cpython that referenced this issue May 29, 2025
Run each example as a subtest in unit tests synthesized by
doctest.DocFileSuite() and doctest.DocTestSuite().
serhiy-storchaka added a commit that referenced this issue May 31, 2025
)

Run each example as a subtest in unit tests synthesized by
doctest.DocFileSuite() and doctest.DocTestSuite().

Add the doctest.DocTestRunner.report_skip() method.
@serhiy-storchaka
Copy link
Member Author

#134890 makes each example to be reported as a subtest, but the main goal -- control doctestі verbosity through unittest and libregrtest -- has not yet been achieved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
stdlib Python modules in the Lib dir tests Tests in the Lib/test dir type-feature A feature request or enhancement
Projects
Status: In Progress
Development

No branches or pull requests

2 participants
@serhiy-storchaka and others