Skip to content

gh-62965: Add debugging support to unittest (--debug --pm --pdb --trace) #99169

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 22 commits into
base: main
Choose a base branch
from

Conversation

kxrob
Copy link

@kxrob kxrob commented Nov 6, 2022

Note: The --debug mode (break run with original exception for seamless external post-mortem handling - e.g. for IDEs, interactive, super-runners ...) was originally inspired by #23900 . Some test code was cherry-picked. Yet the implementation here became completely different, as the primitive TestCase.debug() method cannot really provide compatible test control flow consistent with modern features.

The other debugging modes (with in-line callback way of execution) were inspired by the original discussion in #62965 and pytest.

@gpshead
Copy link
Member

gpshead commented Nov 6, 2022

at first glance this is likely good, i'll need to dive into the case or suite code to understand the postmortem hook stuff better. having some other eyeballs on this would be good as well.

@gpshead gpshead self-requested a review November 6, 2022 17:49
@kxrob
Copy link
Author

kxrob commented Nov 6, 2022

Test code I used to try things out:

import sys, traceback, gc
import unittest

class TC(unittest.TestCase):
    def clInstanceA(self):
        print("TC.clInstanceA", self)
    def clInstanceB(self):
        print("TC.clInstanceB", self)
        ##CLIB / 0
    def clInstanceC(self):
        print("TC.clInstanceC", self)
        ##CLIC / 0
    def setUp(self):
        print("TC.setUp", self)
        self.addCleanup(self.clInstanceA)
        self.addCleanup(self.clInstanceB)
        self.addCleanup(self.clInstanceC)
        ##0 / 0
    def tearDown(self):
        print("TC.tearDown", self)
        ##9 / 0
    @classmethod
    def clClass(self):
        print("TC.clClass", self)
        CLC / 0
    @classmethod
    def setUpClass(cls):
        print("TC.setUpClass", cls)
        cls.addClassCleanup(cls.clClass)
        ##breakpoint()
        ##SUC / 0
    @classmethod
    def tearDownClass(cls):
        print("TC.tearDownClass", cls)
        y = unittest.case._AutoDelRunner(lambda: print("-- delframe TDC"))
        TDC / 0
        print("TC.tearDownClassB", cls)

    def test_0(self):
        print("hello 0")
    def test_1(self):
        print("hello 1")
        self.assertTrue(0)
    def test_2(self):
        print("hello 2")
        2 / 0
    def test_3(self):
        print("hello 3")
        with self.subTest():
            3 / 0  # subTest
        print("hello 3B")

def tearDownModule():
        print("testdbg.tearDownModule")
        ##TDM / 0

x = unittest.case._AutoDelRunner(lambda: print("Del X"))

def clear_sysltb():
    print("clear_sysltb", sys.last_traceback, sys.last_value)
    sys.last_traceback = sys.last_value = sys.last_type = None
    ##gc.collect()
    print("clear_sysltb_EXIT", sys.last_traceback, sys.last_value)

if __name__ == "__main__":
    import atexit
    atexit.register(clear_sysltb)
    sys.stderr = sys.stdout  # to work synchronously w/o '-u' in piped run
    unittest.main()

kxrob added 4 commits November 8, 2022 20:34
Clear pm_cleanup when due to avoid running doCleanups() a 2nd time
when frame is recycled post-mortem during `--debug`
and in doClassCleanups(), doModuleCleanups().

Known issue: Upon errors within setUpClass/tearDownClass & ...Module
during '--debug' crash mode (but not in-line post-mortem handling)
the potentially remaining class/module tearDowns & cleanups won't
be run during final tb frame recycling (unlike with case bound
errors). Organizing this fully for rather unlikely use-cases would
require more complex code and perhaps impact readability
disproportionally. After all '--debug' is a rather invasive device.
Not clear if this is better.

Pro:
* uniform, anyway necessary in _addClassOrModuleLevelException()
* no signature change of TestCase.run()
* little less code
* custom resultclass could alternatively trigger debug mode

Con: ?
Closures in pm_* functions don't introduce ref cycles here,
no need for using default parameter hack.

Allow doCleanups() to continue (again) delayed upon pm crash.

Don't call _handle_debug_exception() for skips.

Break ref cycle in _handle_debug_exception() when "--debug"
(Have both `exc` value and `info` tuple parameters again)
'q' / do_quit terminates the test run immediately unlike
'c' / do_continue.

Useful when the issue is spotted and / or some bug triggers
many consecutive errors.
This also supports KeyboardInterrupt / DebuggerQuit etc. away
from '--debug' mode.
The exception also holds an extra ref besides the top error handler frame.

The extra call & ref chain also makes the execution order robust
against a wrong cleanup order of the tb stack frame locals - e.g. by
traceback.clear_frames().
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants