gh-93696: Locate frozen module source with file #93697

SnoopJ · 2022-06-10T20:31:13Z

See #93696, this patch allows pdb to locate the source for frozen stdlib modules. With the patch applied, the output of pdb is as expected:

$ git describe && ./python -V
v3.11.0b1-358-gc91afc1c0a
Python 3.12.0a0
$ ./python -m pdb repro.py 
> /home/snoopjedi/repos/cpython/repro.py(2)<module>()
-> import importlib._bootstrap
(Pdb) n
> /home/snoopjedi/repos/cpython/repro.py(5)<module>()
-> importlib._bootstrap._resolve_name("os", ".", 1)
(Pdb) s
--Call--
> <frozen importlib._bootstrap>(1037)_resolve_name()
(Pdb) l
1032        def __exit__(self, exc_type, exc_value, exc_traceback):
1033            """Release the import lock regardless of any raised exceptions."""
1034            _imp.release_lock()
1035 
1036 
1037 -> def _resolve_name(name, package, level):
1038        """Resolve a relative module name to an absolute one."""
1039        bits = package.rsplit('.', level - 1)
1040        if len(bits) < level:
1041            raise ImportError('attempted relative import beyond top-level package')
1042        base = bits[0]

I am not particularly familiar with the frozen module system so I'm not sure if there are edge cases where this will produce undesirable behavior, but thought that I would send it upstream just in case 😅

Issue: pdb cannot find source code for frozen stdlib modules #93696

SnoopJ · 2022-06-10T22:37:15Z

Edit: I have now added a test to this PR, see follow-up comment

I did not include an additional test with this PR because I don't think there's a good way to generate a dummy frozen module for use in the test and it seems inappropriate to test against one of the stdlib modules. All the same, here's what the test would look like for the current version of importlib._bootstrap, in case there's some clever testing trick I missed out here.

click to see patch

diff --git a/Lib/test/test_pdb.py b/Lib/test/test_pdb.py
index 0141b739c2..894f3e9099 100644
--- a/Lib/test/test_pdb.py
+++ b/Lib/test/test_pdb.py
@@ -532,6 +532,40 @@ def test_list_commands():
     (Pdb) continue
     """
 
+def test_frozen_list():
+    '''Test the list command on frozen stdlib modules
+
+    >>> def test_function():
+    ...     import importlib._bootstrap
+    ...     import pdb; pdb.Pdb(nosigint=True, readrc=False).set_trace()
+    ...     importlib._bootstrap._resolve_name("os", ".", 1)
+
+    >>> with PdbTestInput([  # doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE
+    ...     'step',
+    ...     'list',
+    ...     'continue',
+    ... ]):
+    ...     test_function()
+    > <doctest test.test_pdb.test_frozen_list[0]>(4)test_function()
+    -> importlib._bootstrap._resolve_name("os", ".", 1)
+    (Pdb) step
+    --Call--
+    > <frozen importlib._bootstrap>(1037)_resolve_name()
+    (Pdb) list
+    1032            def __exit__(self, exc_type, exc_value, exc_traceback):
+    1033                """Release the import lock regardless of any raised exceptions."""
+    1034                _imp.release_lock()
+    1035
+    1036
+    1037 ->     def _resolve_name(name, package, level):
+    1038            """Resolve a relative module name to an absolute one."""
+    1039            bits = package.rsplit('.', level - 1)
+    1040            if len(bits) < level:
+    1041                raise ImportError('attempted relative import beyond top-level package')
+    1042            base = bits[0]
+    (Pdb) continue
+    '''
+
 def test_pdb_whatis_command():
     """Test the whatis command

SnoopJ · 2022-06-12T22:13:16Z

Updated with a test that creates a module that suitably fakes the behavior of functions inside of frozen modules, in the sense that co_filename on the contained function starts with '<frozen', and the __file__ attribute of the module will point to the "real" source file that pdb should use for resolving list

Lib/test/test_pdb.py

Misc/NEWS.d/next/Core and Builtins/2022-06-10-16-37-44.gh-issue-93696.65BI2R.rst

Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>

ghost · 2022-07-17T18:37:28Z

All commit authors signed the Contributor License Agreement.

Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>

kumaraditya303

LGTM

SnoopJ · 2022-09-29T18:45:33Z

@ericsnowcurrently does this patch need any changes?

SnoopJ · 2022-10-24T20:06:57Z

@JelleZijlstra generously offered to have a look-see at this during the 3.11 release stream, dropping this here to ping him :)

JelleZijlstra · 2022-10-25T01:54:50Z

Lib/test/test_pdb.py

+            list
+            quit
+        """
+        with open('gh93696.py', 'w') as f:


A little weird that we're writing into the CWD, but that appears to be what the other tests in this file are doing too.

JelleZijlstra · 2022-10-25T02:00:11Z

Lib/pdb.py

+        # this workaround can be removed with the closure of gh-89815
+        if filename.startswith("<frozen"):
+            if "__file__" in self.curframe.f_globals:
+                filename = self.curframe.f_globals["__file__"]


What if __file__ is not a string? It looks like get_file_breaks will crash in that case. We should probably just ignore __file__ in that case.

Hmm, that's a good point. I would think it's a small edge, but it's easy to be more cautious about it.

Yes, I think it's important for a debugger to be resilient in the face of unusual program state.

miss-islington · 2022-10-25T13:22:56Z

Thanks @SnoopJ for the PR, and @JelleZijlstra for merging it 🌮🎉.. I'm working now to backport this PR to: 3.10, 3.11.
🐍🍒⛏🤖

…3697) Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com> (cherry picked from commit d91de28) Co-authored-by: James Gerity <snoopjedi@gmail.com>

bedevere-bot · 2022-10-25T13:23:07Z

GH-98653 is a backport of this pull request to the 3.11 branch.

…3697) Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com> (cherry picked from commit d91de28) Co-authored-by: James Gerity <snoopjedi@gmail.com>

bedevere-bot · 2022-10-25T13:23:09Z

GH-98654 is a backport of this pull request to the 3.10 branch.

Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com> (cherry picked from commit d91de28) Co-authored-by: James Gerity <snoopjedi@gmail.com>

Locate frozen module source with __file__

c91afc1

bedevere-bot added the awaiting review label Jun 10, 2022

Add NEWS entry

4baf238

SnoopJ added 2 commits June 12, 2022 14:32

Merge branch 'main' into pythongh-93696-pdb-find-frozen-source

a9e9c7a

Add test for listing source for frozen modules

ff51687

SnoopJ added 4 commits June 27, 2022 10:27

Merge branch 'main' into pythongh-93696-pdb-find-frozen-source

e58c4dc

Merge branch 'main' into pythongh-93696-pdb-find-frozen-source

793f2bf

Add issue reference to frozen_list test

e88d3a5

Merge branch 'main' into pythongh-93696-pdb-find-frozen-source

47eae63

kumaraditya303 self-requested a review July 16, 2022 06:52

kumaraditya303 reviewed Jul 17, 2022

View reviewed changes

Lib/test/test_pdb.py Outdated Show resolved Hide resolved

kumaraditya303 reviewed Jul 17, 2022

View reviewed changes

Misc/NEWS.d/next/Core and Builtins/2022-06-10-16-37-44.gh-issue-93696.65BI2R.rst Outdated Show resolved Hide resolved

Use assertIn() instead of assertTrue()

116c26b

Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>

SnoopJ and others added 2 commits July 17, 2022 14:38

Link to pdb module docs in news fragment

163f3df

Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com>

Merge branch 'main' into pythongh-93696-pdb-find-frozen-source

c716862

kumaraditya303 approved these changes Jul 18, 2022

View reviewed changes

bedevere-bot added awaiting core review and removed awaiting review labels Jul 18, 2022

kumaraditya303 requested a review from ericsnowcurrently July 18, 2022 13:43

kumaraditya303 added type-bug An unexpected behavior, bug, or error topic-importlib needs backport to 3.11 only security fixes needs backport to 3.10 only security fixes labels Jul 18, 2022

Clarify that pythongh-89815 is a superceding issue

55f2a0f

Merge branch 'main' into pythongh-93696-pdb-find-frozen-source

3facf2f

JelleZijlstra reviewed Oct 25, 2022

View reviewed changes

Only apply filename workaround for string attribute

a2589c9

JelleZijlstra merged commit d91de28 into python:main Oct 25, 2022

bedevere-bot removed the awaiting core review label Oct 25, 2022

bedevere-bot removed the needs backport to 3.11 only security fixes label Oct 25, 2022

bedevere-bot removed the needs backport to 3.10 only security fixes label Oct 25, 2022

SnoopJ deleted the gh-93696-pdb-find-frozen-source branch October 25, 2022 13:23

gaogaotiantian mentioned this pull request Apr 24, 2025

gh-93696: Fixed the breakpoint display error for frozen modules #132862

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-93696: Locate frozen module source with file #93697

gh-93696: Locate frozen module source with file #93697

SnoopJ commented Jun 10, 2022 •

edited by bedevere-bot

Loading

SnoopJ commented Jun 10, 2022 •

edited

Loading

SnoopJ commented Jun 12, 2022 •

edited

Loading

ghost commented Jul 17, 2022 •

edited by ghost

Loading

kumaraditya303 left a comment

SnoopJ commented Sep 29, 2022

SnoopJ commented Oct 24, 2022

JelleZijlstra Oct 25, 2022

JelleZijlstra Oct 25, 2022 •

edited

Loading

SnoopJ Oct 25, 2022

JelleZijlstra Oct 25, 2022

miss-islington commented Oct 25, 2022

bedevere-bot commented Oct 25, 2022

bedevere-bot commented Oct 25, 2022

gh-93696: Locate frozen module source with __file__ #93697

gh-93696: Locate frozen module source with __file__ #93697

Conversation

SnoopJ commented Jun 10, 2022 • edited by bedevere-bot Loading

SnoopJ commented Jun 10, 2022 • edited Loading

Edit: I have now added a test to this PR, see follow-up comment

SnoopJ commented Jun 12, 2022 • edited Loading

ghost commented Jul 17, 2022 • edited by ghost Loading

kumaraditya303 left a comment

Choose a reason for hiding this comment

SnoopJ commented Sep 29, 2022

SnoopJ commented Oct 24, 2022

JelleZijlstra Oct 25, 2022

Choose a reason for hiding this comment

JelleZijlstra Oct 25, 2022 • edited Loading

Choose a reason for hiding this comment

SnoopJ Oct 25, 2022

Choose a reason for hiding this comment

JelleZijlstra Oct 25, 2022

Choose a reason for hiding this comment

miss-islington commented Oct 25, 2022

bedevere-bot commented Oct 25, 2022

bedevere-bot commented Oct 25, 2022

gh-93696: Locate frozen module source with file #93697

gh-93696: Locate frozen module source with file #93697

SnoopJ commented Jun 10, 2022 •

edited by bedevere-bot

Loading

SnoopJ commented Jun 10, 2022 •

edited

Loading

SnoopJ commented Jun 12, 2022 •

edited

Loading

ghost commented Jul 17, 2022 •

edited by ghost

Loading

JelleZijlstra Oct 25, 2022 •

edited

Loading