gh-132737: Support profiling modules that import __main___ #132738
Conversation
aneeshdurg
commented
Apr 19, 2025
•
- Issue: cProfile cannot run code that pickles objects defined in __main__ #132737
|
All commit authors signed the Contributor License Agreement. |
|
Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool. If this change has little impact on Python users, wait for a maintainer to apply the |
Misc/NEWS.d/next/Core_and_Builtins/2025-04-19-18-07-34.gh-issue-132737.9mW1il.rst
Outdated
Show resolved
Hide resolved
Misc/NEWS.d/next/Core_and_Builtins/2025-04-19-18-07-34.gh-issue-132737.9mW1il.rst
Outdated
Show resolved
Hide resolved
|
This is not going to work because you introduced extra function calls in cProfile record, which would confuse people. Try a simple I can work on this (kind of the responsible person for the module), or if you really want to take a swing at the issue I can give you some direction. |
|
I'd be happy to take a swing at it! Any pointers on what to look into? |
|
So Also, you need a regression test in this case. You need to write a test that fails with the original implementation, and passes after. Hopefully it won't be this complicated (I know it's not long, but I don't believe dataclass or even pickle is required to reproduce this issue). It's okay to use pickle if that's the easiest way. |
|
I was able to get my repro working by updating main instead! I'll add regression tests soon. |
…e-132737.9mW1il.rst Co-authored-by: Stan Ulbrych <89152624+StanFromIreland@users.noreply.github.com>
|
You can't do it in |
Lib/cProfile.py
Outdated
| try: | ||
| runctx(code, globs, None, options.outfile, options.sort) | ||
| runctx(code, __main__.__dict__, None, options.outfile, options.sort) |
There was a problem hiding this comment.
Why do you need to restore it? You are exiting the program anyway. Also this is not ideal either. This will include all the global variables to the script that is being profiled. We want print(locals()) to be basically the same with or without the profiler.
There was a problem hiding this comment.
The issue is to juggle the global variables needed by the call to (and implementation of) runctx. It would be a lot easier if I could split cprofile into a module where __main__ only has the main function. Is that something I can do?
if not, it's still possible, just trickier/messier.
There was a problem hiding this comment.
No you can't split it into its own module :( that's too much a change. I think the correct way to go is to make the full runctx path independent of any global variables.
There was a problem hiding this comment.
Found a much cleaner fix - just ensure that the "main" function isn't run in the __main__ namespace.
There was a problem hiding this comment.
Hmm, I'm not sure if this is too hacky. I did not find such pattern in other code. It looks like an acceptable solution but I don't know if there will be implications. I want to ask @vstinner about this as he probably knows a lot of interesting usages.
There was a problem hiding this comment.
It seems like this hack works as expected. But it's strange and surprising :-)
I tried but failed (test fails) to inject a new __main__ module in sys.modules and leave the cProfile module unchanged:
diff --git a/Lib/cProfile.py b/Lib/cProfile.py
index 6253755a9df..abc03fc61eb 100644
--- a/Lib/cProfile.py
+++ b/Lib/cProfile.py
@@ -166,6 +166,7 @@ def main():
'run_module': runpy.run_module,
'modname': args[0]
}
+ modname = args[0]
else:
progname = args[0]
sys.path.insert(0, os.path.dirname(progname))
@@ -181,14 +182,18 @@ def main():
'__cached__': None,
'__builtins__': __builtins__,
}
+ modname = spec.name
+
# cmd has to run in __main__ namespace (or imports from __main__ will
# break). Clear __main__ and replace with the globals provided.
- import __main__
- __main__.__dict__.clear()
- __main__.__dict__.update(globs)
+ import __main__ as cProfileMain
+ new_main = type(cProfileMain)(modname)
+ new_main.__dict__.clear()
+ new_main.__dict__.update(globs)
+ sys.modules['__main__'] = new_main
try:
- runctx(code, __main__.__dict__, None, options.outfile, options.sort)
+ runctx(code, new_main.__dict__, None, options.outfile, options.sort)
except BrokenPipeError as exc:
# Prevent "Exception ignored" during interpreter shutdown.
sys.stdout = NoneThere was a problem hiding this comment.
Isn't this hack what pdb already does implicitly?
There was a problem hiding this comment.
Line 2449 in de9deb7
Ah, it's actually explicitly documented
There was a problem hiding this comment.
@vstinner @gaogaotiantian I did a bit more poking around and I think I managed to get rid of any of the hacky-ness. cProfile's main remains untouched, and in the case where we execute a file, I create a new module, set it as main, and ensure that the globals dict is the dict of the new module. The tests pass.
There was a problem hiding this comment.
I think the current solution looks much less hacky.
Misc/NEWS.d/next/Core_and_Builtins/2025-04-19-18-07-34.gh-issue-132737.9mW1il.rst
Outdated
Show resolved
Hide resolved
…e-132737.9mW1il.rst Co-authored-by: Tian Gao <gaogaotiantian@hotmail.com>
|
I'll merge this in a day or two if no one has objections. |