Skip to content

gh-102567: Add -X importtime=2 for logging an importtime message for already-loaded modules #118655

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

noahbkim
Copy link
Contributor

@noahbkim noahbkim commented May 6, 2024

As mentioned in the issue:

While -X importtime is incredibly useful for analyzing module import times, by design, it doesn't log anything if an imported module has already been loaded. -X importtime=2 would provide additional output for every module that's already been loaded:

The updated example (-Ximporttime=2):

>>> import uuid
import time: cached    | cached     | builtins
import time: cached    | cached     | linecache
import time: cached    | cached     |   _io
import time: cached    | cached     |   os
import time: cached    | cached     |   sys
import time: cached    | cached     |   enum
import time:       594 |        594 |   _uuid
import time:      2428 |       3022 | uuid

With -Ximporttime:

>>> import uuid
import time:       351 |        351 |   _uuid
import time:       843 |       1194 | uuid

Discussion: https://discuss.python.org/t/x-importtrace-to-supplement-x-importtime-for-loaded-modules/23882/5
Prior email chain: https://mail.python.org/archives/list/python-ideas@python.org/thread/GEISYQ5BXWGKT33RWF77EOSOMMMFUBUS/

@Eclips4
Copy link
Member

Eclips4 commented May 6, 2024

Hello!
This PR needs to add:

  1. A NEWS entry
  2. A whatsnew entry
  3. A new paragraph in the Doc/cmdline for the -X option.
  4. New test case in the Lib/test/test_cmd_line

@noahbkim
Copy link
Contributor Author

noahbkim commented May 6, 2024

Hello! This PR needs to add:

  1. A NEWS entry
  2. A whatsnew entry
  3. A new paragraph in the Doc/cmdline for the -X option.
  4. New test case in the Lib/test/test_cmd_line

Thanks for the quick feedback. I've committed a first pass of 1-3. As for tests: there appear to be no importtime tests (likely due to the fact that any simple implementation would be super noisy as other parts of the interpreter change). Would it be sufficient to do some rudimentary check i.e. import foo then ensure a corresponding foo row appears in the output?

Thanks again!

@Eclips4
Copy link
Member

Eclips4 commented May 6, 2024

Hello! This PR needs to add:

  1. A NEWS entry
  2. A whatsnew entry
  3. A new paragraph in the Doc/cmdline for the -X option.
  4. New test case in the Lib/test/test_cmd_line

Thanks for the quick feedback. I've committed a first pass of 1-3. As for tests: there appear to be no importtime tests (likely due to the fact that any simple implementation would be super noisy as other parts of the interpreter change). Would it be sufficient to do some rudimentary check i.e. import foo then ensure a corresponding foo row appears in the output?

Thanks again!

Yes, it would be sufficient :)

@Eclips4
Copy link
Member

Eclips4 commented May 6, 2024

Also, there's a merge conflict. Can you resolve it? (I can do it myself, but it seems like you have turned off this option)

@noahbkim
Copy link
Contributor Author

noahbkim commented May 7, 2024

Also, there's a merge conflict. Can you resolve it? (I can do it myself, but it seems like you have turned off this option)

Done, just added tests as well

@noahbkim
Copy link
Contributor Author

@Eclips4 who should I ping to get final review? Is it alright if I @ the blamed reviewers?

@Eclips4
Copy link
Member

Eclips4 commented May 28, 2024

@Eclips4 who should I ping to get final review? Is it alright if I @ the blamed reviewers?

I guess @vstinner is the right person to review this 😄
However, you should move your changes from whatsnew/3.13.rst to whatsnew/3.14.rst since 3.13 branch is in feature-freeze mode, and this feature will appear only in the 3.14 version.

@Eclips4
Copy link
Member

Eclips4 commented May 28, 2024

However, you should move your changes from whatsnew/3.13.rst to whatsnew/3.14.rst since 3.13 branch is in feature-freeze mode, and this feature will appear only in the 3.14 version.

The same applies for the versionchanged directives.

@ghost
Copy link

ghost commented Sep 30, 2024

All commit authors signed the Contributor License Agreement.
CLA signed

@noahbkim noahbkim force-pushed the feature/import-cache-2 branch 2 times, most recently from 4f627c5 to 0281267 Compare September 30, 2024 22:15
@noahbkim
Copy link
Contributor Author

Apologies for the noise, I had to amend some of the commits with the correct email to pick up the right CLA.

@noahbkim noahbkim force-pushed the feature/import-cache-2 branch from 80aba50 to 0281267 Compare September 30, 2024 22:26
@noahbkim
Copy link
Contributor Author

noahbkim commented Nov 4, 2024

Pinging @serhiy-storchaka for re-review; I intend to leave the mypy errors as-is because this import try/except mechanism is already used elsewhere in the stdlib.

@noahbkim
Copy link
Contributor Author

LGTM! Before merging, we need to know @pablogsal and @ambv opinion on change in Lib/_pyrepl/unix_console.py

Re-pinging for quick review!

@picnixz
Copy link
Member

picnixz commented Jan 11, 2025

@noahbkim Can you make the CI pass on Windows and macOS? the logs are no more available so I can't see what happened.

@noahbkim
Copy link
Contributor Author

noahbkim commented Jan 13, 2025

All checks except for mypy should pass. The issue reported by mypy is the following idiom

try:
    import posix
except ImportError:
    posix = None

Since this is used pretty widely throughout the standard library, I figured it should be left as is and the warning ignored. I could suppress the warning instead.

Update: woah, I lied, something has changed since my last push. I'll fix these failures today.

@picnixz
Copy link
Member

picnixz commented Jan 13, 2025

Since this is used pretty widely throughout the standard library, I figured it should be left as is and the warning ignored. I could suppress the warning instead.

You can mypy-ignore it I think. Otherwise, future modifications in that file would also be flagged, right?

@python-cla-bot
Copy link

python-cla-bot bot commented Apr 14, 2025

All commit authors signed the Contributor License Agreement.

CLA signed

@noahbkim noahbkim force-pushed the feature/import-cache-2 branch from 60d94b3 to 2abb3d8 Compare April 14, 2025 17:25
@noahbkim
Copy link
Contributor Author

noahbkim commented May 1, 2025

@AA-Turner merged your changes, happy to do anything else necessary, apologies for the inconvenience of the organization branch. Not sure where your comment went (I saw it in my email), but thanks for helping push this through.

@AA-Turner
Copy link
Member

AA-Turner commented May 1, 2025

Thanks @noahbkim, no worries! Are you able to make the changes to config_set_import_time etc to reserve values >=2?

We also recently changed our CLA bot, please can you click on the red 'not signed' and ensure everything is up to date?

A

@noahbkim
Copy link
Contributor Author

noahbkim commented May 1, 2025

I think I've messed up my branch, I'm going to attempt to squash everything so I can overwrite my commit email correctly.

@AA-Turner
Copy link
Member

Ok, please ping for a review when you've sorted the branch out! If it's easier, feel free to just open a new PR, but a force push here should also work.

A

@noahbkim noahbkim force-pushed the feature/import-cache-2 branch from e53dea1 to 0ed040c Compare May 1, 2025 20:24
@noahbkim
Copy link
Contributor Author

noahbkim commented May 1, 2025

@AA-Turner I've squashed my changes and given everything a second pass.

It looks like I caused some kind of regression in test_embed.py, but none of the expected invariants make sense to me (I'm not sure why we expect import_time to be set to 1 in any of the listed circumstances). Looking through the blame was not particularly illuminating. My suspicion is that these tests have been migrated/updated several times without regard for what they're actually testing (or perhaps implicit behavior surrounding -Ximporttime has changed since the test suite's addition?).

Zeroing import_time in said cases returns everything to passing, but a second pair of eyes on that particular bit would be greatly appreciated.

I have about 6 tests failing locally, one of which prevents me from compiling with --enable-optimizations (test_json) but I can't imagine those are related. I'll check if my merge base is behind + fix the CI lint issue.

@noahbkim noahbkim force-pushed the feature/import-cache-2 branch from 0ed040c to 62d09cd Compare May 1, 2025 20:31
@noahbkim noahbkim force-pushed the feature/import-cache-2 branch from 62d09cd to 58d07be Compare May 2, 2025 14:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants