-
-
Notifications
You must be signed in to change notification settings - Fork 7.9k
Lazy import of private modules #12781
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
93fb605
to
1761a03
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Don't normally like local imports, but this seems like a compelling argument and is much simpler than using a lazy proxy.
Just importing
|
Out of interest, do you have a number for how much time this will save? |
It's not really easy to read the whole thing, but the flamegraph in #11546 doesn't seem to indicate that these imports are particularly significant, except maybe the urllib one you have downstream. Importing NumPy seems to dominate by a fair fair bit, unless you've already tackled that one? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Per #12781 (comment) I think if we're going that route there should be at least a test showing that importing matplotlib.figure and matplotlib.backend_bases (in a separate subprocess) does not lead to any of these extension modules being loaded.
Otherwise it's too easy to break that property...
@anntzer wrote:
Have you tried it with this PR though? @ImportanceOfBeingErnest wrote:
When importing matplotlib in pyodide, the import time goes from 5.9s to 5.2s (when added with a few others that involve non-underscore-prefixed-but-probably-not-intentionally-public things). This is largely due to the slowness of dynamically linking extension modules on the WebAssembly platform. I would expect the delta on a native platform to be much smaller. (See pyodide/pyodide#251). @QuLogic wrote:
Things change when the cost of importing extension modules is higher. Particularly the Raw data: https://gist.github.com/mdboom/3d5d86e217f8f650f33a85a39cd835cc But, yes, I would be a big win to not import I haven't tackled numpy, but I would consider that something that probably needs to be dealt with in numpy. Lazy importing all of numpy for matplotlib seems unfeasible. |
@anntzer: I've added a test here to ensure this "sticks". It can easily be extended in the future if we want to "blacklist" more modules for lazy loading. This also gave me the opportunity to measure on a native build. As expected, it's pretty minor: 251424 vs 281676 us. |
cae5a92
to
b02a8ec
Compare
Duh, sorry :) Thanks for the test. |
Would also lazy loading |
1198453
to
466c4a5
Compare
I did try inlining urllib imports in #11546 (comment) and actually saw no improvements (despite what the flamegraph says). |
9aedbba
to
d4dbe51
Compare
This is passing CI and ready to merge. |
I can confirm that |
5f88fb2
to
c0e1b34
Compare
@mdboom Took the liberty of rebasing and force-pushing, hope you do not mind. |
This has some real test errors. Presumably missing imports? |
Looks like I missed one on the rebase, fixed now. |
…781-on-v3.1.x Backport PR #12781 on branch v3.1.x (Lazy import of private modules)
PR Summary
Not all platforms have blazing fast dynamic linkers, therefore a lot of import (startup) time can be saved by not importing extension modules that we don't have to.
There are some more aggressive uses of lazy loading that are possible, but this is just the stuff that's already declared private and thus users who are importing these things at this path have had "fair warning". If acceptable, I can provide a follow-on PR that deprecates and then removes additional things.
PR Checklist