Skip to content

Force all deserialized objects to the oldest GC generation #19681

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
+14 −0
Open

Force all deserialized objects to the oldest GC generation #19681

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions mypy/build.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3326,8 +3326,22 @@ def process_graph(graph: Graph, manager: BuildManager) -> None:
#
# TODO: see if it's possible to determine if we need to process only a
# _subset_ of the past SCCs instead of having to process them all.
if platform.python_implementation() == "CPython":
# When deserializing cache we create huge amount of new objects, so even
# with our generous GC thresholds, GC is still doing a lot of pointless
# work searching for garbage. So, we temporarily disable it when
# processing fresh SCCs, and then move all the new objects to the oldest
# generation with the freeze()/unfreeze() trick below. This is arguably
# a hack, but it gives huge performance wins for large third-party
# libraries, like torch.
gc.collect()
gc.disable()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we get here multiple times, if there are multiple dirty sub-DAGs? If yes, do you think it'll be a problem?

A quick workaround would be to do this only at most N times per run (possibly N=1).

for prev_scc in fresh_scc_queue:
process_fresh_modules(graph, prev_scc, manager)
if platform.python_implementation() == "CPython":
gc.freeze()
gc.unfreeze()
gc.enable()
fresh_scc_queue = []
size = len(scc)
if size == 1:
Expand Down
Loading