Skip to content

Avoid grad_fn getting garbage collected in multihooks #102175

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 2 commits into from
Closed

Avoid grad_fn getting garbage collected in multihooks #102175

wants to merge 2 commits into from

Conversation

soulitzer
Copy link
Contributor

@soulitzer soulitzer commented May 24, 2023 •
edited
Loading

Stack from ghstack (oldest at bottom):

Workaround for #102174

@soulitzer soulitzer requested a review from albanD as a code owner May 24, 2023 16:19
@pytorch-bot
Copy link

pytorch-bot bot commented May 24, 2023 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/102175

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 8c789e3:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@soulitzer soulitzer requested a review from awgu May 24, 2023 16:24
@soulitzer soulitzer added release notes: autograd release notes category topic: bug fixes topic category labels May 24, 2023
awgu
awgu approved these changes May 24, 2023
View reviewed changes
Copy link
Collaborator

@awgu awgu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes sense to me! Feel free to wait for @albanD's review.

torch/autograd/graph.py
self.handles = handles
self.grad_fns: Optional[Tuple] = grad_fns
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding Optional[Tuple] typing, when can grad_fns be None?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whoops I meant to set it to None in remove, good catch

soulitzer added a commit that referenced this pull request May 24, 2023
@soulitzer
Avoid grad_fn getting garbage collected in multihooks
484ca3e 
ghstack-source-id: ef6ca6a
Pull Request resolved: #102175
@awgu
Copy link
Collaborator

awgu commented Jun 1, 2023

Hi @albanD. Would it be possible to get a review on this? I would like to add land the improved fine-tuning support for FSDP (#101982) that depends on this.

@soulitzer
Copy link
Contributor Author

soulitzer commented Jun 1, 2023 •
edited
Loading

@awgu this is probably correct, but from discussion with Alban, we probably just want to fix the underlying issue (should be a one line fix). I added a comment on the issue.

@awgu
Copy link
Collaborator

awgu commented Jun 1, 2023

@awgu this is probably correct, but from discussion with Alban, we probably just want to fix the underlying issue (should be a one line fix). I added a comment on the issue.

Oh okay, so will we still land this PR or will there be a separate PR with the proper fix?

@soulitzer
Copy link
Contributor Author

Yeah separate PR, closing this one

@soulitzer soulitzer closed this Jun 2, 2023
@facebook-github-bot facebook-github-bot deleted the gh/soulitzer/209/head branch July 3, 2023 14:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@awgu awgu awgu approved these changes

@albanD albanD Awaiting requested review from albanD

Assignees
No one assigned
Labels
release notes: autograd release notes category topic: bug fixes topic category
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@soulitzer @awgu