Skip to content

Support pin_memory() during CUDA stream capture. #146924

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed

Conversation

galv
Copy link
Collaborator

@galv galv commented Feb 11, 2025

This code previously did not work:

import torch

def test():
    graph = torch.cuda.CUDAGraph()

    with torch.cuda.graph(graph, capture_error_mode="global"):
        data = torch.randn(8)
        data_gpu = torch.randn(8, device="cuda")
        data = data.pin_memory()
        data_gpu.to(data, non_blocking=True)

    graph2 = torch.cuda.CUDAGraph()
    with torch.cuda.graph(graph2, capture_error_mode="global"):
        data2 = torch.randn(8)
        data2_gpu = torch.randn(8, device="cuda")
        data2 = data2.pin_memory()
        data2_gpu.to(data2, non_blocking=True)

if __name__ == "__main__":
    test()

We use events to signal when a particular usage of a pinned host memory block has completed. Every time we call pin_memory(), cudaEventQuery() gets called to see if we we can reuse existing blocks rather than allocating new blocks. cudaEventQuery() is not allowed during stream capture unless we set the thread to relaxed capture mode. (This is safe in this case so long as we make sure that the pinned buffer is live until the cuda graph is destroyed.)

I haven't fully thought this through. I need to make sure that a pinned memory tensor does in fact stay live until its corresponding cuda graph is destroyed. (I haven't done this yet!)

Draft.

Discovered in #146145 (comment)

Not a high priority, but I wanted to start to figure out what proper support might look like.

@galv galv requested review from eqy and syed-ahmed as code owners February 11, 2025 19:22
Copy link

pytorch-bot bot commented Feb 11, 2025

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/146924

Note: Links to docs will display an error until the docs builds have been completed.

❌ 8 New Failures, 1 Unrelated Failure

As of commit c83e708 with merge base 99da439 (image):

NEW FAILURES - The following jobs have failed:

UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@galv
Copy link
Collaborator Author

galv commented Feb 19, 2025

Okay, so this "works" now, but the current problem is that CUDA Graphs don't have a way to "own" their pinned allocations created by CachingHostAllocator.cpp right now. This means that the user must somehow keep pinned allocations alive for the duration of the corresponding cuda graph, which is not a very good UX at all.

@galv
Copy link
Collaborator Author

galv commented Mar 7, 2025

The current code actually works as intended, though it is rudimentary and probably has subtle bugs.

This code previously did not work:

```python
import torch

def test():
    graph = torch.cuda.CUDAGraph()

    with torch.cuda.graph(graph, capture_error_mode="global"):
        data = torch.randn(8)
        data_gpu = torch.randn(8, device="cuda")
        data = data.pin_memory()
        data_gpu.to(data, non_blocking=True)

    graph2 = torch.cuda.CUDAGraph()
    with torch.cuda.graph(graph2, capture_error_mode="global"):
        data2 = torch.randn(8)
        data2_gpu = torch.randn(8, device="cuda")
        data2 = data2.pin_memory()
        data2_gpu.to(data2, non_blocking=True)

if __name__ == "__main__":
    test()
```

We use events to signal when a particular usage of a pinned host
memory block has completed. Every time we call pin_memory(),
cudaEventQuery() gets called to see if we we can reuse existing blocks
rather than allocating new blocks. cudaEventQuery() is not allowed
during stream capture unless we set the thread to relaxed capture
mode. (This is safe in this case so long as we make sure that the
pinned buffer is live until the cuda graph is destroyed.)

I haven't fully thought this through. I need to make sure that a
pinned memory tensor does in fact stay live until its corresponding
cuda graph is destroyed. (I haven't done this yet!)

Draft.

Use external flag only when in stream capture mode.

Initial correct implementation with proper ownership.

Cleanup + working on pools across multiple graphs.

Rudimentary test that capture into same pool across multiple graphs works.
@galv galv force-pushed the add-pinned-memory-stream-capture-supported branch from 28ae6f2 to dc79019 Compare March 7, 2025 20:53
@galv galv added the release notes: cuda release notes category label Mar 7, 2025
Copy link
Contributor

github-actions bot commented May 7, 2025

Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as Stale.
Feel free to remove the Stale label if you feel this was a mistake.
If you are unable to remove the Stale label please contact a maintainer in order to do so.
If you want the bot to never mark this PR stale again, add the no-stale label.
Stale pull requests will automatically be closed after 30 days of inactivity.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants