Skip to content

Enable sample nightly PT2 benchmark on B200 #158011

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 15 commits into from
Closed

Enable sample nightly PT2 benchmark on B200 #158011

wants to merge 15 commits into from

Conversation

huydhn
Copy link
Contributor

@huydhn huydhn commented Jul 10, 2025 •
edited
Loading

@huydhn
Enable nightly PT2 benchmark on B200
7a33ec1 
Signed-off-by: Huy Do <huydhn@gmail.com>
@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Jul 10, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/158011

Note: Links to docs will display an error until the docs builds have been completed.

⏳ No Failures, 25 Pending

As of commit 1fe577b with merge base 4175453 (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@pytorch-bot pytorch-bot bot added the release notes: releng release notes category label Jul 10, 2025
@nWEIdia
Copy link
Collaborator

nWEIdia commented Jul 10, 2025

pytorch/.github/requirements-gha-cache.txt

Line 11 in 4781d72

nvidia-ml-py==11.525.84
has pinned nvidia-ml-py package.

nWEIdia
nWEIdia reviewed Jul 10, 2025
View reviewed changes
@huydhn
Ready to try again
d15e97f 
Signed-off-by: Huy Do <huydhn@gmail.com>
@huydhn
Testing 1 2 3
5144d15 
Signed-off-by: Huy Do <huydhn@gmail.com>
@huydhn huydhn requested a review from malfet July 11, 2025 17:24
@huydhn huydhn marked this pull request as ready for review July 11, 2025 17:24
@huydhn huydhn requested a review from a team as a code owner July 11, 2025 17:24
nWEIdia
nWEIdia approved these changes Jul 11, 2025
View reviewed changes
Copy link
Collaborator

@nWEIdia nWEIdia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the great work! I see the job https://github.com/pytorch/pytorch/actions/runs/16211997928/job/45775083219 was successful after running a lot of benchmark tests.

atalman
atalman approved these changes Jul 11, 2025
View reviewed changes
Copy link
Contributor

@atalman atalman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@huydhn
Run smaller set of models
5f52091 
Signed-off-by: Huy Do <huydhn@gmail.com>
@huydhn huydhn mentioned this pull request Jul 12, 2025
Merged
huydhn added 3 commits July 11, 2025 22:50
@huydhn
Only for dashboard run
0458357 
Signed-off-by: Huy Do <huydhn@gmail.com>
@huydhn
Spurious updates
9ab2118 
Signed-off-by: Huy Do <huydhn@gmail.com>
@huydhn
Still too slow?
85c2630 
Signed-off-by: Huy Do <huydhn@gmail.com>
huydhn added a commit to pytorch/test-infra that referenced this pull request Jul 15, 2025
@huydhn
[ez] Adding results from B200 benchmark to TorchInductor dashboard (#…
a8af85f 
…6923)

TSIA, running from sample benchmark from
pytorch/pytorch#158011

## Preview


https://torchci-git-fork-huydhn-add-b200-benchmark-fbopensource.vercel.app/benchmark/compilers?dashboard=torchinductor&startTime=Sat%2C%2005%20Jul%202025%2005%3A38%3A21%20GMT&stopTime=Sat%2C%2012%20Jul%202025%2005%3A38%3A21%20GMT&granularity=hour&mode=inference&dtype=bfloat16&deviceName=cuda%20(b200)&lBranch=enable-b200-benchmark&lCommit=5144d1552a4cd59d007f42c3744044cb30b044ed&rBranch=enable-b200-benchmark&rCommit=5144d1552a4cd59d007f42c3744044cb30b044ed

cc @nWEIdia

Signed-off-by: Huy Do <huydhn@gmail.com>
@nWEIdia
Copy link
Collaborator

nWEIdia commented Jul 29, 2025

@huydhn Have you had a chance to look into why the torchbench still took 12+ hours?

huydhn added 6 commits July 29, 2025 15:57
@huydhn
Merge branch 'main' into enable-b200-benchmark
ef3033a 
Signed-off-by: Huy Do <huydhn@gmail.com>
@huydhn
Spurious update
ebca68d 
Signed-off-by: Huy Do <huydhn@gmail.com>
@huydhn
Prebuild domain libraries
a291e3e 
Signed-off-by: Huy Do <huydhn@gmail.com>
@huydhn
Test linux.dgx.b200
1738e33 
Signed-off-by: Huy Do <huydhn@gmail.com>
@huydhn
Use linux.gdx.b200 runner name
cb91f09 
Signed-off-by: Huy Do <huydhn@gmail.com>
@huydhn
Wrong syntax
55d1d43 
Signed-off-by: Huy Do <huydhn@gmail.com>
@huydhn
Use bigger build runner
9d7e234 
Signed-off-by: Huy Do <huydhn@gmail.com>
@nWEIdia nWEIdia mentioned this pull request Jul 30, 2025
Closed
@huydhn huydhn changed the title Enable nightly PT2 benchmark on B200 Enable sample nightly PT2 benchmark on B200 Aug 1, 2025
@huydhn
Copy link
Contributor Author

huydhn commented Aug 1, 2025

@pytorchbot merge -f 'Unblock _linux_test workflow on b200`

Per the discussion with @nWEIdia, I will go ahead and land this change because it unblocks _linux_test workflow on b200. The TorchBench benchmark job itself still doesn't work https://github.com/pytorch/pytorch/actions/runs/16615101382/job/47008262828, but at least it runs (and times out). I will deal with this in another PR.

@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Aug 1, 2025

❌ 🤖 pytorchbot command failed:

Got EOF while in a quoted string```
Try `@pytorchbot --help` for more info.

@huydhn
Copy link
Contributor Author

huydhn commented Aug 1, 2025

@pytorchbot merge -f 'Unblock _linux_test workflow on b200'

Per the discussion with @nWEIdia, I will go ahead and land this change because it unblocks _linux_test workflow on b200. The TorchBench benchmark job itself still doesn't work https://github.com/pytorch/pytorch/actions/runs/16615101382/job/47008262828, but at least it runs (and times out). I will deal with this in another PR.

@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

facebook-github-bot pushed a commit to pytorch/benchmark that referenced this pull request Aug 2, 2025
@huydhn @facebook-github-bot
Enable sample nightly PT2 benchmark on B200 (#158011)
b4dd068 
Summary:
Per the discussion with nWEIdia, this resumes the work on pytorch/pytorch#157870 to enable PT2 benchmark on B200

### Testing

https://github.com/pytorch/pytorch/actions/runs/16615101382

X-link: pytorch/pytorch#158011
Approved by: https://github.com/nWEIdia, https://github.com/atalman

Reviewed By: yangw-dev

Differential Revision: D79490413

fbshipit-source-id: 0202249d62bd3011a1e60925b2f8442ddfdcbff7
@nWEIdia nWEIdia mentioned this pull request Aug 4, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@atalman atalman atalman approved these changes

@nWEIdia nWEIdia nWEIdia approved these changes

@malfet malfet Awaiting requested review from malfet

Assignees
No one assigned
Labels
ciflow/inductor Merged module: dynamo release notes: releng release notes category
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@huydhn @nWEIdia @pytorchmergebot @atalman