Still run TritonBundler with BundledAOTAutogradCache, save autotune results #158048

jamesjwu · 2025-07-10T17:52:29Z

Stack from ghstack (oldest at bottom):

-> Still run TritonBundler with BundledAOTAutogradCache, save autotune results #158048

When running BundledAOTAutogradCache with precompile, we still need to run triton bundling so that the precompiled CompiledFxGraph has triton cuda kernels. We also pre save the autotune results in the precompile artifact.

It would be even better to pre trim the cuda kernels on save and apply them, which we can work on later.

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @chenyang78 @kadeng @muchulee8 @amjames @chauhang @aakhundov @coconutruben @Lucaskabela

[ghstack-poisoned]

pytorch-bot · 2025-07-10T17:52:33Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/158048

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

linux.aws.h100.8 instance is down, potentially longer queue on linux.aws.h100

✅ You can merge normally! (3 Unrelated Failures)

As of commit a9297da with merge base 813c76b ():

FLAKY - The following job failed but was likely due to flakiness present on trunk:

trunk / linux-jammy-rocm-py3.10 / test (default, 1, 2, linux.rocm.gpu.2) (gh) (similar failure)
test_nn.py::TestNNDeviceTypeCUDA::test_avg_pool_large_tensor2_cuda

BROKEN TRUNK - The following job failed but was present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

inductor / cuda12.8-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu) (gh) (trunk failure)
torchrec_dlrm

UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:

pull / cuda12.8-py3.10-gcc9-sm75 / test (pr_time_benchmarks, 1, 1, linux.g4dn.metal.nvidia.gpu, unstable) (gh) (#153987)
MISSING REGRESSION TEST

This comment was automatically generated by Dr. CI and updates every 15 minutes.

…esults ghstack-source-id: 0c9801d Pull-Request: #158048

… autotune results" When running BundledAOTAutogradCache with precompile, we still need to run triton bundling so that the precompiled CompiledFxGraph has triton cuda kernels. We also pre save the autotune results in the precompile artifact. It would be even better to pre trim the cuda kernels on save and apply them, which we can work on later. cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy chenyang78 kadeng muchulee8 amjames chauhang aakhundov [ghstack-poisoned]

…esults ghstack-source-id: 6f11e45 Pull-Request: #158048

jamesjwu · 2025-07-18T18:13:20Z

@pytorchbot merge

pytorchmergebot · 2025-07-18T18:15:25Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2025-07-19T00:13:44Z

The merge job was canceled or timed out. This most often happen if two merge requests were issued for the same PR, or if merge job was waiting for more than 6 hours for tests to finish. In later case, please do not hesitate to reissue the merge command
For more information see pytorch-bot wiki.

jamesjwu · 2025-07-20T14:05:34Z

@pytorchbot merge

pytorchmergebot · 2025-07-20T14:07:29Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2025-07-20T14:07:47Z

Merge failed

Reason: 1 jobs have failed, first few of them are: trunk / linux-jammy-rocm-py3.10 / test (default, 2, 2, linux.rocm.gpu.2)

Details for Dev Infra team

Raised by workflow job

jamesjwu · 2025-07-21T13:27:32Z

@pytorchbot merge -i

pytorchmergebot · 2025-07-21T13:29:56Z

Merge started

Your change will be merged while ignoring the following 3 checks: pull / linux-jammy-py3.13-clang12 / test (dynamo_wrapped, 2, 3, linux.2xlarge), inductor / cuda12.8-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, linux.g5.4xlarge.nvidia.gpu), trunk / linux-jammy-rocm-py3.10 / test (default, 2, 2, linux.rocm.gpu.2)

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

jeffdaily · 2025-07-21T20:42:35Z

This PR is causing ROCm CI failures. Issuing a revert.

https://hud.pytorch.org/failure?name=rocm%20%2F%20linux-jammy-rocm-py3.10%20%2F%20test%20(default%2C%202%2C%206%2C%20linux.rocm.gpu.2)&jobName=undefined&failureCaptures=dynamo%2Ftest_package.py%3A%3ATestPackage%3A%3Atest_automatic_dynamo_autotune_cache_device_cuda

jeffdaily · 2025-07-21T20:43:26Z

@pytorchbot revert -c nosignal -m "rocm failures due to unit test introduced in this PR, but no pre-merge signal available"

pytorchmergebot · 2025-07-21T20:45:06Z

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

…totune results (#158048)" This reverts commit 8e57cdb. Reverted #158048 on behalf of https://github.com/jeffdaily due to rocm failures due to unit test introduced in this PR, but no pre-merge signal available ([comment](#158048 (comment)))

pytorchmergebot · 2025-07-21T20:45:25Z

@jamesjwu your PR has been successfully reverted.

[ghstack-poisoned]

…esults ghstack-source-id: 7794a10 Pull-Request: #158048

jamesjwu · 2025-07-22T14:04:22Z

The given test now is properly skipped in ROCM, and the rocm failure seems to be unrelated to this PR.

@pytorchbot merge

pytorchmergebot · 2025-07-22T14:06:31Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

Update

d7ad45b

[ghstack-poisoned]

pytorch-bot bot added ciflow/inductor module: dynamo module: inductor labels Jul 10, 2025

jamesjwu added a commit that referenced this pull request Jul 10, 2025

Still run TritonBundler with BundledAOTAutogradCache, save autotune r…

26a372c

…esults ghstack-source-id: 0c9801d Pull-Request: #158048

jamesjwu mentioned this pull request Jul 10, 2025

[precompile] Increment frame and add compile ids when loading packages #158028

Closed

jamesjwu added the topic: not user facing topic category label Jul 15, 2025

jamesjwu requested review from zhxchen17 and oulgen July 15, 2025 17:33

jamesjwu added a commit that referenced this pull request Jul 17, 2025

Still run TritonBundler with BundledAOTAutogradCache, save autotune r…

c59fc1a

…esults ghstack-source-id: 6f11e45 Pull-Request: #158048

zhxchen17 approved these changes Jul 17, 2025

View reviewed changes

This was referenced Jul 17, 2025

[Precompile] [easy] API For Editable PrecompileCacheArtifacts #158586

Closed

Recheck Autotune cache on Precompile serialization to prune compilation results #158656

Open

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Jul 18, 2025

pytorchmergebot added the merging label Jul 18, 2025

pytorchmergebot removed the merging label Jul 20, 2025

pytorchmergebot added the merging label Jul 21, 2025

pytorchmergebot closed this in 8e57cdb Jul 21, 2025

pytorchmergebot added the Merged label Jul 21, 2025

pytorchmergebot removed the merging label Jul 21, 2025

pytorchmergebot added Reverted ci-no-td Do not run TD on this PR labels Jul 21, 2025

pytorchmergebot reopened this Jul 21, 2025

Fix ROCM

a9297da

[ghstack-poisoned]

jamesjwu added a commit that referenced this pull request Jul 21, 2025

Still run TritonBundler with BundledAOTAutogradCache, save autotune r…

8042587

…esults ghstack-source-id: 7794a10 Pull-Request: #158048

pytorchmergebot added the merging label Jul 22, 2025

pytorchmergebot closed this in 1b772de Jul 22, 2025

pytorchmergebot removed the merging label Jul 22, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Still run TritonBundler with BundledAOTAutogradCache, save autotune results #158048

Still run TritonBundler with BundledAOTAutogradCache, save autotune results #158048

Uh oh!

jamesjwu commented Jul 10, 2025 •

edited

Loading

Uh oh!

pytorch-bot bot commented Jul 10, 2025 •

edited

Loading

Uh oh!

jamesjwu commented Jul 18, 2025

Uh oh!

pytorchmergebot commented Jul 18, 2025

Uh oh!

pytorchmergebot commented Jul 19, 2025

Uh oh!

jamesjwu commented Jul 20, 2025

Uh oh!

pytorchmergebot commented Jul 20, 2025

Uh oh!

pytorchmergebot commented Jul 20, 2025

Uh oh!

jamesjwu commented Jul 21, 2025

Uh oh!

pytorchmergebot commented Jul 21, 2025

Uh oh!

jeffdaily commented Jul 21, 2025

Uh oh!

jeffdaily commented Jul 21, 2025

Uh oh!

pytorchmergebot commented Jul 21, 2025

Uh oh!

pytorchmergebot commented Jul 21, 2025

Uh oh!

jamesjwu commented Jul 22, 2025

Uh oh!

pytorchmergebot commented Jul 22, 2025

Uh oh!

Uh oh!

Still run TritonBundler with BundledAOTAutogradCache, save autotune results #158048

Still run TritonBundler with BundledAOTAutogradCache, save autotune results #158048

Uh oh!

Conversation

jamesjwu commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/158048

❗ 1 Active SEVs

✅ You can merge normally! (3 Unrelated Failures)

Uh oh!

jamesjwu commented Jul 18, 2025

Uh oh!

pytorchmergebot commented Jul 18, 2025

Merge started

Uh oh!

pytorchmergebot commented Jul 19, 2025

Uh oh!

jamesjwu commented Jul 20, 2025

Uh oh!

pytorchmergebot commented Jul 20, 2025

Merge started

Uh oh!

pytorchmergebot commented Jul 20, 2025

Merge failed

Uh oh!

jamesjwu commented Jul 21, 2025

Uh oh!

pytorchmergebot commented Jul 21, 2025

Merge started

Uh oh!

jeffdaily commented Jul 21, 2025

Uh oh!

jeffdaily commented Jul 21, 2025

Uh oh!

pytorchmergebot commented Jul 21, 2025

Uh oh!

pytorchmergebot commented Jul 21, 2025

Uh oh!

jamesjwu commented Jul 22, 2025

Uh oh!

pytorchmergebot commented Jul 22, 2025

Merge started

Uh oh!

Uh oh!

jamesjwu commented Jul 10, 2025 •

edited

Loading

pytorch-bot bot commented Jul 10, 2025 •

edited

Loading