Move inductor workflows focal (ubuntu 20.04) -> jammy (ubuntu 22.04) #154153

atalman · 2025-05-22T21:40:31Z

Trying to fix: #154157

pytorch-bot · 2025-05-22T21:40:35Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/154153

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 3 New Failures, 26 Pending, 3 Unrelated Failures

As of commit f5fa877 with merge base 413664b ():

NEW FAILURES - The following jobs have failed:

docker-builds / docker-build (linux.12xlarge, pytorch-linux-jammy-py3-clang12-executorch) (gh)
Final attempt failed. Child_process exited with error code 1
inductor / linux-jammy-cpu-py3.9-gcc11-inductor / test (cpu_inductor_torchbench, 2, 2, linux.8xlarge.amx) (gh)
opacus_cifar10
inductor / linux-jammy-cpu-py3.9-gcc11-inductor / test (dynamic_cpu_inductor_torchbench, 2, 2, linux.8xlarge.amx) (gh)
opacus_cifar10

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

periodic / linux-focal-cuda12.6-py3-gcc11-slow-gradcheck / test (default, 1, 8, linux.g5.4xlarge.nvidia.gpu, module:slowgradcheck) (gh) (trunk failure)
test_matmul_cuda.py::TestMatmulCudaCUDA::test_cublas_addmm_reduced_precision_size_10000_backend_cublaslt_cuda_float16
periodic / linux-focal-cuda12.6-py3-gcc11-slow-gradcheck / test (default, 2, 8, linux.g5.4xlarge.nvidia.gpu, module:slowgradcheck) (gh) (trunk failure)
inductor/test_max_autotune.py::TestMaxAutotune::test_max_autotune_decompose_k_dynamic_input

UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:

Build manywheel docker images for s390x / build-docker-cpu-s390x (gh) (#154074)
Process completed with exit code 1.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

huydhn · 2025-05-23T01:29:49Z

TIL, we don't need to manually create new ECR entries anymore, which is super nice

focal->jammy [CI] migrate focal to jammy fix Revert "fix" This reverts commit d70b171cdd5e2a7286aa4b80fd8e7ee67b9e867c.

atalman · 2025-05-27T11:51:54Z

@pytorchmergebot merge -f "all required signal is green"

pytorchmergebot · 2025-05-27T11:53:27Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

malfet · 2025-05-27T19:21:51Z

@pytorchbot revert -m "Broke inductor tests, see https://hud.pytorch.org/hud/pytorch/pytorch/b8452e55bc50109a14add60397fee6c85486ae8d/1?per_page=50&name_filter=inductor_torchbench&mergeEphemeralLF=true" -c nosignal

pytorchmergebot · 2025-05-27T19:23:24Z

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

… 22.04) (#154153)" This reverts commit 2bd95f3. Reverted #154153 on behalf of https://github.com/malfet due to Broke inductor tests, see https://hud.pytorch.org/hud/pytorch/pytorch/b8452e55bc50109a14add60397fee6c85486ae8d/1?per_page=50&name_filter=inductor_torchbench&mergeEphemeralLF=true ([comment](#154153 (comment)))

pytorchmergebot · 2025-05-27T19:23:31Z

@atalman your PR has been successfully reverted.

atalman · 2025-05-27T19:33:20Z

Looks like it broke opacus_cifar10: #154446

davidberard98 · 2025-05-27T20:01:17Z

maybe this is just a new release of opacus? https://pypi.org/project/opacus/#history

desertfire · 2025-05-27T20:03:22Z

The fact this is an eager failure is pretty suspicious given the nature of your PR,

2025-05-27T17:08:45.4796194Z cuda eval  opacus_cifar10                     
2025-05-27T17:08:45.4810045Z Traceback (most recent call last):
2025-05-27T17:08:45.4810789Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1940, in validate_model
2025-05-27T17:08:45.4811457Z     self.model_iter_fn(model, example_inputs)
2025-05-27T17:08:45.4812106Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 460, in forward_pass
2025-05-27T17:08:45.4812855Z     return mod(*inputs)
2025-05-27T17:08:45.4813593Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1767, in _wrapped_call_impl
2025-05-27T17:08:45.4814246Z     return self._call_impl(*args, **kwargs)
2025-05-27T17:08:45.4814837Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1778, in _call_impl
2025-05-27T17:08:45.4815433Z     return forward_call(*args, **kwargs)
2025-05-27T17:08:45.4816057Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/opacus/grad_sample/grad_sample_module.py", line 149, in forward
2025-05-27T17:08:45.4816686Z     return self._module(*args, **kwargs)
2025-05-27T17:08:45.4817305Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1767, in _wrapped_call_impl
2025-05-27T17:08:45.4817938Z     return self._call_impl(*args, **kwargs)
2025-05-27T17:08:45.4818524Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1778, in _call_impl
2025-05-27T17:08:45.4819114Z     return forward_call(*args, **kwargs)
2025-05-27T17:08:45.4819698Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/resnet.py", line 285, in forward
2025-05-27T17:08:45.4820282Z     return self._forward_impl(x)
2025-05-27T17:08:45.4820870Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/resnet.py", line 270, in _forward_impl
2025-05-27T17:08:45.4821473Z     x = self.relu(x)
2025-05-27T17:08:45.4822039Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1767, in _wrapped_call_impl
2025-05-27T17:08:45.4822671Z     return self._call_impl(*args, **kwargs)
2025-05-27T17:08:45.4823271Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1778, in _call_impl
2025-05-27T17:08:45.4823870Z     return forward_call(*args, **kwargs)
2025-05-27T17:08:45.4824766Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/activation.py", line 135, in forward
2025-05-27T17:08:45.4825391Z     return F.relu(input, inplace=self.inplace)
2025-05-27T17:08:45.4825952Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/functional.py", line 1711, in relu
2025-05-27T17:08:45.4826490Z     result = torch.relu_(input)
2025-05-27T17:08:45.4828147Z RuntimeError: Output 0 of BackwardHookFunctionBackward is a view and is being modified inplace. This view was created inside a custom Function (or because an input was returned as-is) and the autograd logic to handle view+inplace would override the custom backward associated with the custom Function, leading to incorrect gradients. This behavior is forbidden. You can fix this by cloning the output of the custom Function.

atalman · 2025-05-27T20:36:31Z

This does look like opacus update:

Previous passing PR: https://github.com/pytorch/pytorch/actions/runs/15272660608/job/42954922171#step:22:1478
used opacus==1.5.3

atalman · 2025-05-27T23:14:28Z

@pytorchmergebot merge -f "lint is green, this is a reland"

pytorchmergebot · 2025-05-27T23:16:01Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

…4437) Fixes #154157 Inductor Workflows where moved from focal to jammy here: #154153 Pull Request resolved: #154437 Approved by: https://github.com/Skylion007, https://github.com/cyyever, https://github.com/davidberard98, https://github.com/huydhn

…orch#154437) Fixes pytorch#154157 Inductor Workflows where moved from focal to jammy here: pytorch#154153 Pull Request resolved: pytorch#154437 Approved by: https://github.com/Skylion007, https://github.com/cyyever, https://github.com/davidberard98, https://github.com/huydhn

Move inductor workflow from focal to jammy

4524a9c

atalman requested review from a team and jeffdaily as code owners May 22, 2025 21:40

pytorch-bot bot added the topic: not user facing topic category label May 22, 2025

atalman mentioned this pull request May 22, 2025

Move all CI/CD workflows from focal to jammy #154157

Closed

atalman changed the title ~~[DRAFT] Move inductor workflow from focal to jammy~~ [WIP] Move inductor workflow from focal to jammy May 22, 2025

atalman added ciflow/inductor ciflow/periodic Trigger jobs ran periodically on master (periodic.yml) on the PR labels May 23, 2025

atalman changed the title ~~[WIP] Move inductor workflow from focal to jammy~~ Move inductor workflows from focal (ubuntu 20.04) -> jammy (ubuntu 22.04) May 23, 2025

atalman changed the title ~~Move inductor workflows from focal (ubuntu 20.04) -> jammy (ubuntu 22.04)~~ Move inductor workflows focal (ubuntu 20.04) -> jammy (ubuntu 22.04) May 23, 2025

Skylion007 approved these changes May 23, 2025

View reviewed changes

next_iter

4c0e74e

focal->jammy [CI] migrate focal to jammy fix Revert "fix" This reverts commit d70b171cdd5e2a7286aa4b80fd8e7ee67b9e867c.

atalman force-pushed the inductor_workflows_try_move branch from d70b171 to 4c0e74e Compare May 23, 2025 15:45

huydhn approved these changes May 23, 2025

View reviewed changes

atalman added 3 commits May 23, 2025 09:04

fix

c18ec34

periodic

bf53861

missed

f5fa877

atalman added the ciflow/h100 label May 26, 2025

cyyever approved these changes May 27, 2025

View reviewed changes

pytorchmergebot added the merging label May 27, 2025

pytorchmergebot added the Merged label May 27, 2025

pytorchmergebot closed this in 2bd95f3 May 27, 2025

pytorchmergebot removed the merging label May 27, 2025

This was referenced May 27, 2025

[CI] Migrate focal (ubuntu 20.04) images to jammy (ubuntu 22.04) #154437

Closed

Change result for opacus_cifar10 inductor_torchbench_inference #154445

Closed

pytorchmergebot added Reverted ci-no-td Do not run TD on this PR labels May 27, 2025

pytorchmergebot reopened this May 27, 2025

atalman mentioned this pull request May 27, 2025

[Inductor] opacus_cifar10 test returns eager_fail_to_run after opacus 1.5.4 update #154446

Closed

pytorchmergebot added the merging label May 27, 2025

pytorchmergebot closed this in fe082c5 May 27, 2025

pytorchmergebot removed the merging label May 27, 2025

atalman mentioned this pull request Jun 3, 2025

Triton pin update for PyTorch 2.8 / Triton 3.4 #154206

Closed

23 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Move inductor workflows focal (ubuntu 20.04) -> jammy (ubuntu 22.04) #154153

Move inductor workflows focal (ubuntu 20.04) -> jammy (ubuntu 22.04) #154153

Uh oh!

atalman commented May 22, 2025 •

edited

Loading

Uh oh!

pytorch-bot bot commented May 22, 2025 •

edited

Loading

Uh oh!

huydhn commented May 23, 2025

Uh oh!

atalman commented May 27, 2025

Uh oh!

pytorchmergebot commented May 27, 2025

Uh oh!

malfet commented May 27, 2025

Uh oh!

pytorchmergebot commented May 27, 2025

Uh oh!

pytorchmergebot commented May 27, 2025

Uh oh!

atalman commented May 27, 2025

Uh oh!

davidberard98 commented May 27, 2025

Uh oh!

desertfire commented May 27, 2025

Uh oh!

atalman commented May 27, 2025

Uh oh!

atalman commented May 27, 2025

Uh oh!

pytorchmergebot commented May 27, 2025

Uh oh!

Uh oh!

Move inductor workflows focal (ubuntu 20.04) -> jammy (ubuntu 22.04) #154153

Move inductor workflows focal (ubuntu 20.04) -> jammy (ubuntu 22.04) #154153

Uh oh!

Conversation

atalman commented May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/154153

❌ 3 New Failures, 26 Pending, 3 Unrelated Failures

Uh oh!

huydhn commented May 23, 2025

Uh oh!

atalman commented May 27, 2025

Uh oh!

pytorchmergebot commented May 27, 2025

Merge started

Uh oh!

malfet commented May 27, 2025

Uh oh!

pytorchmergebot commented May 27, 2025

Uh oh!

pytorchmergebot commented May 27, 2025

Uh oh!

atalman commented May 27, 2025

Uh oh!

davidberard98 commented May 27, 2025

Uh oh!

desertfire commented May 27, 2025

Uh oh!

atalman commented May 27, 2025

Uh oh!

atalman commented May 27, 2025

Uh oh!

pytorchmergebot commented May 27, 2025

Merge started

Uh oh!

Uh oh!

atalman commented May 22, 2025 •

edited

Loading

pytorch-bot bot commented May 22, 2025 •

edited

Loading