[CI] Migrate focal (ubuntu 20.04) images to jammy (ubuntu 22.04) #154437

atalman · 2025-05-27T17:57:33Z

Fixes #154157

Inductor Workflows where moved from focal to jammy here: #154153

cc @H-Huang @awgu @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k

pytorch-bot · 2025-05-27T17:57:37Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/154437

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 84 Pending

As of commit 569ad47 with merge base 523b637 ():

NEW FAILURE - The following job has failed:

docker-builds / docker-build (linux.12xlarge, pytorch-linux-jammy-py3-clang12-executorch) (gh)
Final attempt failed. Child_process exited with error code 1

UNSTABLE - The following jobs are marked as unstable, possibly due to flakiness on trunk:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

atalman · 2025-05-27T18:00:00Z

.ci/docker/build.sh

 fi

 if [ -n "${UBUNTU_VERSION}" ]; then
  OS="ubuntu"
-elif [ -n "${CENTOS_VERSION}" ]; then
-  OS="centos"


There should be no CentOS in CI/CD anymore

atalman · 2025-05-27T18:00:49Z

.ci/docker/build.sh

@@ -370,14 +361,6 @@ esac

 tmp_tag=$(basename "$(mktemp -u)" | tr '[:upper:]' '[:lower:]')

-#when using cudnn version 8 install it separately from cuda
-if [[ "$image" == *cuda*  && ${OS} == "ubuntu" ]]; then
-  IMAGE_NAME="nvidia/cuda:${CUDA_VERSION}-cudnn${CUDNN_VERSION}-devel-ubuntu${UBUNTU_VERSION}"


Nvidia images used on Focal builds. For Jammy builds these images are not used anymore

huydhn · 2025-05-31T00:04:38Z

@pytorchbot rebase -b main

ZainRizvi

What do you think about splitting this into mulitple separate PRs to derisk the switch:

PR 1: Builds the jammy docker images for all desired configs
PR 2: Switches over all workflows to use the jammy images
PR 3: Stops building the focal docker images

It'll cut down on the blast radius in case something goes wrong. For example, it'll let us make sure that all jammy docker builds succeed before we start taking CI dependencies on them.

pytorchmergebot · 2025-05-31T00:06:10Z

@pytorchbot started a rebase job onto refs/remotes/origin/main. Check the current status here

pytorchmergebot · 2025-05-31T00:06:14Z

Successfully rebased focal_jammy onto refs/remotes/origin/main, please pull locally before adding more changes (for example, via git checkout focal_jammy && git pull --rebase)

atalman · 2025-06-02T22:25:45Z

@ZainRizvi I believe if the docker build + CI is successful its a lot easier to do in one shot as was done here: #154153

Building Docker images by itself are not really useful, since we most likely need to change them once we start migrating jobs over. To minimize the blast radius we can probably try to migrate in smaller chunks

atalman · 2025-06-03T16:19:57Z

@pytorchmergebot rebase -b main

pytorchmergebot · 2025-06-03T16:21:36Z

@pytorchbot started a rebase job onto refs/remotes/origin/main. Check the current status here

pytorchmergebot · 2025-06-03T16:21:39Z

Successfully rebased focal_jammy onto refs/remotes/origin/main, please pull locally before adding more changes (for example, via git checkout focal_jammy && git pull --rebase)

huydhn

LGTM! Do you think we could just keep one version in CI going forward? for example, everything is using 22.04 (jammy) now, then we will upgrade it all to 24.04 (the next LTS)

atalman · 2025-06-05T12:31:12Z

Following errors alredy exist on trunk:
test_linalg.py::TestLinalgCUDA::test_svd_memory_allocation_cuda_complex128 GH job link HUD commit link

and

test_matmul_cuda.py::TestMatmulCudaCUDA::test_cublas_addmm_reduced_precision_size_10000_backend_cublaslt_cuda_float16 GH job link HUD commit link

atalman · 2025-06-05T15:22:05Z

@pytorchmergebot merge -f "lint is green other jobs where already tested"

pytorchmergebot · 2025-06-05T15:23:44Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

…orch#154437) Fixes pytorch#154157 Inductor Workflows where moved from focal to jammy here: pytorch#154153 Pull Request resolved: pytorch#154437 Approved by: https://github.com/Skylion007, https://github.com/cyyever, https://github.com/davidberard98, https://github.com/huydhn

malfet · 2025-06-06T14:56:18Z

@pytorchbot revert -m "I could be wrong, but looks like it broke slow jobs, see https://hud.pytorch.org/hud/pytorch/pytorch/b0fbbef1361ccaab8a5aec8e7cd62150e7b361de/1?per_page=50&name_filter=slow&mergeEphemeralLF=true" -c nosignal

pytorchmergebot · 2025-06-06T14:58:03Z

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

pytorchmergebot · 2025-06-06T14:58:14Z

Reverting PR 154437 failed

Reason: Command git -C /home/runner/work/pytorch/pytorch revert --no-edit cd361fc247a9abdbe9851867e31ac3cefcff299e returned non-zero exit code 1

Auto-merging .ci/docker/build.sh
CONFLICT (content): Merge conflict in .ci/docker/build.sh
Auto-merging .github/workflows/docker-builds.yml
CONFLICT (content): Merge conflict in .github/workflows/docker-builds.yml
error: could not revert cd361fc247a... [CI] Migrate focal (ubuntu 20.04) images to jammy (ubuntu 22.04) (#154437)
hint: After resolving the conflicts, mark them with
hint: "git add/rm <pathspec>", then run
hint: "git revert --continue".
hint: You can instead skip this commit with "git revert --skip".
hint: To abort and get back to the state before "git revert",
hint: run "git revert --abort".
hint: Disable this message with "git config set advice.mergeConflict false"

Details for Dev Infra team

Raised by workflow job

…orch#154437) Fixes pytorch#154157 Inductor Workflows where moved from focal to jammy here: pytorch#154153 Pull Request resolved: pytorch#154437 Approved by: https://github.com/Skylion007, https://github.com/cyyever, https://github.com/davidberard98, https://github.com/huydhn

atalman requested review from zou3519, Chillee, kshitij12345, a team and jeffdaily as code owners May 27, 2025 17:57

pytorch-bot bot added oncall: distributed Add this issue/PR to distributed oncall triage queue topic: not user facing topic category labels May 27, 2025

atalman commented May 27, 2025

View reviewed changes

Skylion007 approved these changes May 27, 2025

View reviewed changes

cyyever approved these changes May 28, 2025

View reviewed changes

ZainRizvi reviewed May 31, 2025

View reviewed changes

pytorchmergebot force-pushed the focal_jammy branch from 29019c2 to 75c6e1b Compare May 31, 2025 00:06

atalman force-pushed the focal_jammy branch 2 times, most recently from 56c6445 to 2ad7c7d Compare June 2, 2025 23:17

atalman added ciflow/trunk Trigger trunk jobs on your pull request ciflow/periodic Trigger jobs ran periodically on master (periodic.yml) on the PR ciflow/slow labels Jun 2, 2025

pytorchmergebot force-pushed the focal_jammy branch from 2ad7c7d to 011c753 Compare June 3, 2025 16:21

atalman force-pushed the focal_jammy branch from 011c753 to 21a44d7 Compare June 3, 2025 23:09

davidberard98 approved these changes Jun 4, 2025

View reviewed changes

huydhn approved these changes Jun 4, 2025

View reviewed changes

huydhn added the keep-going Don't stop on first failure, keep running tests until the end label Jun 4, 2025

atalman force-pushed the focal_jammy branch from 21a44d7 to 7d68b28 Compare June 5, 2025 12:43

atalman added 3 commits June 5, 2025 06:36

Migrate focal to jammy

adc5270

clang12

fb6016f

fix

18dc5ce

atalman force-pushed the focal_jammy branch from 7d68b28 to 18dc5ce Compare June 5, 2025 13:36

be

569ad47

pytorchmergebot added the merging label Jun 5, 2025

pytorchmergebot added the Merged label Jun 5, 2025

pytorchmergebot closed this in cd361fc Jun 5, 2025

pytorchmergebot removed the merging label Jun 5, 2025

huydhn mentioned this pull request Jun 6, 2025

Try running test_foreach sequentially #155366

Draft

[CI] Migrate focal (ubuntu 20.04) images to jammy (ubuntu 22.04) #154437

[CI] Migrate focal (ubuntu 20.04) images to jammy (ubuntu 22.04) #154437

Uh oh!

Conversation

atalman commented May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/154437

❌ 1 New Failure, 84 Pending

Uh oh!

atalman May 27, 2025

Choose a reason for hiding this comment

Uh oh!

atalman May 27, 2025

Choose a reason for hiding this comment

Uh oh!

huydhn commented May 31, 2025

Uh oh!

ZainRizvi left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pytorchmergebot commented May 31, 2025

Uh oh!

pytorchmergebot commented May 31, 2025

Uh oh!

atalman commented Jun 2, 2025

Uh oh!

atalman commented Jun 3, 2025

Uh oh!

pytorchmergebot commented Jun 3, 2025

Uh oh!

pytorchmergebot commented Jun 3, 2025

Uh oh!

huydhn left a comment

Choose a reason for hiding this comment

Uh oh!

atalman commented Jun 5, 2025

Uh oh!

atalman commented Jun 5, 2025

Uh oh!

pytorchmergebot commented Jun 5, 2025

Merge started

Uh oh!

malfet commented Jun 6, 2025

Uh oh!

pytorchmergebot commented Jun 6, 2025

Uh oh!

pytorchmergebot commented Jun 6, 2025

Reverting PR 154437 failed

Uh oh!

Uh oh!

atalman commented May 27, 2025 •

edited

Loading

pytorch-bot bot commented May 27, 2025 •

edited

Loading

ZainRizvi left a comment •

edited

Loading