Skip to content

Commit 70174b6

Browse files
ezyangezyang
committed
Update
[ghstack-poisoned]
2 parents ddf14a1 + e9c3490 commit 70174b6

File tree

2,466 files changed

+103131
-93197
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

2,466 files changed

+103131
-93197
lines changed

‎.bazelrc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@ build --cxxopt=--std=c++17
22
build --copt=-I.
33
# Bazel does not support including its cc_library targets as system
44
# headers. We work around this for generated code
5-
# (e.g. c10/macros/cmake_macros.h) by making the generated directory a
5+
# (e.g. torch/headeronly/macros/cmake_macros.h) by making the generated directory a
66
# system include path.
77
build --copt=-isystem --copt bazel-out/k8-fastbuild/bin
88
build --copt=-isystem --copt bazel-out/darwin-fastbuild/bin

‎.ci/aarch64_linux/aarch64_ci_build.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ set -eux -o pipefail
44
GPU_ARCH_VERSION=${GPU_ARCH_VERSION:-}
55

66
if [[ "$GPU_ARCH_VERSION" == *"12.9"* ]]; then
7-
export TORCH_CUDA_ARCH_LIST="9.0;10.0;12.0"
7+
export TORCH_CUDA_ARCH_LIST="8.0;9.0;10.0;12.0"
88
fi
99

1010
SCRIPTPATH="$( cd -- "$(dirname "$0")" >/dev/null 2>&1 ; pwd -P )"

‎.ci/aarch64_linux/build_aarch64_wheel.py

Lines changed: 4 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -438,9 +438,7 @@ def build_torchvision(
438438
)
439439
build_vars += f"BUILD_VERSION={version}.dev{build_date}"
440440
elif build_version is not None:
441-
build_vars += (
442-
f"BUILD_VERSION={build_version} PYTORCH_VERSION={branch[1:].split('-')[0]}"
443-
)
441+
build_vars += f"BUILD_VERSION={build_version} PYTORCH_VERSION={branch[1:].split('-', maxsplit=1)[0]}"
444442
if host.using_docker():
445443
build_vars += " CMAKE_SHARED_LINKER_FLAGS=-Wl,-z,max-page-size=0x10000"
446444

@@ -495,9 +493,7 @@ def build_torchdata(
495493
)
496494
build_vars += f"BUILD_VERSION={version}.dev{build_date}"
497495
elif build_version is not None:
498-
build_vars += (
499-
f"BUILD_VERSION={build_version} PYTORCH_VERSION={branch[1:].split('-')[0]}"
500-
)
496+
build_vars += f"BUILD_VERSION={build_version} PYTORCH_VERSION={branch[1:].split('-', maxsplit=1)[0]}"
501497
if host.using_docker():
502498
build_vars += " CMAKE_SHARED_LINKER_FLAGS=-Wl,-z,max-page-size=0x10000"
503499

@@ -553,9 +549,7 @@ def build_torchtext(
553549
)
554550
build_vars += f"BUILD_VERSION={version}.dev{build_date}"
555551
elif build_version is not None:
556-
build_vars += (
557-
f"BUILD_VERSION={build_version} PYTORCH_VERSION={branch[1:].split('-')[0]}"
558-
)
552+
build_vars += f"BUILD_VERSION={build_version} PYTORCH_VERSION={branch[1:].split('-', maxsplit=1)[0]}"
559553
if host.using_docker():
560554
build_vars += " CMAKE_SHARED_LINKER_FLAGS=-Wl,-z,max-page-size=0x10000"
561555

@@ -613,9 +607,7 @@ def build_torchaudio(
613607
)
614608
build_vars += f"BUILD_VERSION={version}.dev{build_date}"
615609
elif build_version is not None:
616-
build_vars += (
617-
f"BUILD_VERSION={build_version} PYTORCH_VERSION={branch[1:].split('-')[0]}"
618-
)
610+
build_vars += f"BUILD_VERSION={build_version} PYTORCH_VERSION={branch[1:].split('-', maxsplit=1)[0]}"
619611
if host.using_docker():
620612
build_vars += " CMAKE_SHARED_LINKER_FLAGS=-Wl,-z,max-page-size=0x10000"
621613

‎.ci/caffe2/test.sh

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ source "$(dirname "${BASH_SOURCE[0]}")/common.sh"
55

66
if [[ ${BUILD_ENVIRONMENT} == *onnx* ]]; then
77
pip install click mock tabulate networkx==2.0
8-
pip -q install --user "file:///var/lib/jenkins/workspace/third_party/onnx#egg=onnx"
8+
pip -q install "file:///var/lib/jenkins/workspace/third_party/onnx#egg=onnx"
99
fi
1010

1111
# Skip tests in environments where they are not built/applicable
@@ -147,8 +147,8 @@ export DNNL_MAX_CPU_ISA=AVX2
147147
if [[ "${SHARD_NUMBER:-1}" == "1" ]]; then
148148
# TODO(sdym@meta.com) remove this when the linked issue resolved.
149149
# py is temporary until https://github.com/Teemu/pytest-sugar/issues/241 is fixed
150-
pip install --user py==1.11.0
151-
pip install --user pytest-sugar
150+
pip install py==1.11.0
151+
pip install pytest-sugar
152152
# NB: Warnings are disabled because they make it harder to see what
153153
# the actual erroring test is
154154
"$PYTHON" \

‎.ci/docker/README.md

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,3 +36,104 @@ See `build.sh` for valid build environments (it's the giant switch).
3636
# Set flags (see build.sh) and build image
3737
sudo bash -c 'TRITON=1 ./build.sh pytorch-linux-bionic-py3.8-gcc9 -t myimage:latest
3838
```
39+
40+
## [Guidance] Adding a New Base Docker Image
41+
42+
### Background
43+
44+
The base Docker images in directory `.ci/docker/` are built by the `docker-builds.yml` workflow. Those images are used throughout the PyTorch CI/CD pipeline. You should only create or modify a base Docker image if you need specific environment changes or dependencies before building PyTorch on CI.
45+
46+
1. **Automatic Rebuilding**:
47+
- The Docker image building process is triggered automatically when changes are made to files in the `.ci/docker/*` directory
48+
- This ensures all images stay up-to-date with the latest dependencies and configurations
49+
50+
2. **Image Reuse in PyTorch Build Workflows** (example: linux-build):
51+
- The images generated by `docker-builds.yml` are reused in `_linux-build.yml` through the `calculate-docker-image` step
52+
- The `_linux-build.yml` workflow:
53+
- Pulls the Docker image determined by the `calculate-docker-image` step
54+
- Runs a Docker container with that image
55+
- Executes `.ci/pytorch/build.sh` inside the container to build PyTorch
56+
57+
3. **Usage in Test Workflows** (example: linux-test):
58+
- The same Docker images are also used in `_linux-test.yml` for running tests
59+
- The `_linux-test.yml` workflow follows a similar pattern:
60+
- It uses the `calculate-docker-image` step to determine which Docker image to use
61+
- It pulls the Docker image and runs a container with that image
62+
- It installs the wheels from the artifacts generated by PyTorch build jobs
63+
- It executes test scripts (like `.ci/pytorch/test.sh` or `.ci/pytorch/multigpu-test.sh`) inside the container
64+
65+
### Understanding File Purposes
66+
67+
#### `.ci/docker/build.sh` vs `.ci/pytorch/build.sh`
68+
- **`.ci/docker/build.sh`**:
69+
- Used for building base Docker images
70+
- Executed by the `docker-builds.yml` workflow to pre-build Docker images for CI
71+
- Contains configurations for different Docker build environments
72+
73+
- **`.ci/pytorch/build.sh`**:
74+
- Used for building PyTorch inside a Docker container
75+
- Called by workflows like `_linux-build.yml` after the Docker container is started
76+
- Builds PyTorch wheels and other artifacts
77+
78+
#### `.ci/docker/ci_commit_pins/` vs `.github/ci_commit_pins`
79+
- **`.ci/docker/ci_commit_pins/`**:
80+
- Used for pinning dependency versions during base Docker image building
81+
- Ensures consistent environments for building PyTorch
82+
- Changes here trigger base Docker image rebuilds
83+
84+
- **`.github/ci_commit_pins`**:
85+
- Used for pinning dependency versions during PyTorch building and tests
86+
- Ensures consistent dependencies for PyTorch across different builds
87+
- Used by build scripts running inside Docker containers
88+
89+
### Step-by-Step Guide for Adding a New Base Docker Image
90+
91+
#### 1. Add Pinned Commits (If Applicable)
92+
93+
We use pinned commits for build stability. The `nightly.yml` workflow checks and updates pinned commits for certain repository dependencies daily.
94+
95+
If your new Docker image needs a library installed from a specific pinned commit or built from source:
96+
97+
1. Add the repository you want to track in `nightly.yml` and `merge-rules.yml`
98+
2. Add the initial pinned commit in `.ci/docker/ci_commit_pins/`. The text filename should match the one defined in step 1
99+
100+
#### 2. Configure the Base Docker Image
101+
1. **Add new Base Docker image configuration** (if applicable):
102+
103+
Add the configuration in `.ci/docker/build.sh`. For example:
104+
```bash
105+
pytorch-linux-jammy-cuda12.8-cudnn9-py3.12-gcc11-new1)
106+
CUDA_VERSION=12.8.1
107+
ANACONDA_PYTHON_VERSION=3.12
108+
GCC_VERSION=11
109+
VISION=yes
110+
KATEX=yes
111+
UCX_COMMIT=${_UCX_COMMIT}
112+
UCC_COMMIT=${_UCC_COMMIT}
113+
TRITON=yes
114+
NEW_ARG_1=yes
115+
;;
116+
```
117+
118+
2. **Add build arguments to Docker build command**:
119+
120+
If you're introducing a new argument to the Docker build, make sure to add it in the Docker build step in `.ci/docker/build.sh`:
121+
```bash
122+
docker build \
123+
....
124+
--build-arg "NEW_ARG_1=${NEW_ARG_1}"
125+
```
126+
127+
3. **Update Dockerfile logic**:
128+
129+
Update the Dockerfile to use the new argument. For example, in `ubuntu/Dockerfile`:
130+
```dockerfile
131+
ARG NEW_ARG_1
132+
# Set up environment for NEW_ARG_1
133+
RUN if [ -n "${NEW_ARG_1}" ]; then bash ./do_something.sh; fi
134+
```
135+
136+
4. **Add the Docker configuration** in `.github/workflows/docker-builds.yml`:
137+
138+
The `docker-builds.yml` workflow pre-builds the Docker images whenever changes occur in the `.ci/docker/` directory. This includes the
139+
pinned commit updates.

0 commit comments

Comments
 (0)