[build] modernize build-backend: `setuptools.build_meta:legacy` -> `setuptools.build_meta` #155998

XuehaiPan · 2025-06-14T17:45:32Z

Stack from ghstack (oldest at bottom):

[build] bootstrap git repo for build for non-git-clone archive #157432
[build] remove upper version pin for setuptools<80.0 #156049
[build] modernize build-frontend: python setup.py develop/install -> [uv ]pip install --no-build-isolation [-e ]. #156027
-> [build] modernize build-backend: setuptools.build_meta:__legacy__ -> setuptools.build_meta #155998
[AOTI] Fix AOT inductor CMake build dependency order #157557

Change build-system.build-backend: setuptools.build_meta:__legacy__ -> setuptools.build_meta. Also, move static package info from setup.py to pyproject.toml.

Now the repo can be installed from source via pip command instead of python setup.py develop:

python -m pip install --verbose --editable .

python -m pip install --verbose --no-build-isolation --editable .

In addition, the SDist is also buildable:

python -m build --sdist
python -m install dist/torch-*.tar.gz  # build from source using SDist

Note that we should build the SDist with a fresh git clone if we will upload the output to PyPI. Because all files under third_party will be included in the SDist. The SDist file will be huge if the git submodules are initialized.

cc @seemethere @malfet @osalpekar @atalman

[ghstack-poisoned]

pytorch-bot · 2025-06-14T17:45:34Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/155998

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit fc23e9c with merge base a6fab82 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

…> `setuptools.build_meta` Change `build-system.build-backend`: `setuptools.build_meta:__legacy__` -> `setuptools.build_meta`. Also, move static package info from `setup.py` to `pyproject.toml`. Now the repo can be installed from source via `pip` command instead of `python setup.py develop`: ```bash python -m pip install --verbose --editable . python -m pip install --verbose --no-build-isolation --editable . ``` In addition, the SDist is also buildable: ```bash python -m build --sdist python -m install dist/torch-*.tar.gz # build from source using SDist ``` ghstack-source-id: 6dbd393 Pull-Request: pytorch#155998

pyproject.toml

Skylion007 · 2025-06-14T17:54:53Z

MANIFEST.in

+include WORKSPACE
+include requirements.txt
+include version.txt
+include .gitmodules


Buillding from sdist works with only .gitmodules?

I guess .gitmodules is not required because we will need to build under a git clone anyway.

One solution is to ship .gitmodules in the SDist and initialize an empty git folder to allow submodule cloning:

# in setup.py if not os.path.exists(os.path.join(cwd, '.git')): subprocesses.check_call(['git', 'init'], cwd=cwd) subprocesses.check_call(['git', 'add', '.'], cwd=cwd) subprocesses.check_call(['git', 'commit', '-m', 'init for building from sdist'], cwd=cwd) subprocesses.check_call(['git', 'submodule', 'update', '--init', '--recursive'], cwd=cwd)

The sdist should include all submodules. It looks to me like this is already included (since third_party is explicitly included) in gh-152098. .gitmodules should then not be required.

MANIFEST.in

XuehaiPan · 2025-06-14T17:57:28Z

MANIFEST.in

+graft torchgen
+# FIXME: torch-xla codegen build failure if include this file in wheel
+exclude torchgen/BUILD.bazel


torch_xla/csrc/BUILD genrule command (from target //torch_xla/csrc:gen_lazy_tensor) cannot import torchgen if this file is present in site-packages/torchgen/BUILD.bazel. cc @qihqi

The following fix will not resolve the import issue.

Fix torchgen imports in codegen xla#9310

CI failure before pytorch/xla#9310: https://hud.pytorch.org/pr/pytorch/pytorch/153538#43672158222
CI failure after pytorch/xla#9310: https://hud.pytorch.org/pr/pytorch/pytorch/153538#44066727613

…> `setuptools.build_meta` Change `build-system.build-backend`: `setuptools.build_meta:__legacy__` -> `setuptools.build_meta`. Also, move static package info from `setup.py` to `pyproject.toml`. Now the repo can be installed from source via `pip` command instead of `python setup.py develop`: ```bash python -m pip install --verbose --editable . python -m pip install --verbose --no-build-isolation --editable . ``` In addition, the SDist is also buildable: ```bash python -m build --sdist python -m install dist/torch-*.tar.gz # build from source using SDist ``` ghstack-source-id: 6dbd393 Pull-Request: pytorch#155998

Skylion007 · 2025-06-14T18:20:35Z

@XuehaiPan We are sure this doesn't break s3cache, right (with the temporary build dir)?

Skylion007 · 2025-06-14T18:21:42Z

uv also support --no-build-isolation flag in pyproject.toml (unlike pip). Maybe we should add an entry for it.

Skylion007 · 2025-06-14T18:22:58Z

pyproject.toml

@@ -38,9 +16,56 @@ requires = [
    "typing-extensions>=4.10.0",
    "requests",
 ]
-# Use legacy backend to import local packages in setup.py
-build-backend = "setuptools.build_meta:__legacy__"
+build-backend = "setuptools.build_meta"


Ultimately, this still builds under build isolation which is a big gotcha for the CI. Can we migrate to legacy scikit-build? That at least does a lot of stuff to effectively disable build isolation by default.

I'm not sure I understand this comment. It doesn't seem like a good idea to migrate to another legacy tool. Build isolation can be confusing for users when they're not that experienced, but for better or worse it is how Python packaging works. I don't see a reason to try hard to subvert that.

Scikit-build-core works too, but I'm not sure how stable the setup tools shim is for it.

Skylion007 · 2025-06-14T18:23:33Z

setup.py

-BUILD_LIBTORCH_WHL = os.getenv("BUILD_LIBTORCH_WHL", "0") == "1"
-BUILD_PYTHON_ONLY = os.getenv("BUILD_PYTHON_ONLY", "0") == "1"
-
+# Also update `project.requires-python` in pyproject.toml when changing this


It would be great if we could detect build isolation and raise a warning

I don't think it's possible, nor is it desirable - it's perfectly valid for example for a packager to build with python -m build --wheel.

malfet · 2025-07-03T16:45:16Z

@pytorchbot revert -m "Broke inductor_cpp, wrapper see https://hud.pytorch.org/hud/pytorch/pytorch/e472daa80963aae389089f9dc324b04261e2a5ef/1?per_page=50&name_filter=inductor_cpp_" -c nosignal

pytorchmergebot · 2025-07-03T16:46:53Z

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

…acy__` -> `setuptools.build_meta` (#155998)" This reverts commit 404008e. Reverted #155998 on behalf of https://github.com/malfet due to Broke inductor_cpp, wrapper see https://hud.pytorch.org/hud/pytorch/pytorch/e472daa80963aae389089f9dc324b04261e2a5ef/1?per_page=50&name_filter=inductor_cpp_ ([comment](#155998 (comment)))

pytorchmergebot · 2025-07-03T16:47:10Z

@XuehaiPan your PR has been successfully reverted.

[ghstack-poisoned]

…> `setuptools.build_meta` Change `build-system.build-backend`: `setuptools.build_meta:__legacy__` -> `setuptools.build_meta`. Also, move static package info from `setup.py` to `pyproject.toml`. Now the repo can be installed from source via `pip` command instead of `python setup.py develop`: ```bash python -m pip install --verbose --editable . python -m pip install --verbose --no-build-isolation --editable . ``` In addition, the SDist is also buildable: ```bash python -m build --sdist python -m install dist/torch-*.tar.gz # build from source using SDist ``` ghstack-source-id: f0c9fda Pull-Request: pytorch#155998

[ghstack-poisoned]

…> `setuptools.build_meta` Change `build-system.build-backend`: `setuptools.build_meta:__legacy__` -> `setuptools.build_meta`. Also, move static package info from `setup.py` to `pyproject.toml`. Now the repo can be installed from source via `pip` command instead of `python setup.py develop`: ```bash python -m pip install --verbose --editable . python -m pip install --verbose --no-build-isolation --editable . ``` In addition, the SDist is also buildable: ```bash python -m build --sdist python -m install dist/torch-*.tar.gz # build from source using SDist ``` ghstack-source-id: 9a2a960 Pull-Request: pytorch#155998

[ghstack-poisoned]

…> `setuptools.build_meta` Change `build-system.build-backend`: `setuptools.build_meta:__legacy__` -> `setuptools.build_meta`. Also, move static package info from `setup.py` to `pyproject.toml`. Now the repo can be installed from source via `pip` command instead of `python setup.py develop`: ```bash python -m pip install --verbose --editable . python -m pip install --verbose --no-build-isolation --editable . ``` In addition, the SDist is also buildable: ```bash python -m build --sdist python -m install dist/torch-*.tar.gz # build from source using SDist ``` ghstack-source-id: 58b5296 Pull-Request: pytorch#155998

[ghstack-poisoned]

…> `setuptools.build_meta` Change `build-system.build-backend`: `setuptools.build_meta:__legacy__` -> `setuptools.build_meta`. Also, move static package info from `setup.py` to `pyproject.toml`. Now the repo can be installed from source via `pip` command instead of `python setup.py develop`: ```bash python -m pip install --verbose --editable . python -m pip install --verbose --no-build-isolation --editable . ``` In addition, the SDist is also buildable: ```bash python -m build --sdist python -m install dist/torch-*.tar.gz # build from source using SDist ``` ghstack-source-id: 3f1f3b1 Pull-Request: pytorch#155998

[ghstack-poisoned]

…> `setuptools.build_meta` Change `build-system.build-backend`: `setuptools.build_meta:__legacy__` -> `setuptools.build_meta`. Also, move static package info from `setup.py` to `pyproject.toml`. Now the repo can be installed from source via `pip` command instead of `python setup.py develop`: ```bash python -m pip install --verbose --editable . python -m pip install --verbose --no-build-isolation --editable . ``` In addition, the SDist is also buildable: ```bash python -m build --sdist python -m install dist/torch-*.tar.gz # build from source using SDist ``` ghstack-source-id: 2e61e35 Pull-Request: pytorch#155998

XuehaiPan · 2025-07-04T15:47:08Z

@pytorchbot merge

pytorchmergebot · 2025-07-04T15:49:13Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

Update

79390a1

[ghstack-poisoned]

XuehaiPan mentioned this pull request Jun 14, 2025

[BE][Easy] pin setuptools>=77 for PEP-639 support #155999

Closed

pytorchbot added the open source label Jun 14, 2025

XuehaiPan requested review from seemethere, malfet, atalman, ezyang and albanD June 14, 2025 17:49

XuehaiPan self-assigned this Jun 14, 2025

Skylion007 reviewed Jun 14, 2025

View reviewed changes

pyproject.toml Show resolved Hide resolved

pyproject.toml Outdated Show resolved Hide resolved

Skylion007 reviewed Jun 14, 2025

View reviewed changes

XuehaiPan commented Jun 14, 2025

View reviewed changes

Skylion007 reviewed Jun 14, 2025

View reviewed changes

XuehaiPan mentioned this pull request Jun 15, 2025

[BE] add a minimal linter to check pyproject.toml consistency #156017

Closed

pytorchmergebot closed this in 404008e Jul 3, 2025

pytorchmergebot removed the merging label Jul 3, 2025

pytorchmergebot added Reverted ci-no-td Do not run TD on this PR labels Jul 3, 2025

pytorchmergebot reopened this Jul 3, 2025

Update

36cdde7

[ghstack-poisoned]

XuehaiPan mentioned this pull request Jul 3, 2025

[AOTI] Fix AOT inductor CMake build dependency order #157557

Closed

XuehaiPan added the ciflow/inductor label Jul 3, 2025

Update

fbd31d9

[ghstack-poisoned]

Update

973de48

[ghstack-poisoned]

Update

8b6d37a

[ghstack-poisoned]

Update

fc23e9c

[ghstack-poisoned]

pytorchmergebot added the merging label Jul 4, 2025

pytorchmergebot closed this in 524e827 Jul 4, 2025

pytorchmergebot removed the merging label Jul 4, 2025

github-actions bot deleted the gh/XuehaiPan/279/head branch August 4, 2025 02:21

[build] modernize build-backend: setuptools.build_meta:__legacy__ -> setuptools.build_meta #155998

[build] modernize build-backend: setuptools.build_meta:__legacy__ -> setuptools.build_meta #155998

Uh oh!

Conversation

XuehaiPan commented Jun 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Jun 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/155998

✅ No Failures

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

XuehaiPan Jun 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Skylion007 commented Jun 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Skylion007 commented Jun 14, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

malfet commented Jul 3, 2025

Uh oh!

pytorchmergebot commented Jul 3, 2025

Uh oh!

pytorchmergebot commented Jul 3, 2025

Uh oh!

XuehaiPan commented Jul 4, 2025

Uh oh!

pytorchmergebot commented Jul 4, 2025

Merge started

Uh oh!

Uh oh!

[build] modernize build-backend: `setuptools.build_meta:legacy` -> `setuptools.build_meta` #155998

[build] modernize build-backend: `setuptools.build_meta:legacy` -> `setuptools.build_meta` #155998

XuehaiPan commented Jun 14, 2025 •

edited

Loading

pytorch-bot bot commented Jun 14, 2025 •

edited

Loading

XuehaiPan Jun 14, 2025 •

edited

Loading

Skylion007 commented Jun 14, 2025 •

edited

Loading