[BE]: Fix NVSHMEM builds, add missing 12.9 dependency and update to latest for 2.8RC #157453

Skylion007 · 2025-07-02T14:12:40Z

Fixed our bad builds of nvshmem, (we were not building or testing before) and also updates to the latest version. Newest versions has critical support for things that would actually make it useful, like bfloat16 and float16 support.

This is a proper fix for: #157411

pytorch-bot · 2025-07-02T14:12:43Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/157453

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit c2d6d7b with merge base 82eefae ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

Skylion007 · 2025-07-02T14:43:05Z

Great Ubuntu is having an outage

atalman · 2025-07-02T15:00:41Z

cc @kwen2501 @ptrblck @nWEIdia @tinglvv @malfet

kwen2501

Thank you so much! LGTM!

kwen2501 · 2025-07-02T17:25:41Z

.ci/docker/common/install_cuda.sh

+  cp -a "libnvshmem/include/"* /usr/local/include/
+  cp -a "libnvshmem/lib/"*     /usr/local/lib/


Oh I was previously detecting NVSHMEM lib from python/site-packages/nvidia.
Maybe I would need to change how we set NVSHMEM_HOME based on this change?

It should be detected either way here? We do use NVSHMEM from the python package anyway at the end for non-static builds.

This is how we currently detect it for building:

pytorch/tools/setup_helpers/cmake.py

Lines 316 to 320 in d5d14ee

# Detect build dependencies from python lib path (in order to set *_HOME variables)

# NVSHMEM

nvshmem_home = py_lib_path + "/nvidia/nvshmem"

if os.path.exists(nvshmem_home):

build_options["NVSHMEM_HOME"] = nvshmem_home

And there is a gate here that stops building torch with NVSHMEM if NVSHMEM_HOME is not set:

pytorch/caffe2/CMakeLists.txt

Lines 998 to 1002 in 0105cd8

if(NOT DEFINED NVSHMEM_HOME)

message(WARNING "NVSHMEM_HOME not found. Please run `pip install nvidia-nvshmem-<version>`, or set NVSHMEM_HOME to its location.")

# Disable nvshmem if NVSHMEM_HOME is not found

set(USE_NVSHMEM FALSE)

endif()

I see two solutions:
(1) We add export NVSHMEM_HOME=/usr/local here, but it looks a bit weird;
(2) We add a findNVSHMEM.cmake as a single place to detect the library and set NVSHMEM_HOME there.

What do you think?

I'd say a findNVSHMEM.cmake

The pip version is fine too obviously, but it didn't seem to work for you other build?

So libnvshmem comes with libnvshmem/lib/cmake/nvshmem/NVSHMEMConfig.cmake, but it doesn't set the NVSHMEM home.

@kwen2501 This more complicated as our current CMake is broken when we build with NVSHMEM because it tries to build for unsupported architectures.

@kwen2501 Also libtorch will be missing it now, right?

Correct. So we'd need to find a way to work for both. (pip install or system install)

@kwen2501 Proper way is just do CMake. It will work pip install thanks the to the rpath changes we made.

Thanks. I added CMake search here: #157513.
Do you mind taking a look? Thank you!

atalman · 2025-07-03T18:00:26Z

@pytorchmergebot merge

pytorchmergebot · 2025-07-03T18:02:15Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

Skylion007 · 2025-07-03T19:44:43Z

@atalman This is needed first as it's not actually able to find the installed NVSHMEM currently: #157513

…ation" Previously we only search for NVSHMEM from pip install location. This PR adds search in system locations deemed default by CMake. Related: #157453 untars NVSHMEM into `/usr/local` on our CI machines. cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k Skylion007 [ghstack-poisoned]

Previously we only search for NVSHMEM from pip install location. This PR adds search in system locations deemed default by CMake. Related: #157453 untars NVSHMEM into `/usr/local` on our CI machines. cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k Skylion007 [ghstack-poisoned]

Previously we only search for NVSHMEM from pip install location. This PR adds search in system locations deemed default by CMake. Related: #157453 untars NVSHMEM into `/usr/local` on our CI machines. Pull Request resolved: #157513 Approved by: https://github.com/atalman, https://github.com/Skylion007

…ation" Previously we only search for NVSHMEM from pip install location. This PR adds search in system locations deemed default by CMake. Related: #157453 untars NVSHMEM into `/usr/local` on our CI machines. cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k Skylion007 [ghstack-poisoned]

Previously we only search for NVSHMEM from pip install location. This PR adds search in system locations deemed default by CMake. Related: #157453 untars NVSHMEM into `/usr/local` on our CI machines. cc H-Huang awgu wanchaol fegin fduwjj wz337 wconstab d4l3k Skylion007 [ghstack-poisoned]

Previously we only search for NVSHMEM from pip install location. This PR adds search in system locations deemed default by CMake. Related: #157453 untars NVSHMEM into `/usr/local` on our CI machines. Pull Request resolved: #157513 Approved by: https://github.com/atalman, https://github.com/Skylion007 (cherry picked from commit 99c1a6b)

kwen2501 · 2025-07-08T05:56:21Z

@pytorchbot cherry-pick --onto release/2.8 -c critical

…atest for 2.8RC (#157453) Fixed our bad builds of nvshmem, (we were not building or testing before) and also updates to the latest version. Newest versions has critical support for things that would actually make it useful, like bfloat16 and float16 support. This is a proper fix for: #157411 Pull Request resolved: #157453 Approved by: https://github.com/kwen2501, https://github.com/atalman (cherry picked from commit a6fab82)

pytorchbot · 2025-07-08T06:01:10Z

Cherry picking #157453

The cherry pick PR is at #157774 and it is recommended to link a critical cherry pick PR with an issue. The following tracker issues are updated:

[v.2.8.0] Release Tracker #156745 (comment)

Details for Dev Infra team

Raised by workflow job

Skylion007 requested review from eqy, atalman and nWEIdia July 2, 2025 14:12

Skylion007 requested review from a team and jeffdaily as code owners July 2, 2025 14:12

pytorch-bot bot added the topic: not user facing topic category label Jul 2, 2025

pytorchbot added the open source label Jul 2, 2025

Skylion007 force-pushed the skylion007/update-nvshmem-3-3-9 branch from 8119930 to 99dabab Compare July 2, 2025 14:17

Skylion007 changed the title ~~[BE]: Fix NVSHMEM builds and update to latest~~ [BE]: Fix NVSHMEM builds, add missing 12.9 dependency and update to latest Jul 2, 2025

Skylion007 force-pushed the skylion007/update-nvshmem-3-3-9 branch 3 times, most recently from c76ed71 to 1b12f69 Compare July 2, 2025 14:32

Skylion007 requested review from malfet and tinglvv July 2, 2025 14:39

Skylion007 added the better-engineering Relatively self-contained tasks for better engineering contributors label Jul 2, 2025

Skylion007 force-pushed the skylion007/update-nvshmem-3-3-9 branch from 1b12f69 to f8899ce Compare July 2, 2025 14:46

Skylion007 changed the title ~~[BE]: Fix NVSHMEM builds, add missing 12.9 dependency and update to latest~~ [BE]: Fix NVSHMEM builds, add missing 12.9 dependency and update to latest for 2.8RC Jul 2, 2025

Skylion007 added this to the 2.8.0 milestone Jul 2, 2025

Skylion007 force-pushed the skylion007/update-nvshmem-3-3-9 branch from f8899ce to c84bbb0 Compare July 2, 2025 14:56

atalman added the ciflow/binaries Trigger all binary build and upload jobs on the PR label Jul 2, 2025

[BE]: Fix NVSHMEM builds and update to latest

c2d6d7b

Skylion007 force-pushed the skylion007/update-nvshmem-3-3-9 branch from c84bbb0 to c2d6d7b Compare July 2, 2025 16:30

kwen2501 approved these changes Jul 2, 2025

View reviewed changes

kwen2501 reviewed Jul 2, 2025

View reviewed changes

This was referenced Jul 2, 2025

[SymmMem] Install NVSHMEM wheel in CI docker #157411

Open

[SymmMem] Find NVSHMEM from system installation #157513

Closed

atalman approved these changes Jul 2, 2025

View reviewed changes

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Jul 3, 2025

pytorchmergebot added the merging label Jul 3, 2025

pytorchmergebot added the Merged label Jul 3, 2025

pytorchmergebot closed this in a6fab82 Jul 3, 2025

pytorchmergebot removed the merging label Jul 3, 2025

pytorchbot mentioned this pull request Jul 8, 2025

[SymmMem] Find NVSHMEM from system installation #157755

Open

pytorchbot mentioned this pull request Jul 8, 2025

[v.2.8.0] Release Tracker #156745

Closed

atalman removed this from the 2.8.0 milestone Jul 11, 2025

		cp -a "libnvshmem/include/"* /usr/local/include/
		cp -a "libnvshmem/lib/"* /usr/local/lib/

	# Detect build dependencies from python lib path (in order to set *_HOME variables)
	# NVSHMEM
	nvshmem_home = py_lib_path + "/nvidia/nvshmem"
	if os.path.exists(nvshmem_home):
	build_options["NVSHMEM_HOME"] = nvshmem_home

	if(NOT DEFINED NVSHMEM_HOME)
	message(WARNING "NVSHMEM_HOME not found. Please run `pip install nvidia-nvshmem-<version>`, or set NVSHMEM_HOME to its location.")
	# Disable nvshmem if NVSHMEM_HOME is not found
	set(USE_NVSHMEM FALSE)
	endif()

[BE]: Fix NVSHMEM builds, add missing 12.9 dependency and update to latest for 2.8RC #157453

[BE]: Fix NVSHMEM builds, add missing 12.9 dependency and update to latest for 2.8RC #157453

Uh oh!

Conversation

Skylion007 commented Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/157453

✅ No Failures

Uh oh!

Skylion007 commented Jul 2, 2025

Uh oh!

atalman commented Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kwen2501 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Skylion007 Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kwen2501 Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

atalman commented Jul 3, 2025

Uh oh!

pytorchmergebot commented Jul 3, 2025

Merge started

Uh oh!

Skylion007 commented Jul 3, 2025

Uh oh!

kwen2501 commented Jul 8, 2025

Uh oh!

pytorchbot commented Jul 8, 2025

Cherry picking #157453

Uh oh!

Uh oh!

Skylion007 commented Jul 2, 2025 •

edited

Loading

pytorch-bot bot commented Jul 2, 2025 •

edited

Loading

atalman commented Jul 2, 2025 •

edited

Loading

Skylion007 Jul 2, 2025 •

edited

Loading

kwen2501 Jul 2, 2025 •

edited

Loading