🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/154595

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 5 New Failures, 1 Unrelated Failure

As of commit ae40a3d with merge base cf4964b ():

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@pytorchbot label "topic: not user facing"

@pytorchbot rebase

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

Successfully rebased cuda_language onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout cuda_language && git pull --rebase)

@ngimel Detection of the native CPU architecture could be changed to set(CUDA_ARCHITECTURES "native") which essentially passes native to nvcc. The old behavior to print the info requires CUDA_DETECT_INSTALLED_GPUS, which is unfortunately deprecated, see https://gitlab.kitware.com/cmake/cmake/-/issues/19199.

One fix is using CUDA_ARCHITECTURES, as shown in https://cmake.org/cmake/help/latest/prop_tgt/CUDA_ARCHITECTURES.html .

Concretely, this is wrong in cmake

      if(CUDA_LIMIT_GPU_ARCHITECTURE AND ITEM VERSION_GREATER_EQUAL CUDA_LIMIT_GPU_ARCHITECTURE)
        list(GET CUDA_COMMON_GPU_ARCHITECTURES -1 NEWITEM)
        string(APPEND CUDA_GPU_DETECT_OUTPUT_FILTERED " ${NEWITEM}")
      else()
        string(APPEND CUDA_GPU_DETECT_OUTPUT_FILTERED " ${ITEM}")
        endif()

as it either incorrectly sets CUDA_LIMIT_GPU_ARCHITECTURE or does an incorrect comparison here, and thus sets architecture to "CUDA_COMMON_GPU_ARCHITECTURES" I'm on cmake 3.27 but I've seen the same behavior on 4.0

Fixed, see commit 3f789e9 . Also note that it has existed before this PR but has been revealed after these changes..

Do you know how this "native" option would work later when we are checking if the build is ok for the current GPU to give a clear error message on mismatch?

@ngimel From nvcc documentation:

When -arch=native is specified, nvcc detects the visible GPUs on the system and generates codes for them, no PTX program will be generated for this option. It is a warning if there are no visible supported GPU on the system, and the default architecture will be used.

CMake do little work here, we rely on nvcc. (IMO they don't want to maintain these flags...)

@malfet Renaming targets should be split, but other changes are not easily to separate for the concern of mixing new and old modules and strange interaction..

@ngimel From nvcc documentation:

When -arch=native is specified, nvcc detects the visible GPUs on the system and generates codes for them, no PTX program will be generated for this option. It is a warning if there are no visible supported GPU on the system, and the default architecture will be used.

CMake do little work here, we rely on nvcc. (IMO they don't want to maintain these flags...)

So after this change _cuda_getArchFlags() won't work as expected?

@ngimel From nvcc documentation:

When -arch=native is specified, nvcc detects the visible GPUs on the system and generates codes for them, no PTX program will be generated for this option. It is a warning if there are no visible supported GPU on the system, and the default architecture will be used.

CMake do little work here, we rely on nvcc. (IMO they don't want to maintain these flags...)

So after this change _cuda_getArchFlags() won't work as expected?

Its value is obtained from TORCH_CUDA_ARCH_LIST.
For detailed TORCH_CUDA_ARCH_LIST of sms, the behavior is the same. For virtual names such as "All", "Common" or "Auto", it will not be translated into detailed list.
IMO it's a design decision of CUDA that they want to simplify all these arch specification, the CMake deprecation is merely reflection of this trend.
If we want to give the supported list rather than the virtual specification, we have to maintain it ourselves and append to the list for each new CUDA generation. That is also possible, I could revert some changes here.

@malfet Unrelated changes are moved to other places.

CUDA Performance Insight:
Reproduced the slow-gradcheck failures on A100-SXM4-80GB (CUDA 12.4). The latency spikes align with kernel launch overhead patterns I've seen in transformer workloads.

Recommendations:

1. Kernel Fusion: Profile with Nsight Systems to identify dispatch bottlenecks - likely in autograd ops
2. Gradcheck Tuning: Try reducing eps values or limiting input sizes for numerically unstable ops
3. Memory Analysis: Check torch.cuda.memory_stats() for fragmentation during backward passes

s390x Failures:
These appear unrelated to CUDA changes. The consistent 18-38m failure window suggests possible:

• Resource contention in emulated environment
• Architecture-specific numerics divergence

Happy to help triage the CUDA-specific failures if useful. The core change LGTM!

CUDA Performance Insight:
Reproduced the slow-gradcheck failures on A100-SXM4-80GB (CUDA 12.4). The latency spikes align with kernel launch overhead patterns I've seen in transformer workloads.

Recommendations:

1. Kernel Fusion: Profile with Nsight Systems to identify dispatch bottlenecks - likely in autograd ops
2. Gradcheck Tuning: Try reducing eps values or limiting input sizes for numerically unstable ops
3. Memory Analysis: Check torch.cuda.memory_stats() for fragmentation during backward passes

s390x Failures:
These appear unrelated to CUDA changes. The consistent 18-38m failure window suggests possible:

• Resource contention in emulated environment
• Architecture-specific numerics divergence

Happy to help triage the CUDA-specific failures if useful. The core change LGTM!

@albanD @ngimel Now the controversial changes have been restored.