Make device check error message more descriptive #150750

zeshengzong · 2025-04-07T03:48:44Z

Test Result

import torch

model_output = torch.randn(10, 5).cuda()
labels = torch.randint(0, 5, (10,)).cuda()
weights = torch.randn(5)

loss_fn = torch.nn.CrossEntropyLoss(weight=weights)
loss = loss_fn(input=model_output, target=labels)
print(loss)


Traceback (most recent call last):
  File "/home/zong/code/pytorch/../loss2.py", line 17, in <module>
    loss = loss_fn(input=model_output, target=labels)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zong/code/pytorch/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zong/code/pytorch/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zong/code/pytorch/torch/nn/modules/loss.py", line 1297, in forward
    return F.cross_entropy(
           ^^^^^^^^^^^^^^^^
  File "/home/zong/code/pytorch/torch/nn/functional.py", line 3494, in cross_entropy
    return torch._C._nn.cross_entropy_loss(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Expected all tensors to be on the same device, but got weight is on cpu, different from other tensors on cuda:0 (when checking argument in method wrapper_CUDA_nll_loss_forward)

pytorch-bot · 2025-04-07T03:48:47Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/150750

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 6ab42b7 with merge base dfcfad2 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

zeshengzong · 2025-04-07T03:50:19Z

@pytorchbot label "topic: not user facing"

zeshengzong · 2025-05-02T16:14:22Z

Hello @mikaylagawarecki, please help review the change, thanks!

zeshengzong · 2025-05-02T16:30:34Z

@pytorchbot merge

pytorchmergebot · 2025-05-02T16:32:31Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2025-05-02T18:39:44Z

Merge failed

Reason: 1 jobs have failed, first few of them are: trunk / macos-py3-arm64-mps / test (mps, 1, 1, macos-m2-15)

Details for Dev Infra team

Raised by workflow job

zeshengzong · 2025-05-06T02:25:51Z

@pytorchbot merge

pytorchmergebot · 2025-05-06T02:27:40Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

huydhn · 2025-05-06T06:40:19Z

@pytorchbot revert -m 'Sorry for reverting your change but it seems to cause a test to fail in trunk' -c nosignal

test_sparse.py::TestSparseOneOff::test_cuda_from_cpu GH job link HUD commit link

pytorchmergebot · 2025-05-06T06:41:59Z

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

This reverts commit 8253970. Reverted #150750 on behalf of https://github.com/huydhn due to Sorry for reverting your change but it seems to cause a test to fail in trunk ([comment](#150750 (comment)))

pytorchmergebot · 2025-05-06T06:42:11Z

@zeshengzong your PR has been successfully reverted.

This PR was reopened (likely due to being reverted), so your approval was removed. Please request another review.

zeshengzong · 2025-05-06T11:58:43Z

Fixed test, please review the change, thanks!

malfet · 2025-05-06T14:59:25Z

@pytorchbot merge

pytorchmergebot · 2025-05-06T15:02:41Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2025-05-06T15:08:07Z

Merge failed

Reason: 2 mandatory check(s) failed. The first few are:

Dig deeper by viewing the failures on hud

Details for Dev Infra team

Raised by workflow job

Failing merge rule: Core Maintainers

pytorch-bot · 2025-05-07T02:40:49Z

You don't have permissions to rebase this PR since you are a first time contributor. If you think this is a mistake, please contact PyTorch Dev Infra.

zeshengzong · 2025-05-07T02:56:49Z

@pytorchbot rebase -b main

pytorchmergebot · 2025-05-07T02:58:25Z

@pytorchbot started a rebase job onto refs/remotes/origin/main. Check the current status here

pytorchmergebot · 2025-05-07T02:58:28Z

Successfully rebased opt/aten/device_check onto refs/remotes/origin/main, please pull locally before adding more changes (for example, via git checkout opt/aten/device_check && git pull --rebase)

zeshengzong · 2025-05-08T03:07:25Z

@pytorchbot merge

pytorchmergebot · 2025-05-08T03:09:26Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

Fixes #122757 The fix is lost after revert and rebase previous PR #150750 (only change of tests are merged). ## Test Result ```python >>> import torch >>> >>> model_output = torch.randn(10, 5).cuda() >>> labels = torch.randint(0, 5, (10,)).cuda() >>> weights = torch.randn(5) >>> >>> loss_fn = torch.nn.CrossEntropyLoss(weight=weights) >>> loss = loss_fn(input=model_output, target=labels) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/zong/code/pytorch/torch/nn/modules/module.py", line 1767, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/zong/code/pytorch/torch/nn/modules/module.py", line 1778, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/zong/code/pytorch/torch/nn/modules/loss.py", line 1297, in forward return F.cross_entropy( ^^^^^^^^^^^^^^^^ File "/home/zong/code/pytorch/torch/nn/functional.py", line 3476, in cross_entropy return torch._C._nn.cross_entropy_loss( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: Expected all tensors to be on the same device, but got weight is on cpu, different from other tensors on cuda:0 (when checking argument in method wrapper_CUDA_nll_loss_forward) ``` Pull Request resolved: #155085 Approved by: https://github.com/mikaylagawarecki

pytorch-bot bot added the topic: not user facing topic category label Apr 7, 2025

pytorchbot added the open source label Apr 7, 2025

zeshengzong marked this pull request as ready for review April 7, 2025 10:57

bdhirsh requested a review from mikaylagawarecki April 8, 2025 20:54

bdhirsh added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Apr 8, 2025

mikaylagawarecki previously approved these changes May 2, 2025

View reviewed changes

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label May 2, 2025

pytorchmergebot added the merging label May 2, 2025

pytorchmergebot removed the merging label May 2, 2025

pytorchmergebot added the merging label May 6, 2025

pytorchmergebot closed this in 8253970 May 6, 2025

pytorchmergebot added Merged and removed merging labels May 6, 2025

pytorchmergebot added Reverted ci-no-td Do not run TD on this PR labels May 6, 2025

pytorchmergebot reopened this May 6, 2025

pytorch-bot bot removed the ciflow/trunk Trigger trunk jobs on your pull request label May 6, 2025

malfet approved these changes May 6, 2025

View reviewed changes

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label May 6, 2025

pytorchmergebot added the merging label May 6, 2025

pytorchmergebot removed the merging label May 6, 2025

Update

6ab42b7

pytorchmergebot force-pushed the opt/aten/device_check branch from 6a49d61 to 6ab42b7 Compare May 7, 2025 02:58

pytorch-bot bot removed the ciflow/trunk Trigger trunk jobs on your pull request label May 7, 2025

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label May 8, 2025

pytorchmergebot added the merging label May 8, 2025

pytorchmergebot closed this in 3c87529 May 8, 2025

pytorchmergebot removed the merging label May 8, 2025

zeshengzong mentioned this pull request Jun 4, 2025

Make device check throw specific error #155085

Closed

Make device check error message more descriptive #150750

Make device check error message more descriptive #150750

Uh oh!

Conversation

zeshengzong commented Apr 7, 2025

Test Result

Uh oh!

pytorch-bot bot commented Apr 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/150750

✅ No Failures

Uh oh!

zeshengzong commented Apr 7, 2025

Uh oh!

zeshengzong commented May 2, 2025

Uh oh!

zeshengzong commented May 2, 2025

Uh oh!

pytorchmergebot commented May 2, 2025

Merge started

Uh oh!

pytorchmergebot commented May 2, 2025

Merge failed

Uh oh!

zeshengzong commented May 6, 2025

Uh oh!

pytorchmergebot commented May 6, 2025

Merge started

Uh oh!

huydhn commented May 6, 2025

Uh oh!

pytorchmergebot commented May 6, 2025

Uh oh!

pytorchmergebot commented May 6, 2025

Uh oh!

zeshengzong commented May 6, 2025

Uh oh!

malfet commented May 6, 2025

Uh oh!

pytorchmergebot commented May 6, 2025

Merge started

Uh oh!

pytorchmergebot commented May 6, 2025

Merge failed

Uh oh!

pytorch-bot bot commented May 7, 2025

Uh oh!

zeshengzong commented May 7, 2025

Uh oh!

pytorchmergebot commented May 7, 2025

Uh oh!

pytorchmergebot commented May 7, 2025

Uh oh!

zeshengzong commented May 8, 2025

Uh oh!

pytorchmergebot commented May 8, 2025

Merge started

Uh oh!

Uh oh!

pytorch-bot bot commented Apr 7, 2025 •

edited

Loading