[C10D] Add check_rng_sync util #160283

wconstab · 2025-08-10T22:23:32Z

Stack from ghstack (oldest at bottom):

-> [C10D] Add check_rng_sync util #160283

Debugs RNG desync by checking the current state on each rank in the group and summarizing the differences if any are detected.

Notes:

used allgather instead of gather since its simpler to do this SPMD rather than add conditional behavior, though I could be convinced we only want to log on rank0.

Usage:
check_rng_sync(generator, group)

Prints something like this:

(cuda):

[rank0]:E0808 ] Generator desync detected:
[rank0]:E0808 ] Ranks    (Seed, Offset) values
[rank0]:E0808 ] -------  -----------------------
[rank0]:E0808 ] 0        (456, 0)
[rank0]:E0808 ] 1        (123, 4)
[rank0]:E0808 ] 2-3      (123, 0)

(cpu):

[rank2]:E0810 ] Generator desync detected:
[rank2]:E0810 ] Ranks      Generator State Hash values
[rank2]:E0810 ] -------  -----------------------------
[rank2]:E0810 ] 0                  7633364531954955665
[rank2]:E0810 ] 1                  8807615394212033278
[rank2]:E0810 ] 2-3               -6150027303226666531

cc @H-Huang @awgu @wanchaol @fegin @fduwjj @wz337 @d4l3k @pragupta

Debugs RNG desync by checking the current state on each rank in the group and summarizing the differences if any are detected. Notes: - used allgather instead of gather since its simpler to do this SPMD rather than add conditional behavior, though I could be convinced we only want to log on rank0. Usage: `check_rng_sync(generator, group)` Prints something like this: (cuda): ``` [rank0]:E0808 ] Generator desync detected: [rank0]:E0808 ] Ranks (Seed, Offset) values [rank0]:E0808 ] ------- ----------------------- [rank0]:E0808 ] 0 (456, 0) [rank0]:E0808 ] 1 (123, 4) [rank0]:E0808 ] 2-3 (123, 0) ``` (cpu): ``` [rank2]:E0810 ] Generator desync detected: [rank2]:E0810 ] Ranks Generator State Hash values [rank2]:E0810 ] ------- ----------------------------- [rank2]:E0810 ] 0 7633364531954955665 [rank2]:E0810 ] 1 8807615394212033278 [rank2]:E0810 ] 2-3 -6150027303226666531 ``` [ghstack-poisoned]

pytorch-bot · 2025-08-10T22:23:36Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/160283

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 225a97f with merge base 556e2a7 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

Debugs RNG desync by checking the current state on each rank in the group and summarizing the differences if any are detected. Notes: - used allgather instead of gather since its simpler to do this SPMD rather than add conditional behavior, though I could be convinced we only want to log on rank0. Usage: `check_rng_sync(generator, group)` Prints something like this: (cuda): ``` [rank0]:E0808 ] Generator desync detected: [rank0]:E0808 ] Ranks (Seed, Offset) values [rank0]:E0808 ] ------- ----------------------- [rank0]:E0808 ] 0 (456, 0) [rank0]:E0808 ] 1 (123, 4) [rank0]:E0808 ] 2-3 (123, 0) ``` (cpu): ``` [rank2]:E0810 ] Generator desync detected: [rank2]:E0810 ] Ranks Generator State Hash values [rank2]:E0810 ] ------- ----------------------------- [rank2]:E0810 ] 0 7633364531954955665 [rank2]:E0810 ] 1 8807615394212033278 [rank2]:E0810 ] 2-3 -6150027303226666531 ``` ghstack-source-id: 3d60739 Pull Request resolved: #160283

Debugs RNG desync by checking the current state on each rank in the group and summarizing the differences if any are detected. Notes: - used allgather instead of gather since its simpler to do this SPMD rather than add conditional behavior, though I could be convinced we only want to log on rank0. Usage: `check_rng_sync(generator, group)` Prints something like this: (cuda): ``` [rank0]:E0808 ] Generator desync detected: [rank0]:E0808 ] Ranks (Seed, Offset) values [rank0]:E0808 ] ------- ----------------------- [rank0]:E0808 ] 0 (456, 0) [rank0]:E0808 ] 1 (123, 4) [rank0]:E0808 ] 2-3 (123, 0) ``` (cpu): ``` [rank2]:E0810 ] Generator desync detected: [rank2]:E0810 ] Ranks Generator State Hash values [rank2]:E0810 ] ------- ----------------------------- [rank2]:E0810 ] 0 7633364531954955665 [rank2]:E0810 ] 1 8807615394212033278 [rank2]:E0810 ] 2-3 -6150027303226666531 ``` cc H-Huang awgu wanchaol fegin fduwjj wz337 d4l3k pragupta [ghstack-poisoned]

Debugs RNG desync by checking the current state on each rank in the group and summarizing the differences if any are detected. Notes: - used allgather instead of gather since its simpler to do this SPMD rather than add conditional behavior, though I could be convinced we only want to log on rank0. Usage: `check_rng_sync(generator, group)` Prints something like this: (cuda): ``` [rank0]:E0808 ] Generator desync detected: [rank0]:E0808 ] Ranks (Seed, Offset) values [rank0]:E0808 ] ------- ----------------------- [rank0]:E0808 ] 0 (456, 0) [rank0]:E0808 ] 1 (123, 4) [rank0]:E0808 ] 2-3 (123, 0) ``` (cpu): ``` [rank2]:E0810 ] Generator desync detected: [rank2]:E0810 ] Ranks Generator State Hash values [rank2]:E0810 ] ------- ----------------------------- [rank2]:E0810 ] 0 7633364531954955665 [rank2]:E0810 ] 1 8807615394212033278 [rank2]:E0810 ] 2-3 -6150027303226666531 ``` [ghstack-poisoned]

ezyang · 2025-08-13T00:04:41Z

test/distributed/test_collective_utils.py

+        # [rank0]:E0808 ] -------  -----------------------
+        # [rank0]:E0808 ] 0        (456, 0)
+        # [rank0]:E0808 ] 1        (123, 4)
+        # [rank0]:E0808 ] 2-3      (123, 0)


Can you not have the function also return a str

Yea, I can do that.

ezyang · 2025-08-13T00:07:54Z

torch/distributed/collective_utils.py

+    all_states = [torch.empty_like(local_state) for _ in range(group.size())]
+    torch.distributed.all_gather(all_states, local_state)
+    seeds_offsets = [
+        (state[:8].view(torch.uint64).item(), state[8:].view(torch.uint64).item())


Remind me what's going on here?

It's seed,offset both uint64_t packed together into a 16-byte, iirc

torch/distributed/collective_utils.py

ezyang

okey dokey

ezyang

okey dokey

Debugs RNG desync by checking the current state on each rank in the group and summarizing the differences if any are detected. Notes: - used allgather instead of gather since its simpler to do this SPMD rather than add conditional behavior, though I could be convinced we only want to log on rank0. Usage: `check_rng_sync(generator, group)` Prints something like this: (cuda): ``` [rank0]:E0808 ] Generator desync detected: [rank0]:E0808 ] Ranks (Seed, Offset) values [rank0]:E0808 ] ------- ----------------------- [rank0]:E0808 ] 0 (456, 0) [rank0]:E0808 ] 1 (123, 4) [rank0]:E0808 ] 2-3 (123, 0) ``` (cpu): ``` [rank2]:E0810 ] Generator desync detected: [rank2]:E0810 ] Ranks Generator State Hash values [rank2]:E0810 ] ------- ----------------------------- [rank2]:E0810 ] 0 7633364531954955665 [rank2]:E0810 ] 1 8807615394212033278 [rank2]:E0810 ] 2-3 -6150027303226666531 ``` [ghstack-poisoned]

Debugs RNG desync by checking the current state on each rank in the group and summarizing the differences if any are detected. Notes: - used allgather instead of gather since its simpler to do this SPMD rather than add conditional behavior, though I could be convinced we only want to log on rank0. Usage: `check_rng_sync(generator, group)` Prints something like this: (cuda): ``` [rank0]:E0808 ] Generator desync detected: [rank0]:E0808 ] Ranks (Seed, Offset) values [rank0]:E0808 ] ------- ----------------------- [rank0]:E0808 ] 0 (456, 0) [rank0]:E0808 ] 1 (123, 4) [rank0]:E0808 ] 2-3 (123, 0) ``` (cpu): ``` [rank2]:E0810 ] Generator desync detected: [rank2]:E0810 ] Ranks Generator State Hash values [rank2]:E0810 ] ------- ----------------------------- [rank2]:E0810 ] 0 7633364531954955665 [rank2]:E0810 ] 1 8807615394212033278 [rank2]:E0810 ] 2-3 -6150027303226666531 ``` ghstack-source-id: a13ee46 Pull Request resolved: #160283

Debugs RNG desync by checking the current state on each rank in the group and summarizing the differences if any are detected. Notes: - used allgather instead of gather since its simpler to do this SPMD rather than add conditional behavior, though I could be convinced we only want to log on rank0. Usage: `check_rng_sync(generator, group)` Prints something like this: (cuda): ``` [rank0]:E0808 ] Generator desync detected: [rank0]:E0808 ] Ranks (Seed, Offset) values [rank0]:E0808 ] ------- ----------------------- [rank0]:E0808 ] 0 (456, 0) [rank0]:E0808 ] 1 (123, 4) [rank0]:E0808 ] 2-3 (123, 0) ``` (cpu): ``` [rank2]:E0810 ] Generator desync detected: [rank2]:E0810 ] Ranks Generator State Hash values [rank2]:E0810 ] ------- ----------------------------- [rank2]:E0810 ] 0 7633364531954955665 [rank2]:E0810 ] 1 8807615394212033278 [rank2]:E0810 ] 2-3 -6150027303226666531 ``` [ghstack-poisoned]

Debugs RNG desync by checking the current state on each rank in the group and summarizing the differences if any are detected. Notes: - used allgather instead of gather since its simpler to do this SPMD rather than add conditional behavior, though I could be convinced we only want to log on rank0. Usage: `check_rng_sync(generator, group)` Prints something like this: (cuda): ``` [rank0]:E0808 ] Generator desync detected: [rank0]:E0808 ] Ranks (Seed, Offset) values [rank0]:E0808 ] ------- ----------------------- [rank0]:E0808 ] 0 (456, 0) [rank0]:E0808 ] 1 (123, 4) [rank0]:E0808 ] 2-3 (123, 0) ``` (cpu): ``` [rank2]:E0810 ] Generator desync detected: [rank2]:E0810 ] Ranks Generator State Hash values [rank2]:E0810 ] ------- ----------------------------- [rank2]:E0810 ] 0 7633364531954955665 [rank2]:E0810 ] 1 8807615394212033278 [rank2]:E0810 ] 2-3 -6150027303226666531 ``` ghstack-source-id: 55f1f0b Pull Request resolved: #160283

wconstab · 2025-08-13T20:25:42Z

@pytorchbot merge

pytorchmergebot · 2025-08-13T20:27:44Z

Merge failed

Reason: This PR needs a release notes: label
If your changes are user facing and intended to be a part of release notes, please use a label starting with release notes:.

If not, please add the topic: not user facing label.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "topic: not user facing"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

Details for Dev Infra team

Raised by workflow job

wconstab · 2025-08-13T20:51:31Z

@pytorchbot merge

pytorchmergebot · 2025-08-13T20:53:21Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorch-bot bot added oncall: distributed Add this issue/PR to distributed oncall triage queue labels Aug 10, 2025

This was referenced Aug 10, 2025

[C10D] add _summarize_ranks util #160284

Open

[DTensor] Decide / Document RNG semantics #159991

Open

wconstab added 4 commits August 11, 2025 16:04

wconstab mentioned this pull request Aug 12, 2025

[DTensor] Make default RNG semantics match user-passed generator #160482

Open

ezyang reviewed Aug 13, 2025

View reviewed changes

torch/distributed/collective_utils.py Outdated Show resolved Hide resolved

ezyang reviewed Aug 13, 2025

View reviewed changes

torch/distributed/collective_utils.py Outdated Show resolved Hide resolved

ezyang approved these changes Aug 13, 2025

View reviewed changes

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Aug 13, 2025

pytorchmergebot added the merging label Aug 13, 2025

pytorchmergebot removed the merging label Aug 13, 2025

wconstab added the release notes: distributed (c10d) release notes category label Aug 13, 2025

pytorchmergebot added the merging label Aug 13, 2025

pytorchmergebot added the Merged label Aug 13, 2025

pytorchmergebot closed this in 6da11d9 Aug 13, 2025

pytorchmergebot removed the merging label Aug 13, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[C10D] Add check_rng_sync util #160283

[C10D] Add check_rng_sync util #160283

wconstab commented Aug 10, 2025 •

edited by pytorch-bot bot

Loading

Uh oh!

pytorch-bot bot commented Aug 10, 2025 •

edited

Loading

Uh oh!

ezyang Aug 13, 2025

Uh oh!

wconstab Aug 13, 2025

Uh oh!

ezyang Aug 13, 2025

Uh oh!

wconstab Aug 13, 2025

Uh oh!

Uh oh!

Uh oh!

ezyang left a comment

Uh oh!

ezyang left a comment

Uh oh!

wconstab commented Aug 13, 2025

Uh oh!

pytorchmergebot commented Aug 13, 2025

Uh oh!

wconstab commented Aug 13, 2025

Uh oh!

pytorchmergebot commented Aug 13, 2025

Uh oh!

Uh oh!

[C10D] Add check_rng_sync util #160283

[C10D] Add check_rng_sync util #160283

Conversation

wconstab commented Aug 10, 2025 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Aug 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/160283

✅ No Failures

Uh oh!

ezyang Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

wconstab Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

ezyang Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

wconstab Aug 13, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ezyang left a comment

Choose a reason for hiding this comment

Uh oh!

ezyang left a comment

Choose a reason for hiding this comment

Uh oh!

wconstab commented Aug 13, 2025

Uh oh!

pytorchmergebot commented Aug 13, 2025

Merge failed

Uh oh!

wconstab commented Aug 13, 2025

Uh oh!

pytorchmergebot commented Aug 13, 2025

Merge started

Uh oh!

Uh oh!

wconstab commented Aug 10, 2025 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Aug 10, 2025 •

edited

Loading