Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: pytorch/pytorch
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: main
Choose a base ref
...
head repository: pytorch/pytorch
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: gh/wconstab/441/orig
Choose a head ref
  • 2 commits
  • 2 files changed
  • 1 contributor

Commits on Aug 10, 2025

  1. [C10D] Add check_rng_sync util 

    Debugs RNG desync by checking the current state on each rank in the group and summarizing the differences if any are detected.

Notes:
- used allgather instead of gather since its simpler to do this SPMD rather than add conditional behavior, though I could be convinced we only want to log on rank0.

Usage:
`check_rng_sync(generator, group)`

Prints something like this:

(cuda):
```
[rank0]:E0808 ] Generator desync detected:
[rank0]:E0808 ] Ranks    (Seed, Offset) values
[rank0]:E0808 ] -------  -----------------------
[rank0]:E0808 ] 0        (456, 0)
[rank0]:E0808 ] 1        (123, 4)
[rank0]:E0808 ] 2-3      (123, 0)
```

(cpu):
```
[rank2]:E0810 ] Generator desync detected:
[rank2]:E0810 ] Ranks      Generator State Hash values
[rank2]:E0810 ] -------  -----------------------------
[rank2]:E0810 ] 0                  7633364531954955665
[rank2]:E0810 ] 1                  8807615394212033278
[rank2]:E0810 ] 2-3               -6150027303226666531
```

ghstack-source-id: 3d60739
Pull Request resolved: #160283
    @wconstab
    wconstab committed Aug 10, 2025
    Configuration menu
    Copy the full SHA
    c2b0c39 View commit details
    Browse the repository at this point in the history

  2. WIP summarize ranks 

    ghstack-source-id: 37921cd
Pull Request resolved: #160284
    @wconstab
    wconstab committed Aug 10, 2025
    Configuration menu
    Copy the full SHA
    85c6506 View commit details
    Browse the repository at this point in the history
Loading