Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

EMAcodebook update #1

Open
GuGuGuDove opened this issue Apr 15, 2024 · 1 comment
Open

EMAcodebook update #1

GuGuGuDove opened this issue Apr 15, 2024 · 1 comment

Comments

@GuGuGuDove
Copy link

GuGuGuDove commented Apr 15, 2024 •
edited
Loading

Hi maxjcohen,

in codebook.py line 186, if the ema_positions got divided by ema_cluster_size, that means the first term in line 183 got divided too, which seems not reasonable? (only the 2nd term in line 183 is the ensemble of several similar encodings so normalization needed but the 1st term is not)

Could you give me some hints? Thx!
@maxjcohen
Copy link
Owner

Hi, and thank you for taking an interest in this repo.

When you say

the ema_positions got divided by ema_cluster_size

I don't quite follow what you mean, as it is not the ema_positions values which are updated themselves, but the codebooks:

self.weight.data = self.ema_positions / self.ema_cluster_size.unsqueeze(-1)

As described in the last page of the original paper (only the arXiv version).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@maxjcohen @GuGuGuDove