Skip to content

Fix tokenizer #383

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Sep 2, 2024
Merged

Fix tokenizer #383

merged 1 commit into from
Sep 2, 2024

Conversation

stduhpf
Copy link
Contributor

@stduhpf stduhpf commented Aug 29, 2024

The type char is signed by default, this causes problems with characters having ASCII code greater than 127.

Fixes #382

Copy link
Contributor

@Green-Sky Green-Sky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very good observation

@leejet
Copy link
Owner

leejet commented Sep 2, 2024

Thank you for your contribution.

@leejet leejet merged commit 14206fd into leejet:master Sep 2, 2024
10 checks passed
stduhpf added a commit to stduhpf/stable-diffusion.cpp that referenced this pull request Nov 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Non-ASCII characters in prompt crashes the tokenizer
3 participants