Skip to content

Automatically set chat format from gguf #1110

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Jan 29, 2024
Merged

Automatically set chat format from gguf #1110

merged 4 commits into from
Jan 29, 2024

Conversation

abetlen
Copy link
Owner

@abetlen abetlen commented Jan 19, 2024 •
edited
Loading

Closes #1096

Okay so for now this is going to be the behaviour / priority of the automatic chat format detection

  1. Use provided chat_handler param
  2. Use provided chat_format param
  3. If model has a gguf chat_template try to string match it against known jinja2 templates and use the matching template's eos token
  4. If model has a gguf chat_template use eos_token for stop generation
  5. Use llama 2 chat format

@abetlen abetlen mentioned this pull request Jan 19, 2024
Closed
@abetlen abetlen merged commit da003d8 into main Jan 29, 2024
@abetlen abetlen deleted the chat-format-from-gguf branch January 31, 2024 20:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Use chat_template from gguf metadata
1 participant
@abetlen