Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JSON schema conversion: ⚡️ faster repetitions, min/maxLength for strings, cap number length #6555

Merged
merged 17 commits into from
Apr 12, 2024
Merged

JSON schema conversion: ⚡️ faster repetitions, min/maxLength for strings, cap number length #6555

Changes from 1 commit
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
2148f24
json: rename python schema converter to make import easier
ochafik Apr 8, 2024
f771a8f
server: skip null json_schema / grammar fields
ochafik Apr 8, 2024
159b883
json: deps management for primitive rules (+ allow null values)
ochafik Apr 8, 2024
a59e943
json: optimize repetitions for minItems/maxItems and regexps: `a{,3}`…
ochafik Apr 8, 2024
07163fb
grammars: add troubleshooting section to readme
ochafik Apr 8, 2024
dcf5d32
json: cap length of numbers to 15 digits before/after decimal point
ochafik Apr 8, 2024
181f984
json: unify all repetition code (w/ or w/o sep)
ochafik Apr 8, 2024
de4e60e
json: support string minLength/maxLength
ochafik Apr 8, 2024
6c885dc
server+json: update server/README w/ result_format
ochafik Apr 8, 2024
3c81e94
nits
ochafik Apr 8, 2024
67a5184
json: fix type error w/ python 3.8
ochafik Apr 8, 2024
9c33ee9
json: fix server/README (json_schema in /completion vs. result_format…
ochafik Apr 9, 2024
ed13d47
json: simplify DOT `{"type": "string", "pattern": "^.$"}`
ochafik Apr 12, 2024
958bdda
Merge remote-tracking branch 'origin/master' into json-faster-repetit…
ochafik Apr 12, 2024
64e3059
json: remove recursion in opt_repetitions (avoids Python stack overflow)
ochafik Apr 12, 2024
ba90d5b
json: rm dead code
ochafik Apr 12, 2024
dfd4eb3
json: rm useless assert & ggml.h import
ochafik Apr 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
grammars: add troubleshooting section to readme
  • Loading branch information
@ochafik
ochafik committed Apr 8, 2024
commit 07163fb627f5a43d48b9698bdeb1322279c3a379
10 changes: 10 additions & 0 deletions grammars/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,3 +89,13 @@ This guide provides a brief overview. Check out the GBNF files in this directory
```
./main -m <model> --grammar-file grammars/some-grammar.gbnf -p 'Some prompt'
```

## Troubleshooting
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like this section. After this gets merged in, I'll write a section on the dangers of left-recursion.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably also document the json->grammar converters here, I'll send that separately


Grammars currently have performance gotchas (see https://github.com/ggerganov/llama.cpp/issues/4218).

### Efficient optional repetitions

A common pattern is to allow repetitions of a pattern `x` up to N times.

While semantically correct, the syntax `x? x? x?.... x?` (with N repetitions) will result in extremely slow inference. Instead, you can write `(x (x (x ... (x)?...)?)?)?` (w/ N-deep nesting)