-
Notifications
You must be signed in to change notification settings - Fork 12.9k
common : add GLM-4.5 tool calling support #15186
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
common : add GLM-4.5 tool calling support #15186
Conversation
- Add COMMON_CHAT_FORMAT_GLM_4_5 format enum - Implement GLM-4.5 tool call parser for <tool_call><arg_key><arg_value> format - Add template detection based on <arg_key> and <arg_value> tags - Fix null content handling in message parsing and serialization - Ensure GLM-4.5 detection runs before Hermes to avoid misidentification This enables tool calling functionality for GLM-4.5 models when using --jinja flag. The parser handles GLM-4.5's XML-like tool call format with key-value argument pairs.
I tried the PR, and it fixes tool calling on GLM 4.5 Air (unsloth version) getting called correctly. |
But its Qwen tool calling issue right? I think once other pending PRs are merged you should not see the issue. |
Yea, I don't think is related to this specific PR. But the problem is shared with this Qwen tool calling issue. |
Hey, quick thought — I might be misunderstanding this, but it looks like this PR will parse GLM’s XML-style tool calls and turn them into JSON If that’s the case, projects like Roo Code (which currently only know how to handle XML tool calls) might suddenly stop recognizing the output from GLM models when running through llama.cpp. Am I right about this? |
Does this template parse the thinking tags correctly? I'm getting my responses inline instead of in the reasoning_content field. |
Very nice! #15162 aims to achieve the same for Qwen3 Coder; only seems more mature/higher quality (using minja and letting it handle quoting/escaping argument strings, storing the jinja template in ./models/templates, having test cases in ./tests/test-chat.cpp,). Maybe @ochafik and @dhandhalyabhavik can sync up/collaborate and bring both PRs in a consistent way forward? |
Hello everyone, thanks for insightful comments, Let me answer all of you, @TNohSam There are two ways to implement tool calling, I have tested Roo Code just now, it is working fine. Both type of function or tool calling will work with the current PR. @jfgonsalves enable @bfroemel sure, @ochafik can you please review my added changes? Help me merge this PR. I would really appreciate. Thank you. |
You can enable There is parser logic common for all models that will do this job. Check out the code here This PR has nothing to do with it. Thank you for pointing it out though. |
I am still having trouble getting llama.cpp to identify the GLM-4.5 chat template. Am I missing something in my command? srv params_from_: Chat format: Hermes 2 Pro ./build/bin/llama-server --model /mnt/home_extend/models/unsloth_GLM-4.5-GGUF/Q5_K_M/GLM-4.5-Q5_K_M-00001-of-00006.gguf --alias glm-4.5 --no-webui --threads 44 --ctx-size 131072 --n-gpu-layers 94 -ot exps=CPU -ub 2048 -b 2048 --temp 0.6 --top-p 1.0 --flash-attn --host 0.0.0.0 --jinja --port 8099 --chat-template-file /mnt/home_extend/models/unsloth_GLM-4.5-GGUF/template.jinja |
Hey @trilog-inc I rebuild and its working fine for me. I got GLM 4.5 in my logs, tool calling also works well.
I hope you have build it correctly using FYI, I have used GLM 4.5 Air for testing. Both should work as I can see both of them have same Arch & jinja template. I have used this command, (re-copy jinja template, there there is a modification recommended by a user)
|
Trying this PR, tools get called and get a response but the model can't continue with them. I do see the right chat format as expected in the logs and did build the PR correctly.
I'm using https://huggingface.co/unsloth/GLM-4.5-Air-GGUF?chat_template=default as the chat template Edit: Using the jinja template in the OP fixed the issue. |
I had similar problem. Not sure what chat template to use |
I copied the template from the OP's edit and used it, and now tool calling is working for me. |
It worked like a charm, thanks |
For claude code / openai, this template would work
This worked with Claude Code and Claude Code Router. |
@dhandhalyabhavik I don't thin the See: #15186 (comment)_ I changed |
Hey @hksdpc255 can you please review again. GLM 4.5 produces Tested working on
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I only have time to do a quick review. The code looks good overall, though I haven't had time to test it.
I think we can merge after @hksdpc255 approve this PR
@dhandhalyabhavik I test it with zed editor. after 2 or 3 tool calling, llama-server crashed with something like:
Screenshot of zed editor: ![]() Then I test it with claude-code combined with claude-code-router, same problem.
![]() The runtime_error seems thrown at: https://github.com/ggml-org/llama.cpp/pull/15186/files#diff-2580689a73e0d42f06cf4b5ed02e7cf87c6fc874343bce9c6f0e1ebf190d95b7R68 |
meaybe... diff --git a/common/chat.cpp b/common/chat.cpp
index 07e65539..a5fc6db4 100644
--- a/common/chat.cpp
+++ b/common/chat.cpp
@@ -1389,9 +1389,8 @@ static void common_chat_parse_glm_4_5(common_chat_msg_parser & builder) {
json parsed;
try {
parsed = json::parse(json_str);
- } catch (const json::parse_error& e) {
- builder.add_content(remaining);
- return;
+ } catch (const json::parse_error &) {
+ throw common_chat_msg_partial_exception("incomplete GLM JSON tool call");
} |
Its seems that the model generates something like:
Then llama-server crash. Here's a request that always triggers a crash: curl 'http://llama.server:port/v1/chat/completions' -H "Content-Type: application/json" -d '{"model":"local","messages":[{"role":"system","content":"You are a highly skilled software engineer with extensive knowledge in many programming languages, frameworks, design patterns, and best practices.\n\n## Communication\n\n1. Be conversational but professional.\n2. Refer to the user in the second person and yourself in the first person.\n3. Format your responses in markdown. Use backticks to format file, directory, function, and class names.\n4. NEVER lie or make things up.\n5. Refrain from apologizing all the time when results are unexpected. Instead, just try your best to proceed or explain the circumstances to the user without apologizing.\n\n## Tool Use\n\n1. Make sure to adhere to the tools schema.\n2. Provide every required argument.\n3. DO NOT use tools to access items that are already available in the context section.\n4. Use only the tools that are currently available.\n5. DO NOT use a tool that is not available just because it appears in the conversation. This means the user turned it off.\n6. NEVER run commands that don'\''t terminate on their own such as web servers (like `npm run start`, `npm run dev`, `python -m http.server`, etc) or file watchers.\n7. Avoid HTML entity escaping - use plain characters instead.\n\n## Searching and Reading\n\nIf you are unsure how to fulfill the user'\''s request, gather more information with tool calls and/or clarifying questions.\n\nIf appropriate, use tool calls to explore the current project, which contains the following root directories:\n\n- `/work`\n\n- Bias towards not asking the user for help if you can find the answer yourself.\n- When providing paths to tools, the path should always start with the name of a project root directory listed above.\n- Before you read or edit a file, you must first find the full path. DO NOT ever guess a file path!\n- When looking for symbols in the project, prefer the `grep` tool.\n- As you learn about the structure of the project, use that information to scope `grep` searches to targeted subtrees of the project.\n- The user might specify a partial file path. If you don'\''t know the full path, use `find_path` (not `grep`) before you read the file.\n\n## Code Block Formatting\n\nWhenever you mention a code block, you MUST use ONLY use the following format:\n```path/to/Something.blah#L123-456\n(code goes here)\n```\nThe `#L123-456` means the line number range 123 through 456, and the path/to/Something.blah\nis a path in the project. (If there is no valid path in the project, then you can use\n/dev/null/path.extension for its path.) This is the ONLY valid way to format code blocks, because the Markdown parser\ndoes not understand the more common ```language syntax, or bare ``` blocks. It only\nunderstands this path-based syntax, and if the path is missing, then it will error and you will have to do it over again.\nJust to be really clear about this, if you ever find yourself writing three backticks followed by a language name, STOP!\nYou have made a mistake. You can only ever put paths after triple backticks!\n<example>\nBased on all the information I'\''ve gathered, here'\''s a summary of how this system works:\n1. The README file is loaded into the system.\n2. The system finds the first two headers, including everything in between. In this case, that would be:\n```path/to/README.md#L8-12\n# First Header\nThis is the info under the first header.\n## Sub-header\n```\n3. Then the system finds the last header in the README:\n```path/to/README.md#L27-29\n## Last Header\nThis is the last header in the README.\n```\n4. Finally, it passes this information on to the next process.\n</example>\n<example>\nIn Markdown, hash marks signify headings. For example:\n```/dev/null/example.md#L1-3\n# Level 1 heading\n## Level 2 heading\n### Level 3 heading\n```\n</example>\nHere are examples of ways you must never render code blocks:\n<bad_example_do_not_do_this>\nIn Markdown, hash marks signify headings. For example:\n```\n# Level 1 heading\n## Level 2 heading\n### Level 3 heading\n```\n</bad_example_do_not_do_this>\nThis example is unacceptable because it does not include the path.\n<bad_example_do_not_do_this>\nIn Markdown, hash marks signify headings. For example:\n```markdown\n# Level 1 heading\n## Level 2 heading\n### Level 3 heading\n```\n</bad_example_do_not_do_this>\nThis example is unacceptable because it has the language instead of the path.\n<bad_example_do_not_do_this>\nIn Markdown, hash marks signify headings. For example:\n # Level 1 heading\n ## Level 2 heading\n ### Level 3 heading\n</bad_example_do_not_do_this>\nThis example is unacceptable because it uses indentation to mark the code block\ninstead of backticks with a path.\n<bad_example_do_not_do_this>\nIn Markdown, hash marks signify headings. For example:\n```markdown\n/dev/null/example.md#L1-3\n# Level 1 heading\n## Level 2 heading\n### Level 3 heading\n```\n</bad_example_do_not_do_this>\nThis example is unacceptable because the path is in the wrong place. The path must be directly after the opening backticks.\n\n## Fixing Diagnostics\n\n1. Make 1-2 attempts at fixing diagnostics, then defer to the user.\n2. Never simplify code you'\''ve written just to solve diagnostics. Complete, mostly correct code is more valuable than perfect code that doesn'\''t solve the problem.\n\n## Debugging\n\nWhen debugging, only make code changes if you are certain that you can solve the problem.\nOtherwise, follow debugging best practices:\n1. Address the root cause instead of the symptoms.\n2. Add descriptive logging statements and error messages to track variable and code state.\n3. Add test functions and statements to isolate the problem.\n\n## Calling External APIs\n\n1. Unless explicitly requested by the user, use the best suited external APIs and packages to solve the task. There is no need to ask the user for permission.\n2. When selecting which version of an API or package to use, choose one that is compatible with the user'\''s dependency management file(s). If no such file exists or if the package is not present, use the latest version that is in your training data.\n3. If an external API requires an API Key, be sure to point this out to the user. Adhere to best security practices (e.g. DO NOT hardcode an API key in a place where it can be exposed)\n\n## System Information\n\nOperating System: linux\nDefault Shell: /bin/bash\n\n"},{"role":"user","content":"create a rust project that implements a commandline tool like \"expr\""},{"role":"assistant","content":"\nI'\''ll help you create a Rust project that implements a command line tool similar to the Unix `expr` command. Let me start by exploring the current project structure.","tool_calls":[{"id":"UShTmFwR6I82RPl8ClgECTASj2mylTbY","type":"function","function":{"name":"list_directory","arguments":"{\"path\":\"/work\"}"}}]},{"role":"tool","content":"\n# Files:\nwork/prompt.txt\n","tool_call_id":"UShTmFwR6I82RPl8ClgECTASj2mylTbY"},{"role":"assistant","content":"\nI'\''ll create a Rust project that implements a command line tool similar to the Unix `expr` command. Let me start by setting up the project structure.","tool_calls":[{"id":"pVUbcOxZS8M7zh1x8N1FHfsjNXRM7eSa","type":"function","function":{"name":"create_directory","arguments":"{\"path\":\"work/expr-cli\"}"}}]},{"role":"tool","content":"Created directory work/expr-cli","tool_call_id":"pVUbcOxZS8M7zh1x8N1FHfsjNXRM7eSa"},{"role":"assistant","content":"\nNow let me create the Cargo.toml file to set up the Rust project:","tool_calls":[{"id":"CK67dYM4Kk9F1xYroM7ok5wAw9XLg2qD","type":"function","function":{"name":"edit_file","arguments":"{\"display_description\":\"Create Cargo.toml for Rust expr-cli project\",\"path\":\"work/expr-cli/Cargo.toml\",\"mode\":\"create\"}"}}]},{"role":"tool","content":"Edited work/expr-cli/Cargo.toml:\n\n```diff\n@@ -1,0 +1,31 @@\n+[package]\n+name = \"expr-cli\"\n+version = \"0.1.0\"\n+edition = \"2021\"\n+authors = [\"Your Name <your.email@example.com>\"]\n+description = \"A command line tool for evaluating expressions, similar to Unix expr\"\n+license = \"MIT\"\n+repository = \"https://github.com/yourusername/expr-cli\"\n+keywords = [\"cli\", \"expr\", \"calculator\", \"math\"]\n+categories = [\"command-line-utilities\"]\n+\n+[dependencies]\n+clap = { version = \"4.4\", features = [\"derive\"] }\n+thiserror = \"1.0\"\n+anyhow = \"1.0\"\n+\n+[dev-dependencies]\n+assert_cmd = \"2.0\"\n+predicates = \"3.0\"\n+\n+[[bin]]\n+name = \"expr-cli\"\n+path = \"src/main.rs\"\n+test = true\n+bench = false\n+\n+[profile.release]\n+opt-level = 3\n+lto = true\n+codegen-units = 1\n+strip = true\n\n```","tool_call_id":"CK67dYM4Kk9F1xYroM7ok5wAw9XLg2qD"}],"stream":true,"max_completion_tokens":64000,"temperature":1.0,"tools":[{"type":"function","function":{"name":"delete_path","description":"Deletes the file or directory (and the directory'\''s contents, recursively) at the specified path in the project, and returns confirmation of the deletion.\n","parameters":{"required":["path"],"properties":{"path":{"description":"The path of the file or directory to delete.\n\n<example>\nIf the project has the following files:\n\n- directory1/a/something.txt\n- directory2/a/things.txt\n- directory3/a/other.txt\n\nYou can delete the first file by providing a path of \"directory1/a/something.txt\"\n</example>","type":"string"}},"type":"object","additionalProperties":false}}},{"type":"function","function":{"name":"now","description":"Returns the current datetime in RFC 3339 format. Only use this tool when the user specifically asks for it or the current task would benefit from knowing the current datetime.","parameters":{"definitions":{"Timezone":{"oneOf":[{"description":"Use UTC for the datetime.","type":"string","const":"utc"},{"description":"Use local time for the datetime.","type":"string","const":"local"}]}},"required":["timezone"],"type":"object","properties":{"timezone":{"description":"The timezone to use for the datetime.","allOf":[{"$ref":"#/definitions/Timezone"}]}},"additionalProperties":false}}},{"type":"function","function":{"name":"copy_path","description":"Copies a file or directory in the project, and returns confirmation that the copy succeeded.\nDirectory contents will be copied recursively (like `cp -r`).\n\nThis tool should be used when it'\''s desirable to create a copy of a file or directory without modifying the original.\nIt'\''s much more efficient than doing this by separately reading and then writing the file or directory'\''s contents,\nso this tool should be preferred over that approach whenever copying is the goal.\n","parameters":{"required":["source_path","destination_path"],"properties":{"source_path":{"description":"The source path of the file or directory to copy.\nIf a directory is specified, its contents will be copied recursively (like `cp -r`).\n\n<example>\nIf the project has the following files:\n\n- directory1/a/something.txt\n- directory2/a/things.txt\n- directory3/a/other.txt\n\nYou can copy the first file by providing a source_path of \"directory1/a/something.txt\"\n</example>","type":"string"},"destination_path":{"description":"The destination path where the file or directory should be copied to.\n\n<example>\nTo copy \"directory1/a/something.txt\" to \"directory2/b/copy.txt\",\nprovide a destination_path of \"directory2/b/copy.txt\"\n</example>","type":"string"}},"type":"object","additionalProperties":false}}},{"type":"function","function":{"name":"edit_file","description":"This is a tool for creating a new file or editing an existing file. For moving or renaming files, you should generally use the `terminal` tool with the '\''mv'\'' command instead.\n\nBefore using this tool:\n\n1. Use the `read_file` tool to understand the file'\''s contents and context\n\n2. Verify the directory path is correct (only applicable when creating new files):\n - Use the `list_directory` tool to verify the parent directory exists and is the correct location\n","parameters":{"definitions":{"EditFileMode":{"type":"string","enum":["edit","create","overwrite"]}},"required":["display_description","path","mode"],"type":"object","properties":{"display_description":{"description":"A one-line, user-friendly markdown description of the edit. This will be\nshown in the UI and also passed to another model to perform the edit.\n\nBe terse, but also descriptive in what you want to achieve with this\nedit. Avoid generic instructions.\n\nNEVER mention the file path in this description.\n\n<example>Fix API endpoint URLs</example>\n<example>Update copyright year in `page_footer`</example>\n\nMake sure to include this field before all the others in the input object\nso that we can display it immediately.","type":"string"},"path":{"description":"The full path of the file to create or modify in the project.\n\nWARNING: When specifying which file path need changing, you MUST\nstart each path with one of the project'\''s root directories.\n\nThe following examples assume we have two root directories in the project:\n- /a/b/backend\n- /c/d/frontend\n\n<example>\n`backend/src/main.rs`\n\nNotice how the file path starts with `backend`. Without that, the path\nwould be ambiguous and the call would fail!\n</example>\n\n<example>\n`frontend/db.js`\n</example>","type":"string"},"mode":{"description":"The mode of operation on the file. Possible values:\n- '\''edit'\'': Make granular edits to an existing file.\n- '\''create'\'': Create a new file if it doesn'\''t exist.\n- '\''overwrite'\'': Replace the entire contents of an existing file.\n\nWhen a file already exists or you just created it, prefer editing\nit as opposed to recreating it from scratch.","allOf":[{"$ref":"#/definitions/EditFileMode"}]}},"additionalProperties":false}}},{"type":"function","function":{"name":"thinking","description":"A tool for thinking through problems, brainstorming ideas, or planning without executing any actions. Use this tool when you need to work through complex problems, develop strategies, or outline approaches before taking action.\n","parameters":{"required":["content"],"properties":{"content":{"description":"Content to think about. This should be a description of what to think about or\na problem to solve.","type":"string"}},"type":"object","additionalProperties":false}}},{"type":"function","function":{"name":"read_file","description":"Reads the content of the given file in the project.\n\n- Never attempt to read a path that hasn'\''t been previously mentioned.\n","parameters":{"required":["path"],"properties":{"path":{"description":"The relative path of the file to read.\n\nThis path should never be absolute, and the first component\nof the path should always be a root directory in a project.\n\n<example>\nIf the project has the following root directories:\n\n- /a/b/directory1\n- /c/d/directory2\n\nIf you want to access `file.txt` in `directory1`, you should use the path `directory1/file.txt`.\nIf you want to access `file.txt` in `directory2`, you should use the path `directory2/file.txt`.\n</example>","type":"string"},"start_line":{"description":"Optional line number to start reading on (1-based index)","type":["integer","null"],"format":"uint32","minimum":0,"default":null},"end_line":{"description":"Optional line number to end reading on (1-based index, inclusive)","type":["integer","null"],"format":"uint32","minimum":0,"default":null}},"description":"If the model requests to read a file whose size exceeds this, then","type":"object","additionalProperties":false}}},{"type":"function","function":{"name":"diagnostics","description":"Get errors and warnings for the project or a specific file.\n\nThis tool can be invoked after a series of edits to determine if further edits are necessary, or if the user asks to fix errors or warnings in their codebase.\n\nWhen a path is provided, shows all diagnostics for that specific file.\nWhen no path is provided, shows a summary of error and warning counts for all files in the project.\n\n<example>\nTo get diagnostics for a specific file:\n{\n \"path\": \"src/main.rs\"\n}\n\nTo get a project-wide diagnostic summary:\n{}\n</example>\n\n<guidelines>\n- If you think you can fix a diagnostic, make 1-2 attempts and then give up.\n- Don'\''t remove code you'\''ve generated just because you can'\''t fix an error. The user can help you fix it.\n</guidelines>\n","parameters":{"properties":{"path":{"description":"The path to get diagnostics for. If not provided, returns a project-wide summary.\n\nThis path should never be absolute, and the first component\nof the path should always be a root directory in a project.\n\n<example>\nIf the project has the following root directories:\n\n- lorem\n- ipsum\n\nIf you wanna access diagnostics for `dolor.txt` in `ipsum`, you should use the path `ipsum/dolor.txt`.\n</example>","type":["string","null"]}},"type":"object","additionalProperties":false}}},{"type":"function","function":{"name":"fetch","description":"Fetches a URL and returns the content as Markdown.\n","parameters":{"required":["url"],"properties":{"url":{"description":"The URL to fetch.","type":"string"}},"type":"object","additionalProperties":false}}},{"type":"function","function":{"name":"create_directory","description":"Creates a new directory at the specified path within the project. Returns confirmation that the directory was created.\n\nThis tool creates a directory and all necessary parent directories (similar to `mkdir -p`). It should be used whenever you need to create new directories within the project.\n","parameters":{"required":["path"],"properties":{"path":{"description":"The path of the new directory.\n\n<example>\nIf the project has the following structure:\n\n- directory1/\n- directory2/\n\nYou can create a new directory by providing a path of \"directory1/new_directory\"\n</example>","type":"string"}},"type":"object","additionalProperties":false}}},{"type":"function","function":{"name":"grep","description":"Searches the contents of files in the project with a regular expression\n\n- Prefer this tool to path search when searching for symbols in the project, because you won'\''t need to guess what path it'\''s in.\n- Supports full regex syntax (eg. \"log.*Error\", \"function\\\\s+\\\\w+\", etc.)\n- Pass an `include_pattern` if you know how to narrow your search on the files system\n- Never use this tool to search for paths. Only search file contents with this tool.\n- Use this tool when you need to find files containing specific patterns\n- Results are paginated with 20 matches per page. Use the optional '\''offset'\'' parameter to request subsequent pages.\n- DO NOT use HTML entities solely to escape characters in the tool parameters.\n","parameters":{"required":["regex"],"properties":{"regex":{"description":"A regex pattern to search for in the entire project. Note that the regex\nwill be parsed by the Rust `regex` crate.\n\nDo NOT specify a path here! This will only be matched against the code **content**.","type":"string"},"include_pattern":{"description":"A glob pattern for the paths of files to include in the search.\nSupports standard glob patterns like \"**/*.rs\" or \"src/**/*.ts\".\nIf omitted, all files in the project will be searched.","type":["string","null"]},"offset":{"description":"Optional starting position for paginated results (0-based).\nWhen not provided, starts from the beginning.","type":"integer","format":"uint32","minimum":0,"default":0},"case_sensitive":{"description":"Whether the regex is case-sensitive. Defaults to false (case-insensitive).","type":"boolean","default":false}},"type":"object","additionalProperties":false}}},{"type":"function","function":{"name":"list_directory","description":"Lists files and directories in a given path. Prefer the `grep` or `find_path` tools when searching the codebase.\n","parameters":{"required":["path"],"properties":{"path":{"description":"The fully-qualified path of the directory to list in the project.\n\nThis path should never be absolute, and the first component\nof the path should always be a root directory in a project.\n\n<example>\nIf the project has the following root directories:\n\n- directory1\n- directory2\n\nYou can list the contents of `directory1` by using the path `directory1`.\n</example>\n\n<example>\nIf the project has the following root directories:\n\n- foo\n- bar\n\nIf you wanna list contents in the directory `foo/baz`, you should use the path `foo/baz`.\n</example>","type":"string"}},"type":"object","additionalProperties":false}}},{"type":"function","function":{"name":"terminal","description":"Executes a shell one-liner and returns the combined output.\n\nThis tool spawns a process using the user'\''s shell, reads from stdout and stderr (preserving the order of writes), and returns a string with the combined output result.\n\nThe output results will be shown to the user already, only list it again if necessary, avoid being redundant.\n\nMake sure you use the `cd` parameter to navigate to one of the root directories of the project. NEVER do it as part of the `command` itself, otherwise it will error.\n\nDo not use this tool for commands that run indefinitely, such as servers (like `npm run start`, `npm run dev`, `python -m http.server`, etc) or file watchers that don'\''t terminate on their own.\n\nRemember that each invocation of this tool will spawn a new shell process, so you can'\''t rely on any state from previous invocations.\n","parameters":{"required":["command","cd"],"properties":{"command":{"description":"The one-liner command to execute.","type":"string"},"cd":{"description":"Working directory for the command. This must be one of the root directories of the project.","type":"string"}},"type":"object","additionalProperties":false}}},{"type":"function","function":{"name":"find_path","description":"Fast file path pattern matching tool that works with any codebase size\n\n- Supports glob patterns like \"**/*.js\" or \"src/**/*.ts\"\n- Returns matching file paths sorted alphabetically\n- Prefer the `grep` tool to this tool when searching for symbols unless you have specific information about paths.\n- Use this tool when you need to find files by name patterns\n- Results are paginated with 50 matches per page. Use the optional '\''offset'\'' parameter to request subsequent pages.\n","parameters":{"required":["glob"],"properties":{"glob":{"description":"The glob to match against every path in the project.\n\n<example>\nIf the project has the following root directories:\n\n- directory1/a/something.txt\n- directory2/a/things.txt\n- directory3/a/other.txt\n\nYou can get back the first two paths by providing a glob of \"*thing*.txt\"\n</example>","type":"string"},"offset":{"description":"Optional starting position for paginated results (0-based).\nWhen not provided, starts from the beginning.","type":"integer","format":"uint","minimum":0,"default":0}},"type":"object","additionalProperties":false}}},{"type":"function","function":{"name":"move_path","description":"Moves or rename a file or directory in the project, and returns confirmation that the move succeeded.\nIf the source and destination directories are the same, but the filename is different, this performs\na rename. Otherwise, it performs a move.\n\nThis tool should be used when it'\''s desirable to move or rename a file or directory without changing its contents at all.\n","parameters":{"required":["source_path","destination_path"],"properties":{"source_path":{"description":"The source path of the file or directory to move/rename.\n\n<example>\nIf the project has the following files:\n\n- directory1/a/something.txt\n- directory2/a/things.txt\n- directory3/a/other.txt\n\nYou can move the first file by providing a source_path of \"directory1/a/something.txt\"\n</example>","type":"string"},"destination_path":{"description":"The destination path where the file or directory should be moved/renamed to.\nIf the paths are the same except for the filename, then this will be a rename.\n\n<example>\nTo move \"directory1/a/something.txt\" to \"directory2/b/renamed.txt\",\nprovide a destination_path of \"directory2/b/renamed.txt\"\n</example>","type":"string"}},"type":"object","additionalProperties":false}}}]}' RAW request body: {
"model": "local",
"messages": [
{
"role": "system",
"content": "You are a highly skilled software engineer with extensive knowledge in many programming languages, frameworks, design patterns, and best practices.\n\n## Communication\n\n1. Be conversational but professional.\n2. Refer to the user in the second person and yourself in the first person.\n3. Format your responses in markdown. Use backticks to format file, directory, function, and class names.\n4. NEVER lie or make things up.\n5. Refrain from apologizing all the time when results are unexpected. Instead, just try your best to proceed or explain the circumstances to the user without apologizing.\n\n## Tool Use\n\n1. Make sure to adhere to the tools schema.\n2. Provide every required argument.\n3. DO NOT use tools to access items that are already available in the context section.\n4. Use only the tools that are currently available.\n5. DO NOT use a tool that is not available just because it appears in the conversation. This means the user turned it off.\n6. NEVER run commands that don't terminate on their own such as web servers (like `npm run start`, `npm run dev`, `python -m http.server`, etc) or file watchers.\n7. Avoid HTML entity escaping - use plain characters instead.\n\n## Searching and Reading\n\nIf you are unsure how to fulfill the user's request, gather more information with tool calls and/or clarifying questions.\n\nIf appropriate, use tool calls to explore the current project, which contains the following root directories:\n\n- `/work`\n\n- Bias towards not asking the user for help if you can find the answer yourself.\n- When providing paths to tools, the path should always start with the name of a project root directory listed above.\n- Before you read or edit a file, you must first find the full path. DO NOT ever guess a file path!\n- When looking for symbols in the project, prefer the `grep` tool.\n- As you learn about the structure of the project, use that information to scope `grep` searches to targeted subtrees of the project.\n- The user might specify a partial file path. If you don't know the full path, use `find_path` (not `grep`) before you read the file.\n\n## Code Block Formatting\n\nWhenever you mention a code block, you MUST use ONLY use the following format:\n```path/to/Something.blah#L123-456\n(code goes here)\n```\nThe `#L123-456` means the line number range 123 through 456, and the path/to/Something.blah\nis a path in the project. (If there is no valid path in the project, then you can use\n/dev/null/path.extension for its path.) This is the ONLY valid way to format code blocks, because the Markdown parser\ndoes not understand the more common ```language syntax, or bare ``` blocks. It only\nunderstands this path-based syntax, and if the path is missing, then it will error and you will have to do it over again.\nJust to be really clear about this, if you ever find yourself writing three backticks followed by a language name, STOP!\nYou have made a mistake. You can only ever put paths after triple backticks!\n<example>\nBased on all the information I've gathered, here's a summary of how this system works:\n1. The README file is loaded into the system.\n2. The system finds the first two headers, including everything in between. In this case, that would be:\n```path/to/README.md#L8-12\n# First Header\nThis is the info under the first header.\n## Sub-header\n```\n3. Then the system finds the last header in the README:\n```path/to/README.md#L27-29\n## Last Header\nThis is the last header in the README.\n```\n4. Finally, it passes this information on to the next process.\n</example>\n<example>\nIn Markdown, hash marks signify headings. For example:\n```/dev/null/example.md#L1-3\n# Level 1 heading\n## Level 2 heading\n### Level 3 heading\n```\n</example>\nHere are examples of ways you must never render code blocks:\n<bad_example_do_not_do_this>\nIn Markdown, hash marks signify headings. For example:\n```\n# Level 1 heading\n## Level 2 heading\n### Level 3 heading\n```\n</bad_example_do_not_do_this>\nThis example is unacceptable because it does not include the path.\n<bad_example_do_not_do_this>\nIn Markdown, hash marks signify headings. For example:\n```markdown\n# Level 1 heading\n## Level 2 heading\n### Level 3 heading\n```\n</bad_example_do_not_do_this>\nThis example is unacceptable because it has the language instead of the path.\n<bad_example_do_not_do_this>\nIn Markdown, hash marks signify headings. For example:\n # Level 1 heading\n ## Level 2 heading\n ### Level 3 heading\n</bad_example_do_not_do_this>\nThis example is unacceptable because it uses indentation to mark the code block\ninstead of backticks with a path.\n<bad_example_do_not_do_this>\nIn Markdown, hash marks signify headings. For example:\n```markdown\n/dev/null/example.md#L1-3\n# Level 1 heading\n## Level 2 heading\n### Level 3 heading\n```\n</bad_example_do_not_do_this>\nThis example is unacceptable because the path is in the wrong place. The path must be directly after the opening backticks.\n\n## Fixing Diagnostics\n\n1. Make 1-2 attempts at fixing diagnostics, then defer to the user.\n2. Never simplify code you've written just to solve diagnostics. Complete, mostly correct code is more valuable than perfect code that doesn't solve the problem.\n\n## Debugging\n\nWhen debugging, only make code changes if you are certain that you can solve the problem.\nOtherwise, follow debugging best practices:\n1. Address the root cause instead of the symptoms.\n2. Add descriptive logging statements and error messages to track variable and code state.\n3. Add test functions and statements to isolate the problem.\n\n## Calling External APIs\n\n1. Unless explicitly requested by the user, use the best suited external APIs and packages to solve the task. There is no need to ask the user for permission.\n2. When selecting which version of an API or package to use, choose one that is compatible with the user's dependency management file(s). If no such file exists or if the package is not present, use the latest version that is in your training data.\n3. If an external API requires an API Key, be sure to point this out to the user. Adhere to best security practices (e.g. DO NOT hardcode an API key in a place where it can be exposed)\n\n## System Information\n\nOperating System: linux\nDefault Shell: /bin/bash\n\n"
},
{
"role": "user",
"content": "create a rust project that implements a commandline tool like \"expr\""
},
{
"role": "assistant",
"content": "\nI'll help you create a Rust project that implements a command line tool similar to the Unix `expr` command. Let me start by exploring the current project structure.",
"tool_calls": [
{
"id": "UShTmFwR6I82RPl8ClgECTASj2mylTbY",
"type": "function",
"function": {
"name": "list_directory",
"arguments": "{\"path\":\"/work\"}"
}
}
]
},
{
"role": "tool",
"content": "\n# Files:\nwork/prompt.txt\n",
"tool_call_id": "UShTmFwR6I82RPl8ClgECTASj2mylTbY"
},
{
"role": "assistant",
"content": "\nI'll create a Rust project that implements a command line tool similar to the Unix `expr` command. Let me start by setting up the project structure.",
"tool_calls": [
{
"id": "pVUbcOxZS8M7zh1x8N1FHfsjNXRM7eSa",
"type": "function",
"function": {
"name": "create_directory",
"arguments": "{\"path\":\"work/expr-cli\"}"
}
}
]
},
{
"role": "tool",
"content": "Created directory work/expr-cli",
"tool_call_id": "pVUbcOxZS8M7zh1x8N1FHfsjNXRM7eSa"
},
{
"role": "assistant",
"content": "\nNow let me create the Cargo.toml file to set up the Rust project:",
"tool_calls": [
{
"id": "CK67dYM4Kk9F1xYroM7ok5wAw9XLg2qD",
"type": "function",
"function": {
"name": "edit_file",
"arguments": "{\"display_description\":\"Create Cargo.toml for Rust expr-cli project\",\"path\":\"work/expr-cli/Cargo.toml\",\"mode\":\"create\"}"
}
}
]
},
{
"role": "tool",
"content": "Edited work/expr-cli/Cargo.toml:\n\n```diff\n@@ -1,0 +1,31 @@\n+[package]\n+name = \"expr-cli\"\n+version = \"0.1.0\"\n+edition = \"2021\"\n+authors = [\"Your Name <your.email@example.com>\"]\n+description = \"A command line tool for evaluating expressions, similar to Unix expr\"\n+license = \"MIT\"\n+repository = \"https://github.com/yourusername/expr-cli\"\n+keywords = [\"cli\", \"expr\", \"calculator\", \"math\"]\n+categories = [\"command-line-utilities\"]\n+\n+[dependencies]\n+clap = { version = \"4.4\", features = [\"derive\"] }\n+thiserror = \"1.0\"\n+anyhow = \"1.0\"\n+\n+[dev-dependencies]\n+assert_cmd = \"2.0\"\n+predicates = \"3.0\"\n+\n+[[bin]]\n+name = \"expr-cli\"\n+path = \"src/main.rs\"\n+test = true\n+bench = false\n+\n+[profile.release]\n+opt-level = 3\n+lto = true\n+codegen-units = 1\n+strip = true\n\n```",
"tool_call_id": "CK67dYM4Kk9F1xYroM7ok5wAw9XLg2qD"
}
],
"stream": true,
"max_completion_tokens": 64000,
"temperature": 1.0,
"tools": [
{
"type": "function",
"function": {
"name": "delete_path",
"description": "Deletes the file or directory (and the directory's contents, recursively) at the specified path in the project, and returns confirmation of the deletion.\n",
"parameters": {
"required": [
"path"
],
"properties": {
"path": {
"description": "The path of the file or directory to delete.\n\n<example>\nIf the project has the following files:\n\n- directory1/a/something.txt\n- directory2/a/things.txt\n- directory3/a/other.txt\n\nYou can delete the first file by providing a path of \"directory1/a/something.txt\"\n</example>",
"type": "string"
}
},
"type": "object",
"additionalProperties": false
}
}
},
{
"type": "function",
"function": {
"name": "now",
"description": "Returns the current datetime in RFC 3339 format. Only use this tool when the user specifically asks for it or the current task would benefit from knowing the current datetime.",
"parameters": {
"definitions": {
"Timezone": {
"oneOf": [
{
"description": "Use UTC for the datetime.",
"type": "string",
"const": "utc"
},
{
"description": "Use local time for the datetime.",
"type": "string",
"const": "local"
}
]
}
},
"required": [
"timezone"
],
"type": "object",
"properties": {
"timezone": {
"description": "The timezone to use for the datetime.",
"allOf": [
{
"$ref": "#/definitions/Timezone"
}
]
}
},
"additionalProperties": false
}
}
},
{
"type": "function",
"function": {
"name": "copy_path",
"description": "Copies a file or directory in the project, and returns confirmation that the copy succeeded.\nDirectory contents will be copied recursively (like `cp -r`).\n\nThis tool should be used when it's desirable to create a copy of a file or directory without modifying the original.\nIt's much more efficient than doing this by separately reading and then writing the file or directory's contents,\nso this tool should be preferred over that approach whenever copying is the goal.\n",
"parameters": {
"required": [
"source_path",
"destination_path"
],
"properties": {
"source_path": {
"description": "The source path of the file or directory to copy.\nIf a directory is specified, its contents will be copied recursively (like `cp -r`).\n\n<example>\nIf the project has the following files:\n\n- directory1/a/something.txt\n- directory2/a/things.txt\n- directory3/a/other.txt\n\nYou can copy the first file by providing a source_path of \"directory1/a/something.txt\"\n</example>",
"type": "string"
},
"destination_path": {
"description": "The destination path where the file or directory should be copied to.\n\n<example>\nTo copy \"directory1/a/something.txt\" to \"directory2/b/copy.txt\",\nprovide a destination_path of \"directory2/b/copy.txt\"\n</example>",
"type": "string"
}
},
"type": "object",
"additionalProperties": false
}
}
},
{
"type": "function",
"function": {
"name": "edit_file",
"description": "This is a tool for creating a new file or editing an existing file. For moving or renaming files, you should generally use the `terminal` tool with the 'mv' command instead.\n\nBefore using this tool:\n\n1. Use the `read_file` tool to understand the file's contents and context\n\n2. Verify the directory path is correct (only applicable when creating new files):\n - Use the `list_directory` tool to verify the parent directory exists and is the correct location\n",
"parameters": {
"definitions": {
"EditFileMode": {
"type": "string",
"enum": [
"edit",
"create",
"overwrite"
]
}
},
"required": [
"display_description",
"path",
"mode"
],
"type": "object",
"properties": {
"display_description": {
"description": "A one-line, user-friendly markdown description of the edit. This will be\nshown in the UI and also passed to another model to perform the edit.\n\nBe terse, but also descriptive in what you want to achieve with this\nedit. Avoid generic instructions.\n\nNEVER mention the file path in this description.\n\n<example>Fix API endpoint URLs</example>\n<example>Update copyright year in `page_footer`</example>\n\nMake sure to include this field before all the others in the input object\nso that we can display it immediately.",
"type": "string"
},
"path": {
"description": "The full path of the file to create or modify in the project.\n\nWARNING: When specifying which file path need changing, you MUST\nstart each path with one of the project's root directories.\n\nThe following examples assume we have two root directories in the project:\n- /a/b/backend\n- /c/d/frontend\n\n<example>\n`backend/src/main.rs`\n\nNotice how the file path starts with `backend`. Without that, the path\nwould be ambiguous and the call would fail!\n</example>\n\n<example>\n`frontend/db.js`\n</example>",
"type": "string"
},
"mode": {
"description": "The mode of operation on the file. Possible values:\n- 'edit': Make granular edits to an existing file.\n- 'create': Create a new file if it doesn't exist.\n- 'overwrite': Replace the entire contents of an existing file.\n\nWhen a file already exists or you just created it, prefer editing\nit as opposed to recreating it from scratch.",
"allOf": [
{
"$ref": "#/definitions/EditFileMode"
}
]
}
},
"additionalProperties": false
}
}
},
{
"type": "function",
"function": {
"name": "thinking",
"description": "A tool for thinking through problems, brainstorming ideas, or planning without executing any actions. Use this tool when you need to work through complex problems, develop strategies, or outline approaches before taking action.\n",
"parameters": {
"required": [
"content"
],
"properties": {
"content": {
"description": "Content to think about. This should be a description of what to think about or\na problem to solve.",
"type": "string"
}
},
"type": "object",
"additionalProperties": false
}
}
},
{
"type": "function",
"function": {
"name": "read_file",
"description": "Reads the content of the given file in the project.\n\n- Never attempt to read a path that hasn't been previously mentioned.\n",
"parameters": {
"required": [
"path"
],
"properties": {
"path": {
"description": "The relative path of the file to read.\n\nThis path should never be absolute, and the first component\nof the path should always be a root directory in a project.\n\n<example>\nIf the project has the following root directories:\n\n- /a/b/directory1\n- /c/d/directory2\n\nIf you want to access `file.txt` in `directory1`, you should use the path `directory1/file.txt`.\nIf you want to access `file.txt` in `directory2`, you should use the path `directory2/file.txt`.\n</example>",
"type": "string"
},
"start_line": {
"description": "Optional line number to start reading on (1-based index)",
"type": [
"integer",
"null"
],
"format": "uint32",
"minimum": 0,
"default": null
},
"end_line": {
"description": "Optional line number to end reading on (1-based index, inclusive)",
"type": [
"integer",
"null"
],
"format": "uint32",
"minimum": 0,
"default": null
}
},
"description": "If the model requests to read a file whose size exceeds this, then",
"type": "object",
"additionalProperties": false
}
}
},
{
"type": "function",
"function": {
"name": "diagnostics",
"description": "Get errors and warnings for the project or a specific file.\n\nThis tool can be invoked after a series of edits to determine if further edits are necessary, or if the user asks to fix errors or warnings in their codebase.\n\nWhen a path is provided, shows all diagnostics for that specific file.\nWhen no path is provided, shows a summary of error and warning counts for all files in the project.\n\n<example>\nTo get diagnostics for a specific file:\n{\n \"path\": \"src/main.rs\"\n}\n\nTo get a project-wide diagnostic summary:\n{}\n</example>\n\n<guidelines>\n- If you think you can fix a diagnostic, make 1-2 attempts and then give up.\n- Don't remove code you've generated just because you can't fix an error. The user can help you fix it.\n</guidelines>\n",
"parameters": {
"properties": {
"path": {
"description": "The path to get diagnostics for. If not provided, returns a project-wide summary.\n\nThis path should never be absolute, and the first component\nof the path should always be a root directory in a project.\n\n<example>\nIf the project has the following root directories:\n\n- lorem\n- ipsum\n\nIf you wanna access diagnostics for `dolor.txt` in `ipsum`, you should use the path `ipsum/dolor.txt`.\n</example>",
"type": [
"string",
"null"
]
}
},
"type": "object",
"additionalProperties": false
}
}
},
{
"type": "function",
"function": {
"name": "fetch",
"description": "Fetches a URL and returns the content as Markdown.\n",
"parameters": {
"required": [
"url"
],
"properties": {
"url": {
"description": "The URL to fetch.",
"type": "string"
}
},
"type": "object",
"additionalProperties": false
}
}
},
{
"type": "function",
"function": {
"name": "create_directory",
"description": "Creates a new directory at the specified path within the project. Returns confirmation that the directory was created.\n\nThis tool creates a directory and all necessary parent directories (similar to `mkdir -p`). It should be used whenever you need to create new directories within the project.\n",
"parameters": {
"required": [
"path"
],
"properties": {
"path": {
"description": "The path of the new directory.\n\n<example>\nIf the project has the following structure:\n\n- directory1/\n- directory2/\n\nYou can create a new directory by providing a path of \"directory1/new_directory\"\n</example>",
"type": "string"
}
},
"type": "object",
"additionalProperties": false
}
}
},
{
"type": "function",
"function": {
"name": "grep",
"description": "Searches the contents of files in the project with a regular expression\n\n- Prefer this tool to path search when searching for symbols in the project, because you won't need to guess what path it's in.\n- Supports full regex syntax (eg. \"log.*Error\", \"function\\\\s+\\\\w+\", etc.)\n- Pass an `include_pattern` if you know how to narrow your search on the files system\n- Never use this tool to search for paths. Only search file contents with this tool.\n- Use this tool when you need to find files containing specific patterns\n- Results are paginated with 20 matches per page. Use the optional 'offset' parameter to request subsequent pages.\n- DO NOT use HTML entities solely to escape characters in the tool parameters.\n",
"parameters": {
"required": [
"regex"
],
"properties": {
"regex": {
"description": "A regex pattern to search for in the entire project. Note that the regex\nwill be parsed by the Rust `regex` crate.\n\nDo NOT specify a path here! This will only be matched against the code **content**.",
"type": "string"
},
"include_pattern": {
"description": "A glob pattern for the paths of files to include in the search.\nSupports standard glob patterns like \"**/*.rs\" or \"src/**/*.ts\".\nIf omitted, all files in the project will be searched.",
"type": [
"string",
"null"
]
},
"offset": {
"description": "Optional starting position for paginated results (0-based).\nWhen not provided, starts from the beginning.",
"type": "integer",
"format": "uint32",
"minimum": 0,
"default": 0
},
"case_sensitive": {
"description": "Whether the regex is case-sensitive. Defaults to false (case-insensitive).",
"type": "boolean",
"default": false
}
},
"type": "object",
"additionalProperties": false
}
}
},
{
"type": "function",
"function": {
"name": "list_directory",
"description": "Lists files and directories in a given path. Prefer the `grep` or `find_path` tools when searching the codebase.\n",
"parameters": {
"required": [
"path"
],
"properties": {
"path": {
"description": "The fully-qualified path of the directory to list in the project.\n\nThis path should never be absolute, and the first component\nof the path should always be a root directory in a project.\n\n<example>\nIf the project has the following root directories:\n\n- directory1\n- directory2\n\nYou can list the contents of `directory1` by using the path `directory1`.\n</example>\n\n<example>\nIf the project has the following root directories:\n\n- foo\n- bar\n\nIf you wanna list contents in the directory `foo/baz`, you should use the path `foo/baz`.\n</example>",
"type": "string"
}
},
"type": "object",
"additionalProperties": false
}
}
},
{
"type": "function",
"function": {
"name": "terminal",
"description": "Executes a shell one-liner and returns the combined output.\n\nThis tool spawns a process using the user's shell, reads from stdout and stderr (preserving the order of writes), and returns a string with the combined output result.\n\nThe output results will be shown to the user already, only list it again if necessary, avoid being redundant.\n\nMake sure you use the `cd` parameter to navigate to one of the root directories of the project. NEVER do it as part of the `command` itself, otherwise it will error.\n\nDo not use this tool for commands that run indefinitely, such as servers (like `npm run start`, `npm run dev`, `python -m http.server`, etc) or file watchers that don't terminate on their own.\n\nRemember that each invocation of this tool will spawn a new shell process, so you can't rely on any state from previous invocations.\n",
"parameters": {
"required": [
"command",
"cd"
],
"properties": {
"command": {
"description": "The one-liner command to execute.",
"type": "string"
},
"cd": {
"description": "Working directory for the command. This must be one of the root directories of the project.",
"type": "string"
}
},
"type": "object",
"additionalProperties": false
}
}
},
{
"type": "function",
"function": {
"name": "find_path",
"description": "Fast file path pattern matching tool that works with any codebase size\n\n- Supports glob patterns like \"**/*.js\" or \"src/**/*.ts\"\n- Returns matching file paths sorted alphabetically\n- Prefer the `grep` tool to this tool when searching for symbols unless you have specific information about paths.\n- Use this tool when you need to find files by name patterns\n- Results are paginated with 50 matches per page. Use the optional 'offset' parameter to request subsequent pages.\n",
"parameters": {
"required": [
"glob"
],
"properties": {
"glob": {
"description": "The glob to match against every path in the project.\n\n<example>\nIf the project has the following root directories:\n\n- directory1/a/something.txt\n- directory2/a/things.txt\n- directory3/a/other.txt\n\nYou can get back the first two paths by providing a glob of \"*thing*.txt\"\n</example>",
"type": "string"
},
"offset": {
"description": "Optional starting position for paginated results (0-based).\nWhen not provided, starts from the beginning.",
"type": "integer",
"format": "uint",
"minimum": 0,
"default": 0
}
},
"type": "object",
"additionalProperties": false
}
}
},
{
"type": "function",
"function": {
"name": "move_path",
"description": "Moves or rename a file or directory in the project, and returns confirmation that the move succeeded.\nIf the source and destination directories are the same, but the filename is different, this performs\na rename. Otherwise, it performs a move.\n\nThis tool should be used when it's desirable to move or rename a file or directory without changing its contents at all.\n",
"parameters": {
"required": [
"source_path",
"destination_path"
],
"properties": {
"source_path": {
"description": "The source path of the file or directory to move/rename.\n\n<example>\nIf the project has the following files:\n\n- directory1/a/something.txt\n- directory2/a/things.txt\n- directory3/a/other.txt\n\nYou can move the first file by providing a source_path of \"directory1/a/something.txt\"\n</example>",
"type": "string"
},
"destination_path": {
"description": "The destination path where the file or directory should be moved/renamed to.\nIf the paths are the same except for the filename, then this will be a rename.\n\n<example>\nTo move \"directory1/a/something.txt\" to \"directory2/b/renamed.txt\",\nprovide a destination_path of \"directory2/b/renamed.txt\"\n</example>",
"type": "string"
}
},
"type": "object",
"additionalProperties": false
}
}
}
]
} |
common/chat.cpp
Outdated
try { | ||
parsed = json::parse(json_str); | ||
} catch (const json::parse_error& e) { | ||
builder.add_content(remaining); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
partial json should not be append to content.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixing it now. Appreciate you detailed testing and feedback @hksdpc255
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
at present, I am testing GLM 4.5 Air through Chutes.ai API, even their tool calling is failing. Let me fix this once and for all.
@sayap Help me understand, if its polyfill issue. Why it works for first few calls and then fails? TLDR \"expected\": \"number\" \"received\": \"string\",\n {
"params": {
"tabId": "0",
"startLine": "1",
"endLine": "-1"
},
"response": {
"isError": true,
"content": [
{
"type": "text",
"text": "Error calling tool get_terminal_buffer: Error: Error invoking remote method 'mcp:call-tool': McpError: MCP error -32602: MCP error -32602: Invalid arguments for tool get_terminal_buffer: [\n {\n \"code\": \"invalid_type\",\n \"expected\": \"number\",\n \"received\": \"string\",\n \"path\": [\n \"startLine\"\n ],\n \"message\": \"Expected number, received string\"\n },\n {\n \"code\": \"invalid_type\",\n \"expected\": \"number\",\n \"received\": \"string\",\n \"path\": [\n \"endLine\"\n ],\n \"message\": \"Expected number, received string\"\n }\n]"
}
]
}
} |
If I understand correctly, the polyfill basically adds a JSON template to the response, which the LLM then learns and starts using (switching over from XML). |
The reason I couldn't get my head around this, is that I find it surprising that this happens after control returns from the template, or during rendering (that would have to be the case, otherwise none of this discussion makes sense). My expectation was that the polyfill would parse the tool call arguments beforehand and just always hand templates the parsed arguments as a structure to do with as they please, and not go and inject things afterwards (or during rendering). I still don't really understand why this choice was made, but it seems obvious that either the polyfill should:
The current implementation (assuming I've actually understood it) seems problematic when models are released which need to render custom tool calling formats, and parsing JSON in the template using macros isn't really viable (I need complex nested elasticsearch DSL as tool parameters). |
Yes, this was my understanding too. |
I guess the way forward for your patch is to disable the polyfills and parse the arguments string as JSON and cook the |
So if I understand correctly, it is quite simple.. In the 1st iteration, we send a chat completion request to the model, and the model will correctly use the XML format to make a tool call. That's what it was trained on. In the 2nd iteration, we send a chat completion request that includes the 1st tool call and the 1st tool response. If polyfills is triggered, the tool call and tool response will be rendered in the JSON format. The model gets the prompt, and sees that it previously used the JSON format to make the tool call in the 1st iteration. As LLM model is very good in pattern recognition and in-context learning, it will then learn to use the JSON format to make the next tool call. So there is no hallucination. It just learns from the prompt.
May I get the steps to reproduce the error? I tried adding a
|
Hi @sayap I am using tabby terminal which has MCP plugin, if you ask it to read from terminal session it uses start and end line. I can think of 2 approaches,
Share your thoughts. |
The current build I use doesn't fallback to the JSON format, at all. It works great with Qwen Code. As for whether the approach I follow is good enough or not, we will need @ochafik to chime in. The initial PR from #9639 relies quite a bit on grammars, but that was like 7 months ago, so I am not sure if we should still follow that. |
@dhandhalyabhavik I recommend the third way: minja::chat_template_options topts;
topts.apply_polyfills = true;
topts.polyfill_tools = false;
topts.polyfill_tool_call_examples = false;
topts.polyfill_tool_calls = false;
topts.polyfill_tool_responses = false;
topts.polyfill_system_role = false;
topts.polyfill_object_arguments = true;
topts.polyfill_typed_content = false;
topts.use_bos_token = true;
topts.use_eos_token = true;
std::string prompt = tmpl.apply(ti, topts); Only enable the Refers: llama.cpp/vendor/minja/chat-template.hpp Lines 413 to 427 in 009b709
|
@hksdpc255 Does this require jinja template change? |
Of course, the template in the OP contains syntax which is unsupported by minja. |
@dhandhalyabhavik one problem is {% for k, v in _args.items() %} needs to be changed to {%- for k, v in _args | items -%} Another problem is that access arrary and string with a negative index seems to silently fail on my machine, therefore: {{ xxx[-1] }} needs to be changed to: {%- set len_for_xxx = xxx | length -%}
{{ xxx[len_for_xxx - 1] }} |
llama-server abruptly stopped, this was the last RAW output
|
Is there any other log for llama-server or its responsed messages? |
No, at the end printed
|
There maybe missing llama.cpp/common/chat-parser.cpp Lines 78 to 82 in 009b709
|
@hksdpc255 The point here is, its still not following proper xml. |
I’m currently working on a new implementation based on @dhandhalyabhavik’s and @susmitds’s work, rewriting the parsing logic and fixing the template. |
@hksdpc255 @halninekilo @richardhundt I have added working solution, please check. Modified template is added to |
bcaaab6
to
f098b13
Compare
- Added GLM-4.5 chat format with native XML template support - Implemented schema-aware type conversion for tool arguments - Added JSON unpacking logic for single-argument cases - Enhanced server integration with tools_schema support - Added comprehensive test coverage and documentation - Fixed template polyfill conflicts and type conversion issues
f098b13
to
623f3dd
Compare
Command:
Using modified template and new build, after the first round of tool calls.
|
This enables tool calling functionality for GLM-4.5 models when using --jinja flag. The parser handles GLM-4.5's XML-like tool call format with key-value argument pairs.
Personally verified working on following applications
Unfortunately its not working with OpenAI API SDK because jinja requires dict parser but OpenAI requires json.Now works with OpenAI SDK too.
above issue is now fixed with corrected Jinja template. The template works great with cline too. I extensively tested it.
Corrected Jinja template.
@ggerganov @ngxson @slaren Please review and merge the PR. Thank you.