Skip to content

Conversation

dhandhalyabhavik
Copy link

@dhandhalyabhavik dhandhalyabhavik commented Aug 8, 2025

  • Add COMMON_CHAT_FORMAT_GLM_4_5 format enum
  • Implement GLM-4.5 tool call parser for <tool_call><arg_key><arg_value> format
  • Add template detection based on <arg_key> and <arg_value> tags
  • Fix null content handling in message parsing and serialization
  • Ensure GLM-4.5 detection runs before Hermes to avoid misidentification

This enables tool calling functionality for GLM-4.5 models when using --jinja flag. The parser handles GLM-4.5's XML-like tool call format with key-value argument pairs.

Personally verified working on following applications

  • Cline
  • Roo Code
  • Kilo Code
  • Cherry Studio (MCP + Tool calling)

Unfortunately its not working with OpenAI API SDK because jinja requires dict parser but OpenAI requires json.

Now works with OpenAI SDK too.
above issue is now fixed with corrected Jinja template. The template works great with cline too. I extensively tested it.

Corrected Jinja template.

[gMASK]<sop>
{%- if tools -%}
<|system|>
# Tools

You may call one or more functions to assist with the user query.

You are provided with function signatures within <tools></tools> XML tags:
<tools>
{% for tool in tools %}
{{ tool | tojson }}
{% endfor %}
</tools>

For each function call, output the function name and arguments within the following XML format:
<tool_call>{function-name}
<arg_key>{arg-key-1}</arg_key>
<arg_value>{arg-value-1}</arg_value>
<arg_key>{arg-key-2}</arg_key>
<arg_value>{arg-value-2}</arg_value>
...
</tool_call>{%- endif -%}
{%- macro visible_text(content) -%}
    {%- if content is string -%}
        {{- content }}
    {%- elif content is iterable and content is not mapping -%}
        {%- for item in content -%}
            {%- if item is mapping and item.type == 'text' -%}
                {{- item.text }}
            {%- elif item is string -%}
                {{- item }}
            {%- endif -%}
        {%- endfor -%}
    {%- else -%}
        {{- content }}
    {%- endif -%}
{%- endmacro -%}
{%- set ns = namespace(last_user_index=-1) %}
{%- for m in messages %}
    {%- if m.role == 'user' %}
        {% set ns.last_user_index = loop.index0 -%}
    {%- endif %}
{%- endfor %}
{% for m in messages %}
{%- if m.role == 'user' -%}<|user|>
{%- set user_content = visible_text(m.content) -%}
{{ user_content }}
{%- if enable_thinking is defined and not enable_thinking -%}
{%- if not user_content.endswith("/nothink") -%}
{{- '/nothink' -}}
{%- endif -%}
{%- endif -%}
{%- elif m.role == 'assistant' -%}
<|assistant|>
{%- set reasoning_content = '' %}
{%- set content = visible_text(m.content) %}
{%- if m.reasoning_content is string %}
    {%- set reasoning_content = m.reasoning_content %}
{%- else %}
    {%- if '</think>' in content %}
        {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
        {%- set content = content.split('</think>')[-1].lstrip('\n') %}
    {%- endif %}
{%- endif %}
{%- if loop.index0 > ns.last_user_index and reasoning_content -%}
{{ '\n<think>' + reasoning_content.strip() +  '</think>'}}
{%- else -%}
{{ '\n<think></think>' }}
{%- endif -%}
{%- if content.strip() -%}
{{ '\n' + content.strip() }}
{%- endif -%}
{% if m.tool_calls %}
{% for tc in m.tool_calls %}
{%- if tc.function %}
    {%- set tc = tc.function %}
{%- endif %}
{{ '\n<tool_call>' + tc.name }}
{% set _args = tc.arguments %}
{% for k, v in _args.items() %}
<arg_key>{{ k }}</arg_key>
<arg_value>{{ v | tojson if v is not string else v }}</arg_value>
{% endfor %}
</tool_call>{% endfor %}
{% endif %}
{%- elif m.role == 'tool' -%}
{%- if m.content is string -%}
{%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
    {{- '<|observation|>' }}
{%- endif %}
{{- '\n<tool_response>\n' }}
{{- m.content }}
{{- '\n</tool_response>' }}
{%- else -%}
<|observation|>{% for tr in m.content %}

<tool_response>
{{ tr.output if tr.output is defined else tr }}
</tool_response>{% endfor -%}
{% endif -%}
{%- elif m.role == 'system' -%}
<|system|>
{{ visible_text(m.content) }}
{%- endif -%}
{%- endfor -%}
{%- if add_generation_prompt -%}
    <|assistant|>{{- '\n<think></think>' if (enable_thinking is defined and not enable_thinking) else '' -}}
{%- endif -%}

@ggerganov @ngxson @slaren Please review and merge the PR. Thank you.

- Add COMMON_CHAT_FORMAT_GLM_4_5 format enum
- Implement GLM-4.5 tool call parser for <tool_call><arg_key><arg_value> format
- Add template detection based on <arg_key> and <arg_value> tags
- Fix null content handling in message parsing and serialization
- Ensure GLM-4.5 detection runs before Hermes to avoid misidentification

This enables tool calling functionality for GLM-4.5 models when using --jinja flag.
The parser handles GLM-4.5's XML-like tool call format with key-value argument pairs.
@ajunca
Copy link

ajunca commented Aug 9, 2025

I tried the PR, and it fixes tool calling on GLM 4.5 Air (unsloth version) getting called correctly.
Then though this other problem #15046 arise.

@dhandhalyabhavik
Copy link
Author

I tried the PR, and it fixes tool calling on GLM 4.5 Air (unsloth version) getting called correctly. Then though this other problem #15046 arise.

But its Qwen tool calling issue right? I think once other pending PRs are merged you should not see the issue.

@ajunca
Copy link

ajunca commented Aug 9, 2025

Yea, I don't think is related to this specific PR. But the problem is shared with this Qwen tool calling issue.

@dhandhalyabhavik
Copy link
Author

dhandhalyabhavik commented Aug 10, 2025

Cline

Works great now with Cline 💪,

image.png

Cherry studio with MCP

Works great with MCP settings too 🔥.

image.png

Kilo code

works great.

image

@TNohSam
Copy link

TNohSam commented Aug 10, 2025

Hey, quick thought — I might be misunderstanding this, but it looks like this PR will parse GLM’s XML-style tool calls and turn them into JSON tool_calls before they reach the client.

If that’s the case, projects like Roo Code (which currently only know how to handle XML tool calls) might suddenly stop recognizing the output from GLM models when running through llama.cpp.

Am I right about this?

@jfgonsalves
Copy link

Does this template parse the thinking tags correctly? I'm getting my responses inline instead of in the reasoning_content field.

@bfroemel
Copy link

Very nice!

#15162 aims to achieve the same for Qwen3 Coder; only seems more mature/higher quality (using minja and letting it handle quoting/escaping argument strings, storing the jinja template in ./models/templates, having test cases in ./tests/test-chat.cpp,). Maybe @ochafik and @dhandhalyabhavik can sync up/collaborate and bring both PRs in a consistent way forward?

@dhandhalyabhavik
Copy link
Author

dhandhalyabhavik commented Aug 11, 2025

Hello everyone, thanks for insightful comments, Let me answer all of you,

@TNohSam There are two ways to implement tool calling,
(1) use instruction following template, write parsing code and parse manually.
(2) OpenAI compatible tool calling where functions or tools are part of their chat object class <--- This is what people refer when they say model supports tool calling

I have tested Roo Code just now, it is working fine. Both type of function or tool calling will work with the current PR.

@jfgonsalves enable reasoning_content via llama-server's flag. Check flags.

@bfroemel sure, @ochafik can you please review my added changes? Help me merge this PR. I would really appreciate. Thank you.

@dhandhalyabhavik
Copy link
Author

@jfgonsalves

You can enable reasoning_content via flag.

There is parser logic common for all models that will do this job. Check out the code here

This PR has nothing to do with it. Thank you for pointing it out though.

check it our here
image

@trilog-inc
Copy link

I am still having trouble getting llama.cpp to identify the GLM-4.5 chat template. Am I missing something in my command?

srv params_from_: Chat format: Hermes 2 Pro

./build/bin/llama-server --model /mnt/home_extend/models/unsloth_GLM-4.5-GGUF/Q5_K_M/GLM-4.5-Q5_K_M-00001-of-00006.gguf --alias glm-4.5 --no-webui --threads 44 --ctx-size 131072 --n-gpu-layers 94 -ot exps=CPU -ub 2048 -b 2048 --temp 0.6 --top-p 1.0 --flash-attn --host 0.0.0.0 --jinja --port 8099 --chat-template-file /mnt/home_extend/models/unsloth_GLM-4.5-GGUF/template.jinja

@dhandhalyabhavik
Copy link
Author

dhandhalyabhavik commented Aug 14, 2025

I am still having trouble getting llama.cpp to identify the GLM-4.5 chat template. Am I missing something in my command?

srv params_from_: Chat format: Hermes 2 Pro

./build/bin/llama-server --model /mnt/home_extend/models/unsloth_GLM-4.5-GGUF/Q5_K_M/GLM-4.5-Q5_K_M-00001-of-00006.gguf --alias glm-4.5 --no-webui --threads 44 --ctx-size 131072 --n-gpu-layers 94 -ot exps=CPU -ub 2048 -b 2048 --temp 0.6 --top-p 1.0 --flash-attn --host 0.0.0.0 --jinja --port 8099 --chat-template-file /mnt/home_extend/models/unsloth_GLM-4.5-GGUF/template.jinja

Hey @trilog-inc

I rebuild and its working fine for me. I got GLM 4.5 in my logs, tool calling also works well.

srv params_from_: Chat format: GLM 4.5

I hope you have build it correctly using clone repo -> switch to common-glm45-tool-calls branch -> build -> test

FYI, I have used GLM 4.5 Air for testing. Both should work as I can see both of them have same Arch & jinja template.

I have used this command, (re-copy jinja template, there there is a modification recommended by a user)

./llama.cpp/build/bin/llama-server -hf unsloth/GLM-4.5-Air-GGUF:IQ2_M --alias GLM-4.5-Air-GPUs -c 60000 --host 0.0.0.0 -np 1 -ngl 999 -ts 72,28 -b 1024 -ub 256 --jinja --chat-template-file template/chat_template.jinja

@feliscat
Copy link

feliscat commented Aug 18, 2025

Trying this PR, tools get called and get a response but the model can't continue with them.

I do see the right chat format as expected in the logs and did build the PR correctly.

srv params_from_: Chat format: GLM 4.5

srv  log_server_r: request: GET /v1/models 192.168.1.56 200

got exception: {"code":500,"message":"Value is not callable: null at row 56, column 70:\n (...) ","type":"server_error"}

srv  log_server_r: request: POST /v1/chat/completions 192.168.1.56 500

I'm using https://huggingface.co/unsloth/GLM-4.5-Air-GGUF?chat_template=default as the chat template

Edit: Using the jinja template in the OP fixed the issue.

@jerrydeng
Copy link

I had similar problem. Not sure what chat template to use

@feliscat
Copy link

@jerrydeng

I copied the template from the OP's edit and used it, and now tool calling is working for me.

@jerrydeng
Copy link

It worked like a charm, thanks

@ggerganov ggerganov requested a review from ochafik August 19, 2025 19:29
@jerrydeng
Copy link

jerrydeng commented Aug 19, 2025

For claude code / openai, this template would work

{%- macro visible_text(content) -%}
    {%- if content is string -%}
        {{- content }}
    {%- elif content is iterable and content is not mapping -%}
        {%- for item in content -%}
            {%- if item is mapping and item.type == 'text' -%}
                {{- item.text }}
            {%- elif item is string -%}
                {{- item }}
            {%- endif -%}
        {%- endfor -%}
    {%- else -%}
        {{- content }}
    {%- endif -%}
{%- endmacro -%}

{#--- SYSTEM + TOOLS (OpenAI-style JSON) ---#}
{%- if tools -%}
<|system|>
You can call functions ("tools") to complete tasks. Tools are provided as JSON below.
Respond NORMALLY unless a tool is needed.
If you decide to call a tool, output EXACTLY ONE LINE containing ONLY this JSON:

{"tool_call":{"name":"<function name>","arguments":{...}}}

Rules:
- The JSON must be strictly valid (double quotes, no trailing commas).
- Put all arguments under "arguments" as an object.
- Do not add commentary or extra text on that line.

TOOLS (OpenAI schema):
{{ tools | tojson }}
{%- endif -%}

{#--- find last user index for think gating ---#}
{%- set ns = namespace(last_user_index=-1) %}
{%- for m in messages %}
  {%- if m.role == 'user' %}
    {%- set user_content = visible_text(m.content) -%}
    {%- if not ("tool_response" in user_content) %}
      {%- set ns.last_user_index = loop.index0 -%}
    {%- endif -%}
  {%- endif %}
{%- endfor %}

{#--- MESSAGE RENDERING ---#}
{%- for m in messages %}

  {%- if m.role == 'system' -%}
<|system|>
{{ visible_text(m.content) }}

  {%- elif m.role == 'user' -%}
<|user|>
{%- set user_content = visible_text(m.content) -%}
{{ user_content }}
{%- if enable_thinking is defined and not enable_thinking -%}
  {%- if not user_content.endswith("/nothink") -%}
/nothink
  {%- endif -%}
{%- endif -%}

  {%- elif m.role == 'assistant' -%}
<|assistant|>
{%- set content = visible_text(m.content) %}
{%- set reasoning_content = '' %}

{# pull <think> block out of prior assistant messages if present #}
{%- if m.reasoning_content is string %}
  {%- set reasoning_content = m.reasoning_content %}
{%- else %}
  {%- if '</think>' in content %}
    {%- set think_parts = content.split('</think>') %}
    {%- if think_parts|length > 1 %}
      {%- set before_end_think = think_parts[0] %}
      {%- set after_end_think = think_parts[1] %}
      {%- set think_start_parts = before_end_think.split('<think>') %}
      {%- if think_start_parts|length > 1 %}
        {%- set reasoning_content = think_start_parts[-1].lstrip('\n') %}
      {%- endif %}
      {%- set content = after_end_think.lstrip('\n') %}
    {%- endif %}
  {%- endif %}
{%- endif %}

{%- if loop.index0 > ns.last_user_index and reasoning_content -%}
<think>{{ reasoning_content.strip() }}</think>
{%- else -%}
<think></think>
{%- endif -%}

{# normal assistant text, if any #}
{%- if content.strip() -%}
{{ '\n' + content.strip() }}
{%- endif -%}

{# tool call ECHO SUPPORT (when upstream passed tool_calls back into history) #}
{%- if m.tool_calls %}
{%- for tc in m.tool_calls %}
{%- set f = tc.function if tc.function else tc %}
{%- set args = f.arguments if f.arguments else {} %}
{{ '\n{"tool_call":{"name":"' ~ f.name ~ '","arguments":' ~ (args if args is string else (args | tojson)) ~ '}}' }}
{%- endfor %}
{%- endif %}

  {%- elif m.role == 'tool' -%}
{# Tool outputs are shown as observations for the model to read #}
<|observation|>
{%- if m.content is string -%}
<tool_response>
{{ m.content }}
</tool_response>
{%- else -%}
{%- for tr in m.content %}
<tool_response>
{{ tr.output if tr.output is defined else tr }}
</tool_response>
{%- endfor -%}
{%- endif -%}

  {%- endif -%}
{%- endfor %}

{#--- generation prompt ---#}
{%- if add_generation_prompt -%}
<|assistant|>
{%- if enable_thinking is defined and not enable_thinking -%}<think></think>{%- endif -%}
{%- endif -%}

This worked with Claude Code and Claude Code Router.

@hksdpc255
Copy link

hksdpc255 commented Aug 20, 2025

@jfgonsalves

You can enable reasoning_content via flag.

There is parser logic common for all models that will do this job. Check out the code here

This PR has nothing to do with it. Thank you for pointing it out though.

check it our here image

@dhandhalyabhavik I don't thin the <think> appears in content is something related to reasoning_content flag.

See: #15186 (comment)_

I changed builder.try_parse_reasoning("<think>", "</think>"); in function common_chat_parse_glm_4_5 to builder.try_parse_reasoning("\n<think>", "</think>");, then the reasoning working with your chat template.

@dhandhalyabhavik
Copy link
Author

Hey @hksdpc255

can you please review again. GLM 4.5 produces \n<think>...</think>\n .. content style response.
The problem was partial generation, so the only way to fix it was by adding support for both type of thinking (more on chat.cpp)

Tested working on

  • Cline
  • Roo Code
  • Kilo Code
  • Cherry Studio (MCP + tool calling)
  • My own multi-agent program (GLM tool calling is == OpenAI tool calling now)

Copy link
Collaborator

@ngxson ngxson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I only have time to do a quick review. The code looks good overall, though I haven't had time to test it.

I think we can merge after @hksdpc255 approve this PR

@halninekilo
Copy link

in case its helpful / relevant: tried this with the template from OP, GLM 4.5 AIR @Q4_K_M and function calling via Cherry Studio and ran into -

image

@hksdpc255
Copy link

hksdpc255 commented Aug 21, 2025

@dhandhalyabhavik I test it with zed editor.

after 2 or 3 tool calling, llama-server crashed with something like:

  last ends with: [
] (0x0a)
  current starts with: [P] (0x50)
terminate called after throwing an instance of 'std::runtime_error'
  what():  Invalid diff: '{
  "tool_calls": [
    {
      "name": "list_directory",
      "arguments": {
        "path": "."
      },
      "id": "hP3d3XmQf1sVjJl9N8mZcY7eK6wQ4bR2"
    }
  ],
  "content": "Perfect! The Rust project has been created. Let me check the directory structure to see what files were generated."
' not found at start of 'Perfect! The Rust project has been created. Let me check the directory structure to see what files were generated.'

Screenshot of zed editor:

image

Then I test it with claude-code combined with claude-code-router, same problem.

  last ends with: [
] (0x0a)
  current starts with: [(] (0x28)
terminate called after throwing an instance of 'std::runtime_error'
  what():  Invalid diff: '{
  "tool_calls": [
    {
      "name": "TodoWrite",
      "arguments": {
        "todos": [
          {
            "content": "Explore the QEMU vvfat.c implementation to understand the full requirements",
            "status": "in_progress"
          },
          {
            "content": "Create new Rust project structure with proper Cargo.toml configuration",
            "status": "pending"
          },
          {
            "content": "Research and identify Rust vfat libraries available for VFAT filesystem handling",
            "status": "pending"
          },
          {
            "content": "Design and implement core block device abstraction layer",
            "status": "pending"
          },
          {
            "content": "Implement vhost-user protocol support for block device operations",
            "status": "pending"
          },
          {
            "content": "Add NBD device support with compilation flag",
            "status": "pending"
          },
          {
            "content": "Implement file operations (read, write, create, delete) matching QEMU's vvfat functionality",
            "status": "pending"
          },
          {
            "content": "Break down complex functionality into modular components",
            "status": "pending"
          },
          {
            "content": "Write comprehensive unit tests for VFAT filesystem operations",
            "status": "pending"
          },
          {
            "content": "Write integration tests for vhost-user interface functionality",
            "status": "pending"
          }
        ]
      },
      "id": "zRQ0uK1y2E7Jv4xM3p6R9nT8sUf2d1gP"
    }
  ],
  "content": "(no content)"
' not found at start of '(no content)'
image2

The runtime_error seems thrown at: https://github.com/ggml-org/llama.cpp/pull/15186/files#diff-2580689a73e0d42f06cf4b5ed02e7cf87c6fc874343bce9c6f0e1ebf190d95b7R68

@halninekilo
Copy link

halninekilo commented Aug 21, 2025

meaybe...

diff --git a/common/chat.cpp b/common/chat.cpp
index 07e65539..a5fc6db4 100644
--- a/common/chat.cpp
+++ b/common/chat.cpp
@@ -1389,9 +1389,8 @@ static void common_chat_parse_glm_4_5(common_chat_msg_parser & builder) {
                 json parsed;
                 try {
                     parsed = json::parse(json_str);
-                } catch (const json::parse_error& e) {
-                    builder.add_content(remaining);
-                    return;
+                } catch (const json::parse_error &) {
+                    throw common_chat_msg_partial_exception("incomplete GLM JSON tool call");
                 }

@hksdpc255
Copy link

Its seems that the model generates something like:

<think>... [omitted by me]</think>
{
  "tool_calls": [
    {
      "name": "edit_file",
      "arguments": {
        "display_description": "Create main.rs with expr-cli implementation",
        "path": "work/expr-cli/src/main.rs",
        "mode": "create"
      },
      "id": "KD7F8xKzL1t7wZ2x9N4JXt8aO3YcR7aD"
    }
  ],
  "content": "\nNow let me create the main source file that implements the expr-cli functionality:"
}

Then llama-server crash.

Here's a request that always triggers a crash:

curl 'http://llama.server:port/v1/chat/completions' -H "Content-Type: application/json" -d '{"model":"local","messages":[{"role":"system","content":"You are a highly skilled software engineer with extensive knowledge in many programming languages, frameworks, design patterns, and best practices.\n\n## Communication\n\n1. Be conversational but professional.\n2. Refer to the user in the second person and yourself in the first person.\n3. Format your responses in markdown. Use backticks to format file, directory, function, and class names.\n4. NEVER lie or make things up.\n5. Refrain from apologizing all the time when results are unexpected. Instead, just try your best to proceed or explain the circumstances to the user without apologizing.\n\n## Tool Use\n\n1. Make sure to adhere to the tools schema.\n2. Provide every required argument.\n3. DO NOT use tools to access items that are already available in the context section.\n4. Use only the tools that are currently available.\n5. DO NOT use a tool that is not available just because it appears in the conversation. This means the user turned it off.\n6. NEVER run commands that don'\''t terminate on their own such as web servers (like `npm run start`, `npm run dev`, `python -m http.server`, etc) or file watchers.\n7. Avoid HTML entity escaping - use plain characters instead.\n\n## Searching and Reading\n\nIf you are unsure how to fulfill the user'\''s request, gather more information with tool calls and/or clarifying questions.\n\nIf appropriate, use tool calls to explore the current project, which contains the following root directories:\n\n- `/work`\n\n- Bias towards not asking the user for help if you can find the answer yourself.\n- When providing paths to tools, the path should always start with the name of a project root directory listed above.\n- Before you read or edit a file, you must first find the full path. DO NOT ever guess a file path!\n- When looking for symbols in the project, prefer the `grep` tool.\n- As you learn about the structure of the project, use that information to scope `grep` searches to targeted subtrees of the project.\n- The user might specify a partial file path. If you don'\''t know the full path, use `find_path` (not `grep`) before you read the file.\n\n## Code Block Formatting\n\nWhenever you mention a code block, you MUST use ONLY use the following format:\n```path/to/Something.blah#L123-456\n(code goes here)\n```\nThe `#L123-456` means the line number range 123 through 456, and the path/to/Something.blah\nis a path in the project. (If there is no valid path in the project, then you can use\n/dev/null/path.extension for its path.) This is the ONLY valid way to format code blocks, because the Markdown parser\ndoes not understand the more common ```language syntax, or bare ``` blocks. It only\nunderstands this path-based syntax, and if the path is missing, then it will error and you will have to do it over again.\nJust to be really clear about this, if you ever find yourself writing three backticks followed by a language name, STOP!\nYou have made a mistake. You can only ever put paths after triple backticks!\n<example>\nBased on all the information I'\''ve gathered, here'\''s a summary of how this system works:\n1. The README file is loaded into the system.\n2. The system finds the first two headers, including everything in between. In this case, that would be:\n```path/to/README.md#L8-12\n# First Header\nThis is the info under the first header.\n## Sub-header\n```\n3. Then the system finds the last header in the README:\n```path/to/README.md#L27-29\n## Last Header\nThis is the last header in the README.\n```\n4. Finally, it passes this information on to the next process.\n</example>\n<example>\nIn Markdown, hash marks signify headings. For example:\n```/dev/null/example.md#L1-3\n# Level 1 heading\n## Level 2 heading\n### Level 3 heading\n```\n</example>\nHere are examples of ways you must never render code blocks:\n<bad_example_do_not_do_this>\nIn Markdown, hash marks signify headings. For example:\n```\n# Level 1 heading\n## Level 2 heading\n### Level 3 heading\n```\n</bad_example_do_not_do_this>\nThis example is unacceptable because it does not include the path.\n<bad_example_do_not_do_this>\nIn Markdown, hash marks signify headings. For example:\n```markdown\n# Level 1 heading\n## Level 2 heading\n### Level 3 heading\n```\n</bad_example_do_not_do_this>\nThis example is unacceptable because it has the language instead of the path.\n<bad_example_do_not_do_this>\nIn Markdown, hash marks signify headings. For example:\n    # Level 1 heading\n    ## Level 2 heading\n    ### Level 3 heading\n</bad_example_do_not_do_this>\nThis example is unacceptable because it uses indentation to mark the code block\ninstead of backticks with a path.\n<bad_example_do_not_do_this>\nIn Markdown, hash marks signify headings. For example:\n```markdown\n/dev/null/example.md#L1-3\n# Level 1 heading\n## Level 2 heading\n### Level 3 heading\n```\n</bad_example_do_not_do_this>\nThis example is unacceptable because the path is in the wrong place. The path must be directly after the opening backticks.\n\n## Fixing Diagnostics\n\n1. Make 1-2 attempts at fixing diagnostics, then defer to the user.\n2. Never simplify code you'\''ve written just to solve diagnostics. Complete, mostly correct code is more valuable than perfect code that doesn'\''t solve the problem.\n\n## Debugging\n\nWhen debugging, only make code changes if you are certain that you can solve the problem.\nOtherwise, follow debugging best practices:\n1. Address the root cause instead of the symptoms.\n2. Add descriptive logging statements and error messages to track variable and code state.\n3. Add test functions and statements to isolate the problem.\n\n## Calling External APIs\n\n1. Unless explicitly requested by the user, use the best suited external APIs and packages to solve the task. There is no need to ask the user for permission.\n2. When selecting which version of an API or package to use, choose one that is compatible with the user'\''s dependency management file(s). If no such file exists or if the package is not present, use the latest version that is in your training data.\n3. If an external API requires an API Key, be sure to point this out to the user. Adhere to best security practices (e.g. DO NOT hardcode an API key in a place where it can be exposed)\n\n## System Information\n\nOperating System: linux\nDefault Shell: /bin/bash\n\n"},{"role":"user","content":"create a rust project that implements a commandline tool like \"expr\""},{"role":"assistant","content":"\nI'\''ll help you create a Rust project that implements a command line tool similar to the Unix `expr` command. Let me start by exploring the current project structure.","tool_calls":[{"id":"UShTmFwR6I82RPl8ClgECTASj2mylTbY","type":"function","function":{"name":"list_directory","arguments":"{\"path\":\"/work\"}"}}]},{"role":"tool","content":"\n# Files:\nwork/prompt.txt\n","tool_call_id":"UShTmFwR6I82RPl8ClgECTASj2mylTbY"},{"role":"assistant","content":"\nI'\''ll create a Rust project that implements a command line tool similar to the Unix `expr` command. Let me start by setting up the project structure.","tool_calls":[{"id":"pVUbcOxZS8M7zh1x8N1FHfsjNXRM7eSa","type":"function","function":{"name":"create_directory","arguments":"{\"path\":\"work/expr-cli\"}"}}]},{"role":"tool","content":"Created directory work/expr-cli","tool_call_id":"pVUbcOxZS8M7zh1x8N1FHfsjNXRM7eSa"},{"role":"assistant","content":"\nNow let me create the Cargo.toml file to set up the Rust project:","tool_calls":[{"id":"CK67dYM4Kk9F1xYroM7ok5wAw9XLg2qD","type":"function","function":{"name":"edit_file","arguments":"{\"display_description\":\"Create Cargo.toml for Rust expr-cli project\",\"path\":\"work/expr-cli/Cargo.toml\",\"mode\":\"create\"}"}}]},{"role":"tool","content":"Edited work/expr-cli/Cargo.toml:\n\n```diff\n@@ -1,0 +1,31 @@\n+[package]\n+name = \"expr-cli\"\n+version = \"0.1.0\"\n+edition = \"2021\"\n+authors = [\"Your Name <your.email@example.com>\"]\n+description = \"A command line tool for evaluating expressions, similar to Unix expr\"\n+license = \"MIT\"\n+repository = \"https://github.com/yourusername/expr-cli\"\n+keywords = [\"cli\", \"expr\", \"calculator\", \"math\"]\n+categories = [\"command-line-utilities\"]\n+\n+[dependencies]\n+clap = { version = \"4.4\", features = [\"derive\"] }\n+thiserror = \"1.0\"\n+anyhow = \"1.0\"\n+\n+[dev-dependencies]\n+assert_cmd = \"2.0\"\n+predicates = \"3.0\"\n+\n+[[bin]]\n+name = \"expr-cli\"\n+path = \"src/main.rs\"\n+test = true\n+bench = false\n+\n+[profile.release]\n+opt-level = 3\n+lto = true\n+codegen-units = 1\n+strip = true\n\n```","tool_call_id":"CK67dYM4Kk9F1xYroM7ok5wAw9XLg2qD"}],"stream":true,"max_completion_tokens":64000,"temperature":1.0,"tools":[{"type":"function","function":{"name":"delete_path","description":"Deletes the file or directory (and the directory'\''s contents, recursively) at the specified path in the project, and returns confirmation of the deletion.\n","parameters":{"required":["path"],"properties":{"path":{"description":"The path of the file or directory to delete.\n\n<example>\nIf the project has the following files:\n\n- directory1/a/something.txt\n- directory2/a/things.txt\n- directory3/a/other.txt\n\nYou can delete the first file by providing a path of \"directory1/a/something.txt\"\n</example>","type":"string"}},"type":"object","additionalProperties":false}}},{"type":"function","function":{"name":"now","description":"Returns the current datetime in RFC 3339 format. Only use this tool when the user specifically asks for it or the current task would benefit from knowing the current datetime.","parameters":{"definitions":{"Timezone":{"oneOf":[{"description":"Use UTC for the datetime.","type":"string","const":"utc"},{"description":"Use local time for the datetime.","type":"string","const":"local"}]}},"required":["timezone"],"type":"object","properties":{"timezone":{"description":"The timezone to use for the datetime.","allOf":[{"$ref":"#/definitions/Timezone"}]}},"additionalProperties":false}}},{"type":"function","function":{"name":"copy_path","description":"Copies a file or directory in the project, and returns confirmation that the copy succeeded.\nDirectory contents will be copied recursively (like `cp -r`).\n\nThis tool should be used when it'\''s desirable to create a copy of a file or directory without modifying the original.\nIt'\''s much more efficient than doing this by separately reading and then writing the file or directory'\''s contents,\nso this tool should be preferred over that approach whenever copying is the goal.\n","parameters":{"required":["source_path","destination_path"],"properties":{"source_path":{"description":"The source path of the file or directory to copy.\nIf a directory is specified, its contents will be copied recursively (like `cp -r`).\n\n<example>\nIf the project has the following files:\n\n- directory1/a/something.txt\n- directory2/a/things.txt\n- directory3/a/other.txt\n\nYou can copy the first file by providing a source_path of \"directory1/a/something.txt\"\n</example>","type":"string"},"destination_path":{"description":"The destination path where the file or directory should be copied to.\n\n<example>\nTo copy \"directory1/a/something.txt\" to \"directory2/b/copy.txt\",\nprovide a destination_path of \"directory2/b/copy.txt\"\n</example>","type":"string"}},"type":"object","additionalProperties":false}}},{"type":"function","function":{"name":"edit_file","description":"This is a tool for creating a new file or editing an existing file. For moving or renaming files, you should generally use the `terminal` tool with the '\''mv'\'' command instead.\n\nBefore using this tool:\n\n1. Use the `read_file` tool to understand the file'\''s contents and context\n\n2. Verify the directory path is correct (only applicable when creating new files):\n   - Use the `list_directory` tool to verify the parent directory exists and is the correct location\n","parameters":{"definitions":{"EditFileMode":{"type":"string","enum":["edit","create","overwrite"]}},"required":["display_description","path","mode"],"type":"object","properties":{"display_description":{"description":"A one-line, user-friendly markdown description of the edit. This will be\nshown in the UI and also passed to another model to perform the edit.\n\nBe terse, but also descriptive in what you want to achieve with this\nedit. Avoid generic instructions.\n\nNEVER mention the file path in this description.\n\n<example>Fix API endpoint URLs</example>\n<example>Update copyright year in `page_footer`</example>\n\nMake sure to include this field before all the others in the input object\nso that we can display it immediately.","type":"string"},"path":{"description":"The full path of the file to create or modify in the project.\n\nWARNING: When specifying which file path need changing, you MUST\nstart each path with one of the project'\''s root directories.\n\nThe following examples assume we have two root directories in the project:\n- /a/b/backend\n- /c/d/frontend\n\n<example>\n`backend/src/main.rs`\n\nNotice how the file path starts with `backend`. Without that, the path\nwould be ambiguous and the call would fail!\n</example>\n\n<example>\n`frontend/db.js`\n</example>","type":"string"},"mode":{"description":"The mode of operation on the file. Possible values:\n- '\''edit'\'': Make granular edits to an existing file.\n- '\''create'\'': Create a new file if it doesn'\''t exist.\n- '\''overwrite'\'': Replace the entire contents of an existing file.\n\nWhen a file already exists or you just created it, prefer editing\nit as opposed to recreating it from scratch.","allOf":[{"$ref":"#/definitions/EditFileMode"}]}},"additionalProperties":false}}},{"type":"function","function":{"name":"thinking","description":"A tool for thinking through problems, brainstorming ideas, or planning without executing any actions. Use this tool when you need to work through complex problems, develop strategies, or outline approaches before taking action.\n","parameters":{"required":["content"],"properties":{"content":{"description":"Content to think about. This should be a description of what to think about or\na problem to solve.","type":"string"}},"type":"object","additionalProperties":false}}},{"type":"function","function":{"name":"read_file","description":"Reads the content of the given file in the project.\n\n- Never attempt to read a path that hasn'\''t been previously mentioned.\n","parameters":{"required":["path"],"properties":{"path":{"description":"The relative path of the file to read.\n\nThis path should never be absolute, and the first component\nof the path should always be a root directory in a project.\n\n<example>\nIf the project has the following root directories:\n\n- /a/b/directory1\n- /c/d/directory2\n\nIf you want to access `file.txt` in `directory1`, you should use the path `directory1/file.txt`.\nIf you want to access `file.txt` in `directory2`, you should use the path `directory2/file.txt`.\n</example>","type":"string"},"start_line":{"description":"Optional line number to start reading on (1-based index)","type":["integer","null"],"format":"uint32","minimum":0,"default":null},"end_line":{"description":"Optional line number to end reading on (1-based index, inclusive)","type":["integer","null"],"format":"uint32","minimum":0,"default":null}},"description":"If the model requests to read a file whose size exceeds this, then","type":"object","additionalProperties":false}}},{"type":"function","function":{"name":"diagnostics","description":"Get errors and warnings for the project or a specific file.\n\nThis tool can be invoked after a series of edits to determine if further edits are necessary, or if the user asks to fix errors or warnings in their codebase.\n\nWhen a path is provided, shows all diagnostics for that specific file.\nWhen no path is provided, shows a summary of error and warning counts for all files in the project.\n\n<example>\nTo get diagnostics for a specific file:\n{\n    \"path\": \"src/main.rs\"\n}\n\nTo get a project-wide diagnostic summary:\n{}\n</example>\n\n<guidelines>\n- If you think you can fix a diagnostic, make 1-2 attempts and then give up.\n- Don'\''t remove code you'\''ve generated just because you can'\''t fix an error. The user can help you fix it.\n</guidelines>\n","parameters":{"properties":{"path":{"description":"The path to get diagnostics for. If not provided, returns a project-wide summary.\n\nThis path should never be absolute, and the first component\nof the path should always be a root directory in a project.\n\n<example>\nIf the project has the following root directories:\n\n- lorem\n- ipsum\n\nIf you wanna access diagnostics for `dolor.txt` in `ipsum`, you should use the path `ipsum/dolor.txt`.\n</example>","type":["string","null"]}},"type":"object","additionalProperties":false}}},{"type":"function","function":{"name":"fetch","description":"Fetches a URL and returns the content as Markdown.\n","parameters":{"required":["url"],"properties":{"url":{"description":"The URL to fetch.","type":"string"}},"type":"object","additionalProperties":false}}},{"type":"function","function":{"name":"create_directory","description":"Creates a new directory at the specified path within the project. Returns confirmation that the directory was created.\n\nThis tool creates a directory and all necessary parent directories (similar to `mkdir -p`). It should be used whenever you need to create new directories within the project.\n","parameters":{"required":["path"],"properties":{"path":{"description":"The path of the new directory.\n\n<example>\nIf the project has the following structure:\n\n- directory1/\n- directory2/\n\nYou can create a new directory by providing a path of \"directory1/new_directory\"\n</example>","type":"string"}},"type":"object","additionalProperties":false}}},{"type":"function","function":{"name":"grep","description":"Searches the contents of files in the project with a regular expression\n\n- Prefer this tool to path search when searching for symbols in the project, because you won'\''t need to guess what path it'\''s in.\n- Supports full regex syntax (eg. \"log.*Error\", \"function\\\\s+\\\\w+\", etc.)\n- Pass an `include_pattern` if you know how to narrow your search on the files system\n- Never use this tool to search for paths. Only search file contents with this tool.\n- Use this tool when you need to find files containing specific patterns\n- Results are paginated with 20 matches per page. Use the optional '\''offset'\'' parameter to request subsequent pages.\n- DO NOT use HTML entities solely to escape characters in the tool parameters.\n","parameters":{"required":["regex"],"properties":{"regex":{"description":"A regex pattern to search for in the entire project. Note that the regex\nwill be parsed by the Rust `regex` crate.\n\nDo NOT specify a path here! This will only be matched against the code **content**.","type":"string"},"include_pattern":{"description":"A glob pattern for the paths of files to include in the search.\nSupports standard glob patterns like \"**/*.rs\" or \"src/**/*.ts\".\nIf omitted, all files in the project will be searched.","type":["string","null"]},"offset":{"description":"Optional starting position for paginated results (0-based).\nWhen not provided, starts from the beginning.","type":"integer","format":"uint32","minimum":0,"default":0},"case_sensitive":{"description":"Whether the regex is case-sensitive. Defaults to false (case-insensitive).","type":"boolean","default":false}},"type":"object","additionalProperties":false}}},{"type":"function","function":{"name":"list_directory","description":"Lists files and directories in a given path. Prefer the `grep` or `find_path` tools when searching the codebase.\n","parameters":{"required":["path"],"properties":{"path":{"description":"The fully-qualified path of the directory to list in the project.\n\nThis path should never be absolute, and the first component\nof the path should always be a root directory in a project.\n\n<example>\nIf the project has the following root directories:\n\n- directory1\n- directory2\n\nYou can list the contents of `directory1` by using the path `directory1`.\n</example>\n\n<example>\nIf the project has the following root directories:\n\n- foo\n- bar\n\nIf you wanna list contents in the directory `foo/baz`, you should use the path `foo/baz`.\n</example>","type":"string"}},"type":"object","additionalProperties":false}}},{"type":"function","function":{"name":"terminal","description":"Executes a shell one-liner and returns the combined output.\n\nThis tool spawns a process using the user'\''s shell, reads from stdout and stderr (preserving the order of writes), and returns a string with the combined output result.\n\nThe output results will be shown to the user already, only list it again if necessary, avoid being redundant.\n\nMake sure you use the `cd` parameter to navigate to one of the root directories of the project. NEVER do it as part of the `command` itself, otherwise it will error.\n\nDo not use this tool for commands that run indefinitely, such as servers (like `npm run start`, `npm run dev`, `python -m http.server`, etc) or file watchers that don'\''t terminate on their own.\n\nRemember that each invocation of this tool will spawn a new shell process, so you can'\''t rely on any state from previous invocations.\n","parameters":{"required":["command","cd"],"properties":{"command":{"description":"The one-liner command to execute.","type":"string"},"cd":{"description":"Working directory for the command. This must be one of the root directories of the project.","type":"string"}},"type":"object","additionalProperties":false}}},{"type":"function","function":{"name":"find_path","description":"Fast file path pattern matching tool that works with any codebase size\n\n- Supports glob patterns like \"**/*.js\" or \"src/**/*.ts\"\n- Returns matching file paths sorted alphabetically\n- Prefer the `grep` tool to this tool when searching for symbols unless you have specific information about paths.\n- Use this tool when you need to find files by name patterns\n- Results are paginated with 50 matches per page. Use the optional '\''offset'\'' parameter to request subsequent pages.\n","parameters":{"required":["glob"],"properties":{"glob":{"description":"The glob to match against every path in the project.\n\n<example>\nIf the project has the following root directories:\n\n- directory1/a/something.txt\n- directory2/a/things.txt\n- directory3/a/other.txt\n\nYou can get back the first two paths by providing a glob of \"*thing*.txt\"\n</example>","type":"string"},"offset":{"description":"Optional starting position for paginated results (0-based).\nWhen not provided, starts from the beginning.","type":"integer","format":"uint","minimum":0,"default":0}},"type":"object","additionalProperties":false}}},{"type":"function","function":{"name":"move_path","description":"Moves or rename a file or directory in the project, and returns confirmation that the move succeeded.\nIf the source and destination directories are the same, but the filename is different, this performs\na rename. Otherwise, it performs a move.\n\nThis tool should be used when it'\''s desirable to move or rename a file or directory without changing its contents at all.\n","parameters":{"required":["source_path","destination_path"],"properties":{"source_path":{"description":"The source path of the file or directory to move/rename.\n\n<example>\nIf the project has the following files:\n\n- directory1/a/something.txt\n- directory2/a/things.txt\n- directory3/a/other.txt\n\nYou can move the first file by providing a source_path of \"directory1/a/something.txt\"\n</example>","type":"string"},"destination_path":{"description":"The destination path where the file or directory should be moved/renamed to.\nIf the paths are the same except for the filename, then this will be a rename.\n\n<example>\nTo move \"directory1/a/something.txt\" to \"directory2/b/renamed.txt\",\nprovide a destination_path of \"directory2/b/renamed.txt\"\n</example>","type":"string"}},"type":"object","additionalProperties":false}}}]}'

RAW request body:

{
    "model": "local",
    "messages": [
        {
            "role": "system",
            "content": "You are a highly skilled software engineer with extensive knowledge in many programming languages, frameworks, design patterns, and best practices.\n\n## Communication\n\n1. Be conversational but professional.\n2. Refer to the user in the second person and yourself in the first person.\n3. Format your responses in markdown. Use backticks to format file, directory, function, and class names.\n4. NEVER lie or make things up.\n5. Refrain from apologizing all the time when results are unexpected. Instead, just try your best to proceed or explain the circumstances to the user without apologizing.\n\n## Tool Use\n\n1. Make sure to adhere to the tools schema.\n2. Provide every required argument.\n3. DO NOT use tools to access items that are already available in the context section.\n4. Use only the tools that are currently available.\n5. DO NOT use a tool that is not available just because it appears in the conversation. This means the user turned it off.\n6. NEVER run commands that don't terminate on their own such as web servers (like `npm run start`, `npm run dev`, `python -m http.server`, etc) or file watchers.\n7. Avoid HTML entity escaping - use plain characters instead.\n\n## Searching and Reading\n\nIf you are unsure how to fulfill the user's request, gather more information with tool calls and/or clarifying questions.\n\nIf appropriate, use tool calls to explore the current project, which contains the following root directories:\n\n- `/work`\n\n- Bias towards not asking the user for help if you can find the answer yourself.\n- When providing paths to tools, the path should always start with the name of a project root directory listed above.\n- Before you read or edit a file, you must first find the full path. DO NOT ever guess a file path!\n- When looking for symbols in the project, prefer the `grep` tool.\n- As you learn about the structure of the project, use that information to scope `grep` searches to targeted subtrees of the project.\n- The user might specify a partial file path. If you don't know the full path, use `find_path` (not `grep`) before you read the file.\n\n## Code Block Formatting\n\nWhenever you mention a code block, you MUST use ONLY use the following format:\n```path/to/Something.blah#L123-456\n(code goes here)\n```\nThe `#L123-456` means the line number range 123 through 456, and the path/to/Something.blah\nis a path in the project. (If there is no valid path in the project, then you can use\n/dev/null/path.extension for its path.) This is the ONLY valid way to format code blocks, because the Markdown parser\ndoes not understand the more common ```language syntax, or bare ``` blocks. It only\nunderstands this path-based syntax, and if the path is missing, then it will error and you will have to do it over again.\nJust to be really clear about this, if you ever find yourself writing three backticks followed by a language name, STOP!\nYou have made a mistake. You can only ever put paths after triple backticks!\n<example>\nBased on all the information I've gathered, here's a summary of how this system works:\n1. The README file is loaded into the system.\n2. The system finds the first two headers, including everything in between. In this case, that would be:\n```path/to/README.md#L8-12\n# First Header\nThis is the info under the first header.\n## Sub-header\n```\n3. Then the system finds the last header in the README:\n```path/to/README.md#L27-29\n## Last Header\nThis is the last header in the README.\n```\n4. Finally, it passes this information on to the next process.\n</example>\n<example>\nIn Markdown, hash marks signify headings. For example:\n```/dev/null/example.md#L1-3\n# Level 1 heading\n## Level 2 heading\n### Level 3 heading\n```\n</example>\nHere are examples of ways you must never render code blocks:\n<bad_example_do_not_do_this>\nIn Markdown, hash marks signify headings. For example:\n```\n# Level 1 heading\n## Level 2 heading\n### Level 3 heading\n```\n</bad_example_do_not_do_this>\nThis example is unacceptable because it does not include the path.\n<bad_example_do_not_do_this>\nIn Markdown, hash marks signify headings. For example:\n```markdown\n# Level 1 heading\n## Level 2 heading\n### Level 3 heading\n```\n</bad_example_do_not_do_this>\nThis example is unacceptable because it has the language instead of the path.\n<bad_example_do_not_do_this>\nIn Markdown, hash marks signify headings. For example:\n    # Level 1 heading\n    ## Level 2 heading\n    ### Level 3 heading\n</bad_example_do_not_do_this>\nThis example is unacceptable because it uses indentation to mark the code block\ninstead of backticks with a path.\n<bad_example_do_not_do_this>\nIn Markdown, hash marks signify headings. For example:\n```markdown\n/dev/null/example.md#L1-3\n# Level 1 heading\n## Level 2 heading\n### Level 3 heading\n```\n</bad_example_do_not_do_this>\nThis example is unacceptable because the path is in the wrong place. The path must be directly after the opening backticks.\n\n## Fixing Diagnostics\n\n1. Make 1-2 attempts at fixing diagnostics, then defer to the user.\n2. Never simplify code you've written just to solve diagnostics. Complete, mostly correct code is more valuable than perfect code that doesn't solve the problem.\n\n## Debugging\n\nWhen debugging, only make code changes if you are certain that you can solve the problem.\nOtherwise, follow debugging best practices:\n1. Address the root cause instead of the symptoms.\n2. Add descriptive logging statements and error messages to track variable and code state.\n3. Add test functions and statements to isolate the problem.\n\n## Calling External APIs\n\n1. Unless explicitly requested by the user, use the best suited external APIs and packages to solve the task. There is no need to ask the user for permission.\n2. When selecting which version of an API or package to use, choose one that is compatible with the user's dependency management file(s). If no such file exists or if the package is not present, use the latest version that is in your training data.\n3. If an external API requires an API Key, be sure to point this out to the user. Adhere to best security practices (e.g. DO NOT hardcode an API key in a place where it can be exposed)\n\n## System Information\n\nOperating System: linux\nDefault Shell: /bin/bash\n\n"
        },
        {
            "role": "user",
            "content": "create a rust project that implements a commandline tool like \"expr\""
        },
        {
            "role": "assistant",
            "content": "\nI'll help you create a Rust project that implements a command line tool similar to the Unix `expr` command. Let me start by exploring the current project structure.",
            "tool_calls": [
                {
                    "id": "UShTmFwR6I82RPl8ClgECTASj2mylTbY",
                    "type": "function",
                    "function": {
                        "name": "list_directory",
                        "arguments": "{\"path\":\"/work\"}"
                    }
                }
            ]
        },
        {
            "role": "tool",
            "content": "\n# Files:\nwork/prompt.txt\n",
            "tool_call_id": "UShTmFwR6I82RPl8ClgECTASj2mylTbY"
        },
        {
            "role": "assistant",
            "content": "\nI'll create a Rust project that implements a command line tool similar to the Unix `expr` command. Let me start by setting up the project structure.",
            "tool_calls": [
                {
                    "id": "pVUbcOxZS8M7zh1x8N1FHfsjNXRM7eSa",
                    "type": "function",
                    "function": {
                        "name": "create_directory",
                        "arguments": "{\"path\":\"work/expr-cli\"}"
                    }
                }
            ]
        },
        {
            "role": "tool",
            "content": "Created directory work/expr-cli",
            "tool_call_id": "pVUbcOxZS8M7zh1x8N1FHfsjNXRM7eSa"
        },
        {
            "role": "assistant",
            "content": "\nNow let me create the Cargo.toml file to set up the Rust project:",
            "tool_calls": [
                {
                    "id": "CK67dYM4Kk9F1xYroM7ok5wAw9XLg2qD",
                    "type": "function",
                    "function": {
                        "name": "edit_file",
                        "arguments": "{\"display_description\":\"Create Cargo.toml for Rust expr-cli project\",\"path\":\"work/expr-cli/Cargo.toml\",\"mode\":\"create\"}"
                    }
                }
            ]
        },
        {
            "role": "tool",
            "content": "Edited work/expr-cli/Cargo.toml:\n\n```diff\n@@ -1,0 +1,31 @@\n+[package]\n+name = \"expr-cli\"\n+version = \"0.1.0\"\n+edition = \"2021\"\n+authors = [\"Your Name <your.email@example.com>\"]\n+description = \"A command line tool for evaluating expressions, similar to Unix expr\"\n+license = \"MIT\"\n+repository = \"https://github.com/yourusername/expr-cli\"\n+keywords = [\"cli\", \"expr\", \"calculator\", \"math\"]\n+categories = [\"command-line-utilities\"]\n+\n+[dependencies]\n+clap = { version = \"4.4\", features = [\"derive\"] }\n+thiserror = \"1.0\"\n+anyhow = \"1.0\"\n+\n+[dev-dependencies]\n+assert_cmd = \"2.0\"\n+predicates = \"3.0\"\n+\n+[[bin]]\n+name = \"expr-cli\"\n+path = \"src/main.rs\"\n+test = true\n+bench = false\n+\n+[profile.release]\n+opt-level = 3\n+lto = true\n+codegen-units = 1\n+strip = true\n\n```",
            "tool_call_id": "CK67dYM4Kk9F1xYroM7ok5wAw9XLg2qD"
        }
    ],
    "stream": true,
    "max_completion_tokens": 64000,
    "temperature": 1.0,
    "tools": [
        {
            "type": "function",
            "function": {
                "name": "delete_path",
                "description": "Deletes the file or directory (and the directory's contents, recursively) at the specified path in the project, and returns confirmation of the deletion.\n",
                "parameters": {
                    "required": [
                        "path"
                    ],
                    "properties": {
                        "path": {
                            "description": "The path of the file or directory to delete.\n\n<example>\nIf the project has the following files:\n\n- directory1/a/something.txt\n- directory2/a/things.txt\n- directory3/a/other.txt\n\nYou can delete the first file by providing a path of \"directory1/a/something.txt\"\n</example>",
                            "type": "string"
                        }
                    },
                    "type": "object",
                    "additionalProperties": false
                }
            }
        },
        {
            "type": "function",
            "function": {
                "name": "now",
                "description": "Returns the current datetime in RFC 3339 format. Only use this tool when the user specifically asks for it or the current task would benefit from knowing the current datetime.",
                "parameters": {
                    "definitions": {
                        "Timezone": {
                            "oneOf": [
                                {
                                    "description": "Use UTC for the datetime.",
                                    "type": "string",
                                    "const": "utc"
                                },
                                {
                                    "description": "Use local time for the datetime.",
                                    "type": "string",
                                    "const": "local"
                                }
                            ]
                        }
                    },
                    "required": [
                        "timezone"
                    ],
                    "type": "object",
                    "properties": {
                        "timezone": {
                            "description": "The timezone to use for the datetime.",
                            "allOf": [
                                {
                                    "$ref": "#/definitions/Timezone"
                                }
                            ]
                        }
                    },
                    "additionalProperties": false
                }
            }
        },
        {
            "type": "function",
            "function": {
                "name": "copy_path",
                "description": "Copies a file or directory in the project, and returns confirmation that the copy succeeded.\nDirectory contents will be copied recursively (like `cp -r`).\n\nThis tool should be used when it's desirable to create a copy of a file or directory without modifying the original.\nIt's much more efficient than doing this by separately reading and then writing the file or directory's contents,\nso this tool should be preferred over that approach whenever copying is the goal.\n",
                "parameters": {
                    "required": [
                        "source_path",
                        "destination_path"
                    ],
                    "properties": {
                        "source_path": {
                            "description": "The source path of the file or directory to copy.\nIf a directory is specified, its contents will be copied recursively (like `cp -r`).\n\n<example>\nIf the project has the following files:\n\n- directory1/a/something.txt\n- directory2/a/things.txt\n- directory3/a/other.txt\n\nYou can copy the first file by providing a source_path of \"directory1/a/something.txt\"\n</example>",
                            "type": "string"
                        },
                        "destination_path": {
                            "description": "The destination path where the file or directory should be copied to.\n\n<example>\nTo copy \"directory1/a/something.txt\" to \"directory2/b/copy.txt\",\nprovide a destination_path of \"directory2/b/copy.txt\"\n</example>",
                            "type": "string"
                        }
                    },
                    "type": "object",
                    "additionalProperties": false
                }
            }
        },
        {
            "type": "function",
            "function": {
                "name": "edit_file",
                "description": "This is a tool for creating a new file or editing an existing file. For moving or renaming files, you should generally use the `terminal` tool with the 'mv' command instead.\n\nBefore using this tool:\n\n1. Use the `read_file` tool to understand the file's contents and context\n\n2. Verify the directory path is correct (only applicable when creating new files):\n   - Use the `list_directory` tool to verify the parent directory exists and is the correct location\n",
                "parameters": {
                    "definitions": {
                        "EditFileMode": {
                            "type": "string",
                            "enum": [
                                "edit",
                                "create",
                                "overwrite"
                            ]
                        }
                    },
                    "required": [
                        "display_description",
                        "path",
                        "mode"
                    ],
                    "type": "object",
                    "properties": {
                        "display_description": {
                            "description": "A one-line, user-friendly markdown description of the edit. This will be\nshown in the UI and also passed to another model to perform the edit.\n\nBe terse, but also descriptive in what you want to achieve with this\nedit. Avoid generic instructions.\n\nNEVER mention the file path in this description.\n\n<example>Fix API endpoint URLs</example>\n<example>Update copyright year in `page_footer`</example>\n\nMake sure to include this field before all the others in the input object\nso that we can display it immediately.",
                            "type": "string"
                        },
                        "path": {
                            "description": "The full path of the file to create or modify in the project.\n\nWARNING: When specifying which file path need changing, you MUST\nstart each path with one of the project's root directories.\n\nThe following examples assume we have two root directories in the project:\n- /a/b/backend\n- /c/d/frontend\n\n<example>\n`backend/src/main.rs`\n\nNotice how the file path starts with `backend`. Without that, the path\nwould be ambiguous and the call would fail!\n</example>\n\n<example>\n`frontend/db.js`\n</example>",
                            "type": "string"
                        },
                        "mode": {
                            "description": "The mode of operation on the file. Possible values:\n- 'edit': Make granular edits to an existing file.\n- 'create': Create a new file if it doesn't exist.\n- 'overwrite': Replace the entire contents of an existing file.\n\nWhen a file already exists or you just created it, prefer editing\nit as opposed to recreating it from scratch.",
                            "allOf": [
                                {
                                    "$ref": "#/definitions/EditFileMode"
                                }
                            ]
                        }
                    },
                    "additionalProperties": false
                }
            }
        },
        {
            "type": "function",
            "function": {
                "name": "thinking",
                "description": "A tool for thinking through problems, brainstorming ideas, or planning without executing any actions. Use this tool when you need to work through complex problems, develop strategies, or outline approaches before taking action.\n",
                "parameters": {
                    "required": [
                        "content"
                    ],
                    "properties": {
                        "content": {
                            "description": "Content to think about. This should be a description of what to think about or\na problem to solve.",
                            "type": "string"
                        }
                    },
                    "type": "object",
                    "additionalProperties": false
                }
            }
        },
        {
            "type": "function",
            "function": {
                "name": "read_file",
                "description": "Reads the content of the given file in the project.\n\n- Never attempt to read a path that hasn't been previously mentioned.\n",
                "parameters": {
                    "required": [
                        "path"
                    ],
                    "properties": {
                        "path": {
                            "description": "The relative path of the file to read.\n\nThis path should never be absolute, and the first component\nof the path should always be a root directory in a project.\n\n<example>\nIf the project has the following root directories:\n\n- /a/b/directory1\n- /c/d/directory2\n\nIf you want to access `file.txt` in `directory1`, you should use the path `directory1/file.txt`.\nIf you want to access `file.txt` in `directory2`, you should use the path `directory2/file.txt`.\n</example>",
                            "type": "string"
                        },
                        "start_line": {
                            "description": "Optional line number to start reading on (1-based index)",
                            "type": [
                                "integer",
                                "null"
                            ],
                            "format": "uint32",
                            "minimum": 0,
                            "default": null
                        },
                        "end_line": {
                            "description": "Optional line number to end reading on (1-based index, inclusive)",
                            "type": [
                                "integer",
                                "null"
                            ],
                            "format": "uint32",
                            "minimum": 0,
                            "default": null
                        }
                    },
                    "description": "If the model requests to read a file whose size exceeds this, then",
                    "type": "object",
                    "additionalProperties": false
                }
            }
        },
        {
            "type": "function",
            "function": {
                "name": "diagnostics",
                "description": "Get errors and warnings for the project or a specific file.\n\nThis tool can be invoked after a series of edits to determine if further edits are necessary, or if the user asks to fix errors or warnings in their codebase.\n\nWhen a path is provided, shows all diagnostics for that specific file.\nWhen no path is provided, shows a summary of error and warning counts for all files in the project.\n\n<example>\nTo get diagnostics for a specific file:\n{\n    \"path\": \"src/main.rs\"\n}\n\nTo get a project-wide diagnostic summary:\n{}\n</example>\n\n<guidelines>\n- If you think you can fix a diagnostic, make 1-2 attempts and then give up.\n- Don't remove code you've generated just because you can't fix an error. The user can help you fix it.\n</guidelines>\n",
                "parameters": {
                    "properties": {
                        "path": {
                            "description": "The path to get diagnostics for. If not provided, returns a project-wide summary.\n\nThis path should never be absolute, and the first component\nof the path should always be a root directory in a project.\n\n<example>\nIf the project has the following root directories:\n\n- lorem\n- ipsum\n\nIf you wanna access diagnostics for `dolor.txt` in `ipsum`, you should use the path `ipsum/dolor.txt`.\n</example>",
                            "type": [
                                "string",
                                "null"
                            ]
                        }
                    },
                    "type": "object",
                    "additionalProperties": false
                }
            }
        },
        {
            "type": "function",
            "function": {
                "name": "fetch",
                "description": "Fetches a URL and returns the content as Markdown.\n",
                "parameters": {
                    "required": [
                        "url"
                    ],
                    "properties": {
                        "url": {
                            "description": "The URL to fetch.",
                            "type": "string"
                        }
                    },
                    "type": "object",
                    "additionalProperties": false
                }
            }
        },
        {
            "type": "function",
            "function": {
                "name": "create_directory",
                "description": "Creates a new directory at the specified path within the project. Returns confirmation that the directory was created.\n\nThis tool creates a directory and all necessary parent directories (similar to `mkdir -p`). It should be used whenever you need to create new directories within the project.\n",
                "parameters": {
                    "required": [
                        "path"
                    ],
                    "properties": {
                        "path": {
                            "description": "The path of the new directory.\n\n<example>\nIf the project has the following structure:\n\n- directory1/\n- directory2/\n\nYou can create a new directory by providing a path of \"directory1/new_directory\"\n</example>",
                            "type": "string"
                        }
                    },
                    "type": "object",
                    "additionalProperties": false
                }
            }
        },
        {
            "type": "function",
            "function": {
                "name": "grep",
                "description": "Searches the contents of files in the project with a regular expression\n\n- Prefer this tool to path search when searching for symbols in the project, because you won't need to guess what path it's in.\n- Supports full regex syntax (eg. \"log.*Error\", \"function\\\\s+\\\\w+\", etc.)\n- Pass an `include_pattern` if you know how to narrow your search on the files system\n- Never use this tool to search for paths. Only search file contents with this tool.\n- Use this tool when you need to find files containing specific patterns\n- Results are paginated with 20 matches per page. Use the optional 'offset' parameter to request subsequent pages.\n- DO NOT use HTML entities solely to escape characters in the tool parameters.\n",
                "parameters": {
                    "required": [
                        "regex"
                    ],
                    "properties": {
                        "regex": {
                            "description": "A regex pattern to search for in the entire project. Note that the regex\nwill be parsed by the Rust `regex` crate.\n\nDo NOT specify a path here! This will only be matched against the code **content**.",
                            "type": "string"
                        },
                        "include_pattern": {
                            "description": "A glob pattern for the paths of files to include in the search.\nSupports standard glob patterns like \"**/*.rs\" or \"src/**/*.ts\".\nIf omitted, all files in the project will be searched.",
                            "type": [
                                "string",
                                "null"
                            ]
                        },
                        "offset": {
                            "description": "Optional starting position for paginated results (0-based).\nWhen not provided, starts from the beginning.",
                            "type": "integer",
                            "format": "uint32",
                            "minimum": 0,
                            "default": 0
                        },
                        "case_sensitive": {
                            "description": "Whether the regex is case-sensitive. Defaults to false (case-insensitive).",
                            "type": "boolean",
                            "default": false
                        }
                    },
                    "type": "object",
                    "additionalProperties": false
                }
            }
        },
        {
            "type": "function",
            "function": {
                "name": "list_directory",
                "description": "Lists files and directories in a given path. Prefer the `grep` or `find_path` tools when searching the codebase.\n",
                "parameters": {
                    "required": [
                        "path"
                    ],
                    "properties": {
                        "path": {
                            "description": "The fully-qualified path of the directory to list in the project.\n\nThis path should never be absolute, and the first component\nof the path should always be a root directory in a project.\n\n<example>\nIf the project has the following root directories:\n\n- directory1\n- directory2\n\nYou can list the contents of `directory1` by using the path `directory1`.\n</example>\n\n<example>\nIf the project has the following root directories:\n\n- foo\n- bar\n\nIf you wanna list contents in the directory `foo/baz`, you should use the path `foo/baz`.\n</example>",
                            "type": "string"
                        }
                    },
                    "type": "object",
                    "additionalProperties": false
                }
            }
        },
        {
            "type": "function",
            "function": {
                "name": "terminal",
                "description": "Executes a shell one-liner and returns the combined output.\n\nThis tool spawns a process using the user's shell, reads from stdout and stderr (preserving the order of writes), and returns a string with the combined output result.\n\nThe output results will be shown to the user already, only list it again if necessary, avoid being redundant.\n\nMake sure you use the `cd` parameter to navigate to one of the root directories of the project. NEVER do it as part of the `command` itself, otherwise it will error.\n\nDo not use this tool for commands that run indefinitely, such as servers (like `npm run start`, `npm run dev`, `python -m http.server`, etc) or file watchers that don't terminate on their own.\n\nRemember that each invocation of this tool will spawn a new shell process, so you can't rely on any state from previous invocations.\n",
                "parameters": {
                    "required": [
                        "command",
                        "cd"
                    ],
                    "properties": {
                        "command": {
                            "description": "The one-liner command to execute.",
                            "type": "string"
                        },
                        "cd": {
                            "description": "Working directory for the command. This must be one of the root directories of the project.",
                            "type": "string"
                        }
                    },
                    "type": "object",
                    "additionalProperties": false
                }
            }
        },
        {
            "type": "function",
            "function": {
                "name": "find_path",
                "description": "Fast file path pattern matching tool that works with any codebase size\n\n- Supports glob patterns like \"**/*.js\" or \"src/**/*.ts\"\n- Returns matching file paths sorted alphabetically\n- Prefer the `grep` tool to this tool when searching for symbols unless you have specific information about paths.\n- Use this tool when you need to find files by name patterns\n- Results are paginated with 50 matches per page. Use the optional 'offset' parameter to request subsequent pages.\n",
                "parameters": {
                    "required": [
                        "glob"
                    ],
                    "properties": {
                        "glob": {
                            "description": "The glob to match against every path in the project.\n\n<example>\nIf the project has the following root directories:\n\n- directory1/a/something.txt\n- directory2/a/things.txt\n- directory3/a/other.txt\n\nYou can get back the first two paths by providing a glob of \"*thing*.txt\"\n</example>",
                            "type": "string"
                        },
                        "offset": {
                            "description": "Optional starting position for paginated results (0-based).\nWhen not provided, starts from the beginning.",
                            "type": "integer",
                            "format": "uint",
                            "minimum": 0,
                            "default": 0
                        }
                    },
                    "type": "object",
                    "additionalProperties": false
                }
            }
        },
        {
            "type": "function",
            "function": {
                "name": "move_path",
                "description": "Moves or rename a file or directory in the project, and returns confirmation that the move succeeded.\nIf the source and destination directories are the same, but the filename is different, this performs\na rename. Otherwise, it performs a move.\n\nThis tool should be used when it's desirable to move or rename a file or directory without changing its contents at all.\n",
                "parameters": {
                    "required": [
                        "source_path",
                        "destination_path"
                    ],
                    "properties": {
                        "source_path": {
                            "description": "The source path of the file or directory to move/rename.\n\n<example>\nIf the project has the following files:\n\n- directory1/a/something.txt\n- directory2/a/things.txt\n- directory3/a/other.txt\n\nYou can move the first file by providing a source_path of \"directory1/a/something.txt\"\n</example>",
                            "type": "string"
                        },
                        "destination_path": {
                            "description": "The destination path where the file or directory should be moved/renamed to.\nIf the paths are the same except for the filename, then this will be a rename.\n\n<example>\nTo move \"directory1/a/something.txt\" to \"directory2/b/renamed.txt\",\nprovide a destination_path of \"directory2/b/renamed.txt\"\n</example>",
                            "type": "string"
                        }
                    },
                    "type": "object",
                    "additionalProperties": false
                }
            }
        }
    ]
}

common/chat.cpp Outdated
try {
parsed = json::parse(json_str);
} catch (const json::parse_error& e) {
builder.add_content(remaining);

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

partial json should not be append to content.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixing it now. Appreciate you detailed testing and feedback @hksdpc255

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

at present, I am testing GLM 4.5 Air through Chutes.ai API, even their tool calling is failing. Let me fix this once and for all.

@dhandhalyabhavik
Copy link
Author

dhandhalyabhavik commented Aug 28, 2025

This is a common pattern. Isn't this string argument parsing part of what the polyfills are doing?

@richardhundt That's exactly the problem. The polyfills is triggered as the fallback when the chat template doesn't support string argument, and subsequently the model will learn to do the JSON tool calling from the context with polyfills tool call. In #14758 (comment), DevStrall started doing that in the 2nd iteration.

@sayap Help me understand, if its polyfill issue. Why it works for first few calls and then fails?
What will happen to non-string arguments like number? I tried using the changes you provided here but when arguments are expected in integer format is fails due to string format. Check this out,

TLDR

\"expected\": \"number\" \"received\": \"string\",\n
{
  "params": {
    "tabId": "0",
    "startLine": "1",
    "endLine": "-1"
  },
  "response": {
    "isError": true,
    "content": [
      {
        "type": "text",
        "text": "Error calling tool get_terminal_buffer: Error: Error invoking remote method 'mcp:call-tool': McpError: MCP error -32602: MCP error -32602: Invalid arguments for tool get_terminal_buffer: [\n  {\n    \"code\": \"invalid_type\",\n    \"expected\": \"number\",\n    \"received\": \"string\",\n    \"path\": [\n      \"startLine\"\n    ],\n    \"message\": \"Expected number, received string\"\n  },\n  {\n    \"code\": \"invalid_type\",\n    \"expected\": \"number\",\n    \"received\": \"string\",\n    \"path\": [\n      \"endLine\"\n    ],\n    \"message\": \"Expected number, received string\"\n  }\n]"
      }
    ]
  }
}

@ehoogeveen-medweb
Copy link

If I understand correctly, the polyfill basically adds a JSON template to the response, which the LLM then learns and starts using (switching over from XML).

@richardhundt
Copy link

richardhundt commented Aug 28, 2025

If I understand correctly, the polyfill basically adds a JSON template to the response, which the LLM then learns and starts using (switching over from XML).

The reason I couldn't get my head around this, is that I find it surprising that this happens after control returns from the template, or during rendering (that would have to be the case, otherwise none of this discussion makes sense). My expectation was that the polyfill would parse the tool call arguments beforehand and just always hand templates the parsed arguments as a structure to do with as they please, and not go and inject things afterwards (or during rendering). I still don't really understand why this choice was made, but it seems obvious that either the polyfill should:

  1. parse the arguments beforehand and templates can then safely iterate over keys and values of the arguments, or
  2. pass in the raw arguments string and provide a fromjson for templates to use.

The current implementation (assuming I've actually understood it) seems problematic when models are released which need to render custom tool calling formats, and parsing JSON in the template using macros isn't really viable (I need complex nested elasticsearch DSL as tool parameters).

@dhandhalyabhavik
Copy link
Author

My expectation was that the polyfill would parse the tool call arguments beforehand and just always hand templates the parsed arguments as a structure to do with as they please, and not go and inject things afterwards (or during rendering).

Yes, this was my understanding too.

@richardhundt
Copy link

@dhandhalyabhavik

My expectation was that the polyfill would parse the tool call arguments beforehand and just always hand templates the parsed arguments as a structure to do with as they please, and not go and inject things afterwards (or during rendering).

Yes, this was my understanding too.

I guess the way forward for your patch is to disable the polyfills and parse the arguments string as JSON and cook the message.tool_calls yourself in C++ space and then basically use the original jinja template (with the minja-safe filters ofc).

@sayap
Copy link

sayap commented Aug 28, 2025

@dhandhalyabhavik

Help me understand, if its polyfill issue. Why it works for first few calls and then fails?

So if I understand correctly, it is quite simple..

In the 1st iteration, we send a chat completion request to the model, and the model will correctly use the XML format to make a tool call. That's what it was trained on.

In the 2nd iteration, we send a chat completion request that includes the 1st tool call and the 1st tool response. If polyfills is triggered, the tool call and tool response will be rendered in the JSON format. The model gets the prompt, and sees that it previously used the JSON format to make the tool call in the 1st iteration. As LLM model is very good in pattern recognition and in-context learning, it will then learn to use the JSON format to make the next tool call.

So there is no hallucination. It just learns from the prompt.

What will happen to non-string arguments like number? I tried using the changes you provided #15186 (comment) but when arguments are expected in integer format is fails due to string format.

May I get the steps to reproduce the error? I tried adding a limit parameter with integer type to the list_files function of the ASCII art python script, and it seems to be working fine:

Starting conversation...
User: Can you add some cute ASCII art to art.txt?

--- Iteration 1 ---
Assistant wants to make 1 tool call(s)
Calling function: list_files with args: {'limit': 20}
Function result: Files in directory '.': ...

@dhandhalyabhavik
Copy link
Author

Hi @sayap

I am using tabby terminal which has MCP plugin, if you ask it to read from terminal session it uses start and end line.
But you know what, lets forget this issue right now.
Tell me, What solution do you propose for this xml to json issue. Model automatically falls back to json.

I can think of 2 approaches,

  1. XML support in polyfills
  2. disable polyfills (but that requires jinja template change)

Share your thoughts.

@sayap
Copy link

sayap commented Aug 28, 2025

The current build I use doesn't fallback to the JSON format, at all. It works great with Qwen Code.

As for whether the approach I follow is good enough or not, we will need @ochafik to chime in. The initial PR from #9639 relies quite a bit on grammars, but that was like 7 months ago, so I am not sure if we should still follow that.

@hksdpc255
Copy link

@dhandhalyabhavik I recommend the third way:

    minja::chat_template_options topts;
    topts.apply_polyfills = true;
    topts.polyfill_tools = false;
    topts.polyfill_tool_call_examples = false;
    topts.polyfill_tool_calls = false;
    topts.polyfill_tool_responses = false;
    topts.polyfill_system_role = false;
    topts.polyfill_object_arguments = true;
    topts.polyfill_typed_content = false;
    topts.use_bos_token = true;
    topts.use_eos_token = true;

    std::string prompt = tmpl.apply(ti, topts);

Only enable the polyfill_object_arguments, which will decode the tool call arguments from json encoded string to json.

Refers:

if (polyfill_object_arguments || polyfill_tool_calls) {
for (auto & tool_call : message.at("tool_calls")) {
if (tool_call["type"] == "function") {
auto & function = tool_call.at("function");
auto & arguments = function.at("arguments");
if (arguments.is_string()) {
try {
arguments = json::parse(arguments.get<std::string>());
} catch (const std::exception & ecvt) {
fprintf(stderr, "Failed to parse arguments: %s\n", ecvt.what());
}
}
}
}
}

@dhandhalyabhavik
Copy link
Author

@hksdpc255 Does this require jinja template change?

@hksdpc255
Copy link

Of course, the template in the OP contains syntax which is unsupported by minja.

@hksdpc255
Copy link

@dhandhalyabhavik one problem is

{% for k, v in _args.items() %}

needs to be changed to

{%- for k, v in _args | items -%}

Another problem is that access arrary and string with a negative index seems to silently fail on my machine, therefore:

{{ xxx[-1] }}

needs to be changed to:

{%- set len_for_xxx = xxx | length -%}
{{ xxx[len_for_xxx - 1] }}

@dhandhalyabhavik
Copy link
Author

llama-server abruptly stopped, this was the last RAW output

=== GLM-4.5 RAW INPUT ===
\n<think></think>\n<tool_call>tabby-exec_command\n·{"command":·"nvidia-smi",·"commandExplanation":·"Check·for·NVI
DIA·GPU·card·information",·"tabId":·"0"}</arg_value><|observation|>                                              
=== END RAW INPUT ===

@hksdpc255
Copy link

Is there any other log for llama-server or its responsed messages?

@dhandhalyabhavik
Copy link
Author

No, at the end printed

[Inferior 1 (process 3110467) detached]
terminate called after throwing an instance of 'std::runtime_error'
  what():  Unexpected content at end of input
run.sh: line 14: 3110467 Aborted 

@hksdpc255
Copy link

There maybe missing builder.consume_rest() somewhere.

void common_chat_msg_parser::finish() {
if (!is_partial_ && pos_ != input_.size()) {
throw std::runtime_error("Unexpected content at end of input");// + input_.substr(pos_));
}
}

@dhandhalyabhavik
Copy link
Author

llama-server abruptly stopped, this was the last RAW output

=== GLM-4.5 RAW INPUT ===
\n<think></think>\n<tool_call>tabby-exec_command\n·{"command":·"nvidia-smi",·"commandExplanation":·"Check·for·NVI
DIA·GPU·card·information",·"tabId":·"0"}</arg_value><|observation|>                                              
=== END RAW INPUT ===

@hksdpc255 The point here is, its still not following proper xml.

@hksdpc255
Copy link

I’m currently working on a new implementation based on @dhandhalyabhavik’s and @susmitds’s work, rewriting the parsing logic and fixing the template.

@dhandhalyabhavik
Copy link
Author

dhandhalyabhavik commented Aug 30, 2025

@hksdpc255 @halninekilo @richardhundt I have added working solution, please check.

Modified template is added to models/templates/glm_4_5.jinja template

opencode screenshot
image

@dhandhalyabhavik dhandhalyabhavik force-pushed the common-glm45-tool-calls branch from bcaaab6 to f098b13 Compare August 30, 2025 13:51
@github-actions github-actions bot added testing Everything test related examples devops improvements to build systems and github actions server labels Aug 30, 2025
- Added GLM-4.5 chat format with native XML template support
- Implemented schema-aware type conversion for tool arguments
- Added JSON unpacking logic for single-argument cases
- Enhanced server integration with tools_schema support
- Added comprehensive test coverage and documentation
- Fixed template polyfill conflicts and type conversion issues
@feliscat
Copy link

feliscat commented Aug 31, 2025

@hksdpc255 @halninekilo @richardhundt I have added working solution, please check

Command:

CUDA_VISIBLE_DEVICES=0 llama.cpp.glm/build/bin/llama-server --jinja --flash-attn -c 8000 -m models/GLM-4.5-Air-GGUF/IQ4_XS/GLM-4.5-Air-IQ4_XS-00001-of-00002.gguf --n-gpu-layers 99 --n-cpu-mo
e 32 --host 0.0.0.0 --port 8099 --chat-template-file llama.cpp.glm/models/templates/glm_4_5.jinja

Using modified template and new build, after the first round of tool calls.

got exception: {"code":500,"message":"Value is not callable: null at row 58, column 52:\n {%- set parts = content.split('') -%}\n {%- set before_first_close = parts | f
irst -%}\n ^\n {%- set inner_parts = before_first_close.rstrip('\n').split('') -%}\n at row 58, column 9:\n {%- set pa
rts = content.split('') -%}\n {%- set before_first_close = parts | first -%}\n ^\n {%- set inner_parts = before_first_close.rstrip('\n').split('') -%}\n
at row 56, column 36:\n{%- else %}\n {%- if '' in content %}\n ^\n {%- set parts = content.split('') -%}\n at row 56, column 5:\n{
%- else %}\n {%- if '' in content %}\n ^\n {%- set parts = content.split('') -%}\n at row 55, column 12:\n {%- set reasoning_content = m.reasoning_content %}
n{%- else %}\n ^\n {%- if '' in content %}\n at row 53, column 1:\n{%- set content = visible_text(m.content) %}\n{%- if m.reasoning_content is string %}\n^\n {%- set
reasoning_content = m.reasoning_content %}\n at row 49, column 35:\n{{- '/nothink' if (enable_thinking is defined and not enable_thinking and not user_content.endswith("/nothink")) else ''
-}}\n{%- elif m.role == 'assistant' -%}\n ^\n<|assistant|>\n at row 45, column 1:\n{% for m in messages %}\n{%- if m.role == 'user' -%}<|user|>\n^\n{%- set
user_content = visible_text(m.content) -%}\n at row 44, column 24:\n{%- endfor %}\n{% for m in messages %}\n ^\n{%- if m.role == 'user' -%}<|user|>\n at row 44, column
1:\n{%- endfor %}\n{% for m in messages %}\n^\n{%- if m.role == 'user' -%}<|user|>\n at row 1, column 1:\n[gMASK]\n^\n{%- if tools -%}\n","type":"server_error"}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
devops improvements to build systems and github actions examples server testing Everything test related
Projects
None yet
Development

Successfully merging this pull request may close these issues.