Ensure that function_call_prompt extends system messages following its current schema
#13243
Conversation
…its current schema Fixes BerriAI#11267
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
| message["content"].append({ | ||
| "type": "text", | ||
| "text": f""" {function_prompt}""", | ||
| "cache_control": {"type": "ephemeral"} |
There was a problem hiding this comment.
why add a cache control flag?
There was a problem hiding this comment.
It likely can be removed. I don't really know how cache control works here, and copied the format of the other messages.
| @@ -3711,7 +3711,14 @@ def function_call_prompt(messages: list, functions: list): | |||
| function_added_to_prompt = False | |||
| for message in messages: | |||
| if "system" in message["role"]: | |||
| message["content"] += f""" {function_prompt}""" | |||
| if isinstance(message["content"], str): | |||
There was a problem hiding this comment.
is this even needed anymore? I thought ollama supported function calling now?
There was a problem hiding this comment.
If it does, LiteLLM does not know about it.
Reproduced with Ollama versions
Initially found it with Ollama version 0.7.0 (already should support tools), today I verified it with the most recent version 0.10.1 too. Calling LiteLLM through Claude (e.g. claude --model local "What is the capital of France?")
I run into this bug with the config:
model_list:
- model_name: "local"
litellm_params:
model: "ollama/hf.co/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:UD-Q4_K_XL"
max_tokens: 32684
repetition_penalty: 1.05
temperature: 0.7
top_k: 20
top_p: 0.8
litellm_settings:
master_key: os.environ/LITELLM_MASTER_KEY
Question
Are you saying that the "bug" is that add_function_to_prompt is set to True?
I've seen that by default it's False, and somewhere the code sets it to True. Did not look into its place yet.
There was a problem hiding this comment.
It could be that the qwen3 model does not support tool calling, and add_function_to_prompt is set in
There was a problem hiding this comment.
It would be nice to remove this added string from the system prompt altogether, because certain Ollama models are trained to provide their own tool response format.
Title
Fixes #11267 - Fix system message format bug when using tools with Ollama models
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
Type
🐛 Bug Fix
Changes
Fixed a critical bug in function_call_prompt() where the code attempted to concatenate strings to system message content without checking the message format.
When using tools, like Claude Code, with Ollama models, system messages can have either string content or structured content (list of objects with type/text/cache_control).
Root Cause:
The function_call_prompt() function in litellm/litellm_core_utils/prompt_templates/factory.py was blindly treating all system message content as strings and attempting string concatenation. However, with Ollama + tools, system messages follow the structured format (list of content objects), causing an AttributeError when trying to concatenate.
Fix Applied:
Testing:
Added comprehensive test case test_system_message_format_issue_reproduction() that:
(The fix was created by a human. Only the PR text was written by AI using the prompt: Please, write a summary PR for issue #11267. The fix is in commit eca86cd, follow the format of @.github/pull_request_template.md)