llama : add gpt-oss #15091
llama : add gpt-oss #15091
Conversation
* llama : add attn sinks * ggml : add attn sinks * cuda : add attn sinks * vulkan : add support for sinks in softmax remove unnecessary return * ggml : add fused swiglu_oai op (#11) * ggml : add fused swiglu_oai op * Update ggml/src/ggml-cpu/ops.cpp Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * update CUDA impl * cont : metal impl * add vulkan impl * test-backend-ops : more test cases, clean up * llama : remove unfused impl * remove extra lines --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> --------- Co-authored-by: slaren <slarengh@gmail.com>
|
The chat template by ggml-org, the tool calling is much better than from others:
|
|
has anyone made it work with tool_calling? |
|
Is anyone else getting gibberish with long context prompts? I get the same results with either of the ggufs I downloaded, I drop in a 40K context prompt and get a response of "GGGGGGGGGGGGGG.....". Smaller prompts work fine. llama-server -m /media//T9/Models/gpt-oss-120b-MXFP4-00001-of-00002.gguf --threads 24 --threads-batch 48 --batch-size 4096 --ubatch-size 4096 --ctx-size 131072 --temp 0.6 --min-p 0.01 --tensor-split 20,20,20,20 --n-gpu-layers 80 --main-gpu 1 --flash-attn --no-mmap --mlock --host 0.0.0.0 --port 5001 llama-server -m /media//T9/Models/gpt-oss-120b-F16.gguf --threads 24 --threads-batch 48 --batch-size 4096 --ubatch-size 4096 --ctx-size 131072 --temp 0.6 --min-p 0.01 --tensor-split 20,20,20,20 --n-gpu-layers 80 --main-gpu 1 --flash-attn --no-mmap --mlock --host 0.0.0.0 --port 5001 |
|
@ggerganov Thank you, ggerganov and everyone else for your expedient and awesome work! Will attention sinks be made available to all GGUF'd models? 🤔 |
|
Could anyone help explain why use |
|
@nachoal it appears there's a lot of new stuff here. At least to me-- but I have not used openai's API with openai before, only local models. Like, there are two kinds of system prompts-- a "system message" and a "developer message". Also there are two types of tools-- "builtin_tools" (python or browser tools) referenced in the system message and function tools (described in the developer message). There is a special format for describing the function tools but I'm guessing MCP would work too. The function tools are called in a separate "commentary" channel from normal reply So different types of output appear in different places in the chat completion. As an example, instead of parsing reasoning_text = response.choices[0].message.reasoning_contentwhere response = client.chat.completions.create( ... )It looks like right now in llama.cpp by default when an assistant tries to use a tool, the <|start|>assistant<|channel|>commentary to=fetch json<|message|>{\"url\":\"https://www.github.com\",\"max_length\":5000}The expected format is is supposed to come after the reasoning like this: <|start|>assistant<|channel|>commentary to=functions.get_weather <|constrain|>json<|message|>{"location":"San Francisco"}<|call|>So the output looks very close but not exactly right from what I am seeing. It's missing the <|call|> I'm sure that in the near future a tool call will be fully extracted by llama.cpp and put in The Harmony dox also discusses "Output by the model which can either be a tool call or a message output. " -- so apparently you can get a message OR a tool call, but apparently not both, which is why the The hacky temporary workaround to this bug to maintain compatibility with other models would be to come up with a regex expression you could use to pull the json toolname and arguments/{output} from the There's a note in this PR that the tool template stuff is a WIP and tool use is still to come, so I guess it may make the most sense to just wait for this to get fixed unless you're really itching to get tools working. Anyone that knows more please correct me as I'm just figuring this out myself! |
|
@fat-tire Appreciate the complete explanation 🙏, I ended up just parsing the strings to try tool calling for now, a bit broken but it works. Thanks! |
|
Has the reasoning_effort not been implemented yet? I'm hosting gpt-oss-20b on llama-server and calling it from the OpenAI API Here is quick sample.
from openai import OpenAI
model = OpenAI(api_key="dummy", base_url="http://127.0.0.1:8080")
completion = model.chat.completions.create(
model="dummy",
messages=[{"role": "user", "content": "Write fizzbuzz in Python"}],
reasoning_effort="high",
)
print(completion) |
|
@nachoal Yup a simple regex pattern on that pattern = r".*\<\|start\|\>assistant\<\|channel\|\>commentary to=(?P<toolname>\w+) json\<\|message\|\>(?P<output>.*)"gets you two match groups, |
|
Using build: llama-b6098-bin-win-cuda-12.4-x64 |
|
@uazure what CPU backend is it loading? |
@nai-kon I noticed the same behavior. I think we should open a defect issue rather than clog this further. |
Added in GPT-OSS PR ggml-org/llama.cpp#15091 --------- Co-authored-by: Xuan-Son Nguyen <son@huggingface.co>
| new_name_gate = self.map_tensor_name(name.replace("gate_up_proj_scales", "gate_proj.weight")) | ||
| new_name_up = self.map_tensor_name(name.replace("gate_up_proj_scales", "up_proj.weight")) |
There was a problem hiding this comment.
Too late, but why was this split? Only adds extra ops on the graph...
There was a problem hiding this comment.
The gate_up tensor is organized in a way that a row of gate is followed by a row of up, aka interleaving. While we can rearrange it to the expected layout for fused op, I think it's easier to just split it into gate and up independently
| struct ggml_tensor * ggml_swiglu_oai( | ||
| struct ggml_context * ctx, | ||
| struct ggml_tensor * a, | ||
| struct ggml_tensor * b, | ||
| float alpha, | ||
| float limit) { | ||
| struct ggml_tensor * result = ggml_glu_impl(ctx, a, b, GGML_GLU_OP_SWIGLU_OAI, false); | ||
| ggml_set_op_params_f32(result, 2, alpha); | ||
| ggml_set_op_params_f32(result, 3, limit); | ||
|
|
||
| return result; | ||
| } |
There was a problem hiding this comment.
Technically this is ggml_swiglu_oai_split.
|
Does anybody see value in adding a simple chat client to upstream llama.cpp in C++ or python3 that we can consolidate on like this: https://github.com/ericcurtin/lm-chat/blob/main/lm-chat.py ? For formats like this new harmony one it can be hard to find simple reference implementations, that are not "from openai_harmony" I guess there sort of is the html client implementation, but I'm not sure how many people are ready to crack that open as it's more that just a simple cli chat client. |
|
My opinion is that efforts should be focused on the existing web interface of the HTTP server. |
A couple off issues with the web interface. It's a UI, so added complexity for a simple reference implementation. It's compressed, which kills all the version control, can't easily see changes, it's like a mystery blob that's committed from time to time. I think it would be better if we committed both: ./tools/server/public/index.html.gz and ./tools/server/public/index.html on changes, at least we could track the changes then. |
|
index.html... hmm, good luck decoding the diff of the transpiled JS code |
My bad, the true sources are there. |
* oai moe * compat with new checkpoint * add attn sink impl * add rope scaling yarn * logits match with latest transformers code * wip chat template * rm trailing space * use ggml_scale_bias * rm redundant is_swa_all * convert interleaved gate_up * graph : fix activation function to match reference (ggml-org#7) * vocab : handle o200k_harmony special tokens * ggml : add attention sinks support (ggml-org#1) * llama : add attn sinks * ggml : add attn sinks * cuda : add attn sinks * vulkan : add support for sinks in softmax remove unnecessary return * ggml : add fused swiglu_oai op (ggml-org#11) * ggml : add fused swiglu_oai op * Update ggml/src/ggml-cpu/ops.cpp Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> * update CUDA impl * cont : metal impl * add vulkan impl * test-backend-ops : more test cases, clean up * llama : remove unfused impl * remove extra lines --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com> --------- Co-authored-by: slaren <slarengh@gmail.com> * repack mxfp4 upon conversion * clean up a bit * enable thinking * add quick hack to render only some special tokens * fix bf16 conversion * remove vocab hack * webui ok * support chat parsing for gpt-oss * fix webui * direct mapping mxfp4, FINALLY * force using mxfp4 * properly use lazy tensor * ggml : add mxfp4 ggml : use e8m0 conversion instead of powf Co-authored-by: Diego Devesa <slarengh@gmail.com> change kvalues_mxfp4 table to match e2m1 (ggml-org#6) metal : remove quantization for now (not used) cuda : fix disabled CUDA graphs due to ffn moe bias vulkan : add support for mxfp4 cont : add cm2 dequant * ggml : add ggml_add_id (ggml-org#13) * ggml : add ggml_add_id * add cuda impl * llama : add weight support check for add_id * perf opt * add vulkan impl * rename cuda files * add metal impl * allow in-place ggml_add_id * llama : keep biases on CPU with --cpu-moe * llama : fix compile error ggml-ci * cuda : add fallback for __nv_cvt_e8m0_to_bf16raw ggml-ci * cleanup ggml-ci * sycl : fix supports_op for MXFP4 ggml-ci * fix Unknown reasoning format * ggml-cpu : fix AVX build ggml-ci * fix hip build ggml-ci * cuda : add mxfp4 dequantization support for cuBLAS ggml-ci * ggml-cpu : fix mxfp4 fallback definitions for some architectures ggml-ci * cuda : fix version required for __nv_cvt_e8m0_to_bf16raw --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co> Co-authored-by: slaren <slarengh@gmail.com>
gpt-oss model support in native MXFP4 format:
ggml_add_idoperator in ggmlUsage:
Model collection: https://huggingface.co/collections/ggml-org/gpt-oss-68923b60bee37414546c70bf
Example command:
llama-server -hf ggml-org/gpt-oss-120b-GGUF -c 0 -fa --jinja --reasoning-format none # Then, access http://localhost:8080Model card
References:
Note to maintainers:
This an initial implementation with pretty much complete support for the CUDA, Vulkan, Metal and CPU backends. The idea is to merge this quicker than usual, in time for the official release today, and later we can work on polishing any potential problems and missing features.
Next PRs: