On the first point this was a pain point I faced as well (#15249)

For HF - they should follow similar path as Jan.ai does for example, a.k.a raw llama.cpp, not some weird forks like ollama.

On technology- yes, on marketing strategy, GTM, and partnership lock - there is nothing like Ollama's way of doing it. I can't open any app without ollama support to be the "local AI" support.

A bit of both.

I think packaging has improved in recent times, windows as an example. I kinda like the containers approach for Linux, it makes sense, I'm not sure how many Linux distros are going to tolerate separate llama-cpp-cuda, llama-cpp-rocm, llama-cpp-vulkan packages as per their packaging guidelines.

I do think at a minimum llama.cpp should document upstream what flags should be used with each backend CUDA, Vulkan, Metal (--threads, --cache-reuse, --flash-attention, etc.), as discussed here:

ollama/ollama#11714 (comment)

People use sub-optimal flags often when using llama.cpp and assume performance isn't up to scratch.

Are you using the GGML_CPU_ALL_VARIANTS option?

Nope. Separate instructions builds for CPU instructions(For example: using -DGGML_AVX512=ON for AVX512 build). We even found people use processors which doesn't even have avx2.

You should look into GGML_BACKEND_DL and GGML_CPU_ALL_VARIANTS, it is designed to solve this problem. Most of our releases use it.

Thank you!

Agreed. If llama-server was nicely packaged on all OSes and could be easily distributed and run everywhere, then I don't see a reason to use any alternative. The benefit will also be that you get the latest version of llama.cpp immediately without waiting for other companies to package it with nice UI.

My ideal would be for the llama.cpp project to ship official installers for llama-server on Mac and Windows and Linux,

An easy way to do this is to make it a PWA. Will work in most major browsers, including Safari, Chrome and its derivatives, Android and iOS, but not Firefox. It just takes a few lines of code to do this.

Hey @simonw, as a matter of fact we are working on new WebUI for llama.cpp and we are planning to put it into a native app as well!

The first step is going to be a release of a new version of WebUI rewritten in Svelte and with a much better UI/UX.

You can track the progress here #14839 (comment)

I am planning to have the Pull Request ready for a review at the beginning of the next week, so stay tuned!

i hope they won't deprecate llama-server CLI for a llama-server GUI, it would be too annoying to use on termux or over SSH

Hey @simonw, as a matter of fact we are working on new WebUI for llama.cpp and we are planning to put it into a native app as well!

The first step is going to be a release of a new version of WebUI rewritten in Svelte and with a much better UI/UX.

You can track the progress here #14839 (comment)

I am planning to have the Pull Request ready for a review at the beginning of the next week, so stay tuned!

I think I'm not that enough new to this, and I can compile llamacpp for a specific GPU, but I don't know how to use the llamacpp server for persistent chat-session data caching. Does it simply not have this feature? Or am I missing the parameter?

It's technically possible, but the size of the package would be quite big.

Easiest way is with an αcτµαlly pδrταblε εxεcµταblε

https://justine.lol/ape.html

The winget package comes with the Vulkan backend.

Does that work reliably with CUDA cards and does it support the same kinds of models (Qwen2.5VL for instance)

My experience with the Vulkan backend on a RTX 3060 has been excellent.

This is the killer feature right here. Id use llama.cpp directly over any wrapper if this was added

I feel like point number 1 is widely undermentioned in this thread. The ease that Llama-Swap adds to being able to bounce around models for various tasks is invaluable. If it weren't for that, I would probably still be using Ollama. It still needs a bit of upfront setup, but once you get a model in, it basically becomes "set it and forget it."

Edit I just realized who I was replying to and felt the need to come back and say I love the YT channel man. Keep up the good work!

Docker Model Runner provides 1 and 2 FWIW... They are trying to grow the community also... It uses llama.cpp upstream, contributes any changes required back... Number 3... I recommend AnythingLLM, but there are many options

llama-run does exist although the syntax to pull from huggingface is a little different (I wrote it). I recommend

docker model run hf.co/ggml-org/gemma-3-270m-it-GGUF

it uses llama.cpp backend.

I think command prompts are better off being done in golang... I used linenoise.cpp for the command prompt, C++ is just not a great language for the user facing stuff...

I'd been using GGML_NATIVE=OFF to build native binaries without features specific to newer CPUs, at least that's what I thought it did 😅 What's the difference between GGML_CPU_ALL_VARIANTS=ON and GGML_NATIVE=OFF?

GGML_NATIVE=ON probes your CPU features at build-time, and builds exactly one CPU backend with those exact features.

GGML_CPU_ALL_VARIANTS=ON builds multiple CPU backends all with different features, such that a maximally performant backend is available for any common CPU. The best backend is dynamically loaded at runtime, hence this requires GGML_BACKEND_DL=ON.

Well I guess it's built against Ubuntu libraries so Ubuntu makes sense. This is a Linux packaging problem in general, if using dynamic linking, you gotta build for each distro. But because things are also packaged as containers, you can just use the containers on pretty much any Linux distro.

If it's indeed build against Ubuntu, then I guess it's luck that it runs on Arch, because ldd ./llama-server shows a list, out of which 25 libs are linked against external libs.
AI says Ubuntu has easily on order of magnitude more users. I was hesitant to try the ubuntu-vulkan build because I thought it wouldn't run on Arch (but the "llama.cpp webui is all you need" mentality convinced me after some time to try it first and maybe build later, if it doesn't work). An Arch build would be still nice or ..arch-ubuntu.. in build name.

What's the difference between cudart-llama-bin-win-cuda-12.4-x64.zip and llama-b6210-bin-win-cuda-12.4-x64.zip? Can't find anything in docs.

Yeah, sometimes we can get lucky between distros and binaries are still somewhat ABI compatible cross-distro, but it's probably not something worth depending on.

Linux has decent backwards compatibility. Since Arch is rolling release, it tends to have newer versions of libraries than systems where executables are built, which helps things work often.

@gesrtewrtwr couldn't you get much better performance by using a quantized Qwen3-30B-A3B? Sorry if I'm missing something. I'm still pretty new to this.

If you install this version of docker:

curl -fsSL https://get.docker.com | sudo bash # installs everything, including docker model runner

https://docs.docker.com/ai/model-runner/get-started/#enable-dmr-in-docker-engine

you'll be back in business with the "docker model runner" server for arm64, (which uses llama-server, but you get model swapping, pushable models to OCI registries, some goodies like this)

If you install this version of docker:

curl -fsSL https://get.docker.com | sudo bash # installs everything, including docker model runner

@ericcurtin thank you for your suggestion, I appreciate it. But for the same reason I don't want to use ollama I also don't want to use docker ;-) I prefer to have less features. But just started to try to compile llama.cpp on my arm64 as suggested by others on the post ;-)

Build worked on raspberrypi, see here #13891 (comment) for the details.

I solved CURL and dependency management. The script now also installs ccache since that build takes so long without it. The only manual thing you have to do is install either cuda or vulkan but that is said in the GUI...
I have not yet added personal fork source for a toolbox repo/branch but that is on the way, changing llama.cpp repo/branch was always available.

This is pretty normal and standard for a C/C++ application that uses libcurl as a dependency, you need the headers and library.

It's a trap for young players, though. You just get told you need curl, so you get curl but that doesn't fix it. Then you google and find a random 2 year old web page to find what you need to do.

You can use --device to select the devices to use.

Thanks, interestingly my integrated graphics aren't listed as Vulkan-capable:

$ ./build/bin/llama-server --list-devices
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 ROCm devices:
  Device 0: AMD Radeon RX 7600, gfx1102 (0x1102), VMM: no, Wave Size: 32
  Device 1: AMD Radeon Graphics, gfx90c:xnack- (0x90c), VMM: no, Wave Size: 64
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 7600 (RADV NAVI33) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
Available devices:
  ROCm0: AMD Radeon RX 7600 (8176 MiB, 8142 MiB free)
  ROCm1: AMD Radeon Graphics (15726 MiB, 15669 MiB free)
  Vulkan0: AMD Radeon RX 7600 (RADV NAVI33) (7936 MiB, 7936 MiB free)

I believe you need to enable them explicitly by setting GGML_VK_VISIBLE_DEVICES=0,1 etc.

Yup, that worked:

$ GGML_VK_VISIBLE_DEVICES=0,1 ./build/bin/llama-server --list-devices
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 ROCm devices:
  Device 0: AMD Radeon RX 7600, gfx1102 (0x1102), VMM: no, Wave Size: 32
  Device 1: AMD Radeon Graphics, gfx90c:xnack- (0x90c), VMM: no, Wave Size: 64
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV RENOIR) (radv) | uma: 1 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 0 | matrix cores: none
ggml_vulkan: 1 = AMD Radeon RX 7600 (RADV NAVI33) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
Available devices:
  ROCm0: AMD Radeon RX 7600 (8176 MiB, 8142 MiB free)
  ROCm1: AMD Radeon Graphics (15726 MiB, 15669 MiB free)
  Vulkan0: AMD Radeon Graphics (RADV RENOIR) (10825 MiB, 10825 MiB free)
  Vulkan1: AMD Radeon RX 7600 (RADV NAVI33) (7936 MiB, 7936 MiB free)