Hi all! reaching out to the community for help in testing a new feature.

PR #15550 (in draft pending testing) introduces a new option --target-bpw, which implements an optimised quant type selection algorithm to automatically determine per-tensor quantisation types to achieve a bits-per-weight (bpw) target, with minimal estimated quality loss. The new function,

• builds a candidate set of quant types (K or IQ types)
• for each layer/tensor, it simulates quantise→dequantise per candidate type, and estimates error using a weighted MSE error function. If the imatrix includes activations, it adds a bias penalty term to better reflect forward‑pass impact, making the error estimation more accurate and thus the quant type selection
• it filters candidates to the pareto frontier (lowest error for a given size), then starts from the smallest bpw mix increasing to larger formats, based on the best error‑reduction per added bit, until the global bpw budget is reached
• returns a map of tensor name → ggml_type overrides, which the main quantisation pass uses. If the minimum achievable BPW already exceeds the target, it returns that minimum.

Based on limited testing, this approach seems to produce, on average, better quality models compared to naive quantisation (i.e. simply running standard llama-quantize with no further optimisations).

Although these are very encouraging results, more testing with different model architectures and sizes is needed before categorically concluding the new functionality consistently yields higher quality models.

The testing protocol I'm using is:

1. Generate* Q8_0, Q6_K, Q5_K_M, Q5_K_S, Q4_K_M, Q4_K_S, Q3_K_L, Q3_K_M, Q3_K_S, IQ4_NL, IQ3_M, and IQ3_S naive quantisations (e.g. llama-quantize --imatrix imatrix-with-activations.gguf LLM-Model-F16.gguf Naive-Quantized-<TYPE>.gguf <type>)
2. Determine each model bits per weight (bpw). This can be easily done by using python llama.cpp/gguf-py/gguf/scripts/gguf_dump.py --markdown BPW-Quantized-<TYPE>.gguf type
3. Generate equivalent quant types by setting --target-bpw to the corresponding bpw values (e.g. llama-quantize --imatrix imatrix-with-activations.gguf --target-bpw <naive bpw> LLM-Model-F16.gguf BPW-Quantized-<TYPE>.gguf <type>)
4. Calculate quality scores via llama-perplexity -m <Naive|BPW>-Quantized-<TYPE>.gguf -f calibration_dataset.txt --kl-divergence-base LLM-Model-F16.logits --kl-divergence

(*) this particular list is just a personal preference. The new function can/should be used with any supported type.

I have tested two small but representative models: Llama-3.2-1B ("classic" transformer architecture) and Huihui-MoE-1.2B-A0.6B (typical Mixture or Experts), and got the following results:

(PPL: the smaller the better; 𝜌PPL: the higher the better; KLD: the smaller the better; In bold: best quality)

Llama-3.2-1B results:

Huihui-MoE-1.2B-A0.6B results:

Anyone wishing to lend a hand, grateful if you could try it and share your results, ideally in a similar format as above (it helps to collate and consolidate), along with comments / feedback and specially bug reports.

Please feel free to test any quantisation types you'd like

IMPORTANT NOTE!

A new format imatrix (gguf) is required for the function to work, and if "activations" are included in the imatrix file, the estimation process will be much more accurate. However, at the time of writing, this is only available by generating the file using #14891 with --activation-statistics and --output-format gguf options: llama-imatrix -m LLM-Model-F16.gguf -f calibration-data.txt --activation-statistics --output-format gguf -o imatrix-with-activations.gguf