Skip to content

server: implementation of v1/completions echo logprobs support #15189

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

server: implementation of v1/completions echo logprobs support #15189

wants to merge 1 commit into from
+239 −15

Conversation

fo40225
Copy link

@fo40225 fo40225 commented Aug 9, 2025 •
edited
Loading

@fo40225
Copy link
Author

fo40225 commented Aug 9, 2025

Related issue:
#8942
#12591
EleutherAI/lm-evaluation-harness#2894

@fo40225 fo40225 force-pushed the v1_completions_echo branch from 8284dc4 to 6817897 Compare August 10, 2025 01:37
@fo40225 fo40225 marked this pull request as ready for review August 10, 2025 01:40
@fo40225 fo40225 requested a review from ngxson as a code owner August 10, 2025 01:40
@fo40225
Copy link
Author

fo40225 commented Aug 10, 2025

llama.cpp b6096 + cherrypick 6817897
llama-server
unsloth/SmolLM2-135M-Instruct-GGUF/SmolLM2-135M-Instruct-Q8_0.gguf
lm-eval[api]==0.4.9.1

PIQA HellaSwag OpenBookQA WinoGrande
Ref 66.3 40.9 34.6 51.3
acc 66.3 35.0 23.0 51.6
acc_norm 66.8 43.0 33.4

Ref:
https://huggingface.co/blog/Kseniase/insidesmol
https://huggingface.co/HuggingFaceTB/SmolLM2-135M

llama.cpp/build/bin/llama-server \
 -fa \
 -m huggingface/unsloth/SmolLM2-135M-Instruct-GGUF/SmolLM2-135M-Instruct-Q8_0.gguf \
 --host 0.0.0.0 \
 --port 12345 \
 --jinja

for task in piqa hellaswag openbookqa winogrande
do
  lm-eval --model gguf \
    --tasks $task \
    --model_args base_url=http://127.0.0.1:12345 \
    --output_path ./SmolLM2-135M-Instruct-Q8_0-$task \
    --log_sample
done

@Green-Sky
Copy link
Collaborator

I guess this kind of works around #231

@jukofyork
Copy link
Collaborator

This looks useful - am I correct in thinking this can be used to get the logprobs of the prompt?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ngxson ngxson Awaiting requested review from ngxson ngxson is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
examples server
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@fo40225 @Green-Sky @jukofyork