This PR extends scripts/server-bench.py with synthetic prompts which are simply random lists of tokens with a configurable random length. Each prompt is then assigned a configurable random number of tokens to generate, the server is instructed to generate this exact number of tokens with n_predict and ignore_eos. This ensures that the server performance can be tested easily and consistently for specific combinations of prompt lengths and generation lengths.

I think there is something wrong with how the server handles ignore_eos. I'm setting it in the JSON but the server does not respect it. Also the way it's being handled seems incorrect, both the slot_params and the sampling_params have a field ignore_eos and they're used inconsistently.

I changed the prompt latency to be based on the latency as observed by the Python script instead of as reported by the server. Long-term we can consider extending the script with support for other APIs/projects. I changed the number of workers back to the number of slots to avoid just measuring the time a worker spends in the queue waiting for a free slot.

I changed the script to pass arguments to the server via environment variables. This way users can pass any arguments to the server without server-bench.py becoming bloated.