llm = Llama(
    model_path="/home/axyo/dev/LLM/models/Meta-Llama-3-8B-Instruct-GGUF-v2/Meta-Llama-3-8B-Instruct-v2.Q5_0.gguf",
    n_gpu_layers=-1,
    seed=8,
    n_ctx=4096,
    logits_all=True,
    kv_overrides={"tokenizer.ggml.eos_token_id": 128002},
)

prompt = """<|start_header_id|>user<|end_header_id|>

What is a dog?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

A dog, also known as Canis lupus familiaris, is a domesticated mammal that belongs to the family Canidae. Dogs are closely related to wolves and share many physical and behavioral characteristics with them.

Dogs are typically characterized by their:

1. Physical appearance: Dogs come in a wide range of sizes, shapes, and coat types, from the tiny Chihuahua to the massive Great Dane. They have a furry coat, a wagging tail, and a distinctive snout.
2. Behavior: Dogs are social animals that thrive on interaction with their human family and other dogs. They are known for their loyalty, affection, and ability to form strong bonds with their owners.
3. Diet: Dogs are omnivores, which means they eat both plants and animals. They are often fed a diet of commercial dog food, which is formulated to meet their nutritional needs.
4. Communication: Dogs communicate primarily through body language and vocalizations, such as barking, whining, and growling.
5. Intelligence: Dogs are highly intelligent animals that are capable of learning a wide range of tasks and commands.

Dogs have been domesticated for thousands of years and have been bred for various purposes, such as:

* Companionship: Many dogs are kept as pets and are valued for their companionship and affection.
* Hunting: Some breeds of dogs, such as retrievers and pointers, are bred for hunting and retrieving game.
* Guarding: Some breeds, such as guard dogs and watchdogs, are bred for their protective instincts and ability to defend their territory.
* Assistance: Some dogs are trained as service animals to assist people with disabilities, such as guide dogs for the blind or hearing dogs for the deaf.

Overall, dogs are beloved animals that bring joy, companionship, and love to many people around the world

Would you like to know more about a specific aspect of dogs, such as their behavior, health, or training?"""

output = llm(
    prompt,
    echo=False,
    logprobs=100,
    max_tokens=1,

    repeat_penalty=1.0, # disable penalties

    top_k=1,
    #top_p=1.0,
    #min_p=-10000000000.0,
    temperature=0,
)
print(output)