Skip to content

Conversation

thoughtp0lice
Copy link
Contributor

In llama_cpp/llama.py, the eval function converts return value of self._ctx.get_logits(), which is a CtypesArray, to list then copy it into self.scores. Here the CtypesArray is directly converted to a numpy array which speeds up the conversion and copying. The speed-up is especially noticeable on smaller models with faster inference time.

@thoughtp0lice thoughtp0lice changed the title improve Llama.eval efficiency Improve Llama.eval efficiency May 23, 2024
@abetlen
Copy link
Owner

abetlen commented May 24, 2024

@thoughtp0lice thank you, that's perfect!

@abetlen abetlen merged commit 5cae104 into abetlen:main May 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants