Apple NPU acceleration integrated into llama.cpp, using MiniCPM-V 4.0 as an example. #15262
+1,039
−19
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Feat ios
feat ios: add clean kv cache
ggerganov
reviewed
Aug 12, 2025
There was a problem hiding this comment.
Generally looks OK. Need to improve encapsulation of the CoreML code (see comments). Would need a review from @ngxson.
Also:
- Use "CoreML" instead of "ANE"
- Would eventually need instructions for generating the CoreML inference code - can add those after the PR is approved
Comment on lines
+98
to
+100
| bool ane_embedding(struct clip_ctx * ctx, int n_threads, const struct clip_image_f32_batch * imgs, float * vec); | ||
| bool ane_resampler(struct clip_ctx * ctx, int n_threads, const struct clip_image_f32_batch * imgs, const float * vit_embedding, float * vec); | ||
|
|
There was a problem hiding this comment.
No need to expose this in the public interface
Comment on lines
+115
to
+117
|
|
||
| // ANE support functions | ||
| void clip_set_ane_model_path(struct clip_ctx * ctx, const char * ane_model_path); |
There was a problem hiding this comment.
We should find a way to avoid this. Maybe we can do something similar to whisper.cpp:
| @@ -82,6 +82,7 @@ struct mtmd_context_params { | |||
| enum ggml_log_level verbosity; | |||
| const char * image_marker; // deprecated, use media_marker instead | |||
| const char * media_marker; | |||
| const char * ane_model_path; // path to ANE model for iOS | |||
There was a problem hiding this comment.
Instead of the term "ane", use the term "coreml" as it is more correct. CoreML models can run not only the Apple Neural Engine, but also on the GPU and CPU.
Comment on lines
+3845
to
+3852
|
|
||
| static int flag = 0; | ||
| static const void* coremlEncoder = NULL; | ||
| static std::string cached_model_path = ""; | ||
|
|
||
| // Check if we need to load a new model | ||
| if (flag == 0 || (ane_model_path && cached_model_path != ane_model_path)) { | ||
| if (coremlEncoder) { |
There was a problem hiding this comment.
Avoid this global state. Figure out a way to move this to the clip context.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As stated in #14983, I have integrated Apple NPU (ANE) acceleration into llama.cpp.
Using MiniCPM-V 4.0 as an example, I will introduce a simple way to use ANE and hope we can discuss a better approach.
Download ane in Hugging Face or Modelscope, If you downloaded the zip file, please unzip it.
Used like mmproj, I added the "--ane" interface. The path is the downloaded ane_minicpmv4_vit_f16.mlmodelc file address.
I tested ANE acceleration on several devices. The benchmark results are as follows:
A point worth noting: The first time ANE is used, there is a loading time and it will be slightly slower. After that, as long as ANE is not updated, it will remain ready and waiting in the system.