Closed
Description
Name and Version
$ build/bin/llama-cli --version
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 4 ROCm devices:
Device 0: AMD Instinct MI100, gfx908:sramecc+:xnack- (0x908), VMM: no, Wave Size: 64
Device 1: Radeon RX 7900 XTX, gfx1100 (0x1100), VMM: no, Wave Size: 32
Device 2: AMD Instinct MI100, gfx908:sramecc+:xnack- (0x908), VMM: no, Wave Size: 64
Device 3: AMD Radeon VII, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64
version: 5098 (64eda5de)
built with cc (GCC) 14.2.1 20250110 (Red Hat 14.2.1-7) for x86_64-redhat-linuxOperating systems
Linux
GGML backends
HIP
Hardware
AMD Ryzen Threadripper PRO 7965WXs,
4 AMD GPU:
- Device 0: AMD Instinct MI100, gfx908
- Device 1: Radeon RX 7900 XTX, gfx1100
- Device 2: AMD Instinct MI100
- Device 3: AMD Radeon VII
128GB DDR5 RAM
4TB NVME
Models
Any model.
Problem description & steps to reproduce
Getting error when offloading ANY model to GPU:
tested llama3.1, llama4, aya, qwen2.5
Error
ROCm error: CUBLAS_STATUS_INTERNAL_ERROR
current device: 2, in function ggml_cuda_mul_mat_batched_cublas at /home/iyanello/Projects/ML/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:1867
hipblasGemmBatchedEx(ctx.cublas_handle(), HIPBLAS_OP_T, HIPBLAS_OP_N, ne01, ne11, ne10, alpha, (const void **) (ptrs_src.get() + 0*ne23), HIPBLAS_R_16F, nb01/nb00, (const void **) (ptrs_src.get() + 1*ne23), HIPBLAS_R_16F, nb11/nb10, beta, ( void **) (ptrs_dst.get() + 0*ne23), cu_data_type, ne01, ne23, cu_compute_type, HIPBLAS_GEMM_DEFAULT)
I tried to test llama-cpp build with ROCm 6.1, 6.2 and 6.3 (current). ROCm 6.0 gives another error - segmentation fault (any size of context).
Here is a command I used to build llama-cpp:
HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \
cmake -S . -B build -DSD_HIPBLAS=ON -DCMAKE_BUILD_TYPE=Release \
&& cmake --build build --config Release -- -j 48Command I used to run model:
./build/bin/llama-server -m ~/Downloads/Meta-Llama-3.1-8B-Instruct-Q4_K_L.gguf -ngl 33 --ctx-size 2048 ./build/bin/llama-server -m ~/Downloads/llama4/meta-llama_Llama-4-Scout-17B-16E-Instruct-Q4_K_M -ngl 48 --ctx-size 2048or
./build/bin/llama-server -m ~/Downloads/llama4/meta-llama_Llama-4-Scout-17B-16E-Instruct-Q4_K_M -ngl 48 --ctx-size 2048There is no problem when it runs on CPU
./build/bin/llama-server -m ~/Downloads/llama4/meta-llama_Llama-4-Scout-17B-16E-Instruct-Q4_K_M -ngl 0First Bad Commit
I'm not sure, but last working version was llama-cpp-python v0.3.1 which uses llama-cpp llama.cpp @ dc22344
Relevant log output
llama.cpp git:(master) ✗ ./build/bin/llama-server -m ~/Downloads/llama4/meta-llama_Llama-4-Scout-17B-16E-Instruct-Q4_K_M -ngl 49 --ctx-size 2048
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 4 ROCm devices:
Device 0: AMD Instinct MI100, gfx908:sramecc+:xnack- (0x908), VMM: no, Wave Size: 64
Device 1: Radeon RX 7900 XTX, gfx1100 (0x1100), VMM: no, Wave Size: 32
Device 2: AMD Instinct MI100, gfx908:sramecc+:xnack- (0x908), VMM: no, Wave Size: 64
Device 3: AMD Radeon VII, gfx906:sramecc+:xnack- (0x906), VMM: no, Wave Size: 64
build: 5098 (64eda5de) with cc (GCC) 14.2.1 20250110 (Red Hat 14.2.1-7) for x86_64-redhat-linux
system info: n_threads = 24, n_threads_batch = 24, total_threads = 48
system_info: n_threads = 24 (n_threads_batch = 24) / 48 | ROCm : NO_VMM = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | OPENMP = 1 | AARCH64_REPACK = 1 |
main: binding port with default address family
main: HTTP server is listening, hostname: 127.0.0.1, port: 8080, http threads: 47
main: loading model
srv load_model: loading model '/home/iyanello/Downloads/llama4/meta-llama_Llama-4-Scout-17B-16E-Instruct-Q4_K_M'
llama_model_load_from_file_impl: using device ROCm0 (AMD Instinct MI100) - 32730 MiB free
llama_model_load_from_file_impl: using device ROCm1 (Radeon RX 7900 XTX) - 24416 MiB free
llama_model_load_from_file_impl: using device ROCm2 (AMD Instinct MI100) - 32730 MiB free
llama_model_load_from_file_impl: using device ROCm3 (AMD Radeon VII) - 16342 MiB free
llama_model_loader: loaded meta data with 49 key-value pairs and 628 tensors from /home/iyanello/Downloads/llama4/meta-llama_Llama-4-Scout-17B-16E-Instruct-Q4_K_M (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = llama4
llama_model_loader: - kv 1: general.type str = model
llama_model_loader: - kv 2: general.name str = Llama 4 Scout 17B 16E Instruct
llama_model_loader: - kv 3: general.finetune str = 16E-Instruct
llama_model_loader: - kv 4: general.basename str = Llama-4-Scout
llama_model_loader: - kv 5: general.size_label str = 17B
llama_model_loader: - kv 6: general.license str = other
llama_model_loader: - kv 7: general.license.name str = llama4
llama_model_loader: - kv 8: general.base_model.count u32 = 1
llama_model_loader: - kv 9: general.base_model.0.name str = Llama 4 Scout 17B 16E
llama_model_loader: - kv 10: general.base_model.0.organization str = Meta Llama
llama_model_loader: - kv 11: general.base_model.0.repo_url str = https://huggingface.co/meta-llama/Lla...
llama_model_loader: - kv 12: general.tags arr[str,5] = ["facebook", "meta", "pytorch", "llam...
llama_model_loader: - kv 13: general.languages arr[str,12] = ["ar", "de", "en", "es", "fr", "hi", ...
llama_model_loader: - kv 14: llama4.block_count u32 = 48
llama_model_loader: - kv 15: llama4.context_length u32 = 10485760
llama_model_loader: - kv 16: llama4.embedding_length u32 = 5120
llama_model_loader: - kv 17: llama4.feed_forward_length u32 = 16384
llama_model_loader: - kv 18: llama4.attention.head_count u32 = 40
llama_model_loader: - kv 19: llama4.attention.head_count_kv u32 = 8
llama_model_loader: - kv 20: llama4.rope.freq_base f32 = 500000.000000
llama_model_loader: - kv 21: llama4.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 22: llama4.expert_count u32 = 16
llama_model_loader: - kv 23: llama4.expert_used_count u32 = 1
llama_model_loader: - kv 24: llama4.attention.key_length u32 = 128
llama_model_loader: - kv 25: llama4.attention.value_length u32 = 128
llama_model_loader: - kv 26: llama4.vocab_size u32 = 202048
llama_model_loader: - kv 27: llama4.rope.dimension_count u32 = 128
llama_model_loader: - kv 28: llama4.interleave_moe_layer_step u32 = 1
llama_model_loader: - kv 29: llama4.expert_feed_forward_length u32 = 8192
llama_model_loader: - kv 30: tokenizer.ggml.model str = gpt2
llama_model_loader: - kv 31: tokenizer.ggml.pre str = llama4
llama_model_loader: - kv 32: tokenizer.ggml.tokens arr[str,202048] = ["À", "Á", "õ", "ö", "÷", "ø", ...
llama_model_loader: - kv 33: tokenizer.ggml.token_type arr[i32,202048] = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv 34: tokenizer.ggml.merges arr[str,439802] = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
llama_model_loader: - kv 35: tokenizer.ggml.bos_token_id u32 = 200000
llama_model_loader: - kv 36: tokenizer.ggml.eos_token_id u32 = 200008
llama_model_loader: - kv 37: tokenizer.ggml.padding_token_id u32 = 201134
llama_model_loader: - kv 38: tokenizer.chat_template str = {{- bos_token }}\n{%- if custom_tools ...
llama_model_loader: - kv 39: tokenizer.ggml.add_bos_token bool = true
llama_model_loader: - kv 40: general.quantization_version u32 = 2
llama_model_loader: - kv 41: general.file_type u32 = 15
llama_model_loader: - kv 42: quantize.imatrix.file str = /models_out/Llama-4-Scout-17B-16E-Ins...
llama_model_loader: - kv 43: quantize.imatrix.dataset str = /training_dir/calibration_datav3.txt
llama_model_loader: - kv 44: quantize.imatrix.entries_count i32 = 528
llama_model_loader: - kv 45: quantize.imatrix.chunks_count i32 = 122
llama_model_loader: - kv 46: split.no u16 = 0
llama_model_loader: - kv 47: split.tensors.count i32 = 628
llama_model_loader: - kv 48: split.count u16 = 0
llama_model_loader: - type f32: 146 tensors
llama_model_loader: - type q8_0: 72 tensors
llama_model_loader: - type q4_K: 241 tensors
llama_model_loader: - type q5_K: 48 tensors
llama_model_loader: - type q6_K: 121 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type = Q4_K - Medium
print_info: file size = 62.90 GiB (5.01 BPW)
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: special tokens cache size = 1135
load: token to piece cache size = 1.3873 MB
print_info: arch = llama4
print_info: vocab_only = 0
print_info: n_ctx_train = 10485760
print_info: n_embd = 5120
print_info: n_layer = 48
print_info: n_head = 40
print_info: n_head_kv = 8
print_info: n_rot = 128
print_info: n_swa = 1
print_info: n_swa_pattern = 4
print_info: n_embd_head_k = 128
print_info: n_embd_head_v = 128
print_info: n_gqa = 5
print_info: n_embd_k_gqa = 1024
print_info: n_embd_v_gqa = 1024
print_info: f_norm_eps = 0.0e+00
print_info: f_norm_rms_eps = 1.0e-05
print_info: f_clamp_kqv = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale = 0.0e+00
print_info: f_attn_scale = 0.0e+00
print_info: n_ff = 16384
print_info: n_expert = 16
print_info: n_expert_used = 1
print_info: causal attn = 1
print_info: pooling type = 0
print_info: rope type = 0
print_info: rope scaling = linear
print_info: freq_base_train = 500000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn = 10485760
print_info: rope_finetuned = unknown
print_info: ssm_d_conv = 0
print_info: ssm_d_inner = 0
print_info: ssm_d_state = 0
print_info: ssm_dt_rank = 0
print_info: ssm_dt_b_c_rms = 0
print_info: model type = 17Bx16E (Scout)
print_info: model params = 107.77 B
print_info: general.name = Llama 4 Scout 17B 16E Instruct
print_info: vocab type = BPE
print_info: n_vocab = 202048
print_info: n_merges = 439802
print_info: BOS token = 200000 '<|begin_of_text|>'
print_info: EOS token = 200008 '<|eot|>'
print_info: PAD token = 201134 '<|finetune_right_pad_id|>'
print_info: LF token = 198 'Ċ'
print_info: FIM PRE token = 200002 '<|fim_prefix|>'
print_info: FIM SUF token = 200004 '<|fim_suffix|>'
print_info: FIM MID token = 200003 '<|fim_middle|>'
print_info: EOG token = 200008 '<|eot|>'
print_info: max token length = 192
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors: offloading 48 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 49/49 layers to GPU
load_tensors: ROCm0 model buffer size = 21208.79 MiB
load_tensors: ROCm1 model buffer size = 14153.71 MiB
load_tensors: ROCm2 model buffer size = 19211.72 MiB
load_tensors: ROCm3 model buffer size = 9275.40 MiB
load_tensors: CPU_Mapped model buffer size = 554.94 MiB
....................................................................................................
llama_context: constructing llama_context
llama_context: n_seq_max = 1
llama_context: n_ctx = 2048
llama_context: n_ctx_per_seq = 2048
llama_context: n_batch = 2048
llama_context: n_ubatch = 512
llama_context: causal_attn = 1
llama_context: flash_attn = 0
llama_context: freq_base = 500000.0
llama_context: freq_scale = 1
llama_context: n_ctx_per_seq (2048) < n_ctx_train (10485760) -- the full capacity of the model will not be utilized
llama_context: ROCm_Host output buffer size = 0.77 MiB
init: kv_size = 2048, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 48, can_shift = 1
init: ROCm0 KV buffer size = 128.00 MiB
init: ROCm1 KV buffer size = 88.00 MiB
init: ROCm2 KV buffer size = 120.00 MiB
init: ROCm3 KV buffer size = 48.00 MiB
llama_context: KV self size = 384.00 MiB, K (f16): 192.00 MiB, V (f16): 192.00 MiB
llama_context: pipeline parallelism enabled (n_copies=4)
llama_context: ROCm0 compute buffer size = 272.02 MiB
llama_context: ROCm1 compute buffer size = 272.02 MiB
llama_context: ROCm2 compute buffer size = 272.02 MiB
llama_context: ROCm3 compute buffer size = 506.65 MiB
llama_context: ROCm_Host compute buffer size = 42.03 MiB
llama_context: graph nodes = 2514
llama_context: graph splits = 5
common_init_from_params: setting dry_penalty_last_n to ctx_size = 2048
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
/home/iyanello/Projects/ML/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:75: ROCm error
ROCm error: CUBLAS_STATUS_INTERNAL_ERROR
current device: 2, in function ggml_cuda_mul_mat_batched_cublas at /home/iyanello/Projects/ML/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:1867
hipblasGemmBatchedEx(ctx.cublas_handle(), HIPBLAS_OP_T, HIPBLAS_OP_N, ne01, ne11, ne10, alpha, (const void **) (ptrs_src.get() + 0*ne23), HIPBLAS_R_16F, nb01/nb00, (const void **) (ptrs_src.get() + 1*ne23), HIPBLAS_R_16F, nb11/nb10, beta, ( void **) (ptrs_dst.get() + 0*ne23), cu_data_type, ne01, ne23, cu_compute_type, HIPBLAS_GEMM_DEFAULT)
[New LWP 130898]
[New LWP 130896]
[New LWP 130895]
[New LWP 130894]
[New LWP 130893]
[New LWP 130892]
[New LWP 130891]
[New LWP 130890]
[New LWP 130889]
[New LWP 130888]
[New LWP 130887]
[New LWP 130886]
[New LWP 130885]
[New LWP 130884]
[New LWP 130883]
[New LWP 130882]
[New LWP 130881]
[New LWP 130880]
[New LWP 130879]
[New LWP 130878]
[New LWP 130877]
[New LWP 130876]
[New LWP 130875]
[New LWP 130874]
[New LWP 130873]
[New LWP 130872]
[New LWP 130871]
[New LWP 130870]
[New LWP 130869]
[New LWP 130868]
[New LWP 130867]
[New LWP 130866]
[New LWP 130865]
[New LWP 130864]
[New LWP 130863]
[New LWP 130862]
[New LWP 130861]
[New LWP 130860]
[New LWP 130859]
[New LWP 130858]
[New LWP 130857]
[New LWP 130856]
[New LWP 130855]
[New LWP 130854]
[New LWP 130853]
[New LWP 130852]
[New LWP 130851]
[New LWP 130850]
[New LWP 130849]
[New LWP 130848]
[New LWP 130846]
This GDB supports auto-downloading debuginfo from the following URLs:
<https://debuginfod.fedoraproject.org/>
Enable debuginfod for this session? (y or [n]) [answered N; input not from terminal]
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007f81c84ea3e3 in wait4 () from /lib64/libc.so.6
#0 0x00007f81c84ea3e3 in wait4 () from /lib64/libc.so.6
#1 0x00007f81cc3cbd36 in ggml_abort () from /home/iyanello/Projects/ML/llama.cpp/build/bin/libggml-base.so
#2 0x00007f81c8acc5e2 in ggml_cuda_error(char const*, char const*, char const*, int, char const*) () from /home/iyanello/Projects/ML/llama.cpp/build/bin/libggml-hip.so
#3 0x00007f81c8ad3fbe in ggml_cuda_mul_mat_batched_cublas(ggml_backend_cuda_context&, ggml_tensor const*, ggml_tensor const*, ggml_tensor*) () from /home/iyanello/Projects/ML/llama.cpp/build/bin/libggml-hip.so
#4 0x00007f81c8ad1909 in ggml_backend_cuda_graph_compute(ggml_backend*, ggml_cgraph*) () from /home/iyanello/Projects/ML/llama.cpp/build/bin/libggml-hip.so
#5 0x00007f81cc3e0e04 in ggml_backend_sched_graph_compute_async () from /home/iyanello/Projects/ML/llama.cpp/build/bin/libggml-base.so
#6 0x00007f81cc5e3f31 in llama_context::graph_compute(ggml_cgraph*, bool) () from /home/iyanello/Projects/ML/llama.cpp/build/bin/libllama.so
#7 0x00007f81cc5e7ad8 in llama_context::decode(llama_batch&) () from /home/iyanello/Projects/ML/llama.cpp/build/bin/libllama.so
#8 0x00007f81cc5e8d7b in llama_decode () from /home/iyanello/Projects/ML/llama.cpp/build/bin/libllama.so
#9 0x0000000000577fa4 in common_init_from_params(common_params&) ()
#10 0x000000000047bebc in server_context::load_model(common_params const&) ()
#11 0x0000000000425e91 in main ()
[Inferior 1 (process 130838) detached]
[1] 130838 IOT instruction (core dumped) ./build/bin/llama-server -m -ngl 49 --ctx-size 2048