Eval bug: Gemma4 E4B MTP drafter crashes at slot init with fatal error in fattn.cu:110

[dboybaker]

Name and Version

version: 9584 (e25a32e)
built with GNU 14.2.0 for Linux x86_64

Operating systems

Linux

GGML backends

CUDA

Hardware

Ryzen 5600x + 2 x RTX 3090

Models

Unsloth's gemma-4-E4B-it-qat-UD-Q4_K_XL.gguf and its mpt drafter mtp-gemma-4-E4B-it.gguf

Problem description & steps to reproduce

E4B + MTP drafter crashes during slot initialization warmup decode with a fatal error in ggml_cuda_flash_attn_ext. E2B with identical config works fine at ~320 t/s. *Crash is affected by --flash-attn on vs off. The bug arises with the addition of flags:

      --model-draft /gemma4-e4b/mtp-gemma-4-E4B-it.gguf
      --spec-type draft-mtp
      --spec-draft-n-max 1

First Bad Commit

PR #24282 introduce support for e4b/e2b

Relevant log output

Logs

0.03.104.831 I common_speculative_impl_draft_mtp: adding speculative implementation 'draft-mtp'
0.03.104.834 I common_speculative_impl_draft_mtp: - n_max=1, n_min=0, p_min=0.00, n_embd=2560, backend_sampling=1
0.03.104.835 I common_speculative_impl_draft_mtp: - gpu_layers=-1, cache_k=f16, cache_v=f16, ctx_tgt=yes, ctx_dft=yes, devices=[default]
/llama.cpp/ggml/src/ggml-cuda/fattn.cu:110: fatal error
/llama.cpp/build/bin/libggml-base.so.0(+0x18665) [0x7fa52615f665]
/llama.cpp/build/bin/libggml-base.so.0(ggml_print_backtrace+0x1df) [0x7fa52615fa3f]
/llama.cpp/build/bin/libggml-base.so.0(ggml_abort+0x11e) [0x7fa52615fbce]
/llama.cpp/build/bin/libggml-cuda.so.0(_Z24ggml_cuda_flash_attn_extR25ggml_backend_cuda_contextP11ggml_tensor+0x2a85) [0x7fa5217e3155]
/llama.cpp/build/bin/libggml-cuda.so.0(+0x235616) [0x7fa521835616]
/llama.cpp/build/bin/libggml-base.so.0(ggml_backend_sched_graph_compute_async+0x827) [0x7fa52617c287]
/llama.cpp/build/bin/libllama.so.0(_ZN13llama_context13graph_computeEP11ggml_cgraphb+0xa1) [0x7fa5258cd2f1]
/llama.cpp/build/bin/libllama.so.0(_ZN13llama_context14process_ubatchERK12llama_ubatch14llm_graph_typeP22llama_memory_context_iR11ggml_status+0xe4) [0x7fa5258cff04]
/llama.cpp/build/bin/libllama.so.0(_ZN13llama_context6decodeERK11llama_batch+0x365) [0x7fa5258d5e55]
/llama.cpp/build/bin/libllama.so.0(llama_decode+0xb) [0x7fa5258d7a3b]
/llama.cpp/build/bin/libllama-common.so.0(_Z25common_context_can_seq_rmP13llama_context+0xc8) [0x7fa525de28c8]
/llama.cpp/build/bin/libllama-server-impl.so(_ZN19server_context_impl10load_modelER13common_params+0xcd4) [0x7fa526972214]
/llama.cpp/build/bin/libllama-server-impl.so(_Z12llama_serveriPPc+0x2f03) [0x7fa5268c57d3]
/lib/x86_64-linux-gnu/libc.so.6(+0x29ca8) [0x7fa526235ca8]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85) [0x7fa526235d65]
/usr/local/bin/llama-server(+0x11b1) [0x556ba39dc1b1] 

[arbv]

Same happened to me on ROCm.

[dextune]

Gemma4 MTP CUDA fallback

Notice: This is a temporary fix. Please use the official patch for future updates.

Summary

Fixed a startup abort in Gemma4Assistant MTP runs on CUDA.

What changed

• Updated ggml/src/ggml-cuda/fattn.cu so the 512-wide flash-attn path has a safe fallback when the GQA-specific MMA route is not applicable.
• This avoids the fatal error path during speculative/MTP context initialization.

Code details

• Edit ggml/src/ggml-cuda/fattn.cu.
• In ggml_cuda_get_best_fattn_kernel():
  • change the case 512: fallback from return BEST_FATTN_KERNEL_NONE; to return BEST_FATTN_KERNEL_TILE;
  • change the case 576: fallback from return BEST_FATTN_KERNEL_NONE; to return BEST_FATTN_KERNEL_TILE;
• In ggml_cuda_flash_attn_ext_mma_f16():
  • replace the bare case 512: call to ggml_cuda_flash_attn_ext_mma_f16_switch_ncols2<512, 512>(ctx, dst);
  • with a conditional branch that computes:
    • max_bias from KQV->op_params
    • use_gqa_opt = mask && max_bias == 0.0f
    • gqa_ratio = Q->ne[2] / K->ne[2]
  • then route:
    • gqa_ratio > 4 -> ggml_cuda_flash_attn_ext_mma_f16_switch_ncols1<512, 512, 8>(ctx, dst);
    • gqa_ratio > 2 -> ggml_cuda_flash_attn_ext_mma_f16_switch_ncols1<512, 512, 4>(ctx, dst);
    • gqa_ratio > 1 -> ggml_cuda_flash_attn_ext_mma_f16_switch_ncols1<512, 512, 2>(ctx, dst);
    • otherwise -> ggml_cuda_flash_attn_ext_mma_f16_switch_ncols2<512, 512>(ctx, dst);
• For the Gemma4 MTP shape that was failing, this keeps the kernel on supported flash-attn variants instead of the aborting path.
• No model architecture code, server CLI parsing, or speculative decoding flow was changed.

Validation

• Rebuilt ggml-cuda.
• Re-ran the same llama-server command.
• Startup now reaches speculative decoding context initialized and server is listening.

Impact

• Scope is limited to CUDA flash-attn dispatch for this MTP/Gemma4 case.
• No model or server CLI changes were required.

Build and run

1. Edit ggml/src/ggml-cuda/fattn.cu.
2. Rebuild the CUDA backend:

   cmake --build build --target ggml-cuda -j8

3. No full clean rebuild is needed unless you also change unrelated source files or CMake settings.

[ZacharyReis]

Confirming this on a third configuration — CUDA, not ROCm — so it isn't backend-specific:

• llama.cpp build 1349 (ac4cddeb0), CUDA 13.3, driver 610.43.02
• RTX 5060 Ti 16 GB (Blackwell, sm_120), built ARCH=120a-real with CUDA graphs

Same fattn.cu:110 abort, raised during the load-time common_context_can_seq_rm probe decode:

ggml/src/ggml-cuda/fattn.cu:110: fatal error
  ggml_cuda_flash_attn_ext → ggml_cuda_graph_evaluate_and_capture
  → llama_decode → common_context_can_seq_rm → server_context_impl::load_model

Target gemma-4-E4B-it-Q5_K_M + draft MTP/gemma-4-E4B-it-BF16-MTP (Unsloth), --spec-type draft-mtp --spec-draft-n-max 3 -fa on.

Zero-rebuild workaround for anyone blocked: -fa off. The identical setup then loads and serves normally — and draft-mtp is genuinely effective on constrained output. On a GBNF/JSON-grammar intent classifier we measured ~0.69 aggregate draft acceptance and a ~39% p50 latency reduction vs the no-draft baseline (both -fa off). So the feature is well worth the fix; @dextune's case 512/576 → BEST_FATTN_KERNEL_TILE fallback matches the dispatch path in this backtrace (we used -fa off rather than the patch, so can't independently vouch for it yet).

[arbv]

Here are my logs (RX7900XTX), FWIW, from the latest docker:

Logs

0.00.139.280 I common_params_print_info: build 9592 (ac4cddeb0) with GNU 13.3.0 for Linux x86_64
0.00.139.283 I log_info: verbosity = 4 (adjust with the `-lv N` CLI arg)
0.00.139.283 I device_info:
0.00.139.293 I   - ROCm0   : AMD Radeon RX 7900 XTX (24560 MiB, 24504 MiB free)
0.00.139.297 I   - CPU     : AMD Ryzen 9 9950X 16-Core Processor (94205 MiB, 94205 MiB free)
0.00.139.319 I system_info: n_threads = 16 (n_threads_batch = 16) / 32 | ROCm : NO_VMM = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | 
0.00.139.577 I srv          init: running without SSL
0.00.140.029 I srv          init: using 31 threads for HTTP server
0.00.140.030 I srv          init: The UI is disabled
0.00.140.030 I srv          init: Use --ui/--no-ui (or deprecated --webui/--no-webui) to enable/disable
0.00.140.279 I srv         start: binding port with default address family
0.00.141.584 I srv  llama_server: loading model
0.00.141.591 I srv    load_model: loading model '/models/unsloth-gemma-4-E4B-it-GGUF/gemma-4-E4B-it-UD-Q8_K_XL.gguf'
0.01.290.245 I srv    load_model: [mtmd] estimated worst-case memory usage of mmproj is 945.51 MiB
0.01.593.413 E llama_init_from_model: failed to initialize the context: Gemma4Assistant requires ctx_other to be set (this is normal during memory fitting)
0.01.652.489 W srv    load_model: [spec] failed to measure draft model memory: failed to create llama_context from model
0.01.652.506 I common_init_result: fitting params to device memory ...
0.01.652.507 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
0.01.652.510 I common_params_fit_impl: getting device memory data for initial parameters:
0.01.962.872 I common_memory_breakdown_print: | memory breakdown [MiB]  | total    free    self   model   context   compute    unaccounted |
0.01.962.874 I common_memory_breakdown_print: |   - ROCm0 (RX 7900 XTX) | 24560 = 24356 + (8882 =  5436 +    2168 +    1278) +       -8678 |
0.01.962.874 I common_memory_breakdown_print: |   - Host                |                  4184 =  3536 +       0 +     648                |
0.02.020.891 I common_params_fit_impl: projected to use 8882 MiB of device memory vs. 24356 MiB of free device memory
0.02.020.894 I common_params_fit_impl: will leave 15473 >= 3505 MiB of free device memory, no changes needed
0.02.020.895 I common_fit_params: successfully fit params to free device memory
0.02.020.898 I common_fit_params: fitting params to free memory took 0.37 seconds
0.02.062.824 I llama_model_loader: loaded meta data with 56 key-value pairs and 720 tensors from /models/unsloth-gemma-4-E4B-it-GGUF/gemma-4-E4B-it-UD-Q8_K_XL.gguf (version GGUF V3 (latest))
0.02.062.838 I llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
0.02.062.842 I llama_model_loader: - kv   0:                       general.architecture str              = gemma4
0.02.062.843 I llama_model_loader: - kv   1:                               general.type str              = model
0.02.062.844 I llama_model_loader: - kv   2:                     general.sampling.top_k i32              = 64
0.02.062.860 I llama_model_loader: - kv   3:                     general.sampling.top_p f32              = 0.950000
0.02.062.861 I llama_model_loader: - kv   4:                      general.sampling.temp f32              = 1.000000
0.02.062.861 I llama_model_loader: - kv   5:                               general.name str              = Gemma-4-E4B-It
0.02.062.861 I llama_model_loader: - kv   6:                           general.basename str              = Gemma-4-E4B-It
0.02.062.862 I llama_model_loader: - kv   7:                       general.quantized_by str              = Unsloth
0.02.062.862 I llama_model_loader: - kv   8:                         general.size_label str              = 7.5B
0.02.062.862 I llama_model_loader: - kv   9:                            general.license str              = apache-2.0
0.02.062.864 I llama_model_loader: - kv  10:                       general.license.link str              = https://ai.google.dev/gemma/docs/gemm...
0.02.062.865 I llama_model_loader: - kv  11:                           general.repo_url str              = https://huggingface.co/unsloth
0.02.062.866 I llama_model_loader: - kv  12:                   general.base_model.count u32              = 1
0.02.062.867 I llama_model_loader: - kv  13:                  general.base_model.0.name str              = Gemma 4 E4B It
0.02.062.867 I llama_model_loader: - kv  14:          general.base_model.0.organization str              = Google
0.02.062.868 I llama_model_loader: - kv  15:              general.base_model.0.repo_url str              = https://huggingface.co/google/gemma-4...
0.02.062.875 I llama_model_loader: - kv  16:                               general.tags arr[str,2]       = ["unsloth", "any-to-any"]
0.02.062.876 I llama_model_loader: - kv  17:                         gemma4.block_count u32              = 42
0.02.062.879 I llama_model_loader: - kv  18:                      gemma4.context_length u32              = 131072
0.02.062.880 I llama_model_loader: - kv  19:                    gemma4.embedding_length u32              = 2560
0.02.062.880 I llama_model_loader: - kv  20:                 gemma4.feed_forward_length u32              = 10240
0.02.062.880 I llama_model_loader: - kv  21:                gemma4.attention.head_count u32              = 8
0.02.062.881 I llama_model_loader: - kv  22:             gemma4.attention.head_count_kv u32              = 2
0.02.062.882 I llama_model_loader: - kv  23:                      gemma4.rope.freq_base f32              = 1000000.000000
0.02.062.883 I llama_model_loader: - kv  24:                  gemma4.rope.freq_base_swa f32              = 10000.000000
0.02.062.884 I llama_model_loader: - kv  25:    gemma4.attention.layer_norm_rms_epsilon f32              = 0.000001
0.02.062.884 I llama_model_loader: - kv  26:                gemma4.attention.key_length u32              = 512
0.02.062.885 I llama_model_loader: - kv  27:              gemma4.attention.value_length u32              = 512
0.02.062.886 I llama_model_loader: - kv  28:             gemma4.final_logit_softcapping f32              = 30.000000
0.02.062.886 I llama_model_loader: - kv  29:            gemma4.attention.sliding_window u32              = 512
0.02.062.887 I llama_model_loader: - kv  30:          gemma4.attention.shared_kv_layers u32              = 18
0.02.062.887 I llama_model_loader: - kv  31:    gemma4.embedding_length_per_layer_input u32              = 256
0.02.062.889 I llama_model_loader: - kv  32:    gemma4.attention.sliding_window_pattern arr[bool,42]     = [true, true, true, true, true, false,...
0.02.062.891 I llama_model_loader: - kv  33:            gemma4.attention.key_length_swa u32              = 256
0.02.062.892 I llama_model_loader: - kv  34:          gemma4.attention.value_length_swa u32              = 256
0.02.062.892 I llama_model_loader: - kv  35:                gemma4.rope.dimension_count u32              = 512
0.02.062.892 I llama_model_loader: - kv  36:            gemma4.rope.dimension_count_swa u32              = 256
0.02.062.893 I llama_model_loader: - kv  37:                       tokenizer.ggml.model str              = gemma4
0.02.075.205 I llama_model_loader: - kv  38:                      tokenizer.ggml.tokens arr[str,262144]  = ["<pad>", "<eos>", "<bos>", "<unk>", ...
0.02.099.484 I llama_model_loader: - kv  39:                      tokenizer.ggml.scores arr[f32,262144]  = [-1000.000000, -1000.000000, -1000.00...
0.02.103.055 I llama_model_loader: - kv  40:                  tokenizer.ggml.token_type arr[i32,262144]  = [3, 1, 3, 3, 3, 1, 1, 1, 1, 1, 1, 1, ...
0.02.126.751 I llama_model_loader: - kv  41:                      tokenizer.ggml.merges arr[str,514906]  = ["\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n", ...
0.02.126.754 I llama_model_loader: - kv  42:                tokenizer.ggml.bos_token_id u32              = 2
0.02.126.754 I llama_model_loader: - kv  43:                tokenizer.ggml.eos_token_id u32              = 106
0.02.126.755 I llama_model_loader: - kv  44:            tokenizer.ggml.unknown_token_id u32              = 3
0.02.126.755 I llama_model_loader: - kv  45:            tokenizer.ggml.padding_token_id u32              = 0
0.02.126.756 I llama_model_loader: - kv  46:               tokenizer.ggml.mask_token_id u32              = 4
0.02.126.758 I llama_model_loader: - kv  47:                    tokenizer.chat_template str              = {%- macro format_parameters(propertie...
0.02.126.771 I llama_model_loader: - kv  48:            tokenizer.ggml.add_space_prefix bool             = false
0.02.126.773 I llama_model_loader: - kv  49:               tokenizer.ggml.add_bos_token bool             = true
0.02.126.773 I llama_model_loader: - kv  50:               general.quantization_version u32              = 2
0.02.126.778 I llama_model_loader: - kv  51:                          general.file_type u32              = 7
0.02.126.779 I llama_model_loader: - kv  52:                      quantize.imatrix.file str              = gemma-4-E4B-it-GGUF/imatrix_unsloth.gguf
0.02.126.780 I llama_model_loader: - kv  53:                   quantize.imatrix.dataset str              = unsloth_calibration_gemma-4-E4B-it.txt
0.02.126.781 I llama_model_loader: - kv  54:             quantize.imatrix.entries_count u32              = 342
0.02.126.782 I llama_model_loader: - kv  55:              quantize.imatrix.chunks_count u32              = 141
0.02.126.783 I llama_model_loader: - type  f32:  423 tensors
0.02.126.784 I llama_model_loader: - type  f16:  104 tensors
0.02.126.785 I llama_model_loader: - type q8_0:  192 tensors
0.02.126.788 I llama_model_loader: - type bf16:    1 tensors
0.02.126.790 I print_info: file format = GGUF V3 (latest)
0.02.126.793 I print_info: file type   = Q8_0
0.02.126.796 I print_info: file size   = 8.10 GiB (9.25 BPW) 
0.02.126.819 I llama_prepare_model_devices: using device ROCm0 (AMD Radeon RX 7900 XTX) (0000:03:00.0) - 24356 MiB free
0.02.269.232 I load: 0 unused tokens
0.02.276.953 W load: control-looking token:    212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
0.02.277.278 W load: control-looking token:     50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden
0.02.281.724 W load: control-looking token:      1 '<eos>' was not control-type; this is probably a bug in the model. its type will be overridden
0.02.294.625 I load: printing all EOG tokens:
0.02.294.626 I load:   - 1 ('<eos>')
0.02.294.627 I load:   - 50 ('<|tool_response>')
0.02.294.627 I load:   - 106 ('<turn|>')
0.02.294.628 I load:   - 212 ('</s>')
0.02.294.630 W load: special_eog_ids contains '<|tool_response>', removing '</s>' token from EOG list
0.02.294.831 I load: special tokens cache size = 24
0.02.306.990 I load: token to piece cache size = 1.9445 MB
0.02.307.003 I print_info: arch                  = gemma4
0.02.307.004 I print_info: vocab_only            = 0
0.02.307.004 I print_info: no_alloc              = 0
0.02.307.015 I print_info: n_ctx_train           = 131072
0.02.307.016 I print_info: n_embd_inp            = 2560
0.02.307.016 I print_info: n_embd                = 2560
0.02.307.016 I print_info: n_embd_out            = 2560
0.02.307.016 I print_info: n_layer               = 42
0.02.307.016 I print_info: n_layer_all           = 42
0.02.307.024 I print_info: n_head                = 8
0.02.307.026 I print_info: n_head_kv             = 2
0.02.307.026 I print_info: n_rot                 = 512
0.02.307.026 I print_info: n_swa                 = 512
0.02.307.027 I print_info: is_swa_any            = 1
0.02.307.027 I print_info: n_embd_head_k         = 512
0.02.307.027 I print_info: n_embd_head_v         = 512
0.02.307.032 I print_info: n_gqa                 = 4
0.02.307.039 I print_info: n_embd_k_gqa          = [512, 512, 512, 512, 512, 1024, 512, 512, 512, 512, 512, 1024, 512, 512, 512, 512, 512, 1024, 512, 512, 512, 512, 512, 1024, 512, 512, 512, 512, 512, 1024, 512, 512, 512, 512, 512, 1024, 512, 512, 512, 512, 512, 1024]
0.02.307.042 I print_info: n_embd_v_gqa          = [512, 512, 512, 512, 512, 1024, 512, 512, 512, 512, 512, 1024, 512, 512, 512, 512, 512, 1024, 512, 512, 512, 512, 512, 1024, 512, 512, 512, 512, 512, 1024, 512, 512, 512, 512, 512, 1024, 512, 512, 512, 512, 512, 1024]
0.02.307.043 I print_info: f_norm_eps            = 0.0e+00
0.02.307.043 I print_info: f_norm_rms_eps        = 1.0e-06
0.02.307.044 I print_info: f_clamp_kqv           = 0.0e+00
0.02.307.045 I print_info: f_max_alibi_bias      = 0.0e+00
0.02.307.045 I print_info: f_logit_scale         = 0.0e+00
0.02.307.046 I print_info: f_attn_scale          = 1.0e+00
0.02.307.046 I print_info: f_attn_value_scale    = 0.0000
0.02.307.048 I print_info: n_ff                  = 10240
0.02.307.048 I print_info: n_expert              = 0
0.02.307.048 I print_info: n_expert_used         = 0
0.02.307.049 I print_info: n_expert_groups       = 0
0.02.307.049 I print_info: n_group_used          = 0
0.02.307.050 I print_info: causal attn           = 1
0.02.307.053 I print_info: pooling type          = -1
0.02.307.056 I print_info: rope type             = 2
0.02.307.057 I print_info: rope scaling          = linear
0.02.307.058 I print_info: freq_base_train       = 1000000.0
0.02.307.058 I print_info: freq_scale_train      = 1
0.02.307.059 I print_info: freq_base_swa         = 10000.0
0.02.307.062 I print_info: freq_scale_swa        = 1
0.02.307.063 I print_info: n_embd_head_k_swa     = 256
0.02.307.063 I print_info: n_embd_head_v_swa     = 256
0.02.307.063 I print_info: n_rot_swa             = 256
0.02.307.064 I print_info: n_ctx_orig_yarn       = 131072
0.02.307.065 I print_info: rope_yarn_log_mul     = 0.0000
0.02.307.066 I print_info: rope_finetuned        = unknown
0.02.307.066 I print_info: model type            = E4B
0.02.307.067 I print_info: model params          = 7.52 B
0.02.307.068 I print_info: general.name          = Gemma-4-E4B-It
0.02.307.071 I print_info: vocab type            = BPE
0.02.307.073 I print_info: n_vocab               = 262144
0.02.307.073 I print_info: n_merges              = 514906
0.02.307.074 I print_info: BOS token             = 2 '<bos>'
0.02.307.075 I print_info: EOS token             = 106 '<turn|>'
0.02.307.076 I print_info: UNK token             = 3 '<unk>'
0.02.307.076 I print_info: PAD token             = 0 '<pad>'
0.02.307.076 I print_info: MASK token            = 4 '<mask>'
0.02.307.077 I print_info: LF token              = 107 '
'
0.02.307.078 I print_info: EOG token             = 1 '<eos>'
0.02.307.079 I print_info: EOG token             = 50 '<|tool_response>'
0.02.307.079 I print_info: EOG token             = 106 '<turn|>'
0.02.307.079 I print_info: max token length      = 93
0.02.307.080 I load_tensors: loading model tensors, this can take a while... (mmap = false, direct_io = false)
0.02.456.611 W warning: failed to mlock 3707764736-byte buffer (after previously locking 0 bytes): Cannot allocate memory
Try increasing RLIMIT_MEMLOCK ('ulimit -l' as root).
0.02.456.626 I load_tensors: offloading output layer to GPU
0.02.456.627 I load_tensors: offloading 41 repeating layers to GPU
0.02.456.627 I load_tensors: offloaded 43/43 layers to GPU
0.02.456.631 I load_tensors:        ROCm0 model buffer size =  5436.85 MiB
0.02.456.632 I load_tensors:    ROCm_Host model buffer size =  3536.00 MiB
........................................................
0.08.825.595 I common_init_result: added <eos> logit bias = -inf
0.08.825.602 I common_init_result: added <|tool_response> logit bias = -inf
0.08.825.603 I common_init_result: added <turn|> logit bias = -inf
0.08.827.377 I llama_context: constructing llama_context
0.08.827.383 I llama_context: n_seq_max     = 2
0.08.827.383 I llama_context: n_ctx         = 131072
0.08.827.383 I llama_context: n_ctx_seq     = 131072
0.08.827.383 I llama_context: n_batch       = 4096
0.08.827.384 I llama_context: n_ubatch      = 2048
0.08.827.384 I llama_context: causal_attn   = 1
0.08.827.384 I llama_context: flash_attn    = auto
0.08.827.385 I llama_context: kv_unified    = true
0.08.827.388 I llama_context: freq_base     = 1000000.0
0.08.827.389 I llama_context: freq_scale    = 1
0.08.827.389 I llama_context: n_rs_seq      = 0
0.08.827.389 I llama_context: n_outputs_max = 4
0.08.828.695 I llama_context:  ROCm_Host  output buffer size =     2.00 MiB
0.08.828.699 I llama_kv_cache_iswa: creating non-SWA KV cache, size = 131072 cells
0.08.829.326 I llama_kv_cache:      ROCm0 KV buffer size =  2048.00 MiB
0.08.835.192 I llama_kv_cache: size = 2048.00 MiB (131072 cells,   4 layers,  2/1 seqs), K (bf16): 1024.00 MiB, V (bf16): 1024.00 MiB
0.08.835.194 I llama_kv_cache: attn_rot_k = 0, n_embd_head_k_all = 512
0.08.835.194 I llama_kv_cache: attn_rot_v = 0, n_embd_head_k_all = 512
0.08.835.195 I llama_kv_cache_iswa: creating     SWA KV cache, size = 3072 cells
0.08.835.332 I llama_kv_cache:      ROCm0 KV buffer size =   120.00 MiB
0.08.835.553 I llama_kv_cache: size =  120.00 MiB (  3072 cells,  20 layers,  2/1 seqs), K (bf16):   60.00 MiB, V (bf16):   60.00 MiB
0.08.835.555 I llama_kv_cache: attn_rot_k = 0, n_embd_head_k_all = 256
0.08.835.555 I llama_kv_cache: attn_rot_v = 0, n_embd_head_k_all = 256
0.08.835.561 I sched_reserve: reserving ...
0.08.836.295 I sched_reserve: Flash Attention was auto, set to enabled
0.08.836.296 I sched_reserve: resolving fused Gated Delta Net support:
0.08.836.590 I sched_reserve: fused Gated Delta Net (autoregressive) enabled
0.08.836.860 I sched_reserve: fused Gated Delta Net (chunked) enabled
0.08.867.539 I sched_reserve:      ROCm0 compute buffer size =  1278.07 MiB
0.08.867.543 I sched_reserve:  ROCm_Host compute buffer size =   648.09 MiB
0.08.867.543 I sched_reserve: graph nodes  = 1863
0.08.867.543 I sched_reserve: graph splits = 2
0.08.867.544 I sched_reserve: reserve took 31.98 ms, sched copies = 1
0.08.867.634 W common_init_from_params: KV cache shifting is not supported for this context, disabling KV cache shifting
0.08.867.651 I srv    load_model: loading draft model '/models/unsloth-gemma-4-E4B-it-GGUF/MTP/gemma-4-E4B-it-Q8_0-MTP.gguf'
0.08.923.682 I llama_model_loader: loaded meta data with 45 key-value pairs and 49 tensors from /models/unsloth-gemma-4-E4B-it-GGUF/MTP/gemma-4-E4B-it-Q8_0-MTP.gguf (version GGUF V3 (latest))
0.08.923.689 I llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
0.08.923.694 I llama_model_loader: - kv   0:                       general.architecture str              = gemma4-assistant
0.08.923.695 I llama_model_loader: - kv   1:                               general.type str              = model
0.08.923.697 I llama_model_loader: - kv   2:                     general.sampling.top_k i32              = 64
0.08.923.701 I llama_model_loader: - kv   3:                     general.sampling.top_p f32              = 0.950000
0.08.923.703 I llama_model_loader: - kv   4:                      general.sampling.temp f32              = 1.000000
0.08.923.704 I llama_model_loader: - kv   5:                               general.name str              = E4B
0.08.923.711 I llama_model_loader: - kv   6:                         general.size_label str              = 78M
0.08.923.713 I llama_model_loader: - kv   7:                            general.license str              = apache-2.0
0.08.923.714 I llama_model_loader: - kv   8:                       general.license.link str              = https://ai.google.dev/gemma/docs/gemm...
0.08.923.741 I llama_model_loader: - kv   9:                               general.tags arr[str,1]       = ["any-to-any"]
0.08.923.743 I llama_model_loader: - kv  10:               gemma4-assistant.block_count u32              = 4
0.08.923.743 I llama_model_loader: - kv  11:            gemma4-assistant.context_length u32              = 131072
0.08.923.744 I llama_model_loader: - kv  12:          gemma4-assistant.embedding_length u32              = 256
0.08.923.745 I llama_model_loader: - kv  13:       gemma4-assistant.feed_forward_length u32              = 2048
0.08.923.745 I llama_model_loader: - kv  14:      gemma4-assistant.attention.head_count u32              = 4
0.08.923.746 I llama_model_loader: - kv  15:   gemma4-assistant.attention.head_count_kv u32              = 2
0.08.923.747 I llama_model_loader: - kv  16:            gemma4-assistant.rope.freq_base f32              = 1000000.000000
0.08.923.749 I llama_model_loader: - kv  17:        gemma4-assistant.rope.freq_base_swa f32              = 10000.000000
0.08.923.749 I llama_model_loader: - kv  18: gemma4-assistant.attention.layer_norm_rms_epsilon f32              = 0.000001
0.08.923.750 I llama_model_loader: - kv  19:      gemma4-assistant.attention.key_length u32              = 512
0.08.923.750 I llama_model_loader: - kv  20:    gemma4-assistant.attention.value_length u32              = 512
0.08.923.751 I llama_model_loader: - kv  21:                          general.file_type u32              = 7
0.08.923.751 I llama_model_loader: - kv  22:  gemma4-assistant.attention.sliding_window u32              = 512
0.08.923.752 I llama_model_loader: - kv  23: gemma4-assistant.attention.shared_kv_layers u32              = 4
0.08.923.753 I llama_model_loader: - kv  24: gemma4-assistant.embedding_length_per_layer_input u32              = 0
0.08.923.756 I llama_model_loader: - kv  25: gemma4-assistant.attention.sliding_window_pattern arr[bool,4]      = [true, true, true, false]
0.08.923.756 I llama_model_loader: - kv  26:  gemma4-assistant.attention.key_length_swa u32              = 256
0.08.923.757 I llama_model_loader: - kv  27: gemma4-assistant.attention.value_length_swa u32              = 256
0.08.923.757 I llama_model_loader: - kv  28:      gemma4-assistant.rope.dimension_count u32              = 512
0.08.923.758 I llama_model_loader: - kv  29:  gemma4-assistant.rope.dimension_count_swa u32              = 256
0.08.923.758 I llama_model_loader: - kv  30:      gemma4-assistant.embedding_length_out u32              = 2560
0.08.923.759 I llama_model_loader: - kv  31:      gemma4-assistant.nextn_predict_layers u32              = 4
0.08.923.760 I llama_model_loader: - kv  32:               general.quantization_version u32              = 2
0.08.923.760 I llama_model_loader: - kv  33:                       tokenizer.ggml.model str              = gemma4
0.08.936.731 I llama_model_loader: - kv  34:                      tokenizer.ggml.tokens arr[str,262144]  = ["<pad>", "<eos>", "<bos>", "<unk>", ...
0.08.961.072 I llama_model_loader: - kv  35:                      tokenizer.ggml.scores arr[f32,262144]  = [-1000.000000, -1000.000000, -1000.00...
0.08.964.660 I llama_model_loader: - kv  36:                  tokenizer.ggml.token_type arr[i32,262144]  = [3, 3, 3, 3, 3, 1, 1, 1, 1, 1, 1, 1, ...
0.08.988.197 I llama_model_loader: - kv  37:                      tokenizer.ggml.merges arr[str,514906]  = ["\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n", ...
0.08.988.201 I llama_model_loader: - kv  38:                tokenizer.ggml.bos_token_id u32              = 2
0.08.988.201 I llama_model_loader: - kv  39:                tokenizer.ggml.eos_token_id u32              = 1
0.08.988.202 I llama_model_loader: - kv  40:            tokenizer.ggml.unknown_token_id u32              = 3
0.08.988.202 I llama_model_loader: - kv  41:            tokenizer.ggml.padding_token_id u32              = 0
0.08.988.203 I llama_model_loader: - kv  42:               tokenizer.ggml.mask_token_id u32              = 4
0.08.988.204 I llama_model_loader: - kv  43:            tokenizer.ggml.add_space_prefix bool             = false
0.08.988.204 I llama_model_loader: - kv  44:               tokenizer.ggml.add_bos_token bool             = true
0.08.988.205 I llama_model_loader: - type  f32:   26 tensors
0.08.988.205 I llama_model_loader: - type q8_0:   23 tensors
0.08.988.207 I print_info: file format = GGUF V3 (latest)
0.08.988.207 I print_info: file type   = Q8_0
0.08.988.210 I print_info: file size   = 79.05 MiB (8.50 BPW) 
0.08.988.235 I llama_prepare_model_devices: using device ROCm0 (AMD Radeon RX 7900 XTX) (0000:03:00.0) - 15470 MiB free
0.09.169.240 I load: 0 unused tokens
0.09.178.575 W load: control-looking token:    212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
0.09.178.897 W load: control-looking token:     50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden
0.09.193.520 I load: printing all EOG tokens:
0.09.193.522 I load:   - 1 ('<eos>')
0.09.193.522 I load:   - 50 ('<|tool_response>')
0.09.193.522 I load:   - 106 ('<turn|>')
0.09.193.523 I load:   - 212 ('</s>')
0.09.193.525 W load: special_eog_ids contains '<|tool_response>', removing '</s>' token from EOG list
0.09.193.691 I load: special tokens cache size = 23
0.09.206.929 I load: token to piece cache size = 1.9445 MB
0.09.206.947 I print_info: arch                  = gemma4-assistant
0.09.206.950 I print_info: vocab_only            = 0
0.09.206.950 I print_info: no_alloc              = 0
0.09.206.950 I print_info: n_ctx_train           = 131072
0.09.206.951 I print_info: n_embd_inp            = 2560
0.09.206.951 I print_info: n_embd                = 256
0.09.206.951 I print_info: n_embd_out            = 2560
0.09.206.952 I print_info: n_layer               = 0
0.09.206.952 I print_info: n_layer_all           = 4
0.09.206.959 I print_info: n_head                = 4
0.09.206.960 I print_info: n_head_kv             = 2
0.09.206.960 I print_info: n_rot                 = 512
0.09.206.961 I print_info: n_swa                 = 512
0.09.206.961 I print_info: is_swa_any            = 1
0.09.206.961 I print_info: n_embd_head_k         = 512
0.09.206.961 I print_info: n_embd_head_v         = 512
0.09.206.962 I print_info: n_gqa                 = 2
0.09.206.965 I print_info: n_embd_k_gqa          = [512, 512, 512, 1024]
0.09.206.966 I print_info: n_embd_v_gqa          = [512, 512, 512, 1024]
0.09.206.967 I print_info: f_norm_eps            = 0.0e+00
0.09.206.968 I print_info: f_norm_rms_eps        = 1.0e-06
0.09.206.968 I print_info: f_clamp_kqv           = 0.0e+00
0.09.206.969 I print_info: f_max_alibi_bias      = 0.0e+00
0.09.206.970 I print_info: f_logit_scale         = 0.0e+00
0.09.206.970 I print_info: f_attn_scale          = 1.0e+00
0.09.206.970 I print_info: f_attn_value_scale    = 0.0000
0.09.206.971 I print_info: n_ff                  = 2048
0.09.206.971 I print_info: n_expert              = 0
0.09.206.971 I print_info: n_expert_used         = 0
0.09.206.972 I print_info: n_expert_groups       = 0
0.09.206.972 I print_info: n_group_used          = 0
0.09.206.973 I print_info: causal attn           = 1
0.09.206.980 I print_info: pooling type          = -1
0.09.206.980 I print_info: rope type             = 2
0.09.206.980 I print_info: rope scaling          = linear
0.09.206.981 I print_info: freq_base_train       = 1000000.0
0.09.206.982 I print_info: freq_scale_train      = 1
0.09.206.982 I print_info: freq_base_swa         = 10000.0
0.09.206.983 I print_info: freq_scale_swa        = 1
0.09.206.983 I print_info: n_embd_head_k_swa     = 256
0.09.206.983 I print_info: n_embd_head_v_swa     = 256
0.09.206.983 I print_info: n_rot_swa             = 256
0.09.206.983 I print_info: n_ctx_orig_yarn       = 131072
0.09.206.983 I print_info: rope_yarn_log_mul     = 0.0000
0.09.206.984 I print_info: rope_finetuned        = unknown
0.09.206.985 I print_info: model type            = ?B
0.09.206.986 I print_info: model params          = 77.99 M
0.09.206.988 I print_info: general.name          = E4B
0.09.206.989 I print_info: vocab type            = BPE
0.09.206.989 I print_info: n_vocab               = 262144
0.09.206.989 I print_info: n_merges              = 514906
0.09.206.990 I print_info: BOS token             = 2 '<bos>'
0.09.206.998 I print_info: EOS token             = 1 '<eos>'
0.09.206.999 I print_info: UNK token             = 3 '<unk>'
0.09.206.999 I print_info: PAD token             = 0 '<pad>'
0.09.206.999 I print_info: MASK token            = 4 '<mask>'
0.09.207.000 I print_info: LF token              = 107 '
'
0.09.207.000 I print_info: EOG token             = 1 '<eos>'
0.09.207.000 I print_info: EOG token             = 50 '<|tool_response>'
0.09.207.000 I print_info: EOG token             = 106 '<turn|>'
0.09.207.001 I print_info: max token length      = 93
0.09.207.002 I load_tensors: loading model tensors, this can take a while... (mmap = false, direct_io = false)
0.09.212.951 W warning: failed to mlock 71303168-byte buffer (after previously locking 0 bytes): Cannot allocate memory
Try increasing RLIMIT_MEMLOCK ('ulimit -l' as root).
0.09.212.956 I load_tensors: offloading output layer to GPU
0.09.212.957 I load_tensors: offloading 3 repeating layers to GPU
0.09.212.957 I load_tensors: offloaded 5/5 layers to GPU
0.09.212.960 I load_tensors:        ROCm0 model buffer size =    79.05 MiB
0.09.212.962 I load_tensors:    ROCm_Host model buffer size =    68.00 MiB
.........
0.09.278.137 I llama_context: constructing llama_context
0.09.278.140 I llama_context: n_seq_max     = 2
0.09.278.140 I llama_context: n_ctx         = 131072
0.09.278.140 I llama_context: n_ctx_seq     = 131072
0.09.278.140 I llama_context: n_batch       = 4096
0.09.278.140 I llama_context: n_ubatch      = 2048
0.09.278.141 I llama_context: causal_attn   = 1
0.09.278.142 I llama_context: flash_attn    = auto
0.09.278.142 I llama_context: kv_unified    = true
0.09.278.145 I llama_context: freq_base     = 1000000.0
0.09.278.146 I llama_context: freq_scale    = 1
0.09.278.146 I llama_context: n_rs_seq      = 0
0.09.278.147 I llama_context: n_outputs_max = 4
0.09.278.596 I llama_context:  ROCm_Host  output buffer size =     2.00 MiB
0.09.278.600 I llama_kv_cache_iswa: creating non-SWA KV cache, size = 131072 cells
0.09.279.417 W llama_kv_cache: layer   3: sharing with layer 41. k = 0x7262aaa00000, v = 0x7262baa00000
0.09.279.425 I llama_kv_cache: size =  512.00 MiB (131072 cells,   1 layers,  2/1 seqs), K (q8_0):  256.00 MiB, V (q8_0):  256.00 MiB
0.09.279.425 I llama_kv_cache: attn_rot_k = 0, n_embd_head_k_all = 512
0.09.279.426 I llama_kv_cache: attn_rot_v = 0, n_embd_head_k_all = 512
0.09.279.427 I llama_kv_cache_iswa: creating     SWA KV cache, size = 3072 cells
0.09.279.449 W llama_kv_cache: layer   0: sharing with layer 40. k = 0x726513800000, v = 0x726513b00000
0.09.279.451 W llama_kv_cache: layer   1: sharing with layer 40. k = 0x726513800000, v = 0x726513b00000
0.09.279.452 W llama_kv_cache: layer   2: sharing with layer 40. k = 0x726513800000, v = 0x726513b00000
0.09.279.454 I llama_kv_cache: size =   18.00 MiB (  3072 cells,   3 layers,  2/1 seqs), K (q8_0):    9.00 MiB, V (q8_0):    9.00 MiB
0.09.279.455 I llama_kv_cache: attn_rot_k = 0, n_embd_head_k_all = 256
0.09.279.455 I llama_kv_cache: attn_rot_v = 0, n_embd_head_k_all = 256
0.09.279.461 I sched_reserve: reserving ...
0.09.279.592 I sched_reserve: Flash Attention was auto, set to enabled
0.09.279.601 I sched_reserve: resolving fused Gated Delta Net support:
0.09.279.636 I sched_reserve: fused Gated Delta Net (autoregressive) enabled
0.09.279.665 I sched_reserve: fused Gated Delta Net (chunked) enabled
0.09.304.114 I sched_reserve:      ROCm0 compute buffer size =  1092.01 MiB
0.09.304.117 I sched_reserve:  ROCm_Host compute buffer size =   564.02 MiB
0.09.304.117 I sched_reserve: graph nodes  = 126
0.09.304.117 I sched_reserve: graph splits = 2
0.09.304.118 I sched_reserve: reserve took 24.66 ms, sched copies = 1
0.09.306.264 I clip_model_loader: model name:   Gemma-4-E4B-It
0.09.306.266 I clip_model_loader: description:  
0.09.306.267 I clip_model_loader: GGUF version: 3
0.09.306.268 I clip_model_loader: alignment:    32
0.09.306.268 I clip_model_loader: n_tensors:    1411
0.09.306.268 I clip_model_loader: n_kv:         38
0.09.306.268 I 
0.09.306.276 I clip_model_loader: has vision encoder
0.09.306.278 I clip_model_loader: has audio encoder
0.09.308.921 I clip_ctx: CLIP using ROCm0 backend
0.09.309.158 I load_hparams: projector:          gemma4v
0.09.309.159 I load_hparams: n_embd:             768
0.09.309.159 I load_hparams: n_head:             12
0.09.309.159 I load_hparams: n_ff:               3072
0.09.309.159 I load_hparams: n_layer:            16
0.09.309.160 I load_hparams: ffn_op:             gelu_quick
0.09.309.160 I load_hparams: projection_dim:     2560
0.09.309.160 I 
--- vision hparams ---
0.09.309.160 I load_hparams: image_size:         224
0.09.309.160 I load_hparams: patch_size:         16
0.09.309.160 I load_hparams: has_llava_proj:     0
0.09.309.161 I load_hparams: minicpmv_version:   0
0.09.309.162 I load_hparams: n_merge:            3
0.09.309.164 I load_hparams: n_wa_pattern: 0
0.09.309.164 I load_hparams: image_min_pixels:   1290240 (custom value)
0.09.309.164 I load_hparams: image_max_pixels:   2580480 (custom value)
0.09.309.164 I 
0.09.309.166 I load_hparams: model size:         944.39 MiB
0.09.309.167 I load_hparams: metadata size:      0.50 MiB
0.09.358.838 I clip_ctx: CLIP using ROCm0 backend
0.09.359.084 I load_hparams: projector:          gemma4a
0.09.359.087 I load_hparams: n_embd:             1024
0.09.359.088 I load_hparams: n_head:             8
0.09.359.088 I load_hparams: n_ff:               4096
0.09.359.088 I load_hparams: n_layer:            12
0.09.359.088 I load_hparams: ffn_op:             gelu_quick
0.09.359.088 I load_hparams: projection_dim:     2560
0.09.359.100 I 
--- audio hparams ---
0.09.359.100 I load_hparams: n_mel_bins:         128
0.09.359.101 I load_hparams: proj_stack_factor:  0
0.09.359.101 I load_hparams: audio_chunk_len:    0
0.09.359.101 I load_hparams: audio_sample_rate:  16000
0.09.359.101 I load_hparams: audio_n_fft:        512
0.09.359.101 I load_hparams: audio_window_len:   320
0.09.359.101 I load_hparams: audio_hop_len:      160
0.09.359.101 I 
0.09.359.104 I load_hparams: model size:         944.39 MiB
0.09.359.105 I load_hparams: metadata size:      0.50 MiB
0.09.419.127 W init_audio: audio input is in experimental stage and may have reduced quality:
    https://github.com/ggml-org/llama.cpp/discussions/13759
0.09.419.160 I srv    load_model: loaded multimodal model, '/models/unsloth-gemma-4-E4B-it-GGUF/mmproj-F16.gguf'
0.09.419.170 I srv    load_model: initializing slots, n_slots = 2
0.09.482.918 I common_speculative_impl_draft_mtp: adding speculative implementation 'draft-mtp'
0.09.482.923 I common_speculative_impl_draft_mtp: - n_max=1, n_min=0, p_min=0.00, n_embd=2560, backend_sampling=1
0.09.482.925 I common_speculative_impl_draft_mtp: - gpu_layers=-2, cache_k=q8_0, cache_v=q8_0, ctx_tgt=yes, ctx_dft=yes, devices=[default]
0.09.483.164 W llama_sampler_backend_support: device 'ROCm0' does not have support for op TOP_K needed for sampler 'top-k'
0.09.483.167 W llama_sampler_backend_support: device 'ROCm0' does not have support for op TOP_K needed for sampler 'top-k'
0.09.483.394 I sched_reserve: reserving ...
0.09.508.860 I sched_reserve:      ROCm0 compute buffer size =  1092.01 MiB
0.09.508.863 I sched_reserve:  ROCm_Host compute buffer size =   564.02 MiB
0.09.508.863 I sched_reserve: graph nodes  = 129
0.09.508.864 I sched_reserve: graph splits = 2
0.09.508.864 I sched_reserve: reserve took 25.47 ms, sched copies = 1
/app/ggml/src/ggml-cuda/template-instances/../fattn-tile.cuh:1321: fatal error
libggml-base.so.0(+0x1c1ab)[0x7266b11b91ab]
libggml-base.so.0(ggml_print_backtrace+0x21c)[0x7266b11b962c]
libggml-base.so.0(ggml_abort+0x15b)[0x7266b11b980b]
/app/libggml-hip.so(_Z34ggml_cuda_flash_attn_ext_tile_caseILi512ELi512EEvR25ggml_backend_cuda_contextP11ggml_tensor+0x65e)[0x7266ab91681e]
/app/libggml-hip.so(+0x23e87a02)[0x7266ab715a02]
/app/libggml-hip.so(+0x23e844b6)[0x7266ab7124b6]
libggml-base.so.0(ggml_backend_sched_graph_compute_async+0x817)[0x7266b11d6f57]
libllama.so.0(_ZN13llama_context13graph_computeEP11ggml_cgraphb+0xa1)[0x7266b1356011]
libllama.so.0(_ZN13llama_context14process_ubatchERK12llama_ubatch14llm_graph_typeP22llama_memory_context_iR11ggml_status+0x114)[0x7266b13587e4]
libllama.so.0(_ZN13llama_context6decodeERK11llama_batch+0x3a1)[0x7266b13600d1]
libllama.so.0(llama_decode+0xf)[0x7266b1361d3f]
libllama-common.so.0(_Z25common_context_can_seq_rmP13llama_context+0xd9)[0x7266b18d04d9]
libllama-server-impl.so(_ZN19server_context_impl10load_modelER13common_params+0xba0)[0x7266b2181c20]
libllama-server-impl.so(_Z12llama_serveriPPc+0x21a6)[0x7266b20d2166]
/lib/x86_64-linux-gnu/libc.so.6(+0x2a1ca)[0x7266b1b641ca]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x8b)[0x7266b1b6428b]
/app/llama-server(+0x12a5)[0x610d4c8c22a5]

[arbv]

I wish I could test Vulkan backend, but everything is broken: #24474

[kstoykov]

I also experience the same issue on CUDA 12. Interesting enough the drafter of E2B is working fine.

I tried Vulkan on Radeon 890M and it is working fine.

[philippjess]

Can reproduce on RTX 4090 (SM89, CUDA), master build from 2026-06-11, with the same models (unsloth gemma-4-E4B-it-qat-UD-Q4_K_XL.gguf + mtp-gemma-4-E4B-it.gguf, --spec-type draft-mtp --spec-draft-n-max 4, --parallel 2 --ctx-size 256000). Same abort at ggml/src/ggml-cuda/fattn.cu:110 during slot init.

On my hardware the flash-attention setting does change the outcome (unlike OP's report on 2× 3090):

--flash-attn | KV cache (target) | Result -- | -- | -- auto | q8_0 | crash at fattn.cu:110 auto | f16 | crash at fattn.cu:110 off | f16 | works — MTP runs, ~150 → ~350 t/s

So -fa auto's support check passes (presumably evaluated against the main model graph), but the MTP draft graph — which has cross-layer KV sharing (llama_kv_cache: layer 0-3: sharing with layer 40/41) — dispatches an attention op that lands in the unimplemented branch of the MMA kernel switch and hits the GGML_ABORT at fattn.cu:110. The crash fires on the warmup/probe decode during slot init (common_context_can_seq_rm → llama_decode). The backtrace is frame-identical across both crashing configs (q8_0 and f16 KV), so it's the same code path regardless of cache type.

Backtrace (-fa auto, f16 KV cache):

/app/ggml/src/ggml-cuda/fattn.cu:110: fatal error
libggml-base.so.0(ggml_abort+0x15b)
libggml-cuda.so(ggml_cuda_flash_attn_ext(ggml_backend_cuda_context&, ggml_tensor*)+0x38b5)
libggml-cuda.so(+0x27eeb5)
libggml-base.so.0(ggml_backend_sched_graph_compute_async+0x82f)
libllama.so.0(llama_context::graph_compute(ggml_cgraph*, bool)+0xa1)
libllama.so.0(llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&)+0x112)
libllama.so.0(llama_context::decode(llama_batch const&)+0x375)
libllama.so.0(llama_decode+0xf)
libllama-common.so.0(common_context_can_seq_rm(llama_context*)+0xd9)
libllama-server-impl.so(server_context_impl::load_model(common_params&)+0xdc4)
libllama-server-impl.so(llama_server(int, char**)+0x2dd3)

Two side notes for anyone else hitting this:

• --fit must also be off, otherwise loading fails earlier in the memory-fitting pass with Gemma4Assistant requires ctx_other to be set (that part is Eval bug: MTP with fit causes crash #24117 / Eval bug: Gemma4 MTP #24350, a separate bug).
• The -fa off workaround rules out quantized V-cache (it requires FA), so it has to be paired with f16 KV.

Even once the missing kernel is implemented, -fa auto selecting a path that aborts at runtime seems like its own bug — auto should validate against the draft/MTP graph too, or fall back instead of aborting.

[ozelentok]

For CUDA this appears to be caused by the change in #21768 which assumes that FlashAttention kernels with a head size of 512 and GQA ratios of 1 or 2 are not used, and therefore disables them to reduce compilation time.

Reverting commit ff5ef82 CUDA: skip compilation of superfluous FA kernels resolved the issue for me.

(I previously commented in #24400, but it is likely a duplicate of this issue.)

[localyouser]

My environment:
OS: Ubuntu 24.04 LTS
GPU: RTX3050 (Ampere) <- Only using this
　+ GTX1660Super (Turing) <- Not using this
CUDA-toolkit: 13.1.2-1
llama.cpp:
version: 9574 (bd5e5d252)
built with GNU 13.3.0 for Linux x86_64

Under this configuration, I reverted the changes from the relevant PR, recompiled, and executed the code. The results showed the following behavior. It seems there may be another underlying issue when FlashAttn operates.

--flash-attn	KV cache (target)	Result
auto	q8_0	crash at fattn-tile.cuh:1321: fatal error
auto	f16	crash at fattn-tile.cuh:1321: fatal error
off	f16	works — MTP runs, ~45tps

0.00.060.403 W Setting 'enable_thinking' via --chat-template-kwargs is deprecated. Use --reasoning on / --reasoning off instead.
0.00.060.592 I log_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.060.594 I device_info:
0.00.272.627 I   - CUDA0   : NVIDIA GeForce RTX 3050 (7841 MiB, 7495 MiB free)
0.00.564.825 I   - CUDA1   : NVIDIA GeForce GTX 1660 SUPER (5746 MiB, 2 MiB free)
0.00.564.838 I   - CPU     : Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz (48083 MiB, 48083 MiB free)
0.00.564.944 I system_info: n_threads = 4 (n_threads_batch = 4) / 28 | CUDA : ARCHS = 750,800 | FORCE_MMQ = 1 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 |
0.00.564.952 I srv  llama_server: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
0.00.565.053 I srv          init: using 27 threads for HTTP server
0.00.565.237 I srv         start: binding port with default address family
0.00.566.364 I srv  llama_server: loading model
0.00.566.370 I srv    load_model: loading model '/opt/llama/models/gemma-4-E4B-it-qat-UD-Q4_K_XL.gguf'
0.01.565.827 W load: control-looking token:    212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
0.01.566.483 W load: control-looking token:     50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden
0.01.599.615 W load: special_eog_ids contains '<|tool_response>', removing '</s>' token from EOG list
0.03.009.242 W llama_context: n_ctx_seq (65536) < n_ctx_train (131072) -- the full capacity of the model will not be utilized
0.03.074.710 I srv    load_model: loading draft model '/opt/llama/models/mtp-gemma-4-E4B-it.gguf'
0.04.430.325 W load: control-looking token:    212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
0.04.430.985 W load: control-looking token:     50 '<|tool_response>' was not control-type; this is probably a bug in the model. its type will be overridden
0.04.464.946 W load: special_eog_ids contains '<|tool_response>', removing '</s>' token from EOG list
0.04.779.141 W llama_context: n_ctx_seq (65536) < n_ctx_train (131072) -- the full capacity of the model will not be utilized
0.04.781.988 W llama_kv_cache: layer   3: sharing with layer 41. k = 0x7e2daa000000, v = 0x7e2db2000000
0.04.782.028 W llama_kv_cache: layer   0: sharing with layer 40. k = 0x7e2d77f00000, v = 0x7e2d78180000
0.04.782.030 W llama_kv_cache: layer   1: sharing with layer 40. k = 0x7e2d77f00000, v = 0x7e2d78180000
0.04.782.031 W llama_kv_cache: layer   2: sharing with layer 40. k = 0x7e2d77f00000, v = 0x7e2d78180000
0.05.095.193 I srv    load_model: initializing slots, n_slots = 4
0.05.456.093 I common_speculative_impl_draft_mtp: adding speculative implementation 'draft-mtp'
0.05.456.101 I common_speculative_impl_draft_mtp: - n_max=3, n_min=0, p_min=0.00, n_embd=2560, backend_sampling=1
0.05.456.103 I common_speculative_impl_draft_mtp: - gpu_layers=-1, cache_k=f16, cache_v=f16, ctx_tgt=yes, ctx_dft=yes, devices=[default]
/home/aiuser/llama.cpp/ggml/src/ggml-cuda/template-instances/../fattn-tile.cuh:1321: fatal error
/home/aiuser/llama.cpp/build_dev/bin/libggml-base.so.0(+0x1c1ab)[0x7e2e3314e1ab]
/home/aiuser/llama.cpp/build_dev/bin/libggml-base.so.0(ggml_print_backtrace+0x21c)[0x7e2e3314e62c]
/home/aiuser/llama.cpp/build_dev/bin/libggml-base.so.0(ggml_abort+0x15b)[0x7e2e3314e80b]
/home/aiuser/llama.cpp/build_dev/bin/libggml-cuda.so.0(_Z34ggml_cuda_flash_attn_ext_tile_caseILi512ELi512EEvR25ggml_backend_cuda_contextP11ggml_tensor+0x52b)[0x7e2e2e75447b]
/home/aiuser/llama.cpp/build_dev/bin/libggml-cuda.so.0(+0x2a30f1)[0x7e2e2e4a30f1]
/home/aiuser/llama.cpp/build_dev/bin/libggml-base.so.0(ggml_backend_sched_graph_compute_async+0x817)[0x7e2e3316bca7]
/home/aiuser/llama.cpp/build_dev/bin/libllama.so.0(_ZN13llama_context13graph_computeEP11ggml_cgraphb+0xa1)[0x7e2e320df011]
/home/aiuser/llama.cpp/build_dev/bin/libllama.so.0(_ZN13llama_context14process_ubatchERK12llama_ubatch14llm_graph_typeP22llama_memory_context_iR11ggml_status+0x114)[0x7e2e320e17e4]
/home/aiuser/llama.cpp/build_dev/bin/libllama.so.0(_ZN13llama_context6decodeERK11llama_batch+0x3a1)[0x7e2e320e90d1]
/home/aiuser/llama.cpp/build_dev/bin/libllama.so.0(llama_decode+0xf)[0x7e2e320ead3f]
/home/aiuser/llama.cpp/build_dev/bin/libllama-common.so.0(_Z25common_context_can_seq_rmP13llama_context+0xd9)[0x7e2e325fcdf9]
/home/aiuser/llama.cpp/build_dev/bin/libllama-server-impl.so(_ZN19server_context_impl10load_modelER13common_params+0xb2a)[0x7e2e33377c8a]
/home/aiuser/llama.cpp/build_dev/bin/libllama-server-impl.so(_Z12llama_serveriPPc+0x21a6)[0x7e2e332c87b6]
/lib/x86_64-linux-gnu/libc.so.6(+0x2a1ca)[0x7e2e32a2a1ca]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x8b)[0x7e2e32a2a28b]
./llama-server(+0x12a5)[0x5959119862a5]
Aborted (core dumped)

For CUDA this appears to be caused by the change in #21768 which assumes that FlashAttention kernels with a head size of 512 and GQA ratios of 1 or 2 are not used, and therefore disables them to reduce compilation time.

Reverting commit ff5ef82 CUDA: skip compilation of superfluous FA kernels resolved the issue for me.

(I previously commented in #24400, but it is likely a duplicate of this issue.)

[arbv]

Current workaround: switching to Vulkan.