Eval bug: llama.cpp-b9568/ggml/src/ggml-cuda/fattn.cu:579: fatal error

[macafeeee]

Name and Version

./llama-server --version
version: b9568 (unknown)
built with GNU 14.2.0 for Linux x86_64

compile with -DGGML_CUDA=ON -DGGML_CUDA_USE_FLASH_ATTENTION=ON -DGGML_CUDA_FA_ALL_QUANTS=ON -DGGML_CUDA_NCCL=ON

Operating systems

Linux

GGML backends

CUDA

Hardware

Intel(R) Xeon(R) Gold 6338 X2
PCIE Nvidia V100 32G X4
Debian 13
NVIDIA-SMI 550.163.01
Driver Version: 550.163.01
CUDA Version: 12.4

Models

gemma-4-31B-it-F16 with MTP-gemma-4-31B-it-assistant-F16

Problem description & steps to reproduce

./llama-server -m /models/gemma-4-31B-it-F16.gguf --mmproj /models/mmproj-F16.gguf --spec-draft-model /models/MTP-gemma4-31B-it-assistant-F16.gguf --fit off --split-mode tensor -t 8 -fa on --no-mmap --mlock --ctx-size 131072 -ngl all -ub 2048 -b 2048 --prio 3 --host 0.0.0.0 --port 8080
After several inferences, llama.cpp crashes randomly. The detailed error message is as follows.

First Bad Commit

No response

Relevant log output

Logs

llamacpp log:
[46009] /home/llamacpp/llama.cpp-b9568/ggml/src/ggml-cuda/fattn.cu:579: fatal error
[46009] /home/llamacpp/llama.cpp-b9568/build/bin/libggml-base.so.0(+0x18665) [0x7fbfdb553665]
[46009] /home/llamacpp/llama.cpp-b9568/build/bin/libggml-base.so.0(ggml_print_backtrace+0x1df) [0x7fbfdb553a3f]
[46009] /home/llamacpp/llama.cpp-b9568/build/bin/libggml-base.so.0(ggml_abort+0x11e) [0x7fbfdb553bce]
[46009] /home/llamacpp/llama.cpp-b9568/build/bin/libggml-cuda.so.0(_Z24ggml_cuda_flash_attn_extR25ggml_backend_cuda_contextP11ggml_tensor+0xa77) [0x7fbfd0ffcb27]
[46009] /home/llamacpp/llama.cpp-b9568/build/bin/libggml-cuda.so.0(+0x24b4db) [0x7fbfd104b4db]
[46009] /home/llamacpp/llama.cpp-b9568/build/bin/libggml-base.so.0(+0x46974) [0x7fbfdb581974]
[46009] /home/llamacpp/llama.cpp-b9568/build/bin/libggml-base.so.0(ggml_backend_sched_graph_compute_async+0x827) [0x7fbfdb570047]
[46009] /home/llamacpp/llama.cpp-b9568/build/bin/libllama.so.0(_ZN13llama_context13graph_computeEP11ggml_cgraphb+0xa1) [0x7fbfda6cd2f1]
[46009] /home/llamacpp/llama.cpp-b9568/build/bin/libllama.so.0(_ZN13llama_context14process_ubatchERK12llama_ubatch14llm_graph_typeP22llama_memory_context_iR11ggml_status+0xe4) [0x7fbfda6cff04]
[46009] /home/llamacpp/llama.cpp-b9568/build/bin/libllama.so.0(_ZN13llama_context6decodeERK11llama_batch+0x365) [0x7fbfda6d5e55]
[46009] /home/llamacpp/llama.cpp-b9568/build/bin/libllama.so.0(llama_decode+0xb) [0x7fbfda6d7a3b]
[46009] /home/llamacpp/llama.cpp-b9568/build/bin/libllama-common.so.0(_ZN33common_speculative_impl_draft_mtp5draftERSt6vectorI31common_speculative_draft_paramsSaIS1_EE+0xce) [0x7fbfdac9ffde]
[46009] /home/llamacpp/llama.cpp-b9568/build/bin/libllama-common.so.0(_Z24common_speculative_draftP18common_speculative+0xc9) [0x7fbfdac95ac9]
[46009] /home/llamacpp/llama.cpp-b9568/build/bin/libllama-server-impl.so(_ZN19server_context_impl12update_slotsEv+0x55c) [0x7fbfdb75dddc]
[46009] /home/llamacpp/llama.cpp-b9568/build/bin/libllama-server-impl.so(_ZN12server_queue10start_loopEl+0x1efa) [0x7fbfdb7f37ba]
[46009] /home/llamacpp/llama.cpp-b9568/build/bin/libllama-server-impl.so(_Z12llama_serveriPPc+0x343a) [0x7fbfdb6bd24a]
[46009] /lib/x86_64-linux-gnu/libc.so.6(+0x29ca8) [0x7fbfdb035ca8]
[46009] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85) [0x7fbfdb035d65]
[46009] /home/llamacpp/llamacpp/llama-server(+0x11b1) [0x555b2f5151b1]
2.41.647.215 E srv operator(): http client error: Failed to read connection

dmesg log:
NVRM: Xid (PCI:0000:31:00): 13, pid='', name=, Graphics SM Warp Exception on (GPC 5, TPC 0, SM 0): Out Of Range Address
NVRM: Xid (PCI:0000:31:00): 13, pid='', name=, Graphics SM Global Exception on (GPC 5, TPC 0, SM 0): Multiple Warp Errors
NVRM: Xid (PCI:0000:31:00): 13, pid='', name=, Graphics Exception: ESR 0x52c730=0x107000e 0x52c734=0x24 0x52c728=0x4c1eb72 0x52c72c=0x174
NVRM: Xid (PCI:0000:31:00): 13, pid='', name=, Graphics SM Warp Exception on (GPC 5, TPC 0, SM 1): Out Of Range Address
NVRM: Xid (PCI:0000:31:00): 13, pid='', name=, Graphics SM Global Exception on (GPC 5, TPC 0, SM 1): Multiple Warp Errors
NVRM: Xid (PCI:0000:31:00): 13, pid='', name=, Graphics Exception: ESR 0x52c7b0=0x104000e 0x52c7b4=0x24 0x52c7a8=0x4c1eb72 0x52c7ac=0x174
NVRM: Xid (PCI:0000:31:00): 13, pid='', name=, Graphics SM Warp Exception on (GPC 5, TPC 1, SM 0): Out Of Range Address
NVRM: Xid (PCI:0000:31:00): 13, pid='', name=, Graphics SM Global Exception on (GPC 5, TPC 1, SM 0): Multiple Warp Errors
NVRM: Xid (PCI:0000:31:00): 13, pid='', name=, Graphics Exception: ESR 0x52cf30=0x107000e 0x52cf34=0x24 0x52cf28=0x4c1eb72 0x52cf2c=0x174
NVRM: Xid (PCI:0000:31:00): 13, pid='', name=, Graphics SM Warp Exception on (GPC 5, TPC 2, SM 0): Out Of Range Address
NVRM: Xid (PCI:0000:31:00): 13, pid='', name=, Graphics SM Global Exception on (GPC 5, TPC 2, SM 0): Multiple Warp Errors
NVRM: Xid (PCI:0000:31:00): 13, pid='', name=, Graphics Exception: ESR 0x52d730=0x106000e 0x52d734=0x24 0x52d728=0x4c1eb72 0x52d72c=0x174
NVRM: Xid (PCI:0000:31:00): 13, pid='', name=, Graphics SM Warp Exception on (GPC 5, TPC 2, SM 1): Out Of Range Address
NVRM: Xid (PCI:0000:31:00): 13, pid='', name=, Graphics SM Global Exception on (GPC 5, TPC 2, SM 1): Multiple Warp Errors
NVRM: Xid (PCI:0000:31:00): 13, pid='', name=, Graphics Exception: ESR 0x52d7b0=0x105000e 0x52d7b4=0x24 0x52d7a8=0x4c1eb72 0x52d7ac=0x174
NVRM: Xid (PCI:0000:31:00): 13, pid='', name=, Graphics SM Warp Exception on (GPC 5, TPC 3, SM 0): Out Of Range Address
NVRM: Xid (PCI:0000:31:00): 13, pid='', name=, Graphics SM Global Exception on (GPC 5, TPC 3, SM 0): Multiple Warp Errors
NVRM: Xid (PCI:0000:31:00): 13, pid='', name=, Graphics Exception: ESR 0x52df30=0x105000e 0x52df34=0x24 0x52df28=0x4c1eb72 0x52df2c=0x174
NVRM: Xid (PCI:0000:31:00): 13, pid='', name=, Graphics SM Warp Exception on (GPC 5, TPC 4, SM 0): Out Of Range Address
NVRM: Xid (PCI:0000:31:00): 13, pid='', name=, Graphics SM Global Exception on (GPC 5, TPC 4, SM 0): Multiple Warp Errors
NVRM: Xid (PCI:0000:31:00): 13, pid='', name=, Graphics Exception: ESR 0x52e730=0x104000e 0x52e734=0x24 0x52e728=0x4c1eb72 0x52e72c=0x174
NVRM: Xid (PCI:0000:31:00): 13, pid='', name=, Graphics SM Warp Exception on (GPC 5, TPC 5, SM 0): Out Of Range Address
NVRM: Xid (PCI:0000:31:00): 13, pid='', name=, Graphics SM Global Exception on (GPC 5, TPC 5, SM 0): Multiple Warp Errors
NVRM: Xid (PCI:0000:31:00): 13, pid='', name=, Graphics Exception: ESR 0x52ef30=0x106000e 0x52ef34=0x24 0x52ef28=0x4c1eb72 0x52ef2c=0x174
NVRM: Xid (PCI:0000:31:00): 13, pid=13253, name=llama-server, Graphics Exception: ChID 0008, Class 0000c3c0, Offset 00000510, Data 00419e84

[so-dimm]

I'm seeing the same crash 'ggml-cuda\fattn.cu:579: fatal error' with gemma-4-31B-it on llama-server(version b9553).

The crash occurs with the combination of -sm tensor and --spec-type draft-mtp.
Both prompt processing and context checkpoint creation (when enabled) complete successfully, but the system crashes upon entering the decode phase.

This looks like a regression introduced after May 22, as the mtp was stable up to commit a03120c of PR #23398

Logs showing the crash right after create_check or prompt processing

Log 1: -ctxcp 0

I slot launch_slot_: id  1 | task 1301 | processing task, is_child = 0
W slot update_slots: id  1 | task 1301 | forcing full prompt re-processing due to lack of cache data (likely due to SWA or hybrid/recurrent memory)
I slot print_timing: id  1 | task 1301 | prompt processing, n_tokens =   4096, progress = 0.33, t =   3.49 s / 1173.80 tokens per second
[...]
I slot print_timing: id  1 | task 1301 | prompt processing, n_tokens =  12498, progress = 1.00, t =  10.48 s / 1192.91 tokens per second
D:\a\llama.cpp\llama.cpp\ggml\src\ggml-cuda\fattn.cu:579: fatal error

Log 2: -ctxcp 1

I slot launch_slot_: id  1 | task 328 | processing task, is_child = 0
W slot create_check: id  1 | task 328 | erasing old context checkpoint (pos_min = 63756, pos_max = 66315, n_tokens = 66316, size = 425.013 MiB)
I slot create_check: id  1 | task 328 | created context checkpoint 1 of 1 (pos_min = 64780, pos_max = 67121, n_tokens = 67122, size = 425.013 MiB)
D:\a\llama.cpp\llama.cpp\ggml\src\ggml-cuda\fattn.cu:579: fatal error

[mike472826]

the same for me (when i use -sm layer it works but very slow):

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 575.57.08 Driver Version: 575.57.08 CUDA Version: 12.9 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 3090 On | 00000000:01:00.0 Off | N/A |
| 0% 37C P8 17W / 200W | 21MiB / 24576MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA GeForce RTX 3090 On | 00000000:02:00.0 Off | N/A |
| 0% 39C P8 24W / 200W | 96MiB / 24576MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+

ubuntu 22.04

Available llama.cpp builds:

build_20260607_213328_379ac66
build_20260608_183128_1705d43
build_20260609_150753_d6d0ce8 <-- active

CUDA_VISIBLE_DEVICES=0,1 /home/user/llama.cpp/build/bin/llama-server
-m gemma-4-31B-it-Q8_0_unsloth/unsloth_gemma-4-31B-it-Q8_0.gguf
--no-mmproj
--model-draft gemma-4-31B-it-assistant-GGUF-unsloth/gemma-4-31B-it-MTP-Q8_0.gguf
--spec-type draft-mtp --spec-draft-n-max 4
--port 8080
--host 0.0.0.0
-a 'gemma-4-31b-it','llm'
-np 1
-c 131072
-fit off
--cache-ram -1
--jinja
--chat-template-file gemma-4-jinja/chat_template.jinja
--temp 1.0
--top-p 0.95
--top-k 64
-kvu
-b 2048
-ub 512
-rea on
-fa on
-ngl 99
-sm tensor \

2.22.194.271 I slot print_timing: id 0 | task 1001 | n_decoded = 2997, tg = 71.98 t/s
2.25.206.148 I slot print_timing: id 0 | task 1001 | n_decoded = 3161, tg = 70.79 t/s
2.28.249.287 I slot print_timing: id 0 | task 1001 | n_decoded = 3304, tg = 69.27 t/s
2.29.317.001 I slot print_timing: id 0 | task 1001 | prompt eval time = 3142.73 ms / 2674 tokens ( 1.18 ms per token, 850.85 tokens per second)
2.29.317.004 I slot print_timing: id 0 | task 1001 | eval time = 48762.09 ms / 3349 tokens ( 14.56 ms per token, 68.68 tokens per second)
2.29.317.004 I slot print_timing: id 0 | task 1001 | total time = 51904.82 ms / 6023 tokens
2.29.317.005 I slot print_timing: id 0 | task 1001 | graphs reused = 1745
2.29.317.006 I slot print_timing: id 0 | task 1001 | draft acceptance = 0.81760 ( 2564 accepted / 3136 generated)
2.29.317.014 I statistics draft-mtp: #calls(b,g,a) = 3 1776 1776, #gen drafts = 1776, #acc drafts = 1647, #gen tokens = 7104, #acc tokens = 5864, dur(b,g,a) = 0.010, 17729.036, 2.841 ms
2.29.317.203 I slot release: id 0 | task 1001 | stop processing: n_tokens = 7718, truncated = 0
2.29.317.229 I srv update_slots: all slots are idle
2.38.326.096 I srv params_from_: Chat format: peg-gemma4
2.38.336.486 I slot get_availabl: id 0 | task -1 | selected slot by LCP similarity, sim_best = 0.563 (> 0.100 thold), f_keep = 0.566
2.38.336.517 I reasoning-budget: activated, budget=2147483647 tokens
2.38.336.518 I reasoning-budget: deactivated (natural end)
2.38.336.546 I slot launch_slot_: id 0 | task 1791 | processing task, is_child = 0
2.38.336.555 I slot update_slots: id 0 | task 1791 | Checking checkpoint with [2820, 4355] against 3342...
2.38.489.173 W slot update_slots: id 0 | task 1791 | restored context checkpoint (pos_min = 2820, pos_max = 4355, n_tokens = 4356, n_past = 4355, size = 800.013 MiB)
2.41.797.517 I slot print_timing: id 0 | task 1791 | prompt processing, n_tokens = 3384, progress = 1.00, t = 3.46 s / 977.76 tokens per second
2.42.152.952 I slot create_check: id 0 | task 1791 | created context checkpoint 4 of 32 (pos_min = 6203, pos_max = 7738, n_tokens = 7739, size = 800.013 MiB)
2.42.208.271 I slot print_timing: id 0 | task 1791 | prompt processing, n_tokens = 3394, progress = 1.00, t = 3.87 s / 876.61 tokens per second
/home/user/llama.cpp/ggml/src/ggml-cuda/fattn.cu:579: fatal error
[New LWP 92867]

[annagorshunova]

This may be related to graph reuse when using both --split-mode tensor and --spec-type draft-mtp. In our case, export LLAMA_GRAPH_REUSE_DISABLE=1 seems to help, although it reduces generation speed.

Issue #24314 on ROCm/HIP may also be relevant here, since the symptoms look similar. Its patch appears to disable graph reuse.

[mike472826]

no crash for more than hour of continous work with LLAMA_GRAPH_REUSE_DISABLE=1 , there is slowness in tg (60tps) but until the bug is fixed it allows me to do agentic work in opencode up to 131072ctx, thanks!

This may be related to graph reuse when using both --split-mode tensor and --spec-type draft-mtp. In our case, export LLAMA_GRAPH_REUSE_DISABLE=1 seems to help, although it reduces generation speed.

Issue #24314 on ROCm/HIP may also be relevant here, since the symptoms look similar. Its patch appears to disable graph reuse.

[philpax]

I also encountered this yesterday (2x3090, MTP, tensor-split) and decided I'd put Fable 5 on resolving it overnight, while trying to avoid the LLAMA_GRAPH_REUSE_DISABLE tax.

This patch (which is not submit-ready) fixes the bug for me. I am intentionally not opening a PR, but this may help unblock y'all and/or lead to a human-written PR being submitted.

Here's its analysis (for what it's worth, based on my rather-dated understanding of the llama.cpp codebase, this checks out):

Root cause: The fattn.cu:579 aborts with Gemma 4 31B + MTP assistant + -sm tensor are caused by the meta backend executing cached per-device shadow subgraphs whose external-view tensors have been destroyed and repopulated by another backend instance. In ggml-backend-meta.cpp, shard registrations for external views live in a rotating pair of per-buffer containers (stc_compute[2]), while shadow subgraphs are cached per-backend-instance keyed on cgraph uid.

Since #23398, an MTP assistant context shares its target's KV cache buffers, so two backend instances rotate the same slot pair: when one instance rebuilds twice while the other reuses a uid-matched graph (e.g. target does prompt-then-generate while the drafter coasts), the second rebuild flips back onto the slot the coasting instance's cached subgraphs still reference, resets it, and refills it with the rebuilder's views.

The coasting instance then computes FA nodes whose K/V srcs resolve to the other model's cache views — BEST_FATTN_KERNEL_NONE abort when the shapes are selector-illegal (Gemma 4's head-512 layers require gqa_opt_applies, which is why this model crashes where others would silently compute wrong attention). Diagnostic dumps of the dying node consistently showed e.g. src[1] = "cache_v_l0 (view)" ne=[2048,4608] (a raw stream view of the target's cache) inside a drafter FA node.

Why detection can't fix it: We first tried validating before reuse (generation counters on the slots, identity snapshots on registrations, consistency checks on FA nodes pre-compute) — all insufficient, because graph arenas rebuild in place deterministically: a stale registration at a recycled address is byte-identical to a fresh one, including src pointer values, so staleness is purely ownership/temporal information and invisible to any content inspection. This is essentially the existing FIXME in the file ("tie the lifetime of external views to the meta backend executing the graph") manifesting as a crash.

The fix: We implemented that TODO:

1. the compute containers move from the buffer context to the backend instance, flipped/reset only by that instance's own rebuilds, making cross-instance invalidation structurally impossible
2. since buffer.init_tensor has no owner identity, it no longer registers compute tensors — registrations are created lazily during graph_compute where the executing instance is known
3. graph-external accesses (set/get/cpy_tensor) use persistent thread-local scratch pools with per-entry identity validation (sound there because each pool is single-owner — and persistence matters: per-call shard re-creation cost ~0.4 ms × ~12 calls/token, ~20% throughput)
4. split_state_cache evicts the mismatched entry instead of clearing wholesale, which under lazy registration destroyed memoization and caused a stack-overflow via unbounded recursion.

With the patch, the user's previously-reliable crash reproducer runs clean for hours at throughput parity with master (~87 t/s vs ~67 t/s for the naive always-rebuild mitigation, 2×3090, Gemma 4 31B QAT Q4_K_XL + MTP n_max=2). Workarounds for unpatched builds, in cost order: LLAMA_GRAPH_REUSE_DISABLE=1, or -sm layer.