Vocoder decode runs on CPU despite Vulkan backend: backend/buffer mismatch in load_tensor_data_from_file

[staubsauger]

Summary

The AudioTokenizerDecoder (WavTokenizer vocoder) runs entirely on CPU despite AudioTokenizerDecoder backend: Vulkan0 being reported. Code generation is fully GPU-accelerated (~1.5s), but vocoder decode remains at ~17s regardless of backend, quantization, or GPU used.

The root cause has been identified and confirmed: a backend/buffer mismatch in gguf_loader.cpp causes the scheduler to fall back to CPU for all vocoder ops.

Environment

• OS: NixOS 25.11 (Linux 6.19.8-xanmod1)
• GPUs: AMD Radeon RX 9070 XT + AMD Radeon AI PRO R9700 (both RDNA4 / gfx1201)
• Vulkan driver: RADV GFX1201 (Mesa), Vulkan 1.4.341
• ROCm version: 7.2.0 (also tested, same results — RTF 11x vs 3.6x with Vulkan)

Benchmark results

Input: "Hallo, ich bin ein Sprachmodell auf deiner Radeon R9700." (~5s audio)

Backend	Code generation	Vocoder decode	Total	RTF
ROCm (HIP), f16	42446 ms	18070 ms	60516 ms	11.0x
Vulkan, f16	1743 ms	17016 ms	18760 ms	3.6x
Vulkan, q4_k / q8_0	1513 ms	18348 ms	19861 ms	3.5x

Code generation is 24x faster with Vulkan. Vocoder decode is unchanged across all backends, GPUs, and quantization levels.

Root cause (confirmed)

The bug: load_tensor_data_from_file creates a temporary backend

In src/gguf_loader.cpp, load_tensor_data_from_file allocates tensor memory using a temporary backend that it immediately frees:

// gguf_loader.cpp
ggml_backend_t backend = ggml_backend_init_by_type(preferred_backend_type, nullptr);
// ...
buffer = ggml_backend_alloc_ctx_tensors(model_ctx, backend);
// ... load data ...
ggml_backend_free(backend);  // ← temporary backend freed here

Then AudioTokenizerDecoder::load_model() calls init_preferred_backend(), which returns a different backend object (the shared backend from get_shared_backend_state()).

The GGML backend scheduler checks: does the tensor's buffer belong to the current compute backend? Since the buffer was allocated on the freed temporary backend and the compute backend is a different object, the answer is no → CPU fallback for all ops.

Proof via GGML_SCHED_DEBUG=2

$ GGML_SCHED_DEBUG=2 ./build/qwen3-tts-cli -m models -t "Test." -o /dev/null 2>&1 \
    | grep "## SPLIT" | tail -3
SPLIT #0: Vulkan0 # 2 inputs: [inp_code] [inp_pos]   ← code predictor
SPLIT #0: Vulkan0 # 2 inputs: [inp_step_embd] [inp_pos]
SPLIT #0: CPU # 0 inputs                              ← ENTIRE vocoder on CPU

All transformer/code predictor graphs run on Vulkan0. The vocoder graph — the very last split — runs on CPU.

Why the transformer works but the vocoder doesn't

The transformer uses a different loading path in tts_transformer.cpp:

// tts_transformer.cpp — CORRECT
ggml_backend_t backend = init_preferred_backend("TTSTransformer", &error_msg_);
model_.buffer = ggml_backend_alloc_ctx_tensors(model_.ctx, backend);
// buffer and compute backend are the SAME object ✓

The vocoder uses load_tensor_data_from_file which creates its own temporary backend — not the shared backend returned by init_preferred_backend.

Suggested fix

Change load_tensor_data_from_file to use init_preferred_backend / release_preferred_backend instead of ggml_backend_init_by_type / ggml_backend_free, so all components share the same backend object.

Additionally, AudioTokenizerDecoder::load_model() must save the normalized codebook data before ggml_backend_alloc_ctx_tensors (which moves tensors to GPU memory), and re-upload it afterwards — because t->data becomes a GPU pointer after allocation and can no longer be used as a host source.

Expected impact

If the vocoder weights land on the same GPU buffer as the compute backend, the scheduler should dispatch all vocoder ops to Vulkan. Based on the CUDA benchmark in issue #10 (vocoder decode: 899ms on CUDA), the expected improvement is from ~17s to ~1-2s vocoder decode, bringing total RTF below 1.0x (real-time).

Reproduction

# Build with Vulkan
cmake -S ggml -B ggml/build -G Ninja -DGGML_VULKAN=ON \
  -DVulkan_GLSLC_EXECUTABLE="$(which glslc)" -DCMAKE_BUILD_TYPE=Release
cmake --build ggml/build --target ggml-vulkan -j$(nproc)
cmake --build ggml/build -j$(nproc)
cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc)
Confirm: last SPLIT is CPU (vocoder)
GGML_SCHED_DEBUG=2 ./build/qwen3-tts-cli -m models -t "Hello." -o /dev/null 2>&1 

| grep "## SPLIT" | tail -3

[SiaoZeng]

Root Cause Found + Fix

I hit the same issue on AMD Radeon 8060S (RDNA 3.5, gfx1151). Tracked it down to the codebook normalization in audio_tokenizer_decoder.cpp.

Root Cause

normalize_codebooks() (line 40-73) casts codebook->data to ggml_fp16_t* and writes directly. When tensors are allocated on the Vulkan backend, tensor->data points to device-mapped memory — direct host writes either segfault (#17) or silently corrupt the data, causing the scheduler to fall back to CPU for all subsequent ops.

The subsequent ggml_backend_tensor_set() (line 334) then tries to re-upload the already-accessed buffer, which triggers the segfault on some drivers.

Fix

Replace the direct tensor->data access with proper ggml_backend_tensor_get → normalize on host → ggml_backend_tensor_set:

auto normalize_and_upload = [](struct ggml_tensor * codebook, struct ggml_tensor * usage) {
    if (!codebook || !usage) return;
    size_t cb_bytes = ggml_nbytes(codebook);
    size_t usage_bytes = ggml_nbytes(usage);

    // Download from GPU to host
    std::vector<uint8_t> cb_host(cb_bytes);
    std::vector<uint8_t> usage_host(usage_bytes);
    ggml_backend_tensor_get(codebook, cb_host.data(), 0, cb_bytes);
    ggml_backend_tensor_get(usage, usage_host.data(), 0, usage_bytes);

    // Normalize on host
    ggml_fp16_t * cb_data = (ggml_fp16_t *)cb_host.data();
    float * usage_data = (float *)usage_host.data();
    int64_t codebook_dim = codebook->ne[0];
    int64_t codebook_size = codebook->ne[1];
    const float epsilon = 1e-5f;

    for (int64_t emb_idx = 0; emb_idx < codebook_size; ++emb_idx) {
        float u = usage_data[emb_idx];
        if (u < epsilon) u = epsilon;
        float inv_u = 1.0f / u;
        for (int64_t dim_idx = 0; dim_idx < codebook_dim; ++dim_idx) {
            int64_t idx = dim_idx + emb_idx * codebook_dim;
            float val = ggml_fp16_to_fp32(cb_data[idx]);
            cb_data[idx] = ggml_fp32_to_fp16(val * inv_u);
        }
    }

    // Upload normalized data back to GPU
    ggml_backend_tensor_set(codebook, cb_host.data(), 0, cb_bytes);
};

normalize_and_upload(model_.vq_first_codebook, model_.vq_first_usage);
for (int i = 0; i < 15; ++i) {
    normalize_and_upload(model_.vq_rest_codebook[i], model_.vq_rest_usage[i]);
}

Results (Radeon 8060S, gfx1151, Q8_0)

Component	Before (CPU fallback)	After (Vulkan)	Speedup
Decoder	2306 ms	656 ms	3.5x
Total	5836 ms	2822 ms	2.1x
RTF	1.31	0.72	—

This should also fix #17 (segfault with Vulkan). Happy to submit a PR if you'd like.