MoE Prefill Streaming & DDTree Batched Verify

[howard0su]

Optimizes MoE (Qwen3.5-35B-A3B) prefill and speculative decode performance through streaming expert loading, GPU-side bitmask fast paths, and batched DDTree verify.

1. MoE Prefill Streaming Engine (5ba80db)

• MoeHybridStreamEngine with mmap + madvise for cold expert DMA
• Batched token-per-expert GPU eval replaces per-token sequential dispatch
• 3× prefill speedup (94.7 tok/s → from ~30 tok/s baseline)

2. VRAM Bitmask Fast Path (3a697f5)

• Per-layer expert_vram_mask[4] + all_routed_are_hot() check
• Skips CPU entirely when all routed experts are GPU-resident (95% of layers)
• eval_moe_hot_only_batched() — pure GPU, no cold partition, no merge

3. Cached Hot-Only Batched Graph (fa7f545)

• CachedHotBatchedGraph avoids per-call graph rebuild for repeated sub-batch sizes
• Minimal benefit on sm_75 due to dispatch-count bottleneck (2560 dispatches)

4. Skip MMQ Sub-Batch on sm_80+ (ea47297)

• mmq_safe_full_batch flag in MoeHybridConfig, set via query_gpu_compute_sm()
• On Ampere+ (sm_80+): eliminates ≤4-token sub-batch workaround entirely
• Reduces dispatches from 2560 → 80 per prefill (32× fewer)

5. Batched MoE Verify for DDTree (ccb1ba6)

• hybrid_forward_batch() processes all DDTree tokens layer-by-layer (like prefill)
• Replaces sequential 22-token × 40-layer loop (880 dispatches → 81 dispatches)
• Both verify and replay paths are now batched

Performance (RTX 2080 Ti, sm_75)

Metric	Before	After
Prefill (168 tok)	~30 tok/s	94.7 tok/s
Hot-only layer hit rate	N/A	95%
DDTree verify dispatches	2640	81

Architecture Notes

• sm_75 still uses sub-batch workaround (MMQ bug); sm_80+ gets full benefit
• HIP/AMD keeps workaround active (gfx1151 has same bug)
• DDTree AL=2.16 on this draft model — AR decode is still faster on sm_75 with constrained VRAM. On sm_80+ with more VRAM headroom, DDTree should break even at AL≈3+.

[cubic-dev-ai]

10 issues found across 18 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="server/src/common/moe_hybrid_stream.h">

<violation number="1" location="server/src/common/moe_hybrid_stream.h:56">
P1: Backend-context mismatch: `stream_expert_sync()` accepts `gpu_backend` but ignores it, while `init()` stores a backend handle in `backend_` that is never used for actual GPU operations. Raw CUDA APIs (`cudaMalloc`/`cudaMemcpy`) bypass the ggml backend, so `gpu_scratch_` is tied to the CUDA context at init time, but `eval_moe_cold_experts_streaming()` may compute on a different `gpu_backend`. This can cause illegal memory access or silent corruption if init and compute use different GPU devices.</violation>
</file>

<file name="server/docs/moe_hybrid.md">

<violation number="1" location="server/docs/moe_hybrid.md:302">
P2: PCIe is incorrectly described as half-duplex for bulk transfers; PCIe links are full-duplex (dual simplex) by design, so the rationale against streaming is technically inaccurate.</violation>
</file>

<file name="server/src/common/gguf_mmap.h">

<violation number="1" location="server/src/common/gguf_mmap.h:205">
P2: Unsigned integer overflow in range validation: `offset + length` on `size_t` can wrap around, causing the clamp to be skipped and an invalid oversized range to be passed to `madvise`/`PrefetchVirtualMemory`. Use `if (length > size_ - offset)` instead.</violation>
</file>

<file name="server/src/qwen35moe/qwen35moe_backend.cpp">

<violation number="1" location="server/src/qwen35moe/qwen35moe_backend.cpp:1409">
P2: Missing non-empty batch guard in `hybrid_forward_batch`: `n_tokens == 0` causes `(size_t)(n_tokens - 1)` to underflow to `SIZE_MAX`, producing UB in the `act_cur.assign(...)` pointer arithmetic.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P1: Backend-context mismatch: stream_expert_sync() accepts gpu_backend but ignores it, while init() stores a backend handle in backend_ that is never used for actual GPU operations. Raw CUDA APIs (cudaMalloc/cudaMemcpy) bypass the ggml backend, so gpu_scratch_ is tied to the CUDA context at init time, but eval_moe_cold_experts_streaming() may compute on a different gpu_backend. This can cause illegal memory access or silent corruption if init and compute use different GPU devices.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/src/common/moe_hybrid_stream.h, line 56:

<comment>Backend-context mismatch: `stream_expert_sync()` accepts `gpu_backend` but ignores it, while `init()` stores a backend handle in `backend_` that is never used for actual GPU operations. Raw CUDA APIs (`cudaMalloc`/`cudaMemcpy`) bypass the ggml backend, so `gpu_scratch_` is tied to the CUDA context at init time, but `eval_moe_cold_experts_streaming()` may compute on a different `gpu_backend`. This can cause illegal memory access or silent corruption if init and compute use different GPU devices.</comment>

<file context>
@@ -0,0 +1,112 @@
+    // tensor view over the scratch memory for each weight matrix.
+    // This is a BLOCKING operation (synchronous DMA). For pipelined usage,
+    // use the async variants below.
+    bool stream_expert_sync(const void * mmap_data, size_t mmap_size,
+                            const LayerExpertRegions & regions,
+                            int expert_id,
</file context>

[cubic-dev-ai]

P2: PCIe is incorrectly described as half-duplex for bulk transfers; PCIe links are full-duplex (dual simplex) by design, so the rationale against streaming is technically inaccurate.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/docs/moe_hybrid.md, line 302:

<comment>PCIe is incorrectly described as half-duplex for bulk transfers; PCIe links are full-duplex (dual simplex) by design, so the rationale against streaming is technically inaccurate.</comment>

<file context>
@@ -0,0 +1,396 @@
+**The fundamental problem**: PCIe 4.0 x16 bandwidth (15.75 GB/s) is **4× lower** than DDR5 memory bandwidth (≈50 GB/s). Loading a single Q4_K_M Qwen expert (gate + up + down ≈ 6 MB) over PCIe takes ~380 µs just for the transfer. Meanwhile, the CPU can complete the entire matmul chain in ~100–150 µs using the same data that's already in L3/RAM.
+
+Additionally:
+- **PCIe is half-duplex for bulk transfers**: You cannot overlap upload of expert weights with download of results efficiently.
+- **GPU kernel launch overhead**: Even after DMA completes, the GPU needs to dispatch a kernel for a single-token matmul on just 1–2 cold experts — an inefficient use of GPU SMs.
+- **Bubble injection**: Streaming cold experts to GPU introduces pipeline bubbles. The GPU must wait for DMA → compute → return result, during which the SMs sit idle or context-switch. With CPU-local compute, the GPU is 100% occupied on hot experts.
</file context>

Suggested change

	- **PCIe is half-duplex for bulk transfers**: You cannot overlap upload of expert weights with download of results efficiently.
	- **PCIe is half-duplex for bulk transfers**: You cannot overlap upload of expert weights with download of results efficiently.
	+ **PCIe copy-engine limits overlap**: Most consumer GPUs have only 1–2 DMA copy engines, so upload of expert weights and download of results contend and cannot fully overlap in practice.

[cubic-dev-ai]

P2: Unsigned integer overflow in range validation: offset + length on size_t can wrap around, causing the clamp to be skipped and an invalid oversized range to be passed to madvise/PrefetchVirtualMemory. Use if (length > size_ - offset) instead.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/src/common/gguf_mmap.h, line 205:

<comment>Unsigned integer overflow in range validation: `offset + length` on `size_t` can wrap around, causing the clamp to be skipped and an invalid oversized range to be passed to `madvise`/`PrefetchVirtualMemory`. Use `if (length > size_ - offset)` instead.</comment>

<file context>
@@ -196,6 +200,26 @@ inline const void * GgufMmap::data() const { return data_; }
 
+inline void GgufMmap::advise_willneed(size_t offset, size_t length) const {
+    if (!data_ || offset >= size_) return;
+    if (offset + length > size_) length = size_ - offset;
+    if (length == 0) return;
+#if defined(_WIN32)
</file context>

Suggested change

	if (offset + length > size_) length = size_ - offset;
	if (length > size_ - offset) length = size_ - offset;

[cubic-dev-ai]

P2: Missing non-empty batch guard in hybrid_forward_batch: n_tokens == 0 causes (size_t)(n_tokens - 1) to underflow to SIZE_MAX, producing UB in the act_cur.assign(...) pointer arithmetic.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At server/src/qwen35moe/qwen35moe_backend.cpp, line 1409:

<comment>Missing non-empty batch guard in `hybrid_forward_batch`: `n_tokens == 0` causes `(size_t)(n_tokens - 1)` to underflow to `SIZE_MAX`, producing UB in the `act_cur.assign(...)` pointer arithmetic.</comment>

<file context>
@@ -1153,6 +1222,247 @@ bool Qwen35MoeBackend::hybrid_forward_one_token(int32_t tok, int kv_pos,
+    if (ffn_hot_alloc) ggml_gallocr_free(ffn_hot_alloc);
+
+    // Store last token hidden state in act_cur
+    act_cur.assign(embed_all.data() + (size_t)(n_tokens - 1) * (size_t)hidden,
+                   embed_all.data() + (size_t)n_tokens * (size_t)hidden);
+
</file context>