Wire Qwen3.5 FP8 linear-attention quant metadata#464
Conversation
|
Warning Review limit reached
Next review available in: 27 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
WalkthroughAdds optional FP8 quantization metadata fields for Qwen3.5 linear-attention projection weights, binds that metadata during model loading with fusion and tensor-parallel slicing checks, uses quant-aware weight views in forward GEMM calls, and adds regression tests covering the wiring. ChangesQwen3.5 linear-attention quant metadata
Related Issues: None referenced. Related PRs: None referenced. Suggested labels: cuda, quantization, model-support Suggested reviewers: None specified. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
driver/cuda/src/model/qwen3_5.cpp (1)
313-353: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winMove the FP8+TP rejection check before the expensive TP slice.
Lw.la_in_proj_qkv_quantand theT > 1rejection (Lines 345-353) are computed/checked afterslice_la_kkv_blocked(*full_qkv, ...)has already allocated a device tensor and issued 3cudaMemcpys (Lines 323-326). Every TP+FP8 misconfiguration pays for that slice/copy before throwing.♻️ Proposed reordering
const int K_dim = cfg.linear_num_key_heads * cfg.linear_key_head_dim; const int V_dim = cfg.linear_num_value_heads * cfg.linear_value_head_dim; + const auto qkv_quant = engine.quant_meta(la + "in_proj_qkv.weight"); + if (T > 1 && qkv_quant.has_value()) { + throw std::runtime_error( + "qwen3_5: TP-sliced FP8 linear_attn.in_proj_qkv " + "requires matching scale slicing"); + } if (T > 1) { w.owned_bf16_buffers.push_back( slice_la_kkv_blocked(*full_qkv, K_dim, V_dim, rank, T)); Lw.la_in_proj_qkv = &w.owned_bf16_buffers.back(); ... } else { Lw.la_in_proj_qkv = full_qkv; ... } Lw.la_in_proj_z = &must(engine, la + "in_proj_z.weight"); Lw.la_in_proj_b = &must(engine, la + "in_proj_b.weight"); Lw.la_in_proj_a = &must(engine, la + "in_proj_a.weight"); - Lw.la_in_proj_qkv_quant = engine.quant_meta(la + "in_proj_qkv.weight"); + Lw.la_in_proj_qkv_quant = qkv_quant; Lw.la_in_proj_z_quant = engine.quant_meta(la + "in_proj_z.weight"); Lw.la_in_proj_b_quant = engine.quant_meta(la + "in_proj_b.weight"); Lw.la_in_proj_a_quant = engine.quant_meta(la + "in_proj_a.weight"); - if (T > 1 && Lw.la_in_proj_qkv_quant.has_value()) { - throw std::runtime_error( - "qwen3_5: TP-sliced FP8 linear_attn.in_proj_qkv " - "requires matching scale slicing"); - }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@driver/cuda/src/model/qwen3_5.cpp` around lines 313 - 353, The FP8+TP rejection in qwen3_5 model loading is checked too late, after the expensive tensor-parallel slicing and device copies have already happened. In the qwen3_5.cpp load path, move the quantization check for Lw.la_in_proj_qkv_quant and the T > 1 rejection ahead of the slice_la_kkv_blocked calls, so the code fails fast before allocating buffers or issuing cudaMemcpy work. Keep the logic near the existing must/maybe tensor lookups and use the same Lw.la_in_proj_qkv_quant symbol to gate the TP-specific slicing branch.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@driver/cuda/src/model/qwen3_5.cpp`:
- Around line 313-353: The FP8+TP rejection in qwen3_5 model loading is checked
too late, after the expensive tensor-parallel slicing and device copies have
already happened. In the qwen3_5.cpp load path, move the quantization check for
Lw.la_in_proj_qkv_quant and the T > 1 rejection ahead of the
slice_la_kkv_blocked calls, so the code fails fast before allocating buffers or
issuing cudaMemcpy work. Keep the logic near the existing must/maybe tensor
lookups and use the same Lw.la_in_proj_qkv_quant symbol to gate the TP-specific
slicing branch.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: ecca32b5-20e3-41b7-9ab0-3e4f59321dcd
📒 Files selected for processing (4)
driver/cuda/src/model/qwen3_5.cppdriver/cuda/src/model/qwen3_5.hppdriver/cuda/src/model/qwen3_5_forward.cppdriver/cuda/tests/test_qwen35_linear_attn_quantmeta.py
2 participants
Sibling PRs for this ticket:
Summary
Validation
Summary by CodeRabbit
New Features
Bug Fixes
Tests