Consolidate MLA kv_b_proj sanitize/shard into shared helpers#1324
Open
scyyh11 wants to merge 1 commit into
Open
Consolidate MLA kv_b_proj sanitize/shard into shared helpers#1324scyyh11 wants to merge 1 commit into
scyyh11 wants to merge 1 commit into
Conversation
Extract the duplicated kv_b_proj-splitting (sanitize) and head-sharding logic from deepseek_v3, deepseek_v32, glm4_moe_lite, kimi_linear and longcat_flash into split_kv_b_proj_weights() and shard_mla_projections() in models/mla.py. Behavior-preserving: the splitter dequantizes, splits and re-quantizes the projections exactly as the per-model code did, so quantized checkpoints still load as QuantizedMultiLinear. It additionally tolerates affine weights saved without biases. shard_mla_projections() raises a clear error when num_heads is not divisible across ranks. Signed-off-by: Bvicii <yizhanhuang2002@gmail.com>
scyyh11
force-pushed
the
mla-sanitize-shard-helpers
branch
from
c584703 to
fb1a981
Compare
Author
|
@angeloskath would you mind taking a look when you get a chance? This is a small, behavior-preserving refactor that pulls the duplicated MLA |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Several MLA models (DeepSeek-V3, DeepSeek-V3.2, GLM-4 MoE Lite, Kimi Linear, LongCat Flash) carry near-identical copies of two pieces of logic:
sanitize()— splitting the fusedkv_b_projweight into the absorbedembed_q/unembed_out(MultiLinear) projections, including the quantized-checkpoint path.shard()— slicing those per-head projections across a distributed group.This PR extracts both into shared helpers in
mlx_lm/models/mla.py(split_kv_b_proj_weightsandshard_mla_projections) and updates the five models to call them, removing ~230 lines of duplicated code.Behavior
Behavior-preserving refactor — no functional change for existing checkpoints:
QuantizedMultiLinear(no silent fall back to full precision).biases.shard_mla_projections()raises a clear error when the head count is not divisible across ranks, instead of silently producing wrong shapes.Forward passes, the KV cache, LoRA/DoRA, AWQ and DeepSeek-V2 are intentionally left untouched.
Tests
test_mla_split_requantizes_affine_weight_without_biases(covers the quantized split round-trip and the no-biasescase).test_mla_sharding_requires_divisible_heads.test_deepseek_v3,test_deepseek_v32, …) pass.