[FEAT] Request for Tensor Parallelism Support – Weight Loading Error in _rotation_weight_loader

[nan2088]

Hi,
Great work on the project!

I’m requesting support for tensor parallelism.

[multiproc_executor.py:800] File "/opt/venv/lib/python3.12/site-packages/vllm/model_executor/models/qwen3_5.py", line 513, in load_weights (Worker pid=194) (Worker_TP1 pid=194) ERROR 03-20 07:48:17 [multiproc_executor.py:800] weight_loader(param, loaded_weight) 
(Worker pid=194) (Worker_TP1 pid=194) ERROR 03-20 07:48:17 [multiproc_executor.py:800] File "/app/paroquant/inference/backends/vllm/plugin.py", line 47, in _rotation_weight_loader 
(Worker pid=194) (Worker_TP1 pid=194) ERROR 03-20 07:48:17 [multiproc_executor.py:800] target.copy_(loaded_weight) 

(Worker pid=194) (Worker_TP1 pid=194) ERROR 03-20 07:48:17 [multiproc_executor.py:800] RuntimeError: The size of tensor a (2048) must match the size of tensor b (4096) at non-singleton dimension 1
 (Worker pid=193) (Worker_TP0 pid=193) INFO 03-20 07:48:17 [multiproc_executor.py:749] Parent process exited, terminating worker

[Whamp]

Additional reproduction details

I can reproduce this consistently, and upgrading vLLM does not resolve it.

Reproduction

Published image:

docker run --rm -it \
  --gpus all \
  --ipc=host \
  -p 8000:8000 \
  -e CUDA_DEVICE_ORDER=PCI_BUS_ID \
  -e CUDA_VISIBLE_DEVICES=0,1 \
  -v /home/will/.cache/huggingface:/root/.cache/huggingface \
  -v /home/will/.cache/paroquant:/root/.cache/paroquant \
  ghcr.io/z-lab/paroquant:serve \
  --model z-lab/Qwen3.5-27B-PARO \
  --tensor-parallel-size 2 \
  --max-model-len 32768

I also rebuilt the ParoQuant image with vllm==0.18.0+cu130, and the same TP failure still occurs.

Actual behavior

Startup fails during model loading with:

File "/app/paroquant/inference/backends/vllm/plugin.py", line 47, in _rotation_weight_loader
    target.copy_(loaded_weight)
RuntimeError: The size of tensor a (3072) must match the size of tensor b (6144) at non-singleton dimension 1

Relevant stack includes:

... vllm/model_executor/models/qwen3_5.py
... weight_loader(param, loaded_weight)
... /app/paroquant/inference/backends/vllm/plugin.py", line 47, in _rotation_weight_loader

Expected behavior

The model should load and serve correctly with --tensor-parallel-size 2 across two GPUs.

What seems to be happening

This looks like the TP-sharded parameter is being populated with the full rotation tensor instead of the per-rank shard.

In other words:

• vLLM creates partitioned params for TP=2
• _rotation_weight_loader copies the full tensor into each target shard
• shape mismatch follows (3072 vs 6144 on my setup)

So the blocker appears to be in ParoQuant’s TP handling for rotation weights, not in Docker or GPU selection.

Additional notes

Single-GPU serving does work for me with:

... --max-model-len 32768 --enforce-eager

So this TP issue seems distinct from the separate single-GPU Qwen3.5/vLLM CUDA-graph issue.

Environment

• GPUs: 2x RTX 3090
• Hidden from container: RTX 3080 Ti
• Docker image tested:
  • ghcr.io/z-lab/paroquant:serve (vLLM 0.17.0)
  • locally rebuilt ParoQuant image with vllm==0.18.0+cu130
• Model: z-lab/Qwen3.5-27B-PARO

If helpful, I can also provide the full logs from both the stock image and the rebuilt vLLM 0.18.0 image.