Question about per-token quant

[Hongbosherlock]

Can you introduce how to perform per-token quantization on o_proj and down_proj exactly?

https://github.com/AniZpZ/AutoSmoothQuant/blob/main/autosmoothquant/layers/nn/linear.py#L310

int8_weight, weight_scale = quantize_per_tensor_absmax(module.weight)
        if act_quant == "per-token":
            alpha = weight_scale

when using per-token, the weight_scale is from quantize_per_tensor_absmax, this is a bit confusing for me.

[AniZpZ]

"per-token" is not applied to weights, it applies to the activations of o_proj and down_proj.

Weights always perform "per-tensor" for now.

[Hongbosherlock]

"per-token" is not applied to weights, it applies to the activations of o_proj and down_proj.

Weights always perform "per-tensor" for now.

how can I perform partial quant like this ?

partial quant 1: only down_proj uses fp16
partial quant 2: both o_proj and down_proj use fp16

vllm-project/vllm#1508 (comment)