MLLMs support？

[hywhyw111]

can kvpress support MLLMs, such as LLaVA, Qwen3-VL?

[maxjeblick]

Hi @hywhyw111!

This is a very interesting question!
From a code perspective, kvpress can support MLLMs - Gemma3ForConditionalGeneration is already supported.

For other VLMs like LLaVA or Qwen3-VL, one would need to add handling for their layer paths

LLaVA: model.language_model.model.layers
Qwen2-VL / Qwen3-VL: model.model.language_model.layers

in https://github.com/NVIDIA/kvpress/blob/main/kvpress/presses/base_press.py#L191.

However, we haven't tested KV cache compression on multimodal inputs yet. One consideration is that image tokens will be compressed along with text tokens, which may affect visual understanding differently than text.

Are you working on a specific use case? We may consider investigating MLLM compression in more detail in the future. One key aspect would be to create a benchmark dataset to track MLLMs performance under compression with multimodal inputs.

[debashishc]

hi @maxjeblick, our lab and I are actively working on MLLM kv compression techniques at the moment. My initial results align with your statement regarding visual tokens being affected differently compared to text tokens - not at a stage to share updates yet but happy to collaborate when it appears in your roadmap.

[maxjeblick]

Thanks for the updates, sounds quite interesting!

Potential next steps would consist in:

• Changing the code to support several vision language models (see above). This should be quick.
• Add a long context MLLM benchmark, e.g. https://huggingface.co/datasets/FreedomIntelligence/MileBench (I haven't had a detailed look into the literature, would thus need more research.).

If you want to contribute a MLLM kv compression technique, we are happy for PRs.