Higher latency for image embeddings for ColModernVBert compared to Colpali 1.3

[aabbhishekksr]

on A100 40GB

Comparing latency for text and image both using colpali 1.3 and colmodernvbert. Lateny is lower on text, and way high on Image.

On further breakdown, noticed the processor in both the models are processing very differently . Colpali is using 1 tile of 448448 wheras colmodernvbert is using 13 tike of 512512

COLPALI

Model Load Time: 11.603 sec

Image Embedding:

• Total images: 10
• Embedding calls: 1
• Avg per call: 835.43 ms
• Avg per image: 83.54 ms
• Total time: 0.835 sec
• Throughput: 11.97 images/sec

Query Embedding:

• Total queries: 4
• Embedding calls: 4
• Avg per call: 40.91 ms
• Avg per query: 40.91 ms
• Total time: 0.164 sec
• Throughput: 24.45 queries/sec

Retrieval (Scoring):

• Avg per retrieval: 2.46 ms
• Total: 0.010 sec

Model Load Time: 4.651 sec

Image Embedding:

• Total images: 10
• Embedding calls: 1
• Avg per call: 4250.66 ms
• Avg per image: 425.07 ms
• Total time: 4.251 sec
• Throughput: 2.35 images/sec

Query Embedding:

• Total queries: 4

• Embedding calls: 4

• Avg per call: 31.51 ms

• Avg per query: 31.51 ms

• Total time: 0.126 sec

• Throughput: 31.73 queries/sec

Retrieval (Scoring):

• Avg per retrieval: 2.99 ms
• Total: 0.012 sec
• Count: 4

[QuentinJGMace]

Hey @aabbhishekksr , you're correct ColModernVBERT in its current version is quite slow at encoding images. That being said it is not a limitation of the model, rather a limitation of the way images are processed in SmolVLM's framework (which we used to build ColModernVBERT).

Here is the main issue:
SmolVLM processing (and thus ColModernVBERT's) doesn't batch images in the SigLip's forward pass but forward them sequentially, which is suboptimal.

We didn't focus on solving that problem since we wanted to give an actionable recipe for building great small visual retrievers. Furthermore image encoding should only be done at indexing time, and thus the thoughput is less important than for queries.
However this problem is fixable (without doing any retraining) by rewriting the image processor, we might focus on that at some point but it's currently not a priority for the maintainers of the repo (namely me) as I'm starting a PHD on completely unrelated subjects.

If you want to work on this task and write a PR fixing this we would be very happy to merge it !

I hope this answer gives you more insights :)

[aabbhishekksr]

HI @QuentinJGMace

For my evaluations, i made some changes to reduce no. of patches created in ModernVBERT (currently 13) to 1 (as in standard colpali). This made image encoding faster , but it reduces the overall retrieval quality. I understand that is trade-off given that this model is meant for CPUs and does really well if we do not factor indexing time latency.

Sure, I can take this (as when i get some weekend time-off) to raise a PR

[ManuelFay]

I am looking into how we can optimize this as well. ModernVBert produces n subimages (as you say 13) per large image so ofc lowering this directly results in improved speed but tradesoff for performance.

I created a few latency benchmarks.
ManuelFay#1

From the latency benchmarks - runtime on a L4 with 32 images of resolution 2048x2028, two bottlenecks clearly emerge:

1) Preprocessing bottleneck

Preprocessing is a major part of end-to-end latency:

• processor_only_batched: ~5.25s for 16 docs (about 3.05 docs/s)
  (measured: 5.2478s @ batch=1, 5.2544s @ batch=4)

This supports trying:

• async preprocessing with a DataLoader + prefetch,
• fast processors (Rust-backed implementations),
• overlap of CPU preprocessing with GPU compute.

2) Vision forward bottleneck

Vision forward is the other dominant block:

• vision_only_preprocessed: ~6.37–6.39s for 16 docs (about 2.50 docs/s)
• while the whole model: model_only_preprocessed: ~6.95–7.00s

This confirms we are primarily bottlenecked by:

• SigLIP vision throughput,
• number of subimages/crops per document.

Reducing subimages is therefore a high-leverage optimization.
For example, running at max resolution around 1024×1024 (instead of larger split-heavy settings) should significantly reduce crop count (up to ~4×), with limited quality drop based on our paper results.

What is not bottlenecking

These components are comparatively very fast:

• connector_only_preprocessed: ~0.039s
• inputs_merger_only_preprocessed: ~0.004–0.015s
• text_only_preprocessed: ~0.690s
• text_model_only_preprocessed: ~0.674–0.685s

So connector, inputs_merger, and text-forward are not the primary bottlenecks right now.

By using a dataloader with multiple workers, we completely remove the processing overhead. Further shortening the image size gets you 4 to 8x speed improvements.

[athrael-soju]

Nice profiling work @ManuelFay, really helpful to see the breakdown.

Would it make sense to track the dataloader optimization and the resolution reduction separately? The dataloader overlap should be essentially free, while lowering resolution may require a trade off between retrieval quality and speed.

@QuentinJGMace Quick question on the sequential SigLIP point. Looking at the ModernVBertModel forward, it looks like pixel_values get reshaped to (batch * num_images, C, H, W) and passed through vision_model in one call. Is the sequential processing you're referring to happening in the Idefics3Processor on the CPU side (the image splitting/resizing), or is there a different code path I'm missing?

[ManuelFay]

Yeah about the batching, we looked into it quickly and posted the response together but probably quite imprecise - things seem batched in the vision encoder as well actually (but on me, I gave Quentin wrong info).

For profiling, I didn't post but I did profile separately and we can in fact with pinned memory, prefetch and multiple workers effectivel erase the preprocessing time. However the model forward pass in the vision model is much tougher (except for resolution).

As Quentin said, the goal here was essentially to be able to study the impact of encoders and have a quick online text embedder to be able to serve on CPU, not to optimize image encoding. Having said that, the subimage cropping strategy is surely suboptimal wrt to something like 2d rope + dynamic res like in QwenVL.

My guess for future work would be that best is to take a pretrained small embedding model and then align a good image encoder like Qwen in the same latent space.