mtmd : add post-decode callback

[ggerganov]

Overview

alt #24520

This resolves the [TAG_MTMD_DRAFT_PROCESSING] TODO for synchronizing the target and draft contexts and avoid including llama-ext.h in server-context.cpp.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES. pi:llama.cpp/Qwen3.6-27B

[ngxson]

Just to confirm, apart from avoid including llama-ext.h, this also make sure decoding the spec always calls common_speculative_process instead of calling llama_decode directly, right?

If so, I agree that the callback added here is acceptable

[ggerganov]

Just to confirm, apart from avoid including llama-ext.h, this also make sure decoding the spec always calls common_speculative_process instead of calling llama_decode directly, right?

Yes, I think that as long as we process the exact same batches both with the target and draft/spec contexts, they should remain synchronized.

There is still some incorrectness when doing multi-modal processing with MTP, but to fix that we have to rework the llama_batch.

[ngxson]

There is still some incorrectness when doing multi-modal processing with MTP, but to fix that we have to rework the llama_batch.

ok I will start working on this today (unless you want to take over it)

[ggerganov]

There is still some incorrectness when doing multi-modal processing with MTP, but to fix that we have to rework the llama_batch.

ok I will start working on this today (unless you want to take over it)

I plan to do a few refactors that have piled-up lately (Metal, memory, tests, ggml backend), but wasn't planning to start on llama_batch yet. So if you want to do it, go ahead.

My general idea for llama_batch is to make a new llama_batch_ext that is oblique (i.e. accessed only via dedicated API). And introduce a respective llama_process(llama_batch_ext * batch) instead of separate encode/decode calls. The new llama_batch_ext would have more information - for example, it should be able to provide both vision and target-model embeddings, so that the Qwen MTP can work correctly.