Can it be used together with the MTP draft model?

[serchcc]

That would be awesome if they can be used together.

[mgeldi]

This is a really important point!
I tested the qwen3.6-27b-Q6_K_XL on my 5090.
It actually ran with turbo4 using a context length of 150k. But once you reach higher context lengths, it becomes pretty slow.

MTP is the last thing needed to elevate local development with "Consumer Hardware" to the next level!

[xyehya]

@mgeldi what were your prompt processing and token gen speeds?
Im on a 5080 and 285K with fast DDR5 RAM. 27B is out of my league. My best best was Qwen3.6-35B-A3B:Q8_K_XL .. with llama.cpp moe offloading to CPU RAM and 150K context also, its actually blazing fast in prompt processing >3k t/s and token generation is around 70 then goes down towards 50s. I am having really good results in coding, troubleshooting, system tweaking, reverse-engineering. But sinces its not dense i though i would better get the best quant.

[claude-eric-steiner]

I just ran some speed tests with MTP.
https://github.com/TheTom/llama-cpp-turboquant.git, branch feature/turboquant-kv-cache, at commit 2b61ea2: TheTom/llama-cpp-turboquant@2b61ea2

The output speed gain is real (and most excitedly far greater for big context sizes), model quality tests I've also ran point to no quality degradation.
But for some reasons beyond my understanding, the input token digestion is severely decreased, which brings the wall time down.
Maybe this is because it's a brand new implementation, and not yet mature enough... It was merged only 19h ago...

Just as a reference: I'm on a RTX 3090

Input tokens	qwen36_35b_a3b_iq4nl_no_mtp	qwen36_35b_a3b_iq4nl_draftmtp2	qwen36_35b_a3b_iq4nl_draftmtp3	qwen36_35b_a3b_iq4nl_draftmtp4
1,024	input 1461.19 tok/s, output 98.36 tok/s, elapsed 2.0s	input 480.90 tok/s, output 115.37 tok/s, elapsed 3.3s	input 471.46 tok/s, output 126.36 tok/s, elapsed 3.2s	input 455.82 tok/s, output 143.21 tok/s, elapsed 3.2s
8,192	input 3277.07 tok/s, output 87.38 tok/s, elapsed 4.0s	input 1568.59 tok/s, output 110.49 tok/s, elapsed 6.4s	input 1626.07 tok/s, output 109.00 tok/s, elapsed 6.2s	input 1671.36 tok/s, output 142.32 tok/s, elapsed 5.8s
32,768	input 2867.22 tok/s, output 61.33 tok/s, elapsed 13.6s	input 1704.29 tok/s, output 100.82 tok/s, elapsed 20.5s	input 1805.61 tok/s, output 119.21 tok/s, elapsed 19.3s	input 1805.14 tok/s, output 128.24 tok/s, elapsed 19.2s
65,536	input 2398.14 tok/s, output 44.47 tok/s, elapsed 30.3s	input 1463.43 tok/s, output 85.90 tok/s, elapsed 46.4s	input 1602.67 tok/s, output 101.13 tok/s, elapsed 42.2s	input 1600.79 tok/s, output 107.77 tok/s, elapsed 42.2s
131,072	input 1813.04 tok/s, output 28.48 tok/s, elapsed 77.0s	input 1194.38 tok/s, output 70.47 tok/s, elapsed 111.7s	input 1294.03 tok/s, output 77.20 tok/s, elapsed 103.1s	input 1294.96 tok/s, output 86.81 tok/s, elapsed 102.9s
196,608	input 1451.00 tok/s, output 20.60 tok/s, elapsed 142.0s	input 1079.42 tok/s, output 57.42 tok/s, elapsed 184.6s	input 1077.91 tok/s, output 64.31 tok/s, elapsed 184.6s	input 1087.32 tok/s, output 69.02 tok/s, elapsed 182.9s

I will dig into the input digestion a bit now to maybe tackle this speed issue...
Seems I see already some problems that could cause this.

I've opened an issue TheTom/llama-cpp-turboquant#147

[mgeldi]

I ran it WITH TQ4_1S and MTP.
I'm getting INSANE token speeds on my 5090 with the Qwen3.6-27B model.
First, the fact that it can BARELY fit (with a bit of a limitted context window of 100k) is insane already.

But I'm getting an average of 120 to 130 tokens per second!

I now also compiled a version with TQ3_1S compression.
That doesn't seem to have this weird kernel accelerated path mentioned in the readme. However:

• full context length of 262k tokens
• still running at 40 to 45 tokens per second

that's with turboquant for KV-Cache-compression and the new "turboquant" compression for the weights.
I COOKED.

The future is local. Screw clouds

[xyehya]

@mgeldi Nice!
which repo are you using the weights tq + mtp?
hows the quality and tool calling

[mgeldi]

@xyehya
I used:
https://github.com/TheTom/llama-cpp-turboquant

the readme describes the paper what all of this is based on.

Essentially, I took an MTP-compatible BF16 version, ran the TQ4_1S quantization (and TQ3_1S) and it just worked as expected!

I'm now looking into what other types of models would perfectly fit on a 5090. TQ4_1S is the future. We just need SLIGHTLY smaller models so we can fit full context on a 5090 with max speed!

[claude-eric-steiner]

@xyehya and @mgeldi
There is a substantial overhead in the current upstream MTP implementation.
This overhead is degrading the input token digestion and therefore the total time from prompt input to final output token delivered.
I have described it here: TheTom/llama-cpp-turboquant#147
There is a fix for this, GPT 5.5 has fixed it for me in 30 minutes 👍 I'm quite happy with the result.
On the same issue page you can see the speed boost that I got with this fix.
You can find the fix on my forked github: https://github.com/claude-eric-steiner/llama-cpp-turboquant/tree/codex/mtp-prefill-hidden-state-fix
Clone that repo and build it, or ask codex/claude code to build it for you (if needed put it in a docker env). But make sure you checkout the codex/mtp-prefill-hidden-state-fix branch first.
I would be interested to see what boost it would give you on a 5090.

Cheers Claude