MTP support?

[hg0428]

MTP would be greatly appreciated.

[leehack]

I see there are some activities around MTP on llama.cpp. Once they have it, I'll look into this.

[hg0428]

I see there are some activities around MTP on llama.cpp. Once they have it, I'll look into this.

It's already added on llama.cpp

[hg0428]

Also I think it would be helpful if you showed what build of Llama.cpp you were using in the readme.

[leehack]

Research note / implementation plan after checking current upstream and local integration points.

Current status

• Upstream llama.cpp merged MTP support in PR llama + spec: MTP Support  ggml-org/llama.cpp#22673 on 2026-05-16.
• Current upstream docs list draft-mtp as a speculative decoding type and show --spec-type draft-mtp --spec-draft-n-max N.
• llamadart still pins native assets to b9159 in hook/build.dart, which predates the MTP merge.
• llamadart-native now has published assets through b9333, including the headers archive and platform bundles, so a native-asset baseline that includes MTP is available.
• GitHub compare ggml-org/llama.cpp b9159...b9333 is 174 commits ahead and includes 255582687 llama + spec: MTP Support (#22673) plus follow-up MTP fixes:
  • prompt decode/logit optimization
  • MTP cleanup
  • backend sampling for the MTP draft path
  • skipping logit computation via inp_out_ids
  • server VRAM leak fix for draft/MTP resources
  • MTP KV-cache draft type fix

Why this is not only a tag bump

The tag bump is necessary but not sufficient.

• Generated Dart bindings are still pre-MTP. llama_context_params in lib/src/backends/llama_cpp/bindings.dart does not yet expose upstream fields such as n_rs_seq and ctx_type, and the generated enum does not expose LLAMA_CONTEXT_TYPE_MTP.
• The current native generation loop in lib/src/backends/llama_cpp/llama_cpp_service.dart samples and decodes one token at a time. MTP needs a speculative loop: draft N tokens, verify [sampled + draft] in one target batch, accept the matching prefix, roll back rejected tails, and stream accepted tokens in order.
• Upstream MTP logic lives in common/speculative.* and server orchestration, not in the public llama.h API alone. llamadart-native currently builds with LLAMA_BUILD_COMMON OFF, and src/llama_dart_wrapper.h only exposes llama_dart_set_log_level.
• Upstream MTP uses staging pre-norm embedding APIs from src/llama-ext.h and expects the target decode path to expose embeddings/logits across prompt/decode batches when speculation is active.
• Prompt-prefix reuse and state persistence need special care because MTP maintains a sidecar draft context/KV state. It is safer to disable or tightly gate those optimizations for the first MTP implementation.
• Multimodal support should be considered a follow-up. Upstream says MTP is compatible with vision, but the local MTMD path needs explicit common_speculative_process-equivalent hooks before we can claim that.

Proposed scope

Initial support should be native, text-only, draft-mtp only.

Out of scope for the first pass:

• Web/WebGPU MTP
• n-gram speculative decoding
• external draft-model speculative decoding
• multimodal MTP
• state persistence with MTP enabled
• prompt-prefix reuse with MTP enabled
• grammar/lazy-grammar support with MTP unless parity is proven

Unsupported combinations should throw LlamaUnsupportedException with a concrete reason instead of silently falling back while reporting success.

Implementation plan

1. Baseline native bump

   • Update _llamaCppTag from b9159 to the newest published llamadart-native release that includes MTP, currently b9333.
   • Run tool/native/sync_native_headers_and_bindings.sh --tag b9333.
   • Fix generated binding fallout, especially llama_context_params layout, llama_context_type, n_rs_seq, and llama_n_rs_seq.
   • Add a README/support-matrix note showing the pinned llama.cpp/native asset tag, since this came up in this issue.
   • Add an Unreleased changelog entry for the native pin/binding refresh.

2. Add a narrow native wrapper in llamadart-native

   • Keep upstream sources unpatched.
   • Add an opaque C ABI wrapper around the pieces of upstream common/speculative.* needed for draft-mtp, or add a small wrapper that links the required common sources privately into libllamadart.
   • Expose only C-safe entry points, for example:
     • create/free MTP sidecar context
     • begin prompt/speculation state
     • report whether target batches need embeddings/logits
     • process target decode batches
     • generate draft token list
     • accept draft prefix and update stats
   • Keep ownership of upstream C++ containers and common_speculative state inside native code; Dart should not port this algorithm.

3. Add Dart API surface

   • Add SpeculativeDecodingParams (name TBD) with an enum that starts with none and draftMtp.
   • Put the option on GenerationParams unless implementation proves it needs to be context-level. Default stays none.
   • Start with conservative knobs:
     • draftTokenMax, default 3 to match current upstream docs
     • optional draftTokenMin, default 0
   • Gate Web and unsupported native assets with typed unsupported errors.

4. Integrate the native generation loop

   • When MTP is enabled, lazily create/cache the MTP sidecar context for the existing target context.
   • During prompt ingest, request the output rows MTP needs. Current text prompt ingest only requests logits for the last prompt token; MTP needs the target hidden state stream mirrored into the draft path.
   • Replace the one-token generation path with an MTP path:
     • sample token from current target logits
     • ask native wrapper for draft tokens
     • decode target batch containing sampled token plus draft tokens
     • process the target batch through the MTP wrapper
     • accept matching prefix
     • roll back rejected tail
     • stream accepted token bytes one by one, preserving stop-sequence checks and cancellation behavior
   • Keep the existing one-token loop untouched for default none.

5. Diagnostics and metrics

   • Extend native performance diagnostics with:
     • draft tokens generated
     • draft tokens accepted
     • draft acceptance rate
   • Log the effective native tag and MTP enablement in debug logs without leaking model paths or signed URLs.

6. Tests and validation

   • Unit tests:
     • config defaults and copyWith behavior
     • unsupported Web/MTP path
     • unsupported MTP with old native symbols/assets
     • incompatible options fail loudly
   • Native symbol test:
     • MTP wrapper symbols resolve
     • regenerated binding exposes LLAMA_CONTEXT_TYPE_MTP, ctx_type, n_rs_seq, llama_n_rs_seq
   • Local-only E2E:
     • Add a run_local_e2e.dart scenario for a small/known MTP GGUF, likely Qwen3.6 MTP.
     • Compare output against non-MTP at deterministic settings where possible.
     • Assert draftTokens > 0, acceptedDraftTokens > 0, and no crash across at least CPU plus one accelerated backend where available.
   • Regression:
     • Existing text generation still passes with default speculation disabled.
     • Prompt reuse/state persistence either explicitly disabled under MTP or covered with dedicated tests before enabling.

Acceptance criteria

• Users can enable native text-only MTP with an explicit public option.
• Default behavior is unchanged when speculation is not enabled.
• llamadart docs state the pinned llama.cpp/native tag and which MTP paths are supported.
• Unsupported platforms/options fail with typed actionable errors.
• MTP reports draft/accepted draft metrics.
• Native macOS VM tests and dart analyze pass.
• A local-only MTP smoke scenario exists and documents required model/backend setup.

References:

• Upstream PR: llama + spec: MTP Support  ggml-org/llama.cpp#22673
• Upstream speculative docs: https://github.com/ggml-org/llama.cpp/blob/master/docs/speculative.md
• Current native assets with MTP-capable baseline: https://github.com/leehack/llamadart-native/releases/tag/b9333

[leehack]

@hg0428 I don't know which model you are working with, but you can try this. This will allow you to use MTP through litert-lm.

[leehack]

Closing as addressed in current main. llama.cpp MTP is now available through SpeculativeDecodingConfig.mtp, including native wrapper symbols, diagnostics, docs, and local smoke and benchmark tooling. Follow up in a new issue if a specific MTP model or backend still fails on current releases.