I saw “MTP round-trip” on a checklist for a Megatron conversion pipeline and had no idea what it meant. Two acronyms, one hyphen, apparently important enough that someone had listed it as a thing to verify.
Working out what it meant took me somewhere I didn’t expect. The interesting part turned out not to be MTP at all — it was the reason speculative decoding works in the first place, which rests on a fact about hardware that I had backwards.
TL;DR
- Generating text is slow because it’s sequential: one full forward pass per token.
- But a forward pass over five tokens costs about the same as over one. Generation is bottlenecked by moving weights, not by arithmetic.
- Speculative decoding exploits that: something cheap drafts k tokens, the big model verifies all of them in one pass.
- It is exact, not an approximation. Same output distribution as normal decoding.
- MTP (Multi-Token Prediction) is one way to produce those drafts — a small module trained into the model itself.
- MTP has two separate lives: a training-time auxiliary loss you can throw away, and an inference-time draft head you can’t.
Why generating text is slow
To produce token N+1, the model needs token N. There’s no way around that ordering — it’s what “language model” means.
So generating 100 tokens means 100 full passes through the network. For a model like GLM-5.2, that’s 78 layers, 100 times over.
The obvious conclusion is that generation is 100 times as expensive as reading the prompt. The obvious conclusion is wrong, and the way it’s wrong is the whole point.
The part that got me
A forward pass processing one token and a forward pass processing five tokens take roughly the same wall-clock time.
I had assumed compute scaled with tokens. It doesn’t, because compute isn’t the bottleneck.
Every forward pass has to read the model’s weights out of memory and into the compute units. That’s hundreds of gigabytes moving across a memory bus, and it happens whether you’re processing one token or fifty. The actual arithmetic on a handful of tokens is rounding error next to the cost of fetching the weights to do it with.
So the accounting looks like this:
| weight reads | |
|---|---|
| 5 tokens, one at a time | 5 |
| 5 tokens, in a single pass | 1 |
Same tokens, same math, five times the memory traffic — purely because of the ordering.
That ratio is the prize. If you could somehow process five tokens at once, you’d get them roughly five times faster. You can’t, because token 3 depends on token 2.
Unless you guess.
Speculative decoding: draft, then verify
The trick is to split generation into two roles.
- Something cheap drafts k tokens ahead. These are guesses.
- The big model does one forward pass over all k positions at once, checking each guess.
- Accept the longest correct prefix. Throw away the rest.
- Repeat.
Step 2 is the whole thing, and it hinges on a property that’s easy to miss: verification is parallel even though generation isn’t.
The sequential dependency exists because you don’t know token 3 until you’ve produced token 2. But in verification you already have candidate tokens 1 through k — the drafter handed them over. So you can lay them all out and check them in a single pass. There’s nothing left to wait for.
Guess three tokens correctly and you’ve produced four tokens for the price of one weight read. Guess wrong on the first one and you fall back to producing one token, which is what you’d have done anyway.
It’s exact, not approximate
This is the part I assumed had a catch, and it doesn’t.
Speculative decoding produces the same output distribution as ordinary decoding. It is not a quality-for-speed trade. Every token in the final output is one the big model itself endorsed — the drafter’s guesses are only ever suggestions, and any suggestion the big model wouldn’t have made is rejected.
The acceptance test is designed so that the surviving tokens are distributed exactly as if the big model had generated them alone. So there’s no accuracy knob to tune and no quality regression to monitor. Either it’s faster or it isn’t.
The only cost of a bad drafter is wasted drafting work. Which means the metric that matters is acceptance rate: what fraction of guesses survive.
Where the drafts come from
Several options, and this is where MTP finally enters.
| Draft source | How it works | Cost |
|---|---|---|
| A separate small model | A 1B model drafts for a 70B one | Train, ship and serve a second model; tokenizers must match |
| N-gram / lookup | Copy candidates straight from the context | Free, but only helps when output repeats input — code editing, summarisation |
| MTP head | A small module trained into the main model | One extra layer |
The separate-model approach came first and has an awkward property: the two models were trained independently, so they don’t necessarily think alike. When the drafter’s instincts diverge from the big model’s, acceptance rate falls, and a drafter whose guesses get rejected is pure overhead.
Which motivates building the drafter into the model.
What MTP actually is
Multi-Token Prediction. GLM-5.2’s config says:
1 | "num_hidden_layers": 78, |
78 layers of main model, plus one extra module whose job is to predict the token after next. That module is MTP.
Two properties make it a good drafter:
It agrees with the main model, because it was trained alongside it and sits on top of the same internal representations. Its guesses are the guesses the main model would plausibly make, which is exactly what acceptance rate rewards. DeepSeek-V3, which uses the same design, reports 85–90% acceptance on the next token and around 1.8× end-to-end throughput.
It’s cheap, because it reuses the main model’s already-computed hidden states. Drafting costs one extra layer, not a second model’s worth of forward pass.
And there’s no second checkpoint to version, deploy or keep in sync. The drafter ships inside the model.
MTP has two separate lives
This is the distinction I’d have got wrong if I hadn’t looked.
At training time, MTP is an auxiliary loss. Forcing the model to predict two tokens ahead rather than one pushes it toward representations that carry more forward-looking information, which improves the main model. Used this way, MTP is disposable — you can drop the module when training finishes and keep the benefit.
At inference time, MTP is the draft head. Used this way, you have to keep it, or you lose the speedup.
Same weights, two unrelated reasons to care about them. Whether you need MTP to survive your pipeline depends entirely on which of these you’re after.
Back to the checklist
Which finally explains why “MTP round-trip” is its own line item.
A training pipeline converts the model between formats: HuggingFace to Megatron to train, Megatron back to HuggingFace to serve. The round-trip is that loop, and the question is whether the MTP module comes out the other side intact.
The reason it needs its own check is that losing it is silent. MTP isn’t on the main forward path — it doesn’t affect what the model says, only how fast it says it. So:
- Conversion completes without error
- Forward-parity tests pass
- Training runs, loss curve looks normal
- Export succeeds
- Serving works, at half the throughput, with nothing in the logs
I looked at the verification script in the pipeline that prompted all this. Step 5 checks the HuggingFace to Megatron weight mapping by running both implementations on the same input and comparing outputs — cosine 0.9936, a clean pass. It also contains this line:
1 | if hasattr(hc, "num_nextn_predict_layers"): hc.num_nextn_predict_layers = 0 |
MTP is switched off for the test. Which is the right call for what that test is for — it isolates the main trunk so a mismatch points somewhere specific. But it means the verified mapping says nothing about MTP, and a forward-parity check couldn’t have caught the omission anyway, because MTP doesn’t touch the forward path being compared.
The verification you’d actually want is different in kind: not “does the model still behave correctly” but “is every weight still here”. Enumerate the tensors before and after, compare names and shapes, and require the numbers to match exactly — a format conversion does no arithmetic, so there’s no floating-point slack to allow for. Closer to an inventory than a test.
What I took from it
I went in thinking MTP was the topic and speculative decoding was context. It’s the reverse: speculative decoding is the idea, and MTP is one implementation of one of its parts.
The thing worth carrying, though, is the hardware fact underneath. The cost of a forward pass barely depends on how many tokens are in it. Once that lands, speculative decoding stops looking like a clever trick and starts looking obvious — you have four free seats in the car, so you may as well guess who’s coming.
It also rhymes with something I ran into autoscaling vLLM replicas: the intuitive cost model was wrong in a way that only shows up when you check what the hardware is actually doing.
