Existing Multi-token prediction (MTP) methods are auxiliary by design: the multi-horizon loss mostly trains modules attached to the backbone, and each added horizon costs another module and another pass. We propose Efficient Causal Self-Recursion (ECHO), which removes both costs. The backbone's own final layer predicts every horizon, so the supervision lands on the parameters that generate the next token, and Dynamic Depth Sampling (DDS) keeps deeper horizons affordable by drawing the training depth with probabilities that decay along depth. Future prediction thus becomes part of the autoregressive objective rather than a task attached to it, and the gains follow on both sides of it. In pretraining, added depth converts into downstream accuracy where auxiliary modules saturate, concentrating on reasoning and code at half the MTP training overhead of training every depth. At inference, the pretrained model is its own drafter: it drafts beyond the depth it was trained for, and keeps its draft-target alignment on the open-ended, high-entropy generation where post-hoc drafters lose theirs. The advantage survives a doubled token budget, a repeated pretraining run, and supervised finetuning.
Figures & tables
Figure 1: Multi-horizon supervision on the backbone’s own final layer buys downstream performance after pre-training and supervised fine-tuning (a) and drafting depth beyond what was trained (b).
Figure 2: Overview of ECHO. The backbone’s final decoder layer L and output head are reused to predict every future horizon. Dynamic Depth Sampling (DDS) draws one prediction depth D∼π per optimizer step, with probabilities that decay along depth. Green blocks denote layers shared with the backbone.
Figure 3: What final-layer sharing buys, in pretraining and at inference. (a) Mean per-task score difference from MTP-1 within each task category, over the fourteen benchmarks of Table 1 , with the category size below each label and the fourteen-task averages in the legend. (b) Accepted length at T=1 with rejection sampling, against the target’s predictive entropy; the warm-start EAGLE-3 drafter is trained from DeepSeek’s MTP baseline, following Kimi Team (2026) .
Figure 4: The advantage persists under deeper sampling, supervised finetuning and a doubled token budget. Mean per-task gain over the matched DeepSeek baseline, by task class: MTP-1 before finetuning, MTP-1-Instruct after it. ( 4(c) ) shows the gap at step 78k and at step 156k, and the error bars are the run-to-run standard deviation from an independent repeat of the 78k recipe, 0.08 on reasoning & code and 0.38 on QA & multiple choice. Doubling the budget moves the reasoning & code gap by 0.01 , a sixth of that spread, whereas the repeat moves the QA & multiple choice gap by 0.54 , more than its own. The reasoning & code advantage therefore survives both conditions and the run-to-run spread, while the QA & multiple choice difference stays inside it.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Benchmark
ECHO-1
ECHO-3
ECHO-1.5 (3)
MTP-1
MTP-3
ECHO-2 (7)
Mathematics
GSM8K
61.49
59.82
63.31
61.64
63.08
61.33
MATH
36.04
36.01
34.64
34.31
33.78
33.56
Reasoning
BBH
46.44
50.02
48.55
46.91
40.18
48.45
BBEH
6.63
6.41
6.09
5.33
6.96
5.43
PIQA
77.57
78.26
78.38
77.68
77.97
78.09
WinoGrande
63.85
64.01
64.40
62.27
63.06
62.59
Appendix
Table 1: Per-benchmark scores at step 78,000 (base models, no EMA).
Baseline
Repeat run
Twice the budget
run 1, step 78k
run 2, step 78k
run 1, step 156k
Benchmark
ECHO
DeepSeek
ECHO
DeepSeek
ECHO
DeepSeek
GSM8K
61.49
61.64
60.05
60.73
64.82
65.05
MATH
36.04
34.31
34.58
35.37
38.49
36.65
BBH
46.44
46.91
48.15
47.79
48.31
48.79
BBEH
6.63
5.33
5.83
5.59
6.18
5.29
Appendix
Table 2: One recipe under two perturbations, both against the same baseline. Baseline is the run reported in Table 1 . Repeat run is an independent pretraining run of the same recipe at the same step. Twice the budget is the baseline run continued to step 156k.
Figure 5: Training loss of ECHO-1 and MTP-1 over the first 78k steps of the baseline run, separated into the backbone’s next-token loss (left) and the MTP loss (right). The logged objective is the weighted sum L=Lbackbone+0.3LMTP , and the left panel is recovered by subtracting the weighted MTP loss from it. Each curve is an exponential moving average over all 78,000 logged steps, with no subsampling; the vertical range is clipped to what the curves occupy after step 12k, so the first few thousand steps run above the frame.
Figure 6: The same two terms across both families, over the steps all five runs share. ECHO-1.5 (3) ’s event files start at step 60k on our side and the first thousand of those go to the moving average’s warm-up, which fixes the window at 61k–78k; the full-run view of the depth-1 pair is Figure 5 . On the backbone term (left) the five lie within 0.006 and the ordering follows the family rather than the depth; the depth-1 term (right) is where added depth shows up.
Category
Benchmark
MTP-Instruct
ECHO-1.5 (3) -Instruct
ECHO-2 (7) -Instruct
Mathematics
GSM8K
61.87
62.70
65.20
MATH
31.05
38.54
38.27
Reasoning
BBH
54.41
54.24
58.06
BBEH
11.30
10.33
10.00
PIQA
74.49
74.38
74.38
WinoGrande
61.17
61.09
61.56
Appendix
Table 3: The fourteen in-scope benchmarks after one round of supervised finetuning. Every cell is scored under the protocol of Table 1 .
Model
Qwen3-4B
LLaMA-3.1-8B
Qwen3-8B
Gemma4-12B
Vicuna-13B
Qwen3-14B
Qwen3-Coder-30B-A3B
LLaMA-3.3-70B
EAGLE-3
✗
✓
✗
✗
✓
✗
✗
✓
DFlash
✓
✓
✓
✗
✗
✗
✓
✗
Domino
✓
✗
✓
✗
✗
✗
✗
✗
DSpark
✓
✗
✓
✓
✗
✓
✗
✗
Appendix
Table 4: Target models evaluated by each method, ordered by model size.
Table 5: Training data and drafter capacity in the DFlash-family papers’ comparisons against EAGLE-3. Drafter parameter counts are computed from released configs.
Figure 7: The depth-2 term at matched pretraining step. ECHO-1.5 (3) samples depth 2 on 39.5% of steps where ECHO-3 trains it on all of them, so the sampled series carries fewer points over this window, 6,721 against 17,000 . Fewer is not biased: at depth 1, which both configurations train every step, the loss on the steps that also sample depth 2 differs from the loss on the steps that do not by +0.0002 , against a standard error of 0.0004 .
Figure 8: Decode speedup at T=1 with rejection sampling, over the same cells as Figure 3(b) . The warm-start EAGLE-3 drafter is trained from DeepSeek’s MTP baseline, following Kimi Team (2026) , and is normalized by the baseline of its own run.
Type
Artifact
License / Terms of Use
Training Data
Pretraining
600B-token corpus
Proprietary (redistribution restricted)
SFT
In-house chat corpus
Proprietary (redistribution restricted)
Model Weights and Checkpoints
Weights
DeepSeek-V3 pretrained weights
DeepSeek Model License
Model
gpt-oss-120b
Apache-2.0
Appendix
Table 6: Licenses and terms of use for the scientific artifacts used in our experiments, including training data, pretrained models, evaluation benchmarks, and third-party software. Repository-level licenses are distinguished from dataset-specific and model-weight licenses where applicable.
Multi-token prediction trains models to forecast several future tokens at once, supporting more efficient generation and richer predictive representations.
Multi-token prediction has been shown to increase data density during training, improve downstream text-generation quality, and serves as the defacto approach for self-speculative decoding. Existing foundation and open source models that use MTP heads commit to a static tree-based attention topology throughout the entire generation sequence, meaning the speculation depth, and thus the compute required during verification, stays constant regardless of the context. This is fundamentally misaligned with the entropy patterns of natural language where low-entropy regions often support reliable multi-step drafting, while high-entropy regions require more conservative speculation. To address this, we propose Entropy-guided Multi-Token Prediction (EntMTP), a training-free scheduler that toggles between tree-based attention topologies from a set of task-specific pareto-optimal trees conditioned on a running estimate of local generation entropy. By matching speculation depth to context predictability, EntMTP maximizes expected accepted-token throughput across the full distribution of generated text without sacrificing generation quality. When evaluated across Humaneval, ShareGPT, GSM8k, and Litbench benchmarks, EntMTP consistently achieves a 1.15x speedup against Hydra and peak speedup of 1.36x against Medusa baselines respectively.
Multi-token prediction (MTP) improves the throughput of autoregressive generation by enabling the language model to draft multiple next tokens per forward pass, while a verification step over draft tokens ensures that token distribution of the backbone is preserved. Every open MTP-family release (MiMo-7B, DeepSeek-V3, Qwen3) trains its heads jointly with the backbone over the full pretraining run of tens of trillions of tokens, thus setting the drafter quality at pretraining time. We ask whether a lightweight post-training pass on target-generated chain-of-thought is enough to reach the same expected throughput speedup on a frozen reasoning model, and study how a serving-time system built on such a checkpoint can be optimized. We present three findings. 1) On a frozen Qwen3-8B with K=3 chained MTP heads, we show that a post-training recipe with plain cross-entropy on ≈2.5B tokens reaches or exceeds the expected speedup of jointly trained MiMo-7B on math, coding and knowledge benchmarks. Our post-training recipe utilizes 103-104× less MTP-training tokens as compared with joint pre-training of MiMO-7B MTP baseline. 2) We propose a chain-aware relaxation of draft token verification rule that allows a bounded drift from backbone language model token distribution. We show that this relaxation lifts expected speedups by +12 to +16% per benchmark while preserving task accuracy. 3) We propose an adaptive controller that dynamically chooses the number of MTP heads to be engaged at inference time and demonstrate recovery of upto 11--14% loss in speedup using fixed maximum MTP draft length.
Multi-token prediction (MTP) can accelerate language-model decoding by drafting several tokens and verifying them in a single target-model pass. However, the best speculation horizon changes with context: longer drafts offer greater amortization when reliable, but can waste compute on suffixes discarded after a rejection. Unlike autoregressive drafting, where a stopping rule can decide token by token, MTP must fix the horizon before drafting, as all heads fire in the same forward pass. We therefore formulate horizon selection as an online control problem, using verification feedback to learn how far the draft can be trusted in each context, and introduce Adaptive Multi-Token Prediction (AMTP): a lightweight controller that requires no retraining and leaves the exact verifier unchanged. To guide horizon selection, we characterize the expected number of tokens committed at each horizon via acceptance survival, the probability that a draft prefix is fully accepted. Before each round, AMTP predicts how far the draft is likely to survive and picks the horizon with the highest expected tokens per second, learning freely from every verification pass. Theoretically, we show that any online learner's prediction error translates into throughput regret against the best horizon for each context, and in turn into a bound on wall-clock completion time. With online gradient descent, this regret is sublinear. Across five draft-target pairings built on Llama-3.1-8B-Instruct with Medusa heads, Meta's MTP models, and Qwen3-4B-Inst-2507-MTP, evaluated on AIME 2025 and GSM8K, AMTP decodes up to 4.1x faster than standard decoding and matches or outperforms the fixed window in 8 of 10 settings, with throughput gains of up to 10%.