cs.CLOct 17, 2026
SaveECHO: Efficient Causal Self-Recursion Beyond Auxiliary Multi-Token Prediction
Organizations: ERNIE Team, Baidu · Independent Researcher
Abstract
Existing Multi-token prediction (MTP) methods are auxiliary by design: the multi-horizon loss mostly trains modules attached to the backbone, and each added horizon costs another module and another pass. We propose Efficient Causal Self-Recursion (ECHO), which removes both costs. The backbone's own final layer predicts every horizon, so the supervision lands on the parameters that generate the next token, and Dynamic Depth Sampling (DDS) keeps deeper horizons affordable by drawing the training depth with probabilities that decay along depth. Future prediction thus becomes part of the autoregressive objective rather than a task attached to it, and the gains follow on both sides of it. In pretraining, added depth converts into downstream accuracy where auxiliary modules saturate, concentrating on reasoning and code at half the MTP training overhead of training every depth. At inference, the pretrained model is its own drafter: it drafts beyond the depth it was trained for, and keeps its draft-target alignment on the open-ended, high-entropy generation where post-hoc drafters lose theirs. The advantage survives a doubled token budget, a repeated pretraining run, and supervised finetuning.
Figures & tables
Figure 1: Multi-horizon supervision on the backbone’s own final layer buys downstream performance after pre-training and supervised fine-tuning (a) and drafting depth beyond what was trained (b).
Figure 2: Overview of ECHO. The backbone’s final decoder layer and output head are reused to predict every future horizon. Dynamic Depth Sampling (DDS) draws one prediction depth per optimizer step, with probabilities that decay along depth. Green blocks denote layers shared with the backbone.
Figure 3: What final-layer sharing buys, in pretraining and at inference. (a) Mean per-task score difference from MTP-1 within each task category, over the fourteen benchmarks of Table 1 , with the category size below each label and the fourteen-task averages in the legend. (b) Accepted length at with rejection sampling, against the target’s predictive entropy; the warm-start EAGLE-3 drafter is trained from DeepSeek’s MTP baseline, following Kimi Team (2026) .
Figure 4: The advantage persists under deeper sampling, supervised finetuning and a doubled token budget. Mean per-task gain over the matched DeepSeek baseline, by task class: MTP-1 before finetuning, MTP-1-Instruct after it. ( 4(c) ) shows the gap at step 78k and at step 156k, and the error bars are the run-to-run standard deviation from an independent repeat of the 78k recipe, on reasoning & code and on QA & multiple choice. Doubling the budget moves the reasoning & code gap by , a sixth of that spread, whereas the repeat moves the QA & multiple choice gap by , more than its own. The reasoning & code advantage therefore survives both conditions and the run-to-run spread, while the QA & multiple choice difference stays inside it.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Category | Benchmark | ECHO-1 | ECHO-3 | ECHO-1.5 (3) | MTP-1 | MTP-3 | ECHO-2 (7) |
| Mathematics | GSM8K | 61.49 | 59.82 | 63.31 | 61.64 | 63.08 | 61.33 |
| MATH | 36.04 | 36.01 | 34.64 | 34.31 | 33.78 | 33.56 | |
| Reasoning | BBH | 46.44 | 50.02 | 48.55 | 46.91 | 40.18 | 48.45 |
| BBEH | 6.63 | 6.41 | 6.09 | 5.33 | 6.96 | 5.43 | |
| PIQA | 77.57 | 78.26 | 78.38 | 77.68 | 77.97 | 78.09 | |
| WinoGrande | 63.85 | 64.01 | 64.40 | 62.27 | 63.06 | 62.59 |
Table 1: Per-benchmark scores at step 78,000 (base models, no EMA).
| Baseline | Repeat run | Twice the budget | ||||
| run 1, step 78k | run 2, step 78k | run 1, step 156k | ||||
| Benchmark | ECHO | DeepSeek | ECHO | DeepSeek | ECHO | DeepSeek |
| GSM8K | 61.49 | 61.64 | 60.05 | 60.73 | 64.82 | 65.05 |
| MATH | 36.04 | 34.31 | 34.58 | 35.37 | 38.49 | 36.65 |
| BBH | 46.44 | 46.91 | 48.15 | 47.79 | 48.31 | 48.79 |
| BBEH | 6.63 | 5.33 | 5.83 | 5.59 | 6.18 | 5.29 |
Table 2: One recipe under two perturbations, both against the same baseline. Baseline is the run reported in Table 1 . Repeat run is an independent pretraining run of the same recipe at the same step. Twice the budget is the baseline run continued to step 156k.
Figure 5: Training loss of ECHO-1 and MTP-1 over the first 78k steps of the baseline run, separated into the backbone’s next-token loss (left) and the MTP loss (right). The logged objective is the weighted sum , and the left panel is recovered by subtracting the weighted MTP loss from it. Each curve is an exponential moving average over all logged steps, with no subsampling; the vertical range is clipped to what the curves occupy after step 12k, so the first few thousand steps run above the frame.
Figure 6: The same two terms across both families, over the steps all five runs share. ECHO-1.5 (3) ’s event files start at step 60k on our side and the first thousand of those go to the moving average’s warm-up, which fixes the window at 61k–78k; the full-run view of the depth-1 pair is Figure 5 . On the backbone term (left) the five lie within and the ordering follows the family rather than the depth; the depth-1 term (right) is where added depth shows up.
| Category | Benchmark | MTP-Instruct | ECHO-1.5 (3) -Instruct | ECHO-2 (7) -Instruct |
| Mathematics | GSM8K | 61.87 | 62.70 | 65.20 |
| MATH | 31.05 | 38.54 | 38.27 | |
| Reasoning | BBH | 54.41 | 54.24 | 58.06 |
| BBEH | 11.30 | 10.33 | 10.00 | |
| PIQA | 74.49 | 74.38 | 74.38 | |
| WinoGrande | 61.17 | 61.09 | 61.56 |
Table 3: The fourteen in-scope benchmarks after one round of supervised finetuning. Every cell is scored under the protocol of Table 1 .
| Model | Qwen3-4B | LLaMA-3.1-8B | Qwen3-8B | Gemma4-12B | Vicuna-13B | Qwen3-14B | Qwen3-Coder-30B-A3B | LLaMA-3.3-70B |
| EAGLE-3 | ✗ | ✓ | ✗ | ✗ | ✓ | ✗ | ✗ | ✓ |
| DFlash | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✓ | ✗ |
| Domino | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ |
| DSpark | ✓ | ✗ | ✓ | ✓ | ✗ | ✓ | ✗ | ✗ |
Table 4: Target models evaluated by each method, ordered by model size.
| Method | Model | Training datasets | Samples | Draft params |
| DFlash | Qwen3-4B / 8B | Nemotron V2, CodeAlpaca, ShareGPT, UltraChat, LongAlign | 800K | 459M |
| EAGLE-3 | Qwen3-4B / 8B | ShareGPT, UltraChat | 530K | 218M |
| Domino | Qwen3-8B | Open-PerfectBlend | 1.42M | 1049M |
| EAGLE-3 | Qwen3-8B | ShareGPT, UltraChat | 530K | 400M |
| DSpark | Qwen3-4B / 8B / 14B | Open-PerfectBlend | 1.3M | 459M / 1049M / 1573M |
| EAGLE-3 | Qwen3-4B / 8B / 14B | Open-PerfectBlend | 1.3M | 218M / 400M / 609M |
Table 5: Training data and drafter capacity in the DFlash-family papers’ comparisons against EAGLE-3. Drafter parameter counts are computed from released configs.
Figure 7: The depth-2 term at matched pretraining step. ECHO-1.5 (3) samples depth 2 on of steps where ECHO-3 trains it on all of them, so the sampled series carries fewer points over this window, against . Fewer is not biased: at depth 1, which both configurations train every step, the loss on the steps that also sample depth 2 differs from the loss on the steps that do not by , against a standard error of .
Figure 8: Decode speedup at with rejection sampling, over the same cells as Figure 3(b) . The warm-start EAGLE-3 drafter is trained from DeepSeek’s MTP baseline, following Kimi Team (2026) , and is normalized by the baseline of its own run.
| Type | Artifact | License / Terms of Use |
| Training Data | ||
| Pretraining | 600B-token corpus | Proprietary (redistribution restricted) |
| SFT | In-house chat corpus | Proprietary (redistribution restricted) |
| Model Weights and Checkpoints | ||
| Weights | DeepSeek-V3 pretrained weights | DeepSeek Model License |
| Model | gpt-oss-120b | Apache-2.0 |
Table 6: Licenses and terms of use for the scientific artifacts used in our experiments, including training data, pretrained models, evaluation benchmarks, and third-party software. Repository-level licenses are distinguished from dataset-specific and model-weight licenses where applicable.
More in this topic
Multi-Token Prediction
Multi-token prediction trains models to forecast several future tokens at once, supporting more efficient generation and richer predictive representations.
See all 42 papers