Looped Transformers improve parameter efficiency by repeatedly applying the same Transformer layers. However, generating each token still involves sequential computation and repeated access to model weights, leading to high decoding latency. Recent pipelined self-speculative decoding methods reduce this delay by drafting tokens from early-loop predictions and asynchronously advancing tokens at different recurrent depths in the same batch, but they do not fully exploit early-loop features for longer lookahead and still spend computation verifying incorrect speculative branches. We introduce \method, a speculative decoding framework for looped transformers that enables earlier drafting of future tokens and assigns complementary roles to different recurrent depths. A lightweight lookahead drafter (LAD) predicts multiple future tokens with early-loop states, and deeper-loop feedback helps decide which draft branches to continue and which to pause, while the full-depth model makes the final token decisions. LoopAhead achieves up to 7.82x decoding speedup over autoregressive decoding while preserving the target model's output distribution.
Figures & tables
Figure 1: Key observations and decoding speedups of LoopAhead. Top left shows Ouro-1.4B’s task accuracy when generating at a fixed loop depth r . Top right shows token agreement between depth- r and full-depth predictions under the same prefixes. The bottom panel shows representative decoding speedups over AR, with percentages indicating throughput gains over LoopSpec.
Figure 2: Illustration of LoopAhead with an Ouro-1.4B decoding trajectory. LAD uses early-loop context and the native anchor to draft future tokens before verification. R2 feedback suspends conflicting branches while retaining their KV states for resumption. Full-depth R4 verification determines acceptance and pruning. The complete traces appear in Appendix E and Figure 13 .
Model
Method
GSM8K
MATH500
HumanEval
MBPP
BBH
Ouro-1.4B
DFlash
2.19 (−24.8%)
2.35 (−11.9%)
1.29 (−60.8%)
1.64 (−45.4%)
3.21 (+20.6%)
LoopSpec
2.92 (—)
2.66 (—)
3.29 (—)
3.00 (—)
2.66 (—)
Ours
4.27 (+46.4%)
3.73 (+40.0%)
4.07 (+23.8%)
4.01 (+33.6%)
4.30 (+61.6%)
Ouro-2.6B
DFlash
2.24 (−30.0%)
2.30 (−18.0%)
1.35 (−60.9%)
1.72 (−46.6%)
3.42 (+19.8%)
LoopSpec
3.19 (—)
2.80 (—)
3.45 (—)
3.22 (—)
2.85 (—)
Ours
5.21 (+63.1%)
4.37 (+55.9%)
4.72 (+36.9%)
4.76 (+47.8%)
5.12 (+79.2%)
Table 1: Greedy decoding speedups ( × ) over AR. Results for LoopAhead are highlighted with shading. Parentheses show throughput changes relative to LoopSpec, with increases in green and decreases in red. Bold marks the best result. Timing and aggregation are detailed in Appendix A.5 .
Figure 3: Recurrent computation profiles on Ouro-1.4B, Ouro-2.6B, Raven-Llama-3.2, and Raven-OLMo-2, pooled over five tasks with 3,023 requests per model. (a) Average loops per output token (relative to AR), D=Srec/(RNout) . (b) Average candidate advances per output token (relative to AR), W=W/(RNout) . (c) Average candidates advanced per loop, Bˉ=W/Srec .
Model
Method
GSM8K
MATH500
HumanEval
MBPP
BBH
Ouro-1.4B
DFlash
1.72 ± 0.03 (−30.4%)
1.94 ± 0.001 (−18.2%)
1.24 ± 0.02 (−56.8%)
1.40 ± 0.03 (−45.8%)
2.15 ± 0.03 (−5.1%)
LoopSpec
2.47 ± 0.02 (—)
2.37 ± 0.001 (—)
2.87 ± 0.01 (—)
2.58 ± 0.02 (—)
2.26 ± 0.02 (—)
Ours
3.03 ± 0.04 (+22.8%)
3.06 ± 0.01 (+28.9%)
3.42 ± 0.02 (+18.9%)
3.00 ± 0.04 (+16.2%)
3.09 ± 0.03 (+36.4%)
Raven-Llama-3.2
DFlash
1.12 ± 0.03 (−52.6%)
1.16 ± 0.01 (−58.2%)
1.03 ± 0.03 (−66.6%)
1.07 ± 0.01 (−61.2%)
2.00 ± 0.06 (−38.5%)
LoopSpec
2.37 ± 0.05 (—)
2.78 ± 0.02 (—)
3.10 ± 0.13 (—)
2.76 ± 0.08 (—)
3.26 ± 0.07 (—)
Ours
3.24 ± 0.03 (+36.8%)
3.60 ± 0.07 (+29.4%)
3.80 ± 0.02 (+22.9%)
3.58 ± 0.02 (+29.9%)
4.32 ± 0.03 (+32.6%)
Table 2: Sampling decoding speedups ( × ) over AR at T=1 and top- p=0.9 , reporting mean ± sample standard deviation over seeds 0, 42, and 271828. Parentheses show average throughput changes relative to LoopSpec under three seeds. Colors and boldface follow Table 1 .
Figure 4: Drafter size and per-loop decode cost. (a) Learned parameters on Ouro-1.4B, Ouro-2.6B, and Raven-Llama-3.2, excluding shared target weights and including R4 residual for Raven, Padd . (b) Average decode time per loop, including drafting and scheduling, on Ouro-1.4B and Raven-Llama-3.2, pooled over five tasks with 3,023 requests per model, tˉrec=Tdec/Srec .
Model
LAD
Sched.
Susp.
Vocab.
GSM8K
MATH500
HumanEval
MBPP
BBH
Ouro-1.4B
×
×
×
—
2.88 ×
2.60 ×
3.24 ×
2.90 ×
2.60 ×
×
✓
✓
—
2.87 ×
2.60 ×
3.23 ×
2.89 ×
2.59 ×
✓
×
×
×
2.89 ×
2.38 ×
2.52 ×
2.60 ×
2.78 ×
✓
✓
✓
×
4.13 ×
3.53 ×
3.90 ×
3.70 ×
4.03 ×
✓
✓
×
✓
3.86 ×
3.19 ×
3.43 ×
3.45 ×
3.71 ×
✓
✓
✓
✓
4.26 ×
3.61 ×
4.01 ×
3.80 ×
4.13 ×
Table 3: Greedy decoding speedups ( × ) over AR on 428 requests per model. Sched. denotes our full scheduling framework, with conflict-based suspension controlled separately by Susp. Vocab. denotes draft vocabulary reduction with 8,192 base tokens. Each model’s last two rows are measured together, varying only suspension with budgets, admission rules, and priorities fixed. Native proposal depths are R1/R2 for Ouro and R4/R8 for Raven. Bold marks the best result per model and task.
Figure 5: Scheduling results with LAD and draft vocabulary reduction held fixed over 428 questions per model across five benchmarks. Bars show W=W/(RNout) , the average number of candidate advances per output token relative to AR, and the dashed line denotes AR. Red annotations indicate the percentage reduction in candidate advances when scheduling is enabled. The tables report candidate advances on subsequently discarded nodes normalized by the same AR reference, Wdisc=Wdisc/(RNout) ; average loops per output token, D=Srec/Nout ; average candidates advanced per loop, Bˉ=W/Srec ; and throughput (thr, in tokens/s).
Table 4: Target checkpoints. Revisions are abbreviated to their first 12 hexadecimal characters.
Benchmark
Examples
Prompt
Max. tokens
GSM8K
1,319
3-shot CoT
512
MATH500
500
4-shot
2,048
HumanEval
164
0-shot
1,024
MBPP
500
0-shot
1,024
BBH
540
3-shot CoT
1,024
Appendix
Table 5: Benchmark settings for our evaluation experiments, including example numbers, prompt format and output budgets.
Source
Train
Dev.
Confirm.
GSM8K train
2,000
128
128
Python CodeAlpaca
2,000
128
128
MATH train
2,000
128
128
BBH custom split
2,025
270
270
UltraChat
1,000
128
128
Total
9,025
782
782
Appendix
Table 6: Training, development, and confirmation prompts used to collect teacher trajectories for model-specific LAD drafters.
Model group
d1
d2
R
K
Bret
Bexe
dguard
Stride
All Ouro checkpoints
1
2
4
3
32
16
2
1
Raven-Llama / OLMo
4
8
32
2
32
16
1
2
Raven-TinyLlama
2
8
32
3
48
32
1
2
Appendix
Table 7: Greedy LoopAhead execution parameters.
Figure 6: LAD token quality by extra-token offset i on five benchmarks. Panels (a,b) show full-depth agreement rate among all generated candidates qiall . Panels (c,d) show acceptance rate among candidates checked during verification aicheck .
Figure 7: Raven-Llama-3.2 task accuracy at fixed exit depths and token agreement with full-depth predictions across five benchmarks, supplementing Figure 1 .
Figure 8: Greedy throughput gains of LoopAhead over LoopSpec across six models and five benchmarks. Points show observed gains and black bars show 95% confidence intervals. The dashed line marks equal throughput.
Figure 9: Sampling throughput gains of LoopAhead over LoopSpec at T=1 and top- p=0.9 . Points show mean gains over three seeds, and black bars show 95% confidence intervals. The dashed line marks equal throughput.
Model
Method
CNN/DailyMail
AlpacaEval
IFEval
MT-Bench
DFlash
0.94 (−63.4%)
1.16 (−62.6%)
1.14 (−63.2%)
1.06 (−61.6%)
LoopSpec
2.56 (—)
3.11 (—)
3.11 (—)
2.75 (—)
Ouro-2.6B
Ours
2.62 (+2.4%)
3.43 (+10.4%)
3.33 (+7.3%)
2.81 (+2.4%)
DFlash
0.94 (−76.5%)
1.33 (−71.2%)
1.20 (−76.7%)
1.17 (−74.0%)
LoopSpec
4.01 (—)
4.60 (—)
5.13 (—)
4.51 (—)
Raven-Llama-3.2
Ours
4.51 (+12.3%)
5.31 (+15.4%)
5.39 (+5.0%)
5.14 (+14.0%)
Appendix
Table 8: Greedy decoding speedups ( × ) over AR on additional generation tasks, measured using pooled decode-only throughput with one pass per method. We evaluate 128 prompts each from CNN/DailyMail, AlpacaEval, and IFEval, and 38 first-turn MT-Bench prompts from writing, roleplay, extraction, and humanities. Shading, parentheses, and colors follow Table 1 .
Figure 10: Effect of draft depth d1 and intermediate-feedback depth d2 on greedy decoding. Cells show the geometric mean of per-task throughput ratios relative to each model’s main configuration. Black boxes mark the configurations chosen for our main results in Section 4.2 .
Figure 11: Decode speedup over AR versus additional LAD draft length K , pooled across five benchmarks. Larger circles mark the main settings. Shading and error bars show 95% confidence intervals.
Figure 12: Distribution of discarded LAD candidates by their executed loop depth rend , with scheduling off or on. Annotations show the fraction reaching full depth. The Raven bin (8,32) sums frequencies at depths strictly between 8 and 32.
Figure 13: Complete LoopSpec and LoopAhead decoding traces for the example in Figure 2 , using Ouro-1.4B under greedy decoding. Rows denote output positions and columns denote global loop steps. LoopAhead produces the eight tokens after prefill in 13 steps, compared with 15 for LoopSpec. Two of the nine LAD calls are shown in detail. Colors and symbols follow Figure 2 .
Looped Transformers achieve strong performance with compact parameter sizes by repeatedly applying a shared stack of Transformer blocks across recurrent depths. However, they incur higher decoding latency than standard Transformer models of comparable parameter size because shared weights are accessed at every recurrent depth. To improve decoding efficiency, self-speculative decoding is particularly well suited to Looped Transformers, as their intermediate recurrent states can directly provide draft predictions without an auxiliary draft model. We therefore propose LoopSpec, a training-free self-speculative decoding framework tailored for Looped Transformers. LoopSpec extracts draft tokens from early recurrent states and operates in a pipelined manner, overlapping draft generation of future tokens with target verification of the current token. To improve draft accuracy without excessive compute overhead, we introduce a selective second proposal from deeper recurrent depth while ensuring lossless decoding under both greedy and sampling regimes. Furthermore, we derive the optimal proposal depths in closed form and show the prediction matches measurement. Across reasoning and coding benchmarks, LoopSpec achieves up to 6.83× inference speedup across diverse Looped Transformers.
Speculative decoding accelerates autoregressive generation in large language models. In each drafting stage, a lightweight draft model proposes tokens that the target model subsequently verifies. With increasingly capable draft models, we find that the target model frequently accepts all tokens produced in a drafting stage. A verification nevertheless follows each drafting stage, resulting in unnecessary target-model forward passes even when drafting could have continued. Adaptive draft length methods decide during decoding how many draft tokens precede a verification, but they raise the speedup only for autoregressive draft models. For a parallel draft model, drafting further requires target-model hidden states for draft tokens that have not been verified. We propose DLoop, a looped form of speculative decoding that adaptively performs multiple drafting stages before verification. DLoop continues drafting while the draft model remains confident and verifies all accumulated draft tokens together. Loop-aware training keeps the draft model reliable in the additional drafting stages by exposing it to its own hidden states for unverified draft tokens. By spending additional draft-model forward passes, DLoop reduces the number of target-model forward passes required for verification. Across diverse speculative decoding methods including EAGLE-3, DFlash, Domino, DSpark, and multi-token prediction modules, DLoop improves the wall-clock speedup by 5 to 41 percent while preserving lossless decoding. Code will be available at https://github.com/naver-ai/DLoop.
Geonmo Gu, Byeongho Heo, HeeJae Jun +4
NAVER AI Lab · Korea University · NAVER AI Search Platform
Looped language models repeatedly apply a weight-shared block to increase effective depth without increasing parameter count, but the resulting T sequential recurrent-block calls per generated token substantially increase decoding latency. To address the issue, we introduce Wavefront Decoding (WFD), a training-free self-speculative decoding framework designed for looped language models. WFD exploits two properties of these architectures: intermediate recurrence outputs provide effective draft predictions, and weight sharing allows token states at different positions and recurrence depths to be processed in one batched recurrent-block call. WFD organizes these mixed-depth states into a diagonal wavefront, continuously drafting new positions at shallow depth while advancing earlier positions toward full-depth verification. Unlike the phase-separated draft-then-verify schedule, WFD therefore concurrently batches drafting and verification within the same recurrent calls, while rejected drafts are corrected using full-depth predictions. Across six Spec-Bench task categories, WFD achieves 2.42x speedup on Ouro-2.6B and 3.54x on Huginn-3.5B over autoregressive decoding, consistently outperforming draft-then-verify. Cross-recurrence KV sharing further reduces wavefront KV traffic and increases WFD's speedup to 4.81x on Huginn-3.5B. The code is available at https://github.com/summerbro-hhj/wavefront-decoding.