LoopAhead: Speculating Ahead for Faster Decoding in Looped Language Models

Zixuan Liu; Nan Xue; Rui Bao; Yuhang Xu; Xiaofeng Cao; Jiangchao Yao; Zhiyong Chen

cs.AIOct 15, 2026

LoopAhead: Speculating Ahead for Faster Decoding in Looped Language Models

Authors: Zixuan Liu, Nan Xue, Rui Bao, Yuhang Xu, Xiaofeng Cao, Jiangchao Yao, Zhiyong Chen

Organizations: Shanghai Jiao Tong University · Tongji University

Abstract

Looped Transformers improve parameter efficiency by repeatedly applying the same Transformer layers. However, generating each token still involves sequential computation and repeated access to model weights, leading to high decoding latency. Recent pipelined self-speculative decoding methods reduce this delay by drafting tokens from early-loop predictions and asynchronously advancing tokens at different recurrent depths in the same batch, but they do not fully exploit early-loop features for longer lookahead and still spend computation verifying incorrect speculative branches. We introduce \method, a speculative decoding framework for looped transformers that enables earlier drafting of future tokens and assigns complementary roles to different recurrent depths. A lightweight lookahead drafter (LAD) predicts multiple future tokens with early-loop states, and deeper-loop feedback helps decide which draft branches to continue and which to pause, while the full-depth model makes the final token decisions. LoopAhead achieves up to 7.82x decoding speedup over autoregressive decoding while preserving the target model's output distribution.

Figures & tables

Figure 1: Key observations and decoding speedups of LoopAhead. Top left shows Ouro-1.4B’s task accuracy when generating at a fixed loop depth

r

. Top right shows token agreement between depth-

r

and full-depth predictions under the same prefixes. The bottom panel shows representative decoding speedups over AR, with percentages indicating throughput gains over LoopSpec.

Figure 2: Illustration of LoopAhead with an Ouro-1.4B decoding trajectory. LAD uses early-loop context and the native anchor to draft future tokens before verification. R2 feedback suspends conflicting branches while retaining their KV states for resumption. Full-depth R4 verification determines acceptance and pruning. The complete traces appear in Appendix E and Figure 13 .

Model	Method	GSM8K	MATH500	HumanEval	MBPP	BBH
Ouro-1.4B	DFlash	2.19 $(-24.8\%)$	2.35 $(-11.9\%)$	1.29 $(-60.8\%)$	1.64 $(-45.4\%)$	3.21 $(+20.6\%)$
	LoopSpec	2.92 (—)	2.66 (—)	3.29 (—)	3.00 (—)	2.66 (—)
	Ours	4.27 $(+46.4\%)$	3.73 $(+40.0\%)$	4.07 $(+23.8\%)$	4.01 $(+33.6\%)$	4.30 $(+61.6\%)$
Ouro-2.6B	DFlash	2.24 $(-30.0\%)$	2.30 $(-18.0\%)$	1.35 $(-60.9\%)$	1.72 $(-46.6\%)$	3.42 $(+19.8\%)$
	LoopSpec	3.19 (—)	2.80 (—)	3.45 (—)	3.22 (—)	2.85 (—)
	Ours	5.21 $(+63.1\%)$	4.37 $(+55.9\%)$	4.72 $(+36.9\%)$	4.76 $(+47.8\%)$	5.12 $(+79.2\%)$

Table 1: Greedy decoding speedups (

\times

) over AR. Results for LoopAhead are highlighted with shading. Parentheses show throughput changes relative to LoopSpec, with increases in green and decreases in red. Bold marks the best result. Timing and aggregation are detailed in Appendix A.5 .

Panel of Figure 3 — Figure 3: Recurrent computation profiles on Ouro-1.4B, Ouro-2.6B, Raven-Llama-3.2, and Raven-OLMo-2, pooled over five tasks with 3,023 requests per model. (a) Average loops per output token (relative to AR), $\widetilde{D}=S_{\mathrm{rec}}/(RN_{\mathrm{out}})$ . (b) Average candidate advances per output token (relative to AR), $\widetilde{W}=W/(RN_{\mathrm{out}})$ . (c) Average candidates advanced per loop, $\bar{B}=W/S_{\mathrm{rec}}$ .

Model	Method	GSM8K	MATH500	HumanEval	MBPP	BBH
Ouro-1.4B	DFlash	1.72 $\pm$ 0.03 $(-30.4\%)$	1.94 $\pm$ 0.001 $(-18.2\%)$	1.24 $\pm$ 0.02 $(-56.8\%)$	1.40 $\pm$ 0.03 $(-45.8\%)$	2.15 $\pm$ 0.03 $(-5.1\%)$
	LoopSpec	2.47 $\pm$ 0.02 (—)	2.37 $\pm$ 0.001 (—)	2.87 $\pm$ 0.01 (—)	2.58 $\pm$ 0.02 (—)	2.26 $\pm$ 0.02 (—)
	Ours	3.03 $\pm$ 0.04 $(+22.8\%)$	3.06 $\pm$ 0.01 $(+28.9\%)$	3.42 $\pm$ 0.02 $(+18.9\%)$	3.00 $\pm$ 0.04 $(+16.2\%)$	3.09 $\pm$ 0.03 $(+36.4\%)$
Raven-Llama-3.2	DFlash	1.12 $\pm$ 0.03 $(-52.6\%)$	1.16 $\pm$ 0.01 $(-58.2\%)$	1.03 $\pm$ 0.03 $(-66.6\%)$	1.07 $\pm$ 0.01 $(-61.2\%)$	2.00 $\pm$ 0.06 $(-38.5\%)$
	LoopSpec	2.37 $\pm$ 0.05 (—)	2.78 $\pm$ 0.02 (—)	3.10 $\pm$ 0.13 (—)	2.76 $\pm$ 0.08 (—)	3.26 $\pm$ 0.07 (—)
	Ours	3.24 $\pm$ 0.03 $(+36.8\%)$	3.60 $\pm$ 0.07 $(+29.4\%)$	3.80 $\pm$ 0.02 $(+22.9\%)$	3.58 $\pm$ 0.02 $(+29.9\%)$	4.32 $\pm$ 0.03 $(+32.6\%)$

Table 2: Sampling decoding speedups (

\times

) over AR at

T=1

and top-

p=0.9

, reporting mean

\pm

sample standard deviation over seeds 0, 42, and 271828. Parentheses show average throughput changes relative to LoopSpec under three seeds. Colors and boldface follow Table 1 .

Panel of Figure 4 — Figure 4: Drafter size and per-loop decode cost. (a) Learned parameters on Ouro-1.4B, Ouro-2.6B, and Raven-Llama-3.2, excluding shared target weights and including R4 residual for Raven, $P_{\mathrm{add}}$ . (b) Average decode time per loop, including drafting and scheduling, on Ouro-1.4B and Raven-Llama-3.2, pooled over five tasks with 3,023 requests per model, $\bar{t}_{\mathrm{rec}}=T_{\mathrm{dec}}/S_{\mathrm{rec}}$ .

Model	LAD	Sched.	Susp.	Vocab.	GSM8K	MATH500	HumanEval	MBPP	BBH
Ouro-1.4B	$\times$	$\times$	$\times$	—	2.88 $\times$	2.60 $\times$	3.24 $\times$	2.90 $\times$	2.60 $\times$
	$\times$	$\checkmark$	$\checkmark$	—	2.87 $\times$	2.60 $\times$	3.23 $\times$	2.89 $\times$	2.59 $\times$
	$\checkmark$	$\times$	$\times$	$\times$	2.89 $\times$	2.38 $\times$	2.52 $\times$	2.60 $\times$	2.78 $\times$
	$\checkmark$	$\checkmark$	$\checkmark$	$\times$	4.13 $\times$	3.53 $\times$	3.90 $\times$	3.70 $\times$	4.03 $\times$
	$\checkmark$	$\checkmark$	$\times$	$\checkmark$	3.86 $\times$	3.19 $\times$	3.43 $\times$	3.45 $\times$	3.71 $\times$
	$\checkmark$	$\checkmark$	$\checkmark$	$\checkmark$	4.26 $\times$	3.61 $\times$	4.01 $\times$	3.80 $\times$	4.13 $\times$

Table 3: Greedy decoding speedups (

\times

) over AR on 428 requests per model. Sched. denotes our full scheduling framework, with conflict-based suspension controlled separately by Susp. Vocab. denotes draft vocabulary reduction with 8,192 base tokens. Each model’s last two rows are measured together, varying only suspension with budgets, admission rules, and priorities fixed. Native proposal depths are R1/R2 for Ouro and R4/R8 for Raven. Bold marks the best result per model and task.

Panel of Figure 5 — Figure 5: Scheduling results with LAD and draft vocabulary reduction held fixed over 428 questions per model across five benchmarks. Bars show $\widetilde{W}=W/(RN_{\mathrm{out}})$ , the average number of candidate advances per output token relative to AR, and the dashed line denotes AR. Red annotations indicate the percentage reduction in candidate advances when scheduling is enabled. The tables report candidate advances on subsequently discarded nodes normalized by the same AR reference, $\widetilde{W}_{\mathrm{disc}}=W_{\mathrm{disc}}/(RN_{\mathrm{out}})$ ; average loops per output token, $D=S_{\mathrm{rec}}/N_{\mathrm{out}}$ ; average candidates advanced per loop, $\bar{B}=W/S_{\mathrm{rec}}$ ; and throughput (thr, in tokens/s).

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Hugging Face checkpoint	Revision
Ouro-1.4B	ByteDance/Ouro-1.4B	574fa66cb8bf
Ouro-2.6B	ByteDance/Ouro-2.6B	1ed04250da1a
Ouro-1.4B-Thinking	ByteDance/Ouro-1.4B-Thinking	3aaa2224253a
Raven-Llama-3.2	smcleish/Recurrent-Llama-3.2-train-recurrence-32	a5f6f126e9d9
Raven-OLMo-2	smcleish/Recurrent-OLMo-2-0425-train-recurrence-32	e1208e9af61d
Raven-TinyLlama	smcleish/Recurrent-TinyLlama-3T-train-recurrence-32	2579b6a0e983

Appendix

Table 4: Target checkpoints. Revisions are abbreviated to their first 12 hexadecimal characters.

Benchmark	Examples	Prompt	Max. tokens
GSM8K	1,319	3-shot CoT	512
MATH500	500	4-shot	2,048
HumanEval	164	0-shot	1,024
MBPP	500	0-shot	1,024
BBH	540	3-shot CoT	1,024

Appendix

Table 5: Benchmark settings for our evaluation experiments, including example numbers, prompt format and output budgets.

Source	Train	Dev.	Confirm.
GSM8K train	2,000	128	128
Python CodeAlpaca	2,000	128	128
MATH train	2,000	128	128
BBH custom split	2,025	270	270
UltraChat	1,000	128	128
Total	9,025	782	782

Appendix

Table 6: Training, development, and confirmation prompts used to collect teacher trajectories for model-specific LAD drafters.

Model group	$d_{1}$	$d_{2}$	$R$	$K$	$B_{\mathrm{ret}}$	$B_{\mathrm{exe}}$	$d_{\mathrm{guard}}$	Stride
All Ouro checkpoints	1	2	4	3	32	16	2	1
Raven-Llama / OLMo	4	8	32	2	32	16	1	2
Raven-TinyLlama	2	8	32	3	48	32	1	2

Appendix

Table 7: Greedy LoopAhead execution parameters.

Panel of Figure 6 — Figure 6: LAD token quality by extra-token offset $i$ on five benchmarks. Panels (a,b) show full-depth agreement rate among all generated candidates $q_{i}^{\mathrm{all}}$ . Panels (c,d) show acceptance rate among candidates checked during verification $a_{i}^{\mathrm{check}}$ .

Panel of Figure 7 — Figure 7: Raven-Llama-3.2 task accuracy at fixed exit depths and token agreement with full-depth predictions across five benchmarks, supplementing Figure 1 .

Panel of Figure 8 — Figure 8: Greedy throughput gains of LoopAhead over LoopSpec across six models and five benchmarks. Points show observed gains and black bars show 95% confidence intervals. The dashed line marks equal throughput.

Panel of Figure 9 — Figure 9: Sampling throughput gains of LoopAhead over LoopSpec at $T=1$ and top- $p=0.9$ . Points show mean gains over three seeds, and black bars show 95% confidence intervals. The dashed line marks equal throughput.

Model	Method	CNN/DailyMail	AlpacaEval	IFEval	MT-Bench
	DFlash	0.94 $(-63.4\%)$	1.16 $(-62.6\%)$	1.14 $(-63.2\%)$	1.06 $(-61.6\%)$
	LoopSpec	2.56 (—)	3.11 (—)	3.11 (—)	2.75 (—)
Ouro-2.6B	Ours	2.62 $(+2.4\%)$	3.43 $(+10.4\%)$	3.33 $(+7.3\%)$	2.81 $(+2.4\%)$
	DFlash	0.94 $(-76.5\%)$	1.33 $(-71.2\%)$	1.20 $(-76.7\%)$	1.17 $(-74.0\%)$
	LoopSpec	4.01 (—)	4.60 (—)	5.13 (—)	4.51 (—)
Raven-Llama-3.2	Ours	4.51 $(+12.3\%)$	5.31 $(+15.4\%)$	5.39 $(+5.0\%)$	5.14 $(+14.0\%)$

Appendix

Table 8: Greedy decoding speedups (

\times

) over AR on additional generation tasks, measured using pooled decode-only throughput with one pass per method. We evaluate 128 prompts each from CNN/DailyMail, AlpacaEval, and IFEval, and 38 first-turn MT-Bench prompts from writing, roleplay, extraction, and humanities. Shading, parentheses, and colors follow Table 1 .

Panel of Figure 10 — Figure 10: Effect of draft depth $d_{1}$ and intermediate-feedback depth $d_{2}$ on greedy decoding. Cells show the geometric mean of per-task throughput ratios relative to each model’s main configuration. Black boxes mark the configurations chosen for our main results in Section 4.2 .

Panel of Figure 11 — Figure 11: Decode speedup over AR versus additional LAD draft length $K$ , pooled across five benchmarks. Larger circles mark the main settings. Shading and error bars show 95% confidence intervals.

Panel of Figure 12 — Figure 12: Distribution of discarded LAD candidates by their executed loop depth $r_{\mathrm{end}}$ , with scheduling off or on. Annotations show the fraction reaching full depth. The Raven bin $(8,32)$ sums frequencies at depths strictly between 8 and 32.

Figure 13: Complete LoopSpec and LoopAhead decoding traces for the example in Figure 2 , using Ouro-1.4B under greedy decoding. Rows denote output positions and columns denote global loop steps. LoopAhead produces the eight tokens after prefill in 13 steps, compared with 15 for LoopSpec. Two of the nine LAD calls are shown in detail. Colors and symbols follow Figure 2 .

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 291 papers

LoopSpec: Pipelined Self-Speculative Decoding for Looped Transformers
Sep 15, 2026SangLyul Cho, Langqing Cui, Sehoon Kim +2Efficient Transformer Inference Speculative Decoding
1Seoul National University · 2KAIST
DLoop: Looped Speculative Decoding
Oct 6, 2026Geonmo Gu, Byeongho Heo, HeeJae Jun +4Speculative Decoding LLM Inference Acceleration
NAVER AI Lab · Korea University · NAVER AI Search Platform
WaveFront Decoding: Parallelized Self-Speculative Decoding for Looped Language Models
Sep 19, 2026Hyeongju Ha, Jae-Joon KimLLM Inference Acceleration Speculative Decoding
Seoul National University

Sep 15, 2026J/K move · Enter open · S save

LoopSpec: Pipelined Self-Speculative Decoding for Looped Transformers

SangLyul Cho, Langqing Cui, Sehoon Kim, Dongsu Han+1

1Seoul National University · 2KAIST

Efficient Transformer Inference Speculative Decoding Recurrent Transformers

Open paper AskPDF arXiv

Looped Transformers achieve strong performance with compact parameter sizes by repeatedly applying a shared stack of Transformer blocks across recurrent depths. However, they incur higher decoding latency than standard Transformer models of comparable parameter size because shared weights are accessed at every recurrent depth. To improve decoding efficiency, self-speculative decoding is particularly well suited to Looped Transformers, as their intermediate recurrent states can directly provide draft predictions without an auxiliary draft model. We therefore propose LoopSpec, a training-free self-speculative decoding framework tailored for Looped Transformers. LoopSpec extracts draft tokens from early recurrent states and operates in a pipelined manner, overlapping draft generation of future tokens with target verification of the current token. To improve draft accuracy without excessive compute overhead, we introduce a selective second proposal from deeper recurrent depth while ensuring lossless decoding under both greedy and sampling regimes. Furthermore, we derive the optimal proposal depths in closed form and show the prediction matches measurement. Across reasoning and coding benchmarks, LoopSpec achieves up to 6.83

\times

inference speedup across diverse Looped Transformers.

Abstract

Speculative Decoding

Explore similar work

LoopSpec: Pipelined Self-Speculative Decoding for Looped Transformers

DLoop: Looped Speculative Decoding

WaveFront Decoding: Parallelized Self-Speculative Decoding for Looped Language Models