Looped Transformers improve parameter efficiency by repeatedly applying the same Transformer layers. During autoregressive decoding, however, generating each token requires several sequential passes through these layers, leading to high decoding latency. Pipelined self-speculative methods use shallow-loop predictions to process tokens at different loops in the same batch, but each token must reach the drafting depth before the next can enter the pipeline. We introduce LoopFill, a training-free speculative decoding algorithm for looped LLM that combines multi-token suffix drafts with shallow-loop predictions in a single draft tree. Both drafters contribute to one tree, so a suffix lookup can immediately extend a shallow prediction and several future tokens can enter the pipeline together. The shared block computes the highest-scoring draft candidates at different loop depths in one batch, and full-depth verification preserves the model's autoregressive output distribution. Across seven benchmarks, LoopFill's average speedup over greedy autoregressive decoding is 5.85× on Ouro-2.6B and 5.13× on Huginn-3.5B. The speedups exceed those of LoopAhead, the strongest baseline, by 40% on Ouro-2.6B and 25% on Huginn-3.5B.
Figures & tables
Figure 1: Loop- d and suffix-tree drafts on Ouro-2.6B. (a) Blue: how often the loop- d draft equals the full-depth output. Orange: additional loop 2 hits where loop 1 misses, and additional loop 3 hits where loops 1 and 2 both miss. (b) Mean length of the correct draft for suffix drafts alone, adding loop-1 draft (+Shallow), and also adding suffix draft after loop-1 draft (+Requery)
Method
Data
Model
Multi-token
Training-free
Data-driven drafting
SuffixDecoding ( Oliaro et al., 2025 )
3
7
3
3
+ EAGLE-3 ( Li et al., 2025 )
3
3
3
7
Model-driven drafting
LoopSpec ( Cho et al., 2026 )
7
3
7
3
WaveFront ( Ha and Kim, 2026 )
7
3
7
3
Table 1: Comparison with existing speculative decoding methods from four perspectives . LoopFill combines multi-token suffix drafts with shallow-loop draft from the target model in a draft tree without training.
Figure 2: (a) Looped LLM architecture. In Ouro ( Zhu et al., 2026 ) , the entire Transformer stack is recurrent, with the embedding and LM head outside the loop. Huginn ( Geiping et al., 2025a ) places the recurrent block between prelude and coda blocks with Transformer layers. (b) Comparing four decoding algorithms (3 loops). Row ok is the position that predicts the k -th output token, with its input token on the left, and each column is one loop step. Autoregressive decoding starts a token after the previous one finishes. LoopSpec starts new token after one loop step, but the wrong draft (e.g. the ) forces a restart. SuffixDecoding verifies multi-token drafts but waits for all three loops in every round. LoopFill (Section 3 ) takes 5 loop steps to finish, achieving 3× speedup.
Figure 3: LoopFill decoding. (a) Suffix retrieval extends the shallow draft to . (b) After step 1, a falls outside the shallow top-2 after like , so both a and its child coffee have a path score of zero. (c) The tree after step 3. Dots show how many loops each node has completed toward predicting the token after its prefix. The numbered badges mark the loop step at which each output token is appended.
Model
Method
B=1
B=2
B=4
B=8
B=16
Huginn-3.5B
AR
5.89 / 1.00 ×
11.11 / 1.00 ×
19.00 / 1.00 ×
32.99 / 1.00 ×
48.34 / 1.00 ×
WaveFront
14.92 / 2.53 ×
24.56 / 2.21 ×
39.76 / 2.09 ×
60.81 / 1.84 ×
80.44 / 1.66 ×
LoopSpec
19.65 / 3.34 ×
33.89 / 3.05 ×
51.03 / 2.69 ×
74.24 / 2.25 ×
89.65 / 1.85 ×
SuffixDecoding
9.45 / 1.60 ×
17.34 / 1.56 ×
30.17 / 1.59 ×
52.01 / 1.58 ×
74.45 / 1.54 ×
DFlash
18.36 / 3.12 ×
33.18 / 2.99 ×
57.47 / 3.03 ×
88.18 / 2.67 ×
117.52 / 2.43 ×
LoopAhead
23.95 / 4.07 ×
35.82 / 3.22 ×
52.21 / 2.75 ×
76.12 / 2.31 ×
100.24 / 2.07 ×
Table 3: SpecBench end-to-end output throughput at different concurrency levels at decoding temperature 1.0 . Each cell gives output tokens/s and the throughput ratio to AR at the same concurrency. Best results are highlighted in bold.
Figure 4: Ablations of LoopFill on Huginn-3.5B, SpecBench, B=16 . Each axis is one SpecBench task, including RAG, Multi-turn Conversation (MT-Bench), Translation (Trans.), Summarization(Sum.), QA and Math. The radial value is the throughput (tok/s). Here d1 and d2 are loops 4 and 8 of T=32 .
Figure 5: Effect of the per-request draft-node budget N at two concurrency levels. The left and middle panels show the percentage change from N=16 in decoding rate (output tokens/s, B=1 ) and end-to-end throughput ( B=16 ), respectively. The right panel shows γ (solid) and loop passes per output token (dashed), divided by its value at N=16,B=16 .
Figure 6: Accepted tokens are mostly precomputed. (a) Cumulative distribution of r/T , where r is the number of loops already completed for the prefix ending in the emitted token. (b) Share of decoding loop steps attributed to First token : nothing can be drafted before decoding starts. Drafted too late : the correct token was added to the draft tree too late to finish in time. Drafted, not computed : it was drafted early enough but was pruned.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Evaluation-set size
Domain
GSM8K ( Cobbe et al., 2021 )
1,319
math
MATH-500 ( Hendrycks et al., 2021 )
500
math
BBH ( Suzgun et al., 2022 )
6,511
reasoning
HumanEval+ ( Liu et al., 2023 )
164
code
MBPP+ ( Liu et al., 2023 )
378
code
Spider ( Yu et al., 2019 )
1034
agentic SQL
Appendix
Table 4: Evaluation-set sizes of the seven benchmarks.
Method
Source
Number of samples
LoopAhead
GSM8K ( Cobbe et al., 2021 )
2,000
CodeAlpaca ( Chaudhary, 2023 )
2,000
MATH ( Hendrycks et al., 2021 )
2,000
Nemotron V1 STEM ( Nathawani et al., 2025 )
2,025
UltraChat ( Ding et al., 2023 )
1,000
Total
9,025
Appendix
Table 5: Dataset for training. Deduplicated against the evaluation set.
Setting
LoopAhead
DFlash
Epochs
3
6
Batch size
64
32
Optimizer
AdamW
AdamW
Adam (β1,β2)
(0.9,0.95)
(0.9,0.999)
Weight decay
0.01
0
Peak learning rate
2×10−4
6×10−4
Appendix
Table 6: Training settings for LoopAhead and DFlash
Figure 7: Ablations of LoopFill on Ouro-2.6B, SpecBench, B=16 . Here d1 and d2 are loops 1 and 2 of T=4 .
Figure 8: Loop passes vs. loop-normalized mean accepted length. The ablation configurations of each model at B=16 . The x -axis is the loop passes per token, the y -axis is the loop-normalized mean accepted length γ . The blue line changes the node budget N with everything else default (orange diamond: N=16 , loop- d1 & d2 drafts, Top-2, suffix budget Q=4 ). Green triangles change Top- k (1, 4) or Q (2, 8). Brown squares use loop- d1 drafts only (Top-1/2/4 at Q=4 , Q=2/8 at Top-2). Using loop- d1 drafts only lowers γ by 29-32% on Huginn-3.5B and 12-16% on Ouro-2.6B while increasing the compute per token by 30-38% and 3-6%.
Figure 9: Sensitivity of γ to suffix weight c . Change in the loop-normalized mean accepted length γ relative to c=1 . B=1 , N=16 , greedy. γ stays within 2% of c=1 for c∈[0.25,1000] on both Ouro models and within 4% for c∈[0.05,1000] on Huginn-3.5B. It drops only when suffix-tree drafts are almost ignored: by 5-12% for Ouro at c≤0.05 and by 5.6% for Huginn-3.5B at c=0.001 . The suffix-tree draft score min(1,cq) saturates for large c , so c=4 and c=1000 behave alike.
Model
Method
B=1
B=2
B=4
B=8
B=16
Ouro-1.4B
AR
10.77 / 1.00 ×
20.75 / 1.00 ×
40.06 / 1.00 ×
76.25 / 1.00 ×
135.65 / 1.00 ×
WaveFront
26.86 / 2.49 ×
51.00 / 2.46 ×
94.74 / 2.36 ×
168.89 / 2.21 ×
272.37 / 2.01 ×
LoopSpec
29.77 / 2.76 ×
56.83 / 2.74 ×
105.23 / 2.63 ×
182.51 / 2.39 ×
287.90 / 2.12 ×
SuffixDecoding
17.55 / 1.63 ×
31.54 / 1.52 ×
61.70 / 1.54 ×
113.74 / 1.49 ×
193.24 / 1.42 ×
DFlash
31.81 / 2.95 ×
59.24 / 2.85 ×
110.13 / 2.75 ×
193.01 / 2.53 ×
289.28 / 2.13 ×
LoopAhead
37.48 / 3.48 ×
71.09 / 3.43 ×
124.88 / 3.12 ×
195.35 / 2.56 ×
279.82 / 2.06 ×
Appendix
Table 7: SpecBench end-to-end output throughput at different concurrency levels for Ouro-1.4B at decoding temperature 1.0. Each cell gives output tokens/s and the throughput ratio to AR at the same concurrency. Best results are highlighted in bold.
Figure 10: Which draft each output token came from. LoopFill on SpecBench. Each bar splits the decode tokens of one model, excluding the token produced by prefill. A token is credited to the draft node it matched when it was verified: a suffix-tree draft, a loop- d1 draft of rank 1 or 2 in the next-token distribution after d1 loops, or a loop- d2 draft first added after d2 loops ( d1 and d2 are loops 1 and 2 of T=4 for Ouro and loops 4 and 8 of T=32 for Huginn). Tokens with no matching draft are produced by full-depth verification alone. Suffix-tree drafts supply 38-43% of the tokens and loop- d1 / d2 drafts 55-57%, the shares at B=1 and B=16 differ by at most 1.2 percentage points. Labels give shares of at least 10%.
Figure 11: Draft sources of the output tokens per SpecBench task at B=16 . Same runs and definitions as Figure 10 . Suffix-tree drafts dominate summarization and RAG (50-61%), while translation relies on loop- d1 / d2 drafts, including 25-30% from loop- d2 drafts, and has the most tokens with no matching draft (6-19%). Labels give shares of at least 8%.
Looped Transformers improve parameter efficiency by repeatedly applying the same Transformer layers. However, generating each token still involves sequential computation and repeated access to model weights, leading to high decoding latency. Recent pipelined self-speculative decoding methods reduce this delay by drafting tokens from early-loop predictions and asynchronously advancing tokens at different recurrent depths in the same batch, but they do not fully exploit early-loop features for longer lookahead and still spend computation verifying incorrect speculative branches. We introduce \method, a speculative decoding framework for looped transformers that enables earlier drafting of future tokens and assigns complementary roles to different recurrent depths. A lightweight lookahead drafter (LAD) predicts multiple future tokens with early-loop states, and deeper-loop feedback helps decide which draft branches to continue and which to pause, while the full-depth model makes the final token decisions. LoopAhead achieves up to 7.82x decoding speedup over autoregressive decoding while preserving the target model's output distribution.
Looped Transformers achieve strong performance with compact parameter sizes by repeatedly applying a shared stack of Transformer blocks across recurrent depths. However, they incur higher decoding latency than standard Transformer models of comparable parameter size because shared weights are accessed at every recurrent depth. To improve decoding efficiency, self-speculative decoding is particularly well suited to Looped Transformers, as their intermediate recurrent states can directly provide draft predictions without an auxiliary draft model. We therefore propose LoopSpec, a training-free self-speculative decoding framework tailored for Looped Transformers. LoopSpec extracts draft tokens from early recurrent states and operates in a pipelined manner, overlapping draft generation of future tokens with target verification of the current token. To improve draft accuracy without excessive compute overhead, we introduce a selective second proposal from deeper recurrent depth while ensuring lossless decoding under both greedy and sampling regimes. Furthermore, we derive the optimal proposal depths in closed form and show the prediction matches measurement. Across reasoning and coding benchmarks, LoopSpec achieves up to 6.83× inference speedup across diverse Looped Transformers.
Speculative decoding accelerates autoregressive generation in large language models. In each drafting stage, a lightweight draft model proposes tokens that the target model subsequently verifies. With increasingly capable draft models, we find that the target model frequently accepts all tokens produced in a drafting stage. A verification nevertheless follows each drafting stage, resulting in unnecessary target-model forward passes even when drafting could have continued. Adaptive draft length methods decide during decoding how many draft tokens precede a verification, but they raise the speedup only for autoregressive draft models. For a parallel draft model, drafting further requires target-model hidden states for draft tokens that have not been verified. We propose DLoop, a looped form of speculative decoding that adaptively performs multiple drafting stages before verification. DLoop continues drafting while the draft model remains confident and verifies all accumulated draft tokens together. Loop-aware training keeps the draft model reliable in the additional drafting stages by exposing it to its own hidden states for unverified draft tokens. By spending additional draft-model forward passes, DLoop reduces the number of target-model forward passes required for verification. Across diverse speculative decoding methods including EAGLE-3, DFlash, Domino, DSpark, and multi-token prediction modules, DLoop improves the wall-clock speedup by 5 to 41 percent while preserving lossless decoding. Code will be available at https://github.com/naver-ai/DLoop.
Geonmo Gu, Byeongho Heo, HeeJae Jun +4
NAVER AI Lab · Korea University · NAVER AI Search Platform