Looped Transformers improve parameter efficiency by repeatedly applying the same Transformer layers. During autoregressive decoding, however, generating each token requires several sequential passes through these layers, leading to high decoding latency. Pipelined self-speculative methods use shallow-loop predictions to process tokens at different loops in the same batch, but each token must reach the drafting depth before the next can enter the pipeline. We introduce LoopFill, a training-free speculative decoding algorithm for looped LLM that combines multi-token suffix drafts with shallow-loop predictions in a single draft tree. Both drafters contribute to one tree, so a suffix lookup can immediately extend a shallow prediction and several future tokens can enter the pipeline together. The shared block computes the highest-scoring draft candidates at different loop depths in one batch, and full-depth verification preserves the model's autoregressive output distribution. Across seven benchmarks, LoopFill's average speedup over greedy autoregressive decoding is 5.85× on Ouro-2.6B and 5.13× on Huginn-3.5B. The speedups exceed those of LoopAhead, the strongest baseline, by 40% on Ouro-2.6B and 25% on Huginn-3.5B.
Figures & tables
Figure 1: Loop- d and suffix-tree drafts on Ouro-2.6B. (a) Blue: how often the loop- d draft equals the full-depth output. Orange: additional loop 2 hits where loop 1 misses, and additional loop 3 hits where loops 1 and 2 both miss. (b) Mean length of the correct draft for suffix drafts alone, adding loop-1 draft (+Shallow), and also adding suffix draft after loop-1 draft (+Requery)
Method
Data
Model
Multi-token
Training-free
Data-driven drafting
SuffixDecoding ( Oliaro et al., 2025 )
3
7
3
3
+ EAGLE-3 ( Li et al., 2025 )
3
3
3
7
Model-driven drafting
LoopSpec ( Cho et al., 2026 )
7
3
7
3
WaveFront ( Ha and Kim, 2026 )
7
3
7
3
Table 1: Comparison with existing speculative decoding methods from four perspectives . LoopFill combines multi-token suffix drafts with shallow-loop draft from the target model in a draft tree without training.
Figure 2: (a) Looped LLM architecture. In Ouro ( Zhu et al., 2026 ) , the entire Transformer stack is recurrent, with the embedding and LM head outside the loop. Huginn ( Geiping et al., 2025a ) places the recurrent block between prelude and coda blocks with Transformer layers. (b) Comparing four decoding algorithms (3 loops). Row ok is the position that predicts the k -th output token, with its input token on the left, and each column is one loop step. Autoregressive decoding starts a token after the previous one finishes. LoopSpec starts new token after one loop step, but the wrong draft (e.g. the ) forces a restart. SuffixDecoding verifies multi-token drafts but waits for all three loops in every round. LoopFill (Section 3 ) takes 5 loop steps to finish, achieving 3× speedup.
Figure 3: LoopFill decoding. (a) Suffix retrieval extends the shallow draft to . (b) After step 1, a falls outside the shallow top-2 after like , so both a and its child coffee have a path score of zero. (c) The tree after step 3. Dots show how many loops each node has completed toward predicting the token after its prefix. The numbered badges mark the loop step at which each output token is appended.
Model
Method
B=1
B=2
B=4
B=8
B=16
Huginn-3.5B
AR
5.89 / 1.00 ×
11.11 / 1.00 ×
19.00 / 1.00 ×
32.99 / 1.00 ×
48.34 / 1.00 ×
WaveFront
14.92 / 2.53 ×
24.56 / 2.21 ×
39.76 / 2.09 ×
60.81 / 1.84 ×
80.44 / 1.66 ×
LoopSpec
19.65 / 3.34 ×
33.89 / 3.05 ×
51.03 / 2.69 ×
74.24 / 2.25 ×
89.65 / 1.85 ×
SuffixDecoding
9.45 / 1.60 ×
17.34 / 1.56 ×
30.17 / 1.59 ×
52.01 / 1.58 ×
74.45 / 1.54 ×
DFlash
18.36 / 3.12 ×
33.18 / 2.99 ×
57.47 / 3.03 ×
88.18 / 2.67 ×
117.52 / 2.43 ×
LoopAhead
23.95 / 4.07 ×
35.82 / 3.22 ×
52.21 / 2.75 ×
76.12 / 2.31 ×
100.24 / 2.07 ×
Table 3: SpecBench end-to-end output throughput at different concurrency levels at decoding temperature 1.0 . Each cell gives output tokens/s and the throughput ratio to AR at the same concurrency. Best results are highlighted in bold.
Figure 4: Ablations of LoopFill on Huginn-3.5B, SpecBench, B=16 . Each axis is one SpecBench task, including RAG, Multi-turn Conversation (MT-Bench), Translation (Trans.), Summarization(Sum.), QA and Math. The radial value is the throughput (tok/s). Here d1 and d2 are loops 4 and 8 of T=32 .
Figure 5: Effect of the per-request draft-node budget N at two concurrency levels. The left and middle panels show the percentage change from N=16 in decoding rate (output tokens/s, B=1 ) and end-to-end throughput ( B=16 ), respectively. The right panel shows γ (solid) and loop passes per output token (dashed), divided by its value at N=16,B=16 .
Figure 6: Accepted tokens are mostly precomputed. (a) Cumulative distribution of r/T , where r is the number of loops already completed for the prefix ending in the emitted token. (b) Share of decoding loop steps attributed to First token : nothing can be drafted before decoding starts. Drafted too late : the correct token was added to the draft tree too late to finish in time. Drafted, not computed : it was drafted early enough but was pruned.
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
Evaluation-set size
Domain
GSM8K ( Cobbe et al., 2021 )
1,319
math
MATH-500 ( Hendrycks et al., 2021 )
500
math
BBH ( Suzgun et al., 2022 )
6,511
reasoning
HumanEval+ ( Liu et al., 2023 )
164
code
MBPP+ ( Liu et al., 2023 )
378
code
Spider ( Yu et al., 2019 )
1034
agentic SQL
Appendix
Table 4: Evaluation-set sizes of the seven benchmarks.
Method
Source
Number of samples
LoopAhead
GSM8K ( Cobbe et al., 2021 )
2,000
CodeAlpaca ( Chaudhary, 2023 )
2,000
MATH ( Hendrycks et al., 2021 )
2,000
Nemotron V1 STEM ( Nathawani et al., 2025 )
2,025
UltraChat ( Ding et al., 2023 )
1,000
Total
9,025
Appendix
Table 5: Dataset for training. Deduplicated against the evaluation set.
Setting
LoopAhead
DFlash
Epochs
3
6
Batch size
64
32
Optimizer
AdamW
AdamW
Adam (β1,β2)
(0.9,0.95)
(0.9,0.999)
Weight decay
0.01
0
Peak learning rate
2×10−4
6×10−4
Appendix
Table 6: Training settings for LoopAhead and DFlash
Figure 7: Ablations of LoopFill on Ouro-2.6B, SpecBench, B=16 . Here d1 and d2 are loops 1 and 2 of T=4 .
Figure 8: Loop passes vs. loop-normalized mean accepted length. The ablation configurations of each model at B=16 . The x -axis is the loop passes per token, the y -axis is the loop-normalized mean accepted length γ . The blue line changes the node budget N with everything else default (orange diamond: N=16 , loop- d1 & d2 drafts, Top-2, suffix budget Q=4 ). Green triangles change Top- k (1, 4) or Q (2, 8). Brown squares use loop- d1 drafts only (Top-1/2/4 at Q=4 , Q=2/8 at Top-2). Using loop- d1 drafts only lowers γ by 29-32% on Huginn-3.5B and 12-16% on Ouro-2.6B while increasing the compute per token by 30-38% and 3-6%.
Figure 9: Sensitivity of γ to suffix weight c . Change in the loop-normalized mean accepted length γ relative to c=1 . B=1 , N=16 , greedy. γ stays within 2% of c=1 for c∈[0.25,1000] on both Ouro models and within 4% for c∈[0.05,1000] on Huginn-3.5B. It drops only when suffix-tree drafts are almost ignored: by 5-12% for Ouro at c≤0.05 and by 5.6% for Huginn-3.5B at c=0.001 . The suffix-tree draft score min(1,cq) saturates for large c , so c=4 and c=1000 behave alike.
Model
Method
B=1
B=2
B=4
B=8
B=16
Ouro-1.4B
AR
10.77 / 1.00 ×
20.75 / 1.00 ×
40.06 / 1.00 ×
76.25 / 1.00 ×
135.65 / 1.00 ×
WaveFront
26.86 / 2.49 ×
51.00 / 2.46 ×
94.74 / 2.36 ×
168.89 / 2.21 ×
272.37 / 2.01 ×
LoopSpec
29.77 / 2.76 ×
56.83 / 2.74 ×
105.23 / 2.63 ×
182.51 / 2.39 ×
287.90 / 2.12 ×
SuffixDecoding
17.55 / 1.63 ×
31.54 / 1.52 ×
61.70 / 1.54 ×
113.74 / 1.49 ×
193.24 / 1.42 ×
DFlash
31.81 / 2.95 ×
59.24 / 2.85 ×
110.13 / 2.75 ×
193.01 / 2.53 ×
289.28 / 2.13 ×
LoopAhead
37.48 / 3.48 ×
71.09 / 3.43 ×
124.88 / 3.12 ×
195.35 / 2.56 ×
279.82 / 2.06 ×
Appendix
Table 7: SpecBench end-to-end output throughput at different concurrency levels for Ouro-1.4B at decoding temperature 1.0. Each cell gives output tokens/s and the throughput ratio to AR at the same concurrency. Best results are highlighted in bold.
Figure 10: Which draft each output token came from. LoopFill on SpecBench. Each bar splits the decode tokens of one model, excluding the token produced by prefill. A token is credited to the draft node it matched when it was verified: a suffix-tree draft, a loop- d1 draft of rank 1 or 2 in the next-token distribution after d1 loops, or a loop- d2 draft first added after d2 loops ( d1 and d2 are loops 1 and 2 of T=4 for Ouro and loops 4 and 8 of T=32 for Huginn). Tokens with no matching draft are produced by full-depth verification alone. Suffix-tree drafts supply 38-43% of the tokens and loop- d1 / d2 drafts 55-57%, the shares at B=1 and B=16 differ by at most 1.2 percentage points. Labels give shares of at least 10%.
Figure 11: Draft sources of the output tokens per SpecBench task at B=16 . Same runs and definitions as Figure 10 . Suffix-tree drafts dominate summarization and RAG (50-61%), while translation relies on loop- d1 / d2 drafts, including 25-30% from loop- d2 drafts, and has the most tokens with no matching draft (6-19%). Labels give shares of at least 8%.