Looped language models such as Ouro and Raven apply one weight-shared block several times per token, so autoregressive decoding (AR) runs every loop of every token in sequence. Pipelined self-speculative decoders overlap these loops by drafting tokens from shallow readouts and verifying them at full depth. Yet every loop of a looped LM ends in a readout of the next token, and draft tokens pass through every depth anyway, so each depth can check them without an extra pass. We introduce LoopSkim, in which the depth of the pipeline is the verifier. Draft tokens copied from the context, as in prompt lookup decoding, enter after the first loops, any later readout that contradicts one replaces it at that depth, and only full-depth readouts commit tokens. LoopSkim fixes the random draws of every position in advance, so each check compares two tokens and decoding stays lossless in exact arithmetic under sampling for any number of checks and draft chains. Without training, its speedup over AR averages 1.73x that of LoopSpec across greedy decoding and sampling, and its checks between the first loops and full depth speed it up on every model. An optional draft head adds a second chain and raises this average to 1.99x.
Figures & tables
Figure 1: (a) A greedy decode of Ouro-1.4B ( R=4 ) with both draft chains: 8 tokens take 7 one-loop steps, against 8×4=32 loops for AR. (b) In step 4, the n-gram chain has more admitted tokens than the head’s and is kept. (c) At T>0 , all readouts at position i add the same Gumbel noise g , so each check compares two tokens.
Figure 2: Geometric mean over a model’s tasks, whiskers: range over three repeats. (a, b) Ratio of speedups over LoopSpec on (a) the main tasks and (b) four held-out datasets, two on Raven-TinyLlama ( Section C.1 ). (c) Throughput over LoopSpec’s schedule run in our engine ( Section C.3 ). Numbers in (a–c): LoopSkim. (d) Loops per output token relative to AR ( D ). LoopAhead: as reported in its Figure 3a ( Section C.5 ).
Model
Method
GSM8K
MATH500
BBH
HumanEval+
MBPP+
Geo. mean
Greedy decoding
LoopSpec
2.88 ± 0.07
3.13 ± 0.06
2.78 ± 0.06
3.19 ± 0.15
2.87 ± 0.02
2.97 ± 0.03
CopyLoop
3.39 ± 0.20 (1.18)
3.31 ± 0.11 (1.06)
3.23 ± 0.12 (1.16)
5.49 ± 0.18 (1.72)
2.74 ± 0.10 (0.95)
3.53 ± 0.08 (1.19)
LoopSkim
3.81 ± 0.02 (1.32)
3.79 ± 0.14 (1.21)
5.44 ± 0.17 (1.96)
6.90 ± 0.22 (2.16)
3.18 ± 0.02 (1.11)
4.44 ± 0.06 (1.49)
Ouro- 1.4B
+ head
5.01 ± 0.02 (1.74)
5.24 ± 0.20 (1.67)
5.20 ± 0.13 (1.87)
8.57 ± 0.16 (2.69)
4.38 ± 0.12 (1.53)
5.52 ± 0.08 (1.86)
LoopSpec
3.27 ± 0.04
3.39 ± 0.04
2.94 ± 0.11
3.23 ± 0.04
3.17 ± 0.14
3.20 ± 0.04
Table 1: Speedup ( × ) over AR in each method’s engine, mean ± standard deviation of three repeats ( Section A.4 ). Parentheses: ratio of speedups over LoopSpec. Bold: fastest per model and column. –: not run, as in LoopSpec. LoopSkim: no training. BBH: heads trained without BBH questions.
Figure 3: (a) Throughput without one component relative to with it, geometric mean over the five main tasks. Intermediate checks: greedy, five models ( Table 7 ). Others: Ouro-1.4B and Raven-Llama. Circles: Ouro. Triangles: Raven. (b) Speedup of LoopSkim + head over our AR, geometric mean over GSM8K, MATH500 and MBPP+ ( Section C.9 ). (c) Ratio of speedups over LoopSpec ( Section C.11 ). Numbers: LoopSkim. Whiskers: range over three repeats.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Data
Prompts
Output limit
Subset
Prompt format
GSM8K ( Cobbe et al., 2021 )
1,319
512
test set
LoopSpec’s
MATH500 ( Hendrycks et al., 2021 ; Lightman et al., 2024 )
500
2,048
all
LoopSpec’s
BBH ( Suzgun et al., 2023 )
540
1,024
first 20 per subtask
LoopSpec’s
HE+ ( Liu et al., 2023 ; Chen et al., 2021 )
164
1,024
all
LoopSpec’s
MBPP+ ( Liu et al., 2023 ; Austin et al., 2021 )
378
1,024
all
LoopSpec’s
SVAMP ( Patel et al., 2021 )
1,000
512
all, train and test
as GSM8K
Appendix
Table 2: Main tasks (top), held-out datasets and other data (middle, Sections C.1 and C.9 ) and the thinking tasks (bottom, Section C.2 ). Prompts: count after filtering. Output limit: maximum number of new tokens. Held-out samples are drawn with seed 24, and every prompt stops at the stop strings of its task.
Setting
Ouro
Raven
Draft tokens per chain G
8
8
Stride d
1
greedy: 2, T>0 : 4
Check depths Λ
1, 2, 3, 4
greedy: 2, 4, 8, 16, 32 T>0 : 4, 8, 16, 32
Tokens per pass
36 (45)
greedy: 144 (153) T>0 : 72 (81)
Head draws, T>0
at 0.7×T
at T
Pruning margin, T>0
3 nats
–
Appendix
Table 3: Default settings. Tokens per pass: (C+R/d−1)(1+G) , with the draft head in parentheses. Pruning margin: readouts below full depth drop tokens more than 3 nats below their top token.
Checks below R
Method
Draft source
Draft chains
Draft tokens / step
Ouro
Raven
On contra- diction
Sampling
LoopSpec
readout
1
1
–
–
–
✓
WaveFront
readout
1
1
–
–
–
✗
DAS-Wave
readout
1
≤ 24
–
–
–
✗
LoopAhead
drafter ∗
tree
≤ 3
2
4, 8 †
paused
✓
CopyLoop
context
1
K
1
4
replaced
✗
Appendix
Table 4: Self-speculative decoders for looped LMs. Checks below R : depths, in loops, at which a check can stop a draft token ( R=4 on Ouro, 32 on Raven). Sampling: evaluated with lossless sampling (✗: not evaluated). –: none. ∗ Trained. † 2, 8 on Raven-TinyLlama. ‡ Plus an optional draft head. § 4, 8, 16 under sampling.
Over LoopSpec
Over CopyLoop
Held-out datasets
Held-out datasets
Model
Method
SVAMP
Olymp.
MMLU-Pro
BCB
Mean
Spec-Bench
SVAMP
Olymp.
MMLU-Pro
BCB
Mean
Spec-Bench
Greedy
Ouro-1.4B
LoopSkim
1.36
1.26
1.39
2.27
1.52
1.46
1.10
1.15
1.42
1.35
1.25
1.18
LoopSkim + head
1.71
1.68
1.44
2.41
1.78
1.49
1.38
1.53
1.47
1.43
1.45
1.20
Ouro-2.6B
LoopSkim
1.36
1.84
1.26
2.13
1.61
1.46
0.99
1.23
1.12
1.21
1.13
1.22
Appendix
Table 5: Ratio of speedups, mean of three repeats. Mean: geometric mean over the held-out datasets. Spec-Bench: geometric mean over its six categories. Olymp.: OlympiadBench. BCB: BigCodeBench.
Model
Method
GSM8K
MATH500
BBH
HumanEval+
MBPP+
Geo. mean
Greedy decoding
LoopSpec
2.88 ± 0.07
3.12 ± 0.06
2.78 ± 0.06
3.19 ± 0.15
2.86 ± 0.01
2.96 ± 0.03
CopyLoop
3.38 ± 0.20 (1.17)
3.22 ± 0.11 (1.03)
3.20 ± 0.12 (1.15)
5.49 ± 0.18 (1.72)
2.68 ± 0.10 (0.94)
3.48 ± 0.08 (1.18)
LoopSkim
3.80 ± 0.02 (1.32)
3.73 ± 0.14 (1.20)
5.43 ± 0.17 (1.95)
6.90 ± 0.22 (2.16)
3.10 ± 0.02 (1.08)
4.40 ± 0.06 (1.49)
Ouro- 1.4B
+ head
5.01 ± 0.02 (1.74)
5.16 ± 0.20 (1.65)
5.19 ± 0.13 (1.87)
8.57 ± 0.16 (2.69)
4.31 ± 0.12 (1.51)
5.48 ± 0.08 (1.85)
LoopSpec
3.27 ± 0.04
3.27 ± 0.05
2.94 ± 0.11
3.23 ± 0.04
3.17 ± 0.14
3.17 ± 0.04
Appendix
Table 6: As Table 1 , without the prompts on which the greedy output of any of its decoders falls into a repetition loop in any repeat. The same prompts are left out at T=1 .
Greedy
T=1 , top-p 0.9
Ouro- 1.4B
Ouro- 2.6B
Raven- Llama
Raven- OLMo
Raven- TinyLlama
Ouro- 1.4B
Raven- Llama
Raw decode throughput (tok/s) in Table 1
SGLang’s AR
141.9
70.8
68.5
42.3
62.8
144.1
68.8
Our AR
164.2
83.3
67.1
60.3
58.9
162.0
66.1
LoopSpec
420.5
226.5
380.3
222.4
373.5
345.1
268.6
LoopSkim
728.8
433.6
692.6
700.0
648.2
608.5
416.5
Appendix
Table 7: Raw throughput in Table 1 (top) and the controls of Section C.3 . Geometric mean over tasks of the per-cell means over three repeats. Ratios in our engine: per repeat, the geometric mean over tasks of the per-cell throughput ratio, then the mean over repeats. ∗ Raven, greedy: stride 4 without the checks, 2 with them.
Admis- sion check: rejected (%)
Inter- mediate depths: replaced
Full depth: replaced
Below full depth (%)
Full depth: rate (%)
Mean depth
Greedy
Ouro-1.4B
33.6
9,493
1,247
88.4
1.10
2.5
Ouro-2.6B
25.3
5,759
719
88.9
0.69
2.5
Raven-Llama
19.9
6,897
207
97.1
0.16
6.7
Raven-OLMo
18.8
13,582
387
97.2
0.18
6.4
Raven-TinyLlama
22.4
2,733
72
97.4
0.14
6.6
Appendix
Table 8: Replacements per check depth of LoopSkim on the prompts of Section C.3 , pooled over each model’s tasks. Rejected: share of the draft tokens checked. Below full depth: share of the later replacements made at an intermediate depth. Rate: full-depth replacements per decoded token. Mean depth: mean check depth, in loops, of the later replacements ( R=4 on Ouro, 32 on Raven). All: pooled over models. All + head: the same for LoopSkim + head. T=1 : seed 0.
Greedy
T=1 , top-p 0.9
Ouro- 1.4B
Ouro- 2.6B
Raven- Llama
Raven- OLMo
Raven- TinyLlama
Ouro- 1.4B
Raven- Llama
Loops per output token, D
LoopSpec
0.2853
0.2725
0.0773
0.0954
0.0840
0.3032
0.1291
CopyLoop
0.1984
0.1475
0.0633
0.0619
0.0815
–
–
LoopSkim
0.1833
0.1498
0.0395
0.0380
0.0511
0.2151
0.0961
LoopSkim + head
0.1299
0.1078
0.0321
0.0310
0.0399
0.1628
0.0769
Appendix
Table 9: The factors of Equation 1 on the prompts of Table 1 , pooled over tasks, and the loops over all token positions ( Section C.5 ). Last block: LoopAhead’s Figures 3a and 3b (greedy). LoopSpec’s time per loop: its official code over SGLang’s AR. Bold: lowest per column in each block but τ .
Ouro-1.4B
Raven-Llama
Removed component
Greedy
T=1
Greedy
T=1
Multi-token admission ( G=0 )
0.521 (0.512–0.526)
0.624 (0.618–0.633)
0.585 (0.582–0.589)
0.681 (0.667–0.693)
Admission check ∗
0.606 (0.603–0.608)
0.629 (0.621–0.638)
0.582 (0.577–0.586)
0.559 (0.543–0.583)
N-gram chain
0.731 (0.726–0.741)
0.793 (0.787–0.803)
0.681 (0.675–0.692)
0.735 (0.715–0.757)
Redrafting after a replacement
0.984 (0.972–0.994)
0.973 (0.968–0.981)
0.982 (0.980–0.986)
0.981 (0.975–0.991)
LoopSkim
Appendix
Table 10: Throughput with one component removed, relative to with it, for LoopSkim + head (top) and LoopSkim (bottom). Geometric mean over tasks, mean of three repeats (range). T=1 : seed 0. ∗ Head chain alone, relative to it with the admission check. † Raven, greedy: stride 4 without the checks, 2 with them. ‡ Stride 2 in both arms. § Noise below full depth drawn independently of full depth’s.
Mode
R
LoopSkim + head / our AR
LoopSpec / our AR
LoopSkim + head / LoopSpec
Greedy
8
4.52
2.36
1.92
16
6.97
3.67
1.90
32
10.91
5.88
1.86
T=1
8
2.52
1.59
1.58
16
4.06
2.43
1.67
32
6.97
3.75
1.86
Appendix
Table 11: Raven-Llama run with R loops. Speedup over our AR of LoopSkim + head and of LoopSpec’s schedule in our engine, and their ratio. Geometric mean over tasks, mean of three repeats.
Our AR
LoopSkim + head
Model
GSM8K
MATH500
BBH
HE+
MBPP+
GSM8K
MATH500
BBH
HE+
MBPP+
Ouro-1.4B
78.5
72.6
69.6
65.9
64.8
79.1
73.2
70.7
66.5
64.8
Ouro-2.6B
86.0
57.4
77.8
73.2
67.7
86.2
56.0
77.4
72.6
69.0
Raven-Llama
51.9
35.4
32.8
11.0
23.8
52.0
35.8
31.5
9.8
24.6
Raven-OLMo
46.1
28.0
28.0
10.4
12.2
45.3
27.6
28.7
11.0
11.6
Raven-TinyLlama
43.7
–
–
14.0
23.0
43.1
–
–
14.0
23.8
Appendix
Table 12: Accuracy (%), each decoder run to its end token or the output limit. T=1 : mean over three seeds.
Greedy
T=1 , top-p 0.9
Ouro- 1.4B
Ouro- 2.6B
Raven- Llama
Raven- OLMo
Raven- TinyLlama
Ouro- 1.4B
Raven- Llama
LoopSpec
2.59
2.75
5.21
5.03
6.07
2.26
4.05
CopyLoop
4.00 (1.54)
4.81 (1.75)
8.28 (1.59)
8.75 (1.74)
8.76 (1.44)
–
–
LoopSkim
4.78 (1.85)
5.64 (2.05)
7.96 (1.53)
10.69 (2.13)
9.35 (1.54)
3.94 (1.74)
5.49 (1.36)
+ head
6.39 (2.47)
7.65 (2.78)
9.17 (1.76)
13.17 (2.62)
11.17 (1.84)
4.91 (2.17)
6.34 (1.57)
LoopSpec in our engine
2.77
2.91
5.22
5.47
6.19
2.35
3.84
Appendix
Table 13: Speedup ( × ) over AR in each method’s engine on an RTX 5090, 100 prompts per cell, mean of three repeats. Parentheses: ratio of speedups over LoopSpec. Bold: fastest per column.
Budgets of 4,096–16,384 tokens
Budget of 32,768 tokens
GSM8K
MATH500
AIME24
AIME25
AIME24
AIME25
MATH500
T=1 , top-p 0.7
LoopSpec / SGLang AR
2.25
1.95
1.60
1.53
1.58
1.53
1.80
LoopSkim/ our AR
3.04
3.20
2.99
2.96
3.05
3.15
3.46
LoopSkim + head/ our AR
3.40
3.63
3.15
3.10
3.31
3.41
4.01
LoopSkim/ LoopSpec
1.35
1.64
1.87
1.93
1.93
2.06
1.92
Appendix
Table 14: Ouro-1.4B-Thinking on LoopSpec’s thinking tasks. Speedup over own AR, mean of three repeats, and ratios of speedups, each the quotient of the two speedups it names. LoopSkim + head: the Ouro-1.4B draft head. Bold: largest speedup and largest ratio over LoopSpec per column and block. D : loops per output token relative to AR.
GSM8K
MATH- 500
BBH
Human- Eval+
MBPP+
Geo. mean
All prompts
LoopSkim
4.40
10.55
6.16
9.98
7.90
7.42
LoopSkim + head
5.56
11.55
6.57
10.24
8.80
8.24
CopyLoop
4.14
6.46
4.83
7.61
6.34
5.74
LoopSpec
3.28
3.74
3.28
4.39
4.14
3.74
Without repetition loops
Appendix
Table 15: Huginn-0125 under greedy decoding. Speedup ( × ) over AR in each method’s engine, mean of three repeats. Geometric mean over tasks per repeat, then the mean. Lower block: without the prompts on which any method’s output falls into a repetition loop. Bold: fastest per column and block.
1K
2K
4K
8K
16K
32K
Geo. mean
Over LoopSpec
Ouro-1.4B
1.44
1.60
1.62
1.65
1.51
1.30
1.51
Ouro-2.6B
1.33
1.39
1.38
1.44
1.54
1.33
1.40
Raven-Llama
1.66
1.85
1.91
1.87
2.08
1.77
1.85
Over CopyLoop
Ouro-1.4B
1.38
1.67
1.79
1.91
1.75
1.50
1.66
Appendix
Table 16: Ratio of speedups of LoopSkim + head over each baseline on PG-19 contexts, greedy decoding with the end token ignored, mean of three repeats. Each speedup is over AR in the method’s own engine, and each baseline runs its fastest configuration per length. Geo. mean: over lengths per repeat, then the mean.
Speculative decoding accelerates autoregressive generation in large language models. In each drafting stage, a lightweight draft model proposes tokens that the target model subsequently verifies. With increasingly capable draft models, we find that the target model frequently accepts all tokens produced in a drafting stage. A verification nevertheless follows each drafting stage, resulting in unnecessary target-model forward passes even when drafting could have continued. Adaptive draft length methods decide during decoding how many draft tokens precede a verification, but they raise the speedup only for autoregressive draft models. For a parallel draft model, drafting further requires target-model hidden states for draft tokens that have not been verified. We propose DLoop, a looped form of speculative decoding that adaptively performs multiple drafting stages before verification. DLoop continues drafting while the draft model remains confident and verifies all accumulated draft tokens together. Loop-aware training keeps the draft model reliable in the additional drafting stages by exposing it to its own hidden states for unverified draft tokens. By spending additional draft-model forward passes, DLoop reduces the number of target-model forward passes required for verification. Across diverse speculative decoding methods including EAGLE-3, DFlash, Domino, DSpark, and multi-token prediction modules, DLoop improves the wall-clock speedup by 5 to 41 percent while preserving lossless decoding. Code will be available at https://github.com/naver-ai/DLoop.
Geonmo Gu, Byeongho Heo, HeeJae Jun +4
NAVER AI Lab · Korea University · NAVER AI Search Platform
Looped Transformers improve parameter efficiency by repeatedly applying the same Transformer layers. During autoregressive decoding, however, generating each token requires several sequential passes through these layers, leading to high decoding latency. Pipelined self-speculative methods use shallow-loop predictions to process tokens at different loops in the same batch, but each token must reach the drafting depth before the next can enter the pipeline. We introduce LoopFill, a training-free speculative decoding algorithm for looped LLM that combines multi-token suffix drafts with shallow-loop predictions in a single draft tree. Both drafters contribute to one tree, so a suffix lookup can immediately extend a shallow prediction and several future tokens can enter the pipeline together. The shared block computes the highest-scoring draft candidates at different loop depths in one batch, and full-depth verification preserves the model's autoregressive output distribution. Across seven benchmarks, LoopFill's average speedup over greedy autoregressive decoding is 5.85× on Ouro-2.6B and 5.13× on Huginn-3.5B. The speedups exceed those of LoopAhead, the strongest baseline, by 40% on Ouro-2.6B and 25% on Huginn-3.5B.
Siqi Zhu, Suozhi Huang, Yi Zhao +1
University of Illinois Urbana-Champaign · Princeton University
Looped language models repeatedly apply a weight-shared block to increase effective depth without increasing parameter count, but the resulting T sequential recurrent-block calls per generated token substantially increase decoding latency. To address the issue, we introduce Wavefront Decoding (WFD), a training-free self-speculative decoding framework designed for looped language models. WFD exploits two properties of these architectures: intermediate recurrence outputs provide effective draft predictions, and weight sharing allows token states at different positions and recurrence depths to be processed in one batched recurrent-block call. WFD organizes these mixed-depth states into a diagonal wavefront, continuously drafting new positions at shallow depth while advancing earlier positions toward full-depth verification. Unlike the phase-separated draft-then-verify schedule, WFD therefore concurrently batches drafting and verification within the same recurrent calls, while rejected drafts are corrected using full-depth predictions. Across six Spec-Bench task categories, WFD achieves 2.42x speedup on Ouro-2.6B and 3.54x on Huginn-3.5B over autoregressive decoding, consistently outperforming draft-then-verify. Cross-recurrence KV sharing further reduces wavefront KV traffic and increases WFD's speedup to 4.81x on Huginn-3.5B. The code is available at https://github.com/summerbro-hhj/wavefront-decoding.