Looped language models repeatedly apply a weight-shared block to increase effective depth without increasing parameter count, but the resulting T sequential recurrent-block calls per generated token substantially increase decoding latency. To address the issue, we introduce Wavefront Decoding (WFD), a training-free self-speculative decoding framework designed for looped language models. WFD exploits two properties of these architectures: intermediate recurrence outputs provide effective draft predictions, and weight sharing allows token states at different positions and recurrence depths to be processed in one batched recurrent-block call. WFD organizes these mixed-depth states into a diagonal wavefront, continuously drafting new positions at shallow depth while advancing earlier positions toward full-depth verification. Unlike the phase-separated draft-then-verify schedule, WFD therefore concurrently batches drafting and verification within the same recurrent calls, while rejected drafts are corrected using full-depth predictions. Across six Spec-Bench task categories, WFD achieves 2.42x speedup on Ouro-2.6B and 3.54x on Huginn-3.5B over autoregressive decoding, consistently outperforming draft-then-verify. Cross-recurrence KV sharing further reduces wavefront KV traffic and increases WFD's speedup to 4.81x on Huginn-3.5B. The code is available at https://github.com/summerbro-hhj/wavefront-decoding.
Figures & tables
Figure 1: Unified Looped LM architecture. Optional blocks/paths are drawn with dashed lines.
Figure 2: Recurrent block scheduling on the position-timestep grid when recurrence depth T=3 and draft depth Td=1 .
Task
Metric
Ouro-2.6B
Huginn-3.5B
AR
DtV
WFD (ours)
AR
DtV
WFD (ours)
Multi-turn Conversation
α
−
0.87
0.87
−
0.90
0.91
TPS
6.63
10.88
14.85
5.31
13.34
17.12
Speedup
1.00 ×
1.64 ×
2.24 ×
1.00 ×
2.51 ×
3.22 ×
Translation
α
−
0.78
0.80
−
0.97
0.97
TPS
6.63
7.05
12.70
5.31
16.85
22.23
Table 1: Throughput on Ouro-2.6B and Huginn-3.5B across Spec-Bench tasks. α denotes the acceptance rate, which is undefined ( − ) for the autoregressive baseline. TPS denotes throughput in tokens per second. Overall aggregates all tasks by total generated tokens over total wall-clock time rather than averaging per-task ratios.
Model
Method
KV share
GSM8K
MATH-500
Accuracy
α
Speedup
Accuracy
α
Speedup
Huginn -3.5B
AR
– (32 slots)
30.64
–
1.00 ×
12.60
–
1.00 ×
WFD
– (32 slots)
31.12 +0.48
0.88
2.54 ×
13.20 +0.60
0.90
2.73 ×
AR
4 slots
31.12
–
1.00 ×
13.30
–
1.00 ×
WFD
4 slots
30.59 -0.53
0.98
4.81 ×
12.80 -0.50
0.96
4.46 ×
AR
1 slot
31.28
–
1.00 ×
12.80
–
1.00 ×
Table 2: Task accuracy and inference speed with and without KV sharing. Speedup is relative to the AR baseline with the same KV-sharing configuration.
Figure 3: Speedup over AR as a function of forced acceptance rate α (1k-token prefill, 512 decode steps). Vertical dashed lines mark each model’s measured α from Table 1 .
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Wavefront state at each timestep under WFD when recurrence depth T=3 and draft depth Td=1 . The wavefront grows during ramp-up ( TS=0,1,2 ) and reaches steady state ( TS=3,4,5 ); after a draft rejection at token position 4, it is flushed and regrows ( TS=6,7 ).
Figure 5: Speedup over AR as a function of batch size with and without KV sharing ( 1,024 -token prefill, 512 decode steps; α fixed at the overall rates from Table 1 ).
Figure 6: Decoding speedup over AR as a function of prefill length with or without KV sharing (batch 1, 512 decode steps, α fixed at the overall rates from Table 1 ).
Figure 7: Speedup over AR versus measured acceptance rate for WFD on Huginn-3.5B (GSM8K with 8-shot CoT, batch 1, A6000, T=32 ), one point per draft depth Td .
Model
Method
KV sharing
Accuracy
Token-level match rate
α
Speedup
GSM8K
Huginn-3.5B
AR
– (32 slots)
28.00
–
–
1.00 ×
WFD
– (32 slots)
28.00+0.00
1.00
0.88
2.18 ×
AR
4 slots
23.00
–
–
1.00 ×
WFD
4 slots
23.00+0.00
0.93
0.98
4.24 ×
AR
1 slot
26.00
–
–
1.00 ×
Appendix
Table 3: Token-level match rate between AR and WFD under fp32 precision; token-level match rate is defined as the fraction of tokens in the generated sequence that agree with the sequence generated by AR using the same KV-sharing configuration. α denotes the acceptance rate.
NVIDIA RTX A6000
NVIDIA B200
Architecture
Ampere
Blackwell
Memory
48 GB GDDR6
180 GB HBM3e
Memory bandwidth β
768 GB/s
7.7 TB/s
Peak bf16 throughput π (dense)
154.8 TFLOP/s
2,250 TFLOP/s
Machine balance π/β
≈202 FLOP/B
≈292 FLOP/B
Appendix
Table 4: Hardware specifications for NVIDIA RTX A6000 ( NVIDIA Corporation, 2022 ) and B200 ( NVIDIA Corporation, 2025 ) . Peak throughput π is dense bf16 tensor throughput. B200 figures are per-GPU values from the HGX B200.
Ouro-2.6B
Huginn-3.5B
Architectural family
full-stack
P/R/C
Parameters
2.6B
3.5B
Hidden size d
2,048
5,280
Unique layers
48
2 / 4 / 2 (P/R/C)
Attention
MHA, 16×128
MHA, 55×96
FFN inner size (SwiGLU)
5,632
17,920
Appendix
Table 5: Model constants for Ouro-2.6B ( Zhu et al., 2025 ) and Huginn-3.5B ( Geiping et al., 2026 ) , taken from the papers and the released checkpoint configurations. The last row gives the number of layer applications per decoded token.
Model
Memory traffic
Roofline TPS
Measured TPS
Attainment
Ouro-2.6b
21.5 GB/tok
35.7 tok/s
6.62 tok/s
18 %
Huginn-3.5b
107.9 GB/tok
7.1 tok/s
5.29 tok/s
74 %
Appendix
Table 6: Roofline-predicted vs. measured autoregressive decoding TPS for Ouro-2.6B and Huginn-3.5B. Predictions assume the memory-bandwidth-bound regime at a context length of 1,024 ; measured values are the overall AR TPS from Table 1 . Attainment is the measured fraction of the roofline bound.
Component
Latency
Share
GPU idle
98.3 ms
65 %
GEMV/GEMM
33.9 ms
23 %
Miscellaneous
16.2 ms
11 %
Attention
2.1 ms
1.4 %
Appendix
Table 7: Per-token latency breakdown of autoregressive decoding on Ouro-2.6B under eager execution. The components sum to 150.5 ms per token which is 6.64 tok/s in throughput, consistent with the measured TPS in Table 6 .
Model
KV sharing
Execution
AR (tok/s)
WFD (tok/s)
Ouro 2.6b
✗
eager mode
6.74
18.72 (2.78 × )
CUDA graph
16.72 (+148%)
38.51 (2.30 × )
Huginn 3.5b
✗
eager mode
5.20
21.23 (4.08 × )
CUDA graph
5.36 (+3%)
27.92 (5.21 × )
✓
eager mode
5.19
30.46 (5.87 × )
CUDA graph
5.36 (+3%)
33.34 (6.22 × )
Appendix
Table 8: Decoding-phase TPS of Ouro-2.6B and Huginn-3.5B with and without CUDA graph capture. To isolate the effect of kernel-launch overhead, WFD is measured at its steady-state width with static shapes ( 1,024 -token prefill, 512 decode steps, batch size 1 ). Percentages give the change in AR TPS over eager mode. Parenthesized multipliers give WFD’s speedup over the AR baseline of the same execution mode. Ouro does not admit KV sharing.
LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.
Speculative decoding accelerates autoregressive generation in large language models. In each drafting stage, a lightweight draft model proposes tokens that the target model subsequently verifies. With increasingly capable draft models, we find that the target model frequently accepts all tokens produced in a drafting stage. A verification nevertheless follows each drafting stage, resulting in unnecessary target-model forward passes even when drafting could have continued. Adaptive draft length methods decide during decoding how many draft tokens precede a verification, but they raise the speedup only for autoregressive draft models. For a parallel draft model, drafting further requires target-model hidden states for draft tokens that have not been verified. We propose DLoop, a looped form of speculative decoding that adaptively performs multiple drafting stages before verification. DLoop continues drafting while the draft model remains confident and verifies all accumulated draft tokens together. Loop-aware training keeps the draft model reliable in the additional drafting stages by exposing it to its own hidden states for unverified draft tokens. By spending additional draft-model forward passes, DLoop reduces the number of target-model forward passes required for verification. Across diverse speculative decoding methods including EAGLE-3, DFlash, Domino, DSpark, and multi-token prediction modules, DLoop improves the wall-clock speedup by 5 to 41 percent while preserving lossless decoding. Code will be available at https://github.com/naver-ai/DLoop.
Geonmo Gu, Byeongho Heo, HeeJae Jun +4
NAVER AI Lab · Korea University · NAVER AI Search Platform
Looped Transformers improve parameter efficiency by repeatedly applying the same Transformer layers. However, generating each token still involves sequential computation and repeated access to model weights, leading to high decoding latency. Recent pipelined self-speculative decoding methods reduce this delay by drafting tokens from early-loop predictions and asynchronously advancing tokens at different recurrent depths in the same batch, but they do not fully exploit early-loop features for longer lookahead and still spend computation verifying incorrect speculative branches. We introduce \method, a speculative decoding framework for looped transformers that enables earlier drafting of future tokens and assigns complementary roles to different recurrent depths. A lightweight lookahead drafter (LAD) predicts multiple future tokens with early-loop states, and deeper-loop feedback helps decide which draft branches to continue and which to pause, while the full-depth model makes the final token decisions. LoopAhead achieves up to 7.82x decoding speedup over autoregressive decoding while preserving the target model's output distribution.
Looped Transformers achieve strong performance with compact parameter sizes by repeatedly applying a shared stack of Transformer blocks across recurrent depths. However, they incur higher decoding latency than standard Transformer models of comparable parameter size because shared weights are accessed at every recurrent depth. To improve decoding efficiency, self-speculative decoding is particularly well suited to Looped Transformers, as their intermediate recurrent states can directly provide draft predictions without an auxiliary draft model. We therefore propose LoopSpec, a training-free self-speculative decoding framework tailored for Looped Transformers. LoopSpec extracts draft tokens from early recurrent states and operates in a pipelined manner, overlapping draft generation of future tokens with target verification of the current token. To improve draft accuracy without excessive compute overhead, we introduce a selective second proposal from deeper recurrent depth while ensuring lossless decoding under both greedy and sampling regimes. Furthermore, we derive the optimal proposal depths in closed form and show the prediction matches measurement. Across reasoning and coding benchmarks, LoopSpec achieves up to 6.83× inference speedup across diverse Looped Transformers.