WaveFront Decoding: Parallelized Self-Speculative Decoding for Looped Language Models

Hyeongju Ha; Jae-Joon Kim

cs.LGSep 19, 2026

WaveFront Decoding: Parallelized Self-Speculative Decoding for Looped Language Models

Authors: Hyeongju Ha, Jae-Joon Kim

Organizations: Seoul National University

Abstract

Looped language models repeatedly apply a weight-shared block to increase effective depth without increasing parameter count, but the resulting T sequential recurrent-block calls per generated token substantially increase decoding latency. To address the issue, we introduce Wavefront Decoding (WFD), a training-free self-speculative decoding framework designed for looped language models. WFD exploits two properties of these architectures: intermediate recurrence outputs provide effective draft predictions, and weight sharing allows token states at different positions and recurrence depths to be processed in one batched recurrent-block call. WFD organizes these mixed-depth states into a diagonal wavefront, continuously drafting new positions at shallow depth while advancing earlier positions toward full-depth verification. Unlike the phase-separated draft-then-verify schedule, WFD therefore concurrently batches drafting and verification within the same recurrent calls, while rejected drafts are corrected using full-depth predictions. Across six Spec-Bench task categories, WFD achieves 2.42x speedup on Ouro-2.6B and 3.54x on Huginn-3.5B over autoregressive decoding, consistently outperforming draft-then-verify. Cross-recurrence KV sharing further reduces wavefront KV traffic and increases WFD's speedup to 4.81x on Huginn-3.5B. The code is available at https://github.com/summerbro-hhj/wavefront-decoding.

Figures & tables

Figure 1: Unified Looped LM architecture. Optional blocks/paths are drawn with dashed lines.

Panel of Figure 2 — Figure 2: Recurrent block scheduling on the position-timestep grid when recurrence depth $T=3$ and draft depth $T_{d}=1$ .

Task	Metric	Ouro-2.6B			Huginn-3.5B
Task	Metric	AR	DtV	WFD (ours)	AR	DtV	WFD (ours)
Multi-turn Conversation	$\alpha$	$-$	0.87	0.87	$-$	0.90	0.91
	TPS	6.63	10.88	14.85	5.31	13.34	17.12
	Speedup	1.00 $\times$	1.64 $\times$	2.24 $\times$	1.00 $\times$	2.51 $\times$	3.22 $\times$
Translation	$\alpha$	$-$	0.78	0.80	$-$	0.97	0.97
	TPS	6.63	7.05	12.70	5.31	16.85	22.23

Table 1: Throughput on Ouro-2.6B and Huginn-3.5B across Spec-Bench tasks.

\alpha

denotes the acceptance rate, which is undefined (

-

) for the autoregressive baseline. TPS denotes throughput in tokens per second. Overall aggregates all tasks by total generated tokens over total wall-clock time rather than averaging per-task ratios.

Model	Method	KV share	GSM8K			MATH-500
Model	Method	KV share	Accuracy	$\alpha$	Speedup	Accuracy	$\alpha$	Speedup
Huginn -3.5B	AR	– (32 slots)	30.64	–	1.00 $\times$	12.60	–	1.00 $\times$
	WFD	– (32 slots)	31.12 +0.48	0.88	2.54 $\times$	13.20 +0.60	0.90	2.73 $\times$
	AR	4 slots	31.12	–	1.00 $\times$	13.30	–	1.00 $\times$
	WFD	4 slots	30.59 -0.53	0.98	4.81 $\times$	12.80 -0.50	0.96	4.46 $\times$
	AR	1 slot	31.28	–	1.00 $\times$	12.80	–	1.00 $\times$

Table 2: Task accuracy and inference speed with and without KV sharing. Speedup is relative to the AR baseline with the same KV-sharing configuration.

Figure 3: Speedup over AR as a function of forced acceptance rate

\alpha

(1k-token prefill, 512 decode steps). Vertical dashed lines mark each model’s measured

\alpha

from Table 1 .

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Wavefront state at each timestep under WFD when recurrence depth

T=3

and draft depth

T_{d}=1

. The wavefront grows during ramp-up (

\text{TS}=0,1,2

) and reaches steady state (

\text{TS}=3,4,5

); after a draft rejection at token position 4, it is flushed and regrows (

\text{TS}=6,7

Figure 5: Speedup over AR as a function of batch size with and without KV sharing (

1{,}024

-token prefill,

512

decode steps;

\alpha

fixed at the overall rates from Table 1 ).

Figure 6: Decoding speedup over AR as a function of prefill length with or without KV sharing (batch 1, 512 decode steps,

\alpha

fixed at the overall rates from Table 1 ).

Figure 7: Speedup over AR versus measured acceptance rate for WFD on Huginn-3.5B (GSM8K with 8-shot CoT, batch 1, A6000,

T=32

), one point per draft depth

T_{d}

Model	Method	KV sharing	Accuracy	Token-level match rate	$\alpha$	Speedup
GSM8K
Huginn-3.5B	AR	– (32 slots)	28.00	–	–	1.00 $\times$
	WFD	– (32 slots)	28.00+0.00	1.00	0.88	2.18 $\times$
	AR	4 slots	23.00	–	–	1.00 $\times$
	WFD	4 slots	23.00+0.00	0.93	0.98	4.24 $\times$
	AR	1 slot	26.00	–	–	1.00 $\times$

Appendix

Table 3: Token-level match rate between AR and WFD under fp32 precision; token-level match rate is defined as the fraction of tokens in the generated sequence that agree with the sequence generated by AR using the same KV-sharing configuration.

\alpha

denotes the acceptance rate.

	NVIDIA RTX A6000	NVIDIA B200
Architecture	Ampere	Blackwell
Memory	48 GB GDDR6	180 GB HBM3e
Memory bandwidth $\beta$	768 GB/s	7.7 TB/s
Peak bf16 throughput $\pi$ (dense)	154.8 TFLOP/s	2,250 TFLOP/s
Machine balance $\pi/\beta$	${\approx}\,202$ FLOP/B	${\approx}\,292$ FLOP/B

Appendix

Table 4: Hardware specifications for NVIDIA RTX A6000 ( NVIDIA Corporation, 2022 ) and B200 ( NVIDIA Corporation, 2025 ) . Peak throughput

\pi

is dense bf16 tensor throughput. B200 figures are per-GPU values from the HGX B200.

	Ouro-2.6B	Huginn-3.5B
Architectural family	full-stack	P/R/C
Parameters	2.6B	3.5B
Hidden size $d$	2,048	5,280
Unique layers	48	2 / 4 / 2 (P/R/C)
Attention	MHA, $16\times 128$	MHA, $55\times 96$
FFN inner size (SwiGLU)	5,632	17,920

Appendix

Table 5: Model constants for Ouro-2.6B ( Zhu et al., 2025 ) and Huginn-3.5B ( Geiping et al., 2026 ) , taken from the papers and the released checkpoint configurations. The last row gives the number of layer applications per decoded token.

Model	Memory traffic	Roofline TPS	Measured TPS	Attainment
Ouro-2.6b	21.5 GB/tok	35.7 tok/s	6.62 tok/s	18 %
Huginn-3.5b	107.9 GB/tok	7.1 tok/s	5.29 tok/s	74 %

Appendix

Table 6: Roofline-predicted vs. measured autoregressive decoding TPS for Ouro-2.6B and Huginn-3.5B. Predictions assume the memory-bandwidth-bound regime at a context length of

1{,}024

; measured values are the overall AR TPS from Table 1 . Attainment is the measured fraction of the roofline bound.

Component	Latency	Share
GPU idle	98.3 ms	65 %
GEMV/GEMM	33.9 ms	23 %
Miscellaneous	16.2 ms	11 %
Attention	2.1 ms	1.4 %

Appendix

Table 7: Per-token latency breakdown of autoregressive decoding on Ouro-2.6B under eager execution. The components sum to

150.5

ms per token which is

6.64

tok/s in throughput, consistent with the measured TPS in Table 6 .

Model	KV sharing	Execution	AR (tok/s)	WFD (tok/s)
Ouro 2.6b	✗	eager mode	6.74	18.72 (2.78 $\times$ )
Ouro 2.6b	✗	CUDA graph	16.72 (+148%)	38.51 (2.30 $\times$ )
Huginn 3.5b	✗	eager mode	5.20	21.23 (4.08 $\times$ )
	✗	CUDA graph	5.36 (+3%)	27.92 (5.21 $\times$ )
	✓	eager mode	5.19	30.46 (5.87 $\times$ )
	✓	CUDA graph	5.36 (+3%)	33.34 (6.22 $\times$ )

Appendix

Table 8: Decoding-phase TPS of Ouro-2.6B and Huginn-3.5B with and without CUDA graph capture. To isolate the effect of kernel-launch overhead, WFD is measured at its steady-state width with static shapes (

1{,}024

-token prefill,

512

decode steps, batch size

1

). Percentages give the change in AR TPS over eager mode. Parenthesized multipliers give WFD’s speedup over the AR baseline of the same execution mode. Ouro does not admit KV sharing.

LLM Inference Acceleration

LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.

See all 1,055 papers

DLoop: Looped Speculative Decoding
Oct 6, 2026Geonmo Gu, Byeongho Heo, HeeJae Jun +4Speculative Decoding LLM Inference Acceleration
NAVER AI Lab · Korea University · NAVER AI Search Platform
LoopAhead: Speculating Ahead for Faster Decoding in Looped Language Models
Oct 15, 2026Zixuan Liu, Nan Xue, Rui Bao +4Speculative Decoding LLM Inference Acceleration
Shanghai Jiao Tong University · Tongji University
LoopSpec: Pipelined Self-Speculative Decoding for Looped Transformers
Sep 15, 2026SangLyul Cho, Langqing Cui, Sehoon Kim +2Efficient Transformer Inference Speculative Decoding
1Seoul National University · 2KAIST

Oct 6, 2026J/K move · Enter open · S save

DLoop: Looped Speculative Decoding

Geonmo Gu, Byeongho Heo, HeeJae Jun, Yoohoon Kang+3

NAVER AI Lab · Korea University · NAVER AI Search Platform

Speculative Decoding LLM Inference Acceleration Autoregressive Decoding

Open paper AskPDF arXiv

Speculative decoding accelerates autoregressive generation in large language models. In each drafting stage, a lightweight draft model proposes tokens that the target model subsequently verifies. With increasingly capable draft models, we find that the target model frequently accepts all tokens produced in a drafting stage. A verification nevertheless follows each drafting stage, resulting in unnecessary target-model forward passes even when drafting could have continued. Adaptive draft length methods decide during decoding how many draft tokens precede a verification, but they raise the speedup only for autoregressive draft models. For a parallel draft model, drafting further requires target-model hidden states for draft tokens that have not been verified. We propose DLoop, a looped form of speculative decoding that adaptively performs multiple drafting stages before verification. DLoop continues drafting while the draft model remains confident and verifies all accumulated draft tokens together. Loop-aware training keeps the draft model reliable in the additional drafting stages by exposing it to its own hidden states for unverified draft tokens. By spending additional draft-model forward passes, DLoop reduces the number of target-model forward passes required for verification. Across diverse speculative decoding methods including EAGLE-3, DFlash, Domino, DSpark, and multi-token prediction modules, DLoop improves the wall-clock speedup by 5 to 41 percent while preserving lossless decoding. Code will be available at https://github.com/naver-ai/DLoop.

Abstract

LLM Inference Acceleration

Explore similar work

DLoop: Looped Speculative Decoding

LoopAhead: Speculating Ahead for Faster Decoding in Looped Language Models

LoopSpec: Pipelined Self-Speculative Decoding for Looped Transformers