Shallow Queries, Mature Values: Depth-Asynchronous Self-Speculation for Looped Transformers

Guanghao Li; Zihan Su; Hao Yu; Jinyang Jiang; Tao Ren; Zehao Li; Feng Lu; Ming Tang

cs.LGSep 28, 2026

Shallow Queries, Mature Values: Depth-Asynchronous Self-Speculation for Looped Transformers

Authors: Guanghao Li, Zihan Su, Hao Yu, Jinyang Jiang, Tao Ren, Zehao Li, Feng Lu, Ming Tang, +1 more

Organizations: Tsinghua University · Southern University of Science and Technology · The University of Hong Kong · Peking University · Shenzhen University of Advanced Technology

Abstract

Looped Transformers reuse a shared block across recurrent depths, making autoregressive decoding expensive because every generated token requires many sequential recurrent passes. Self-speculative decoders reduce this cost by drafting at an early depth and verifying at full depth, but typically bind draft computation to prefix representations from the same recurrent depth. We find that queries and keys approach their final-depth representations earlier than values, and controlled prefix-channel interventions show that mature values substantially improve shallow draft predictions. Motivated by this asymmetry, we introduce Depth-Asynchronous Self-Speculation (DAS), which decouples the depth of draft computation from the depth of verified-prefix representations it reads. Its Mature-V primitive lets shallow queries retrieve full-depth prefix values without additional recurrent computation. We further develop DAS-Wave, which combines depth-asynchronous prefix reads with carried parallel refinement, progressive block growth, and an independent full-depth verifier. Across four recurrent-model checkpoints and mathematics and code workloads, DAS-Wave achieves 4.00--6.96 $\times$ mean throughput speedup over paired full-depth autoregressive decoding in the same inference stack. These results identify prefix-information depth as an effective design axis for recurrent self-speculation.

Figures & tables

Figure 1: DAS-Wave overview. (a) Queries from the current draft state attend to full-depth K/V from the verified, materialized prefix at the same core layer. (b) Active positions are refined in parallel while latent states are carried across rounds and the active window grows. Gold marks newly activated states. The anchor remains fixed, and each readout updates the next candidate position. (c) An independent full-depth target verifies the candidates without using carried draft states. In the illustrated example,

y_{1}

and

y_{2}

are accepted;

y_{3}

is the first mismatch, so the target prediction

\hat{y}_{3}=\argmax_{v}p_{3}(v)

is emitted and the remaining draft suffix is discarded.

Figure 2: Shallow queries, mature values. Top: Q/K/V cosine similarity to the final recurrence (

R=32

), averaged over core layers. Queries and keys approach their final-depth representations earlier than values. Bottom: draft–target token agreement under separate prefix-channel replacements. Deep V uses full-depth prefix V while keeping K aligned with the draft; in sequential drafting, this corresponds to Mature-V. Each model uses 12 GSM8K prompts and 756 evaluated next-token prediction positions.

		GSM8K		MATH-500		HumanEval		MBPP
Model	Method	Speedup	$A$	Speedup	$A$	Speedup	$A$	Speedup	$A$
Huginn	Lookahead	1.83 $\times$	1.1	1.84 $\times$	1.0	1.50 $\times$	0.6	1.40 $\times$	0.5
	PLD	1.54 $\times$	0.7	1.93 $\times$	1.2	1.61 $\times$	0.8	1.35 $\times$	0.5
	Token Recycling	2.81 $\times$	2.1	2.81 $\times$	2.1	2.74 $\times$	2.1	2.56 $\times$	1.8
	SAM-Decoding	1.63 $\times$	0.9	2.50 $\times$	1.9	2.01 $\times$	1.3	1.49 $\times$	0.6
	Self-speculative	1.70 $\times$	5.2	1.82 $\times$	5.7	1.64 $\times$	4.6	1.79 $\times$	5.0

Table 1: End-to-end decoding performance. Speedup is mean throughput relative to paired full-depth AR;

A

is the mean number of accepted draft tokens per verification. Bold marks the largest speedup and accepted length in each setting.

			GSM8K		MATH-500
Model	Procedure	Prefix	Speedup	$A$	Speedup	$A$
Huginn	Sequential	Aligned	1.54 $\times$	2.80	1.85 $\times$	3.69
	Sequential	Deep K/V	2.26 $\times$	4.92	2.42 $\times$	5.35
	Parallel, no carry	Aligned	0.65 $\times$	1.38	1.16 $\times$	3.37
	Parallel, no carry	Deep K/V	0.93 $\times$	2.42	1.06 $\times$	2.95
	Parallel + carry	Aligned	0.71 $\times$	1.61	1.20 $\times$	3.51

Table 2: Fixed-setting component comparisons. Sequential rows use draft depth 4. All parallel rows share the same refinement schedule; no-carry rows reinitialize latent states between rounds, while carried rows retain them. Parallel + carry with Deep K/V uses the full DAS-Wave configuration. Component estimates are measured in their own paired timing sessions.

Figure 3: Refinement cost–acceptance tradeoff. GSM8K diagnostic runs with

J=24

. Top: mean throughput speedup; bottom: accepted draft length

A

. Left varies recurrent updates per round; right varies verifier capacity at the selected update count.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Checkpoint	Prelude	Shared core	Coda	Recurrences
Huginn-0125	2	4	2	32
Raven-Llama-3.2	4	6	4	32
Raven-OLMo-2	4	6	4	32
Raven-TinyLlama	4	8	4	32

Appendix

Table 3: Evaluated recurrent-depth models.

Task	Cap	Prompt convention
GSM8K	512	Eight-shot chain of thought
MATH-500	1024	Four-shot worked solutions
HumanEval	512	Code completion
MBPP	512	Code completion

Appendix

Table 4: Prompt conventions and generation caps.

Method	Huginn	Raven family	Prefix reads
Self-speculative	$d=8,b=8$	$d=4,b=16$	Aligned
Mature-Spec	$d=4,b=8$	$d=4,b=16$	Deep K/V
DAS-Wave	$\gamma=32,k=2$	$\gamma=32,k=1$	Deep K/V

Appendix

Table 5: Frozen neural-draft configurations.

Model	Prefix policy	$d=1$	$d=2$	$d=4$	$d=8$
Huginn	Aligned	0.4722	0.6362	0.7937	0.9127
	Deep V	0.6693	0.7659	0.8955	0.9484
	Deep K	0.4683	0.6045	0.8016	0.8968
	Deep K/V	0.6005	0.7937	0.8995	0.9669
Raven Llama	Aligned	0.8399	0.9034	0.9590	0.9841
	Deep V	0.8796	0.9471	0.9841	0.9947

Appendix

Table 6: Draft–target token agreement under depth-dependent prefix-channel substitution.

Figure 4: Prefix read-depth grid. Huginn draft computation is fixed at $r=2$ . Rows vary verified-prefix K depth $d_{K}$ and columns vary V depth $d_{V}$ ; cells report draft–target token agreement. The four corners $d_{K},d_{V}\in\{2,R\}$ correspond to Aligned, Deep V, Deep K, and Deep K/V. The diagnostic uses 8 prompts and 376 prediction positions, separate from Table 6 .

Model	Task	Deep V [95% CI]	$A$	Deep K/V [95% CI]	$A$
Raven-Llama	GSM8K	3.39 $\times$ [3.25, 3.51]	12.6	3.50 $\times$ [3.37, 3.62]	13.1
	MATH-500	3.54 $\times$ [3.36, 3.70]	13.4	3.70 $\times$ [3.55, 3.83]	14.0
	HumanEval	3.34 $\times$ [3.12, 3.53]	12.0	3.44 $\times$ [3.22, 3.64]	13.1
	MBPP	3.72 $\times$ [3.62, 3.81]	13.3	3.84 $\times$ [3.74, 3.92]	14.7
Raven-OLMo	GSM8K	3.50 $\times$ [3.36, 3.64]	13.0	3.63 $\times$ [3.53, 3.73]	13.5
	MATH-500	3.55 $\times$ [3.40, 3.69]	13.1	3.73 $\times$ [3.60, 3.86]	13.8

Appendix

Table 7: Sequential mature-prefix policies on the Raven checkpoints. Entries report mean speedup [95% CI] and accepted length

A

Model	Procedure	Prefix	GSM8K: $S$ [95% CI]	MATH-500: $S$ [95% CI]
Huginn	Sequential	Aligned	1.54 $\times$ [1.46, 1.63]	1.85 $\times$ [1.69, 2.03]
	Sequential	Deep K/V	2.26 $\times$ [2.16, 2.36]	2.42 $\times$ [2.28, 2.56]
	Parallel, no carry	Aligned	0.65 $\times$ [0.62, 0.68]	1.16 $\times$ [0.93, 1.45]
	Parallel, no carry	Deep K/V	0.93 $\times$ [0.89, 0.97]	1.06 $\times$ [0.98, 1.15]
	Parallel + carry	Aligned	0.71 $\times$ [0.68, 0.75]	1.20 $\times$ [1.00, 1.45]
	Parallel + carry	Deep V	1.07 $\times$ [1.02, 1.14]	1.66 $\times$ [1.43, 1.94]

Appendix

Table 8: Fixed-setting component speedups with 95% confidence intervals. Point estimates correspond to the component-session measurements in Table 2 .

			Huginn		Raven-Llama		Raven-OLMo		Raven-TinyLlama
Carry	K	V	GSM	MATH	GSM	MATH	GSM	MATH	GSM	MATH
0	0	0	2.38	4.37	6.78	9.59	7.73	8.33	9.07	8.46
0	0	1	3.14	4.81	7.98	10.46	9.57	10.08	10.46	9.31
0	1	0	2.11	2.69	6.63	9.04	8.04	7.94	8.72	8.44
0	1	1	3.42	3.95	8.50	10.84	9.50	10.53	10.31	9.65
1	0	0	2.61	4.51	8.60	10.28	9.52	9.43	10.55	10.46

Appendix

Table 9: Complete carry/K/V factorial. Entries report mean committed generation progress

G

at the fixed parallel drafting budget.

Model	Prefix policy	$\gamma$	$J$	$k$	Speedup	$A$
Huginn	Aligned	8	4	2	1.49 $\times$	1.29
Huginn	Deep V	32	6	2	1.84 $\times$	2.28
Raven Llama	Aligned	16	12	1	2.94 $\times$	5.40
Raven Llama	Deep V	16	12	2	3.70 $\times$	8.64

Appendix

Table 10: Independently retuned carried controls on GSM8K.

Model	$k$	$\gamma$	Speedup [95% CI]	$A$
Huginn	1	32	3.95 $\times$ [3.68, 4.24]	11.40
	2	32	4.33 $\times$ [4.11, 4.55]	16.49
	4	32	3.54 $\times$ [3.34, 3.73]	18.89
	8	32	2.37 $\times$ [2.23, 2.50]	19.56
	2	8	1.98 $\times$ [1.93, 2.02]	7.36
	2	16	3.58 $\times$ [3.43, 3.72]	13.67

Appendix

Table 11: One-dimensional refinement-budget sweeps. Entries report mean speedup [95% CI] and accepted length

A

Model	$W_{\mathrm{init}}$	Speedup [95% CI]	Ratio to $W_{\mathrm{init}}{=}2$ [95% CI]	$A$
Huginn	1	4.30 $\times$ [4.08, 4.55]	0.985 [0.964, 1.002]	17.95
	2	4.38 $\times$ [4.13, 4.65]	1.000 [1.000, 1.000]	18.34
	4	4.39 $\times$ [4.15, 4.66]	1.004 [0.981, 1.026]	18.59
	8	4.36 $\times$ [4.09, 4.66]	0.996 [0.964, 1.030]	18.78
Raven Llama	1	5.61 $\times$ [5.19, 6.03]	1.029 [0.975, 1.085]	19.88
	2	5.49 $\times$ [5.09, 5.90]	1.000 [1.000, 1.000]	18.90

Appendix

Table 12: Initial-width sensitivity. Direct paired ratios are relative to the default

W_{\mathrm{init}}=2

Model	Method	GSM8K	MATH-500	HumanEval	MBPP
Huginn	Lookahead	1.83 $\times$ [1.76, 1.90]	1.84 $\times$ [1.71, 1.98]	1.50 $\times$ [1.35, 1.68]	1.40 $\times$ [1.31, 1.51]
	PLD	1.54 $\times$ [1.45, 1.66]	1.93 $\times$ [1.66, 2.24]	1.61 $\times$ [1.29, 1.99]	1.35 $\times$ [1.18, 1.55]
	Token Recycling	2.81 $\times$ [2.72, 2.92]	2.81 $\times$ [2.69, 2.88]	2.74 $\times$ [2.43, 3.08]	2.56 $\times$ [2.39, 2.76]
	SAM-Decoding	1.63 $\times$ [1.51, 1.77]	2.50 $\times$ [1.95, 3.20]	2.01 $\times$ [1.43, 2.79]	1.49 $\times$ [1.30, 1.72]
	Self-speculative	1.70 $\times$ [1.63, 1.76]	1.82 $\times$ [1.73, 1.91]	1.64 $\times$ [1.53, 1.76]	1.79 $\times$ [1.71, 1.88]
	Mature-Spec	2.26 $\times$ [2.16, 2.36]	2.41 $\times$ [2.27, 2.55]	2.35 $\times$ [2.17, 2.52]	2.31 $\times$ [2.18, 2.43]

Appendix

Table 13: Main-result uncertainty. Entries are mean throughput speedup

S

[95% CI] over paired full-depth AR. Token Recycling uses moving-block bootstrap over its fixed request stream; all other intervals use paired prompt bootstrap.

Draft	$d$	$A$	Draft ms/request	ms/token
Self-speculative	8	5.2	4070	50.7
Mature-Spec	4	5.1	2293	36.1

Appendix

Table 14: Huginn/GSM8K sequential proposal-cost diagnostic. Draft time is the cumulative CUDA-event time spent on proposal construction per request, excluding prefill and target verification.

Explore similar work

LoopSpec: Pipelined Self-Speculative Decoding for Looped Transformers
Sep 15, 2026SangLyul Cho, Langqing Cui, Sehoon Kim +2Speculative Decoding Transformer Architectures
1Seoul National University · 2KAIST
ASPIRE: Asynchronous Batched Self-Speculative Decoding for Long-Context LLM Inference
Sep 16, 2026Amir Ziashahabi, Hossein Entezari Zarch, Lei Gao +2Speculative Decoding LLM Inference Optimization
University of Southern California, Los Angeles, CA, USA
Approximate Speculative Decoding
Aug 4, 2026Yuannuo Feng, Zegang Peng, Yuxin Xie +5Speculative Decoding Truncation
School of Integrated Circuit Science and Engineering, Beihang University, Beijing, China · Department of Precision Instrument, Tsinghua University, Beijing, China · Faculty of Engineering, The University of Hong Kong, Hong Kong SAR, China +2

Sep 15, 2026, cs.LGJ/K move · Enter open · S save

LoopSpec: Pipelined Self-Speculative Decoding for Looped Transformers

SangLyul Cho, Langqing Cui, Sehoon Kim, Dongsu Han+1

1Seoul National University · 2KAIST

Speculative Decoding Transformer Architectures Self-Speculative Loop Data Generation

Open paper AskPDF arXiv

Looped Transformers achieve strong performance with compact parameter sizes by repeatedly applying a shared stack of Transformer blocks across recurrent depths. However, they incur higher decoding latency than standard Transformer models of comparable parameter size because shared weights are accessed at every recurrent depth. To improve decoding efficiency, self-speculative decoding is particularly well suited to Looped Transformers, as their intermediate recurrent states can directly provide draft predictions without an auxiliary draft model. We therefore propose LoopSpec, a training-free self-speculative decoding framework tailored for Looped Transformers. LoopSpec extracts draft tokens from early recurrent states and operates in a pipelined manner, overlapping draft generation of future tokens with target verification of the current token. To improve draft accuracy without excessive compute overhead, we introduce a selective second proposal from deeper recurrent depth while ensuring lossless decoding under both greedy and sampling regimes. Furthermore, we derive the optimal proposal depths in closed form and show the prediction matches measurement. Across reasoning and coding benchmarks, LoopSpec achieves up to 6.83

\times

inference speedup across diverse Looped Transformers.

Abstract

Explore similar work

LoopSpec: Pipelined Self-Speculative Decoding for Looped Transformers

ASPIRE: Asynchronous Batched Self-Speculative Decoding for Long-Context LLM Inference

Approximate Speculative Decoding