Shallow Queries, Mature Values: Depth-Asynchronous Self-Speculation for Looped Transformers

Guanghao Li; Zihan Su; Hao Yu; Jinyang Jiang; Tao Ren; Zehao Li; Feng Lu; Ming Tang

cs.LGSep 28, 2026

Shallow Queries, Mature Values: Depth-Asynchronous Self-Speculation for Looped Transformers

Authors: Guanghao Li, Zihan Su, Hao Yu, Jinyang Jiang, Tao Ren, Zehao Li, Feng Lu, Ming Tang, +1 more

Organizations: Tsinghua University · Southern University of Science and Technology · The University of Hong Kong · Peking University · Shenzhen University of Advanced Technology

Abstract

Looped Transformers reuse a shared block across recurrent depths, making autoregressive decoding expensive because every generated token requires many sequential recurrent passes. Self-speculative decoders reduce this cost by drafting at an early depth and verifying at full depth, but typically bind draft computation to prefix representations from the same recurrent depth. We find that queries and keys approach their final-depth representations earlier than values, and controlled prefix-channel interventions show that mature values substantially improve shallow draft predictions. Motivated by this asymmetry, we introduce Depth-Asynchronous Self-Speculation (DAS), which decouples the depth of draft computation from the depth of verified-prefix representations it reads. Its Mature-V primitive lets shallow queries retrieve full-depth prefix values without additional recurrent computation. We further develop DAS-Wave, which combines depth-asynchronous prefix reads with carried parallel refinement, progressive block growth, and an independent full-depth verifier. Across four recurrent-model checkpoints and mathematics and code workloads, DAS-Wave achieves 4.00--6.96 $\times$ mean throughput speedup over paired full-depth autoregressive decoding in the same inference stack. These results identify prefix-information depth as an effective design axis for recurrent self-speculation.

Figures & tables

Figure 1: DAS-Wave overview. (a) Queries from the current draft state attend to full-depth K/V from the verified, materialized prefix at the same core layer. (b) Active positions are refined in parallel while latent states are carried across rounds and the active window grows. Gold marks newly activated states. The anchor remains fixed, and each readout updates the next candidate position. (c) An independent full-depth target verifies the candidates without using carried draft states. In the illustrated example,

y_{1}

and

y_{2}

are accepted;

y_{3}

is the first mismatch, so the target prediction

\hat{y}_{3}=\argmax_{v}p_{3}(v)

is emitted and the remaining draft suffix is discarded.

Figure 2: Shallow queries, mature values. Top: Q/K/V cosine similarity to the final recurrence (

R=32

), averaged over core layers. Queries and keys approach their final-depth representations earlier than values. Bottom: draft–target token agreement under separate prefix-channel replacements. Deep V uses full-depth prefix V while keeping K aligned with the draft; in sequential drafting, this corresponds to Mature-V. Each model uses 12 GSM8K prompts and 756 evaluated next-token prediction positions.

		GSM8K		MATH-500		HumanEval		MBPP
Model	Method	Speedup	$A$	Speedup	$A$	Speedup	$A$	Speedup	$A$
Huginn	Lookahead	1.83 $\times$	1.1	1.84 $\times$	1.0	1.50 $\times$	0.6	1.40 $\times$	0.5
	PLD	1.54 $\times$	0.7	1.93 $\times$	1.2	1.61 $\times$	0.8	1.35 $\times$	0.5
	Token Recycling	2.81 $\times$	2.1	2.81 $\times$	2.1	2.74 $\times$	2.1	2.56 $\times$	1.8
	SAM-Decoding	1.63 $\times$	0.9	2.50 $\times$	1.9	2.01 $\times$	1.3	1.49 $\times$	0.6
	Self-speculative	1.70 $\times$	5.2	1.82 $\times$	5.7	1.64 $\times$	4.6	1.79 $\times$	5.0

Table 1: End-to-end decoding performance. Speedup is mean throughput relative to paired full-depth AR;

A

is the mean number of accepted draft tokens per verification. Bold marks the largest speedup and accepted length in each setting.

			GSM8K		MATH-500
Model	Procedure	Prefix	Speedup	$A$	Speedup	$A$
Huginn	Sequential	Aligned	1.54 $\times$	2.80	1.85 $\times$	3.69
	Sequential	Deep K/V	2.26 $\times$	4.92	2.42 $\times$	5.35
	Parallel, no carry	Aligned	0.65 $\times$	1.38	1.16 $\times$	3.37
	Parallel, no carry	Deep K/V	0.93 $\times$	2.42	1.06 $\times$	2.95
	Parallel + carry	Aligned	0.71 $\times$	1.61	1.20 $\times$	3.51

Table 2: Fixed-setting component comparisons. Sequential rows use draft depth 4. All parallel rows share the same refinement schedule; no-carry rows reinitialize latent states between rounds, while carried rows retain them. Parallel + carry with Deep K/V uses the full DAS-Wave configuration. Component estimates are measured in their own paired timing sessions.

Figure 3: Refinement cost–acceptance tradeoff. GSM8K diagnostic runs with

J=24

. Top: mean throughput speedup; bottom: accepted draft length

A

. Left varies recurrent updates per round; right varies verifier capacity at the selected update count.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Checkpoint	Prelude	Shared core	Coda	Recurrences
Huginn-0125	2	4	2	32
Raven-Llama-3.2	4	6	4	32
Raven-OLMo-2	4	6	4	32
Raven-TinyLlama	4	8	4	32

Appendix

Table 3: Evaluated recurrent-depth models.

Task	Cap	Prompt convention
GSM8K	512	Eight-shot chain of thought
MATH-500	1024	Four-shot worked solutions
HumanEval	512	Code completion
MBPP	512	Code completion

Appendix

Table 4: Prompt conventions and generation caps.

Method	Huginn	Raven family	Prefix reads
Self-speculative	$d=8,b=8$	$d=4,b=16$	Aligned
Mature-Spec	$d=4,b=8$	$d=4,b=16$	Deep K/V
DAS-Wave	$\gamma=32,k=2$	$\gamma=32,k=1$	Deep K/V

Appendix

Table 5: Frozen neural-draft configurations.

Model	Prefix policy	$d=1$	$d=2$	$d=4$	$d=8$
Huginn	Aligned	0.4722	0.6362	0.7937	0.9127
	Deep V	0.6693	0.7659	0.8955	0.9484
	Deep K	0.4683	0.6045	0.8016	0.8968
	Deep K/V	0.6005	0.7937	0.8995	0.9669
Raven Llama	Aligned	0.8399	0.9034	0.9590	0.9841
	Deep V	0.8796	0.9471	0.9841	0.9947

Appendix

Table 6: Draft–target token agreement under depth-dependent prefix-channel substitution.

Figure 4: Prefix read-depth grid. Huginn draft computation is fixed at $r=2$ . Rows vary verified-prefix K depth $d_{K}$ and columns vary V depth $d_{V}$ ; cells report draft–target token agreement. The four corners $d_{K},d_{V}\in\{2,R\}$ correspond to Aligned, Deep V, Deep K, and Deep K/V. The diagnostic uses 8 prompts and 376 prediction positions, separate from Table 6 .

Model	Task	Deep V [95% CI]	$A$	Deep K/V [95% CI]	$A$
Raven-Llama	GSM8K	3.39 $\times$ [3.25, 3.51]	12.6	3.50 $\times$ [3.37, 3.62]	13.1
	MATH-500	3.54 $\times$ [3.36, 3.70]	13.4	3.70 $\times$ [3.55, 3.83]	14.0
	HumanEval	3.34 $\times$ [3.12, 3.53]	12.0	3.44 $\times$ [3.22, 3.64]	13.1
	MBPP	3.72 $\times$ [3.62, 3.81]	13.3	3.84 $\times$ [3.74, 3.92]	14.7
Raven-OLMo	GSM8K	3.50 $\times$ [3.36, 3.64]	13.0	3.63 $\times$ [3.53, 3.73]	13.5
	MATH-500	3.55 $\times$ [3.40, 3.69]	13.1	3.73 $\times$ [3.60, 3.86]	13.8

Appendix

Table 7: Sequential mature-prefix policies on the Raven checkpoints. Entries report mean speedup [95% CI] and accepted length

A

Model	Procedure	Prefix	GSM8K: $S$ [95% CI]	MATH-500: $S$ [95% CI]
Huginn	Sequential	Aligned	1.54 $\times$ [1.46, 1.63]	1.85 $\times$ [1.69, 2.03]
	Sequential	Deep K/V	2.26 $\times$ [2.16, 2.36]	2.42 $\times$ [2.28, 2.56]
	Parallel, no carry	Aligned	0.65 $\times$ [0.62, 0.68]	1.16 $\times$ [0.93, 1.45]
	Parallel, no carry	Deep K/V	0.93 $\times$ [0.89, 0.97]	1.06 $\times$ [0.98, 1.15]
	Parallel + carry	Aligned	0.71 $\times$ [0.68, 0.75]	1.20 $\times$ [1.00, 1.45]
	Parallel + carry	Deep V	1.07 $\times$ [1.02, 1.14]	1.66 $\times$ [1.43, 1.94]

Appendix

Table 8: Fixed-setting component speedups with 95% confidence intervals. Point estimates correspond to the component-session measurements in Table 2 .

			Huginn		Raven-Llama		Raven-OLMo		Raven-TinyLlama
Carry	K	V	GSM	MATH	GSM	MATH	GSM	MATH	GSM	MATH
0	0	0	2.38	4.37	6.78	9.59	7.73	8.33	9.07	8.46
0	0	1	3.14	4.81	7.98	10.46	9.57	10.08	10.46	9.31
0	1	0	2.11	2.69	6.63	9.04	8.04	7.94	8.72	8.44
0	1	1	3.42	3.95	8.50	10.84	9.50	10.53	10.31	9.65
1	0	0	2.61	4.51	8.60	10.28	9.52	9.43	10.55	10.46

Appendix

Table 9: Complete carry/K/V factorial. Entries report mean committed generation progress

G

at the fixed parallel drafting budget.

Model	Prefix policy	$\gamma$	$J$	$k$	Speedup	$A$
Huginn	Aligned	8	4	2	1.49 $\times$	1.29
Huginn	Deep V	32	6	2	1.84 $\times$	2.28
Raven Llama	Aligned	16	12	1	2.94 $\times$	5.40
Raven Llama	Deep V	16	12	2	3.70 $\times$	8.64

Appendix

Table 10: Independently retuned carried controls on GSM8K.

Model	$k$	$\gamma$	Speedup [95% CI]	$A$
Huginn	1	32	3.95 $\times$ [3.68, 4.24]	11.40
	2	32	4.33 $\times$ [4.11, 4.55]	16.49
	4	32	3.54 $\times$ [3.34, 3.73]	18.89
	8	32	2.37 $\times$ [2.23, 2.50]	19.56
	2	8	1.98 $\times$ [1.93, 2.02]	7.36
	2	16	3.58 $\times$ [3.43, 3.72]	13.67

Appendix

Table 11: One-dimensional refinement-budget sweeps. Entries report mean speedup [95% CI] and accepted length

A

Model	$W_{\mathrm{init}}$	Speedup [95% CI]	Ratio to $W_{\mathrm{init}}{=}2$ [95% CI]	$A$
Huginn	1	4.30 $\times$ [4.08, 4.55]	0.985 [0.964, 1.002]	17.95
	2	4.38 $\times$ [4.13, 4.65]	1.000 [1.000, 1.000]	18.34
	4	4.39 $\times$ [4.15, 4.66]	1.004 [0.981, 1.026]	18.59
	8	4.36 $\times$ [4.09, 4.66]	0.996 [0.964, 1.030]	18.78
Raven Llama	1	5.61 $\times$ [5.19, 6.03]	1.029 [0.975, 1.085]	19.88
	2	5.49 $\times$ [5.09, 5.90]	1.000 [1.000, 1.000]	18.90

Appendix

Table 12: Initial-width sensitivity. Direct paired ratios are relative to the default

W_{\mathrm{init}}=2

Model	Method	GSM8K	MATH-500	HumanEval	MBPP
Huginn	Lookahead	1.83 $\times$ [1.76, 1.90]	1.84 $\times$ [1.71, 1.98]	1.50 $\times$ [1.35, 1.68]	1.40 $\times$ [1.31, 1.51]
	PLD	1.54 $\times$ [1.45, 1.66]	1.93 $\times$ [1.66, 2.24]	1.61 $\times$ [1.29, 1.99]	1.35 $\times$ [1.18, 1.55]
	Token Recycling	2.81 $\times$ [2.72, 2.92]	2.81 $\times$ [2.69, 2.88]	2.74 $\times$ [2.43, 3.08]	2.56 $\times$ [2.39, 2.76]
	SAM-Decoding	1.63 $\times$ [1.51, 1.77]	2.50 $\times$ [1.95, 3.20]	2.01 $\times$ [1.43, 2.79]	1.49 $\times$ [1.30, 1.72]
	Self-speculative	1.70 $\times$ [1.63, 1.76]	1.82 $\times$ [1.73, 1.91]	1.64 $\times$ [1.53, 1.76]	1.79 $\times$ [1.71, 1.88]
	Mature-Spec	2.26 $\times$ [2.16, 2.36]	2.41 $\times$ [2.27, 2.55]	2.35 $\times$ [2.17, 2.52]	2.31 $\times$ [2.18, 2.43]

Appendix

Table 13: Main-result uncertainty. Entries are mean throughput speedup

S

[95% CI] over paired full-depth AR. Token Recycling uses moving-block bootstrap over its fixed request stream; all other intervals use paired prompt bootstrap.

Draft	$d$	$A$	Draft ms/request	ms/token
Self-speculative	8	5.2	4070	50.7
Mature-Spec	4	5.1	2293	36.1

Appendix

Table 14: Huginn/GSM8K sequential proposal-cost diagnostic. Draft time is the cumulative CUDA-event time spent on proposal construction per request, excluding prefill and target verification.

Explore similar work

Sep 15, 2026cs.LG

LoopSpec: Pipelined Self-Speculative Decoding for Looped Transformers

Looped Transformers achieve strong performance with compact parameter sizes by repeatedly applying a shared stack of Transformer blocks across recurrent depths. However, they incur higher decoding latency than standard Transformer models of comparable parameter size because shared weights are accessed at every recurrent depth. To improve decoding efficiency, self-speculative decoding is particularly well suited to Looped Transformers, as their intermediate recurrent states can directly provide draft predictions without an auxiliary draft model. We therefore propose LoopSpec, a training-free self-speculative decoding framework tailored for Looped Transformers. LoopSpec extracts draft tokens from early recurrent states and operates in a pipelined manner, overlapping draft generation of future tokens with target verification of the current token. To improve draft accuracy without excessive compute overhead, we introduce a selective second proposal from deeper recurrent depth while ensuring lossless decoding under both greedy and sampling regimes. Furthermore, we derive the optimal proposal depths in closed form and show the prediction matches measurement. Across reasoning and coding benchmarks, LoopSpec achieves up to 6.83

\times

inference speedup across diverse Looped Transformers.

SangLyul Cho, Langqing Cui, Sehoon Kim +2

1Seoul National University · 2KAIST

Speculative Decoding Transformer Architectures Self-Speculative Loop Data Generation

Ask PDF

Sep 16, 2026cs.LG

ASPIRE: Asynchronous Batched Self-Speculative Decoding for Long-Context LLM Inference

Long-context LLM inference is bottlenecked by attention, whose repeated KV-cache reads make decoding memory-bound. Self-speculative decoding alleviates this by drafting tokens with sparse attention and verifying them with full attention, but existing batched methods remain synchronized: all requests in a batch share a single draft-verify schedule, even though the optimal draft length varies widely across requests and changes dynamically within each request. We propose ASPIRE, a non-synchronized batched self-speculative decoding framework built on three components. First, a unified mixed forward allows drafting and verifying requests to coexist in the same batched forward pass, removing the need for global draft-verify phases. Second, a lightweight online speculation scheduler uses per-request acceptance-rate estimates and a batch-aware cost model to let each request independently choose when to verify. Third, an intra-draft refresh layer performs full attention at a single designated layer during drafting, updating the sparse context at every draft step to reduce staleness during drafting. Across three models and five reasoning and long-context benchmarks, ASPIRE achieves

1.70

4.58\times

speedup in decoding throughput over autoregressive baselines and improves average speedup by approximately

27\%

over the strongest prior self-speculative baselines.

Amir Ziashahabi, Hossein Entezari Zarch, Lei Gao +2

University of Southern California, Los Angeles, CA, USA

Speculative Decoding LLM Inference Optimization Asynchronous Execution Strongest Prior Asynchronous Baseline Peak Memory

Ask PDF

Aug 4, 2026cs.LG

Approximate Speculative Decoding

Speculative decoding accelerates autoregressive generation by verifying a draft block with a target model in parallel. Under standard greedy verification, decoding stops at the first draft token that differs from the target argmax, discarding the remaining target-scored suffix. Although accepting such a mismatch changes the decoding trajectory, it can make a contiguous suffix reusable when its tokens remain target-greedy under the realized prefix. In this paper, we introduce \textbf{Approximate Speculative Decoding (ASD)}, a training-free verifier that replaces binary first-mismatch truncation with budgeted longest-prefix selection. ASD accepts selected mismatches subject to a local target-logit regret gate, a per-block exception cap, and a persistent request-level regret budget, then reuses the contiguous target-greedy suffix without additional approximate decisions or target-model forward passes. ASD requires neither a new draft model nor fine-tuning, and exactly reduces to standard greedy verification when the budget is zero. Experiments show that ASD improves fixed-workload throughput by

3.05\%

15.26\%

over matched strict verification and averages a

7.78\%

gain across seven Qwen3-14B + DSpark-14B tasks. On DeepSeek-V4-Flash (284B) with DSpark it also raises verifier-side acceptance by roughly

10\%

16\%

on GSM8K and MATH-500 in an FP4-to-FP8 compatibility setting. The source code is publicly available at: https://github.com/Kissmetothemoon/ASD

Yuannuo Feng, Zegang Peng, Yuxin Xie +5

School of Integrated Circuit Science and Engineering, Beihang University, Beijing, China · Department of Precision Instrument, Tsinghua University, Beijing, China · Faculty of Engineering, The University of Hong Kong, Hong Kong SAR, China +2

Speculative Decoding Truncation Deepseek Greedy Algorithm

Ask PDF