ReaDy: Recurrent States as Dynamic Context for Speculative Drafting in Hybrid Linear Attention Large Language Models

Bohan Sun; Zunhai Su; Xialie Zhuang; Jianjin Li; Ngai Wong; Chuan-Wei Kuo

cs.CLOct 14, 2026

ReaDy: Recurrent States as Dynamic Context for Speculative Drafting in Hybrid Linear Attention Large Language Models

Authors: Bohan Sun, Zunhai Su, Xialie Zhuang, Jianjin Li, Ngai Wong, Chuan-Wei Kuo

Organizations: StartLux · University of Chinese Academy of Sciences · The University of Hong Kong

Abstract

Speculative decoding accelerates autoregressive generation by drafting multiple tokens per step, yet feature-based drafters rely on target hidden features whose contextual emphasis may not align with the needs of subsequent draft positions. Hybrid linear attention (HLA) models maintain compact recurrent states that accumulate historical associations, offering an efficient source of complementary context for speculative drafting. We introduce ReaDy, the first speculative decoding method, to the best of our knowledge, to leverage HLA recurrent states for position-specific context retrieval during drafting. ReaDy equips each draft position with a learned query to retrieve supplementary context directly from the target's committed recurrent states. A lightweight state reader integrates this context into draft representations through residual updates, while low-rank state reads and fused operators keep additional computation and I/O overhead low. Building on DSpark's parallel backbone and sequential prediction head, ReaDy employs a lightweight two-stage adaptation strategy, with reader capacity, state sources, and residual placement examined through ablation studies. Across seven benchmarks spanning mathematics, code generation, and dialogue, ReaDy achieves an average $3.16\times$ decoding speedup over autoregressive decoding with Qwen3.8-27B on a single NVIDIA B200 GPU. It outperforms DSpark on every benchmark, improving throughput by 7.97% and acceptance length by 8.50% on average. Our code is available at https://github.com/StartLuxLabs/ReaDy.

Figures & tables

Figure 1: Decoding speedups of Qwen3.8-27B on a single NVIDIA B200 GPU at temperature zero. Each benchmark is normalized to autoregressive decoding. The Average panel reports the arithmetic mean of per-benchmark speedups. ReaDy improves throughput over DSpark across all seven benchmarks.

Figure 2: ReaDy’s state-conditioned drafting and verification process. (a) The drafter combines target hidden features with position-specific reads from committed recurrent states. Verification determines the accepted prefix and the states available in the next round. (b) Query projection maps each draft representation through a rank-8 bottleneck to the 128-dimensional key space. The value map compresses the state values to rank eight. Position-specific readouts are fused with state-scale information and injected through a residual connection.

Figure 3: State-reader execution before and after optimization. Direct gathering and fused state preparation and feature packing reduce intermediate tensor materialization and redundant data movement. Green blocks indicate optimized operations.

Figure 4: Operator latency before and after optimization. State gathering and state reading achieve latency reductions of 71.8% and 28.1%, respectively.

Qwen3.8-27B Temperature = 0 Draft block size = 7
Task	Benchmark	AR	DSpark			ReaDy (ours)
Task	Benchmark	TPS	TPS	$\times$ AR	$\tau$	TPS	$\times$ AR	$\tau$	vs. DSpark
Math	GSM8K	96.40	362.25	3.76 $\times$	5.28	393.60	4.08 $\boldsymbol{\times}$	5.77	+8.66%
Math	MATH-500	95.99	331.90	3.46 $\times$	4.93	353.23	3.68 $\boldsymbol{\times}$	5.27	+6.43%
Code	HumanEval	95.60	287.95	3.01 $\times$	4.34	310.29	3.25 $\boldsymbol{\times}$	4.70	+7.76%
	MBPP	96.21	296.00	3.08 $\times$	4.34	317.55	3.30 $\boldsymbol{\times}$	4.68	+7.28%

Table 1: Decoding performance on a single NVIDIA B200 GPU. TPS excludes prefill;

\times

AR is the speedup over autoregressive decoding. The final column reports ReaDy’s TPS gain over DSpark. Acceptance length

\tau

is the number of generated tokens per verification step, including target-supplied tokens. The average row uses mean TPS and mean

\tau

; speedups are computed from mean TPS. Best results are bold.

Figure 5: Draft-prefix survival in mathematics, code, and dialogue. At position

j

, the rate is the fraction of fully observable blocks in which all first

j

proposals are accepted, pooled within each benchmark. ReaDy improves survival at every position. The curves show cumulative survival, not acceptance conditional on earlier positions.

Method	TPS	$\tau$	$\Delta$ TPS (%)
DSpark	280.42	4.1739	0.00
DSpark + FT	286.50	4.2732	2.17
ReaDy	302.78	4.5285	7.97

Table 2: Fine-tuning control. FT denotes fine-tuning without state access; gains are relative to DSpark.

Reader rank	TPS	$\tau$	$\Delta$ TPS	$\Delta\tau$
DSpark	280.42	4.1739	0.00	0.00
8	302.78	4.5285	7.97	8.50
16	302.36	4.5298	7.82	8.53
32	302.21	4.5323	7.77	8.59
128 (full)	302.31	4.5404	7.81	8.78

Table 3: Reader rank. Full rank uses 128-dimensional key and value spaces.

Target layer IDs	TPS	$\tau$	$\Delta$ TPS	$\Delta\tau$
DSpark	280.42	4.1739	0.00	0.00
(2)	286.11	4.2763	2.03	2.45
(30)	287.17	4.2929	2.41	2.85
(62)	300.21	4.4862	7.06	7.48
(30, 62)	302.78	4.5285	7.97	8.50
(2, 30, 62)	297.86	4.4614	6.22	6.89

Table 4: State source layers (target layer IDs).

Draft layer IDs	TPS	$\tau$	$\Delta$ TPS	$\Delta\tau$
DSpark	280.42	4.1739	0.00	0.00
(4)	302.78	4.5285	7.97	8.50
(0, 2, 4)	299.93	4.5288	6.96	8.50
(0, 1, 2, 3, 4)	297.35	4.5284	6.04	8.49

Table 5: Injection layers (zero-indexed draft layer IDs).

Injection site	TPS	$\tau$	$\Delta$ TPS	$\Delta\tau$
DSpark	280.42	4.1739	0.00	0.00
Pre-attention	301.94	4.4993	7.68	7.80
Post-attention	302.78	4.5285	7.97	8.50

Table 6: Injection position in the final draft layer.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Source corpus	Reported samples
MetaMathQA	395,000
UltraInteract SFT	288,579
UltraChat	207,865
Orca-Math	200,035
UltraFeedback	187,405
Evol-CodeAlpaca	111,272

Appendix

Table 7: Source-corpus counts reported by the Open-PerfectBlend dataset card, before mixture deduplication. These are not per-source counts for ReaDy’s training subsets.

Setting	Value
Optimizer	AdamW
$(\beta_{1},\beta_{2})$	$(0.9,0.999)$
$\epsilon$	$10^{-8}$
Peak learning rate (stage 1)	$1\times 10^{-4}$
Peak learning rate (stage 2)	$5\times 10^{-5}$
Weight decay	0

Appendix

Table 8: Optimizer settings for the two adaptation stages.

Figure 6: An Alpaca story-summarization example. Green highlights accepted proposed tokens; red marks the first mismatch. DSpark matches two of the seven proposals before its first mismatch, while ReaDy matches all seven. These counts refer only to accepted draft proposals, excluding target-supplied tokens.

LLM Inference Acceleration

LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.

See all 1,080 papers

SpecLA: Efficient Speculative Decoding for Linear-Attention Models
Jul 18, 2026Zhibin Wang, Xuying Han, Zhaohua Yang +5Speculative Decoding LLM Inference Acceleration
State Key Laboratory of Novel Software Technology Nanjing University Nanjing, China · Alibaba Group Hangzhou, China
LongSpark: Efficient speculative decoding with a fixed-cost parallel drafter
Sep 29, 2026Hao-Yuan He, Peng-Fei Liu, Si Shen +1Long-Context Language Model Inference Blockwise Diffusion Language Models
RACER: Retrieval-Augmented Contextual Rapid Speculative Decoding
Apr 16, 2026Zihong Zhang, Zuchao Li, Lefei Zhang +2Speculative Decoding LLM Inference Acceleration
School of Artiﬁcial Intelligence, Wuhan University, Wuhan, China · School of Computer Science, Wuhan University, Wuhan, China · School of Information Management, Wuhan University, Wuhan, China +1

Jul 18, 2026J/K move · Enter open · S save

SpecLA: Efficient Speculative Decoding for Linear-Attention Models

Zhibin Wang, Xuying Han, Zhaohua Yang, Fuliang Liu+4

State Key Laboratory of Novel Software Technology Nanjing University Nanjing, China · Alibaba Group Hangzhou, China

Speculative Decoding LLM Inference Acceleration Linear Attention

Open paper AskPDF arXiv

Linear-attention models replace the growing KV cache with recurrent states, but autoregressive decoding still reads, updates, and writes these states one token at a time. Speculative decoding can reduce this cost by verifying several draft tokens in one target pass, yet existing speculative systems are designed for Transformer KV caches. For stateful linear-attention targets, verification must follow recurrent dependencies across chains and branches, acceptance must update only the accepted state trajectory, and the drafter must avoid submitting candidates that waste stateful verification work. This paper presents SpecLA, a speculative decoding runtime for stateful linear-attention models. SpecLA verifies chains and trees with topology-aware kernels, stores compact factors produced during verification to recover accepted states, and uses confidence pruning plus a target-aligned EAGLE-style drafter to feed useful candidates to the verifier. On an NVIDIA H100 with a public GDN-1.3B target, SpecLA achieves up to 1.70x end-to-end speedup over autoregressive decoding.

Abstract

LLM Inference Acceleration

Explore similar work

SpecLA: Efficient Speculative Decoding for Linear-Attention Models

LongSpark: Efficient speculative decoding with a fixed-cost parallel drafter

RACER: Retrieval-Augmented Contextual Rapid Speculative Decoding