Speculative decoding accelerates autoregressive generation by drafting multiple tokens per step, yet feature-based drafters rely on target hidden features whose contextual emphasis may not align with the needs of subsequent draft positions. Hybrid linear attention (HLA) models maintain compact recurrent states that accumulate historical associations, offering an efficient source of complementary context for speculative drafting. We introduce ReaDy, the first speculative decoding method, to the best of our knowledge, to leverage HLA recurrent states for position-specific context retrieval during drafting. ReaDy equips each draft position with a learned query to retrieve supplementary context directly from the target's committed recurrent states. A lightweight state reader integrates this context into draft representations through residual updates, while low-rank state reads and fused operators keep additional computation and I/O overhead low. Building on DSpark's parallel backbone and sequential prediction head, ReaDy employs a lightweight two-stage adaptation strategy, with reader capacity, state sources, and residual placement examined through ablation studies. Across seven benchmarks spanning mathematics, code generation, and dialogue, ReaDy achieves an average 3.16× decoding speedup over autoregressive decoding with Qwen3.8-27B on a single NVIDIA B200 GPU. It outperforms DSpark on every benchmark, improving throughput by 7.97% and acceptance length by 8.50% on average. Our code is available at https://github.com/StartLuxLabs/ReaDy.
Figures & tables
Figure 1: Decoding speedups of Qwen3.8-27B on a single NVIDIA B200 GPU at temperature zero. Each benchmark is normalized to autoregressive decoding. The Average panel reports the arithmetic mean of per-benchmark speedups. ReaDy improves throughput over DSpark across all seven benchmarks.
Figure 2: ReaDy’s state-conditioned drafting and verification process. (a) The drafter combines target hidden features with position-specific reads from committed recurrent states. Verification determines the accepted prefix and the states available in the next round. (b) Query projection maps each draft representation through a rank-8 bottleneck to the 128-dimensional key space. The value map compresses the state values to rank eight. Position-specific readouts are fused with state-scale information and injected through a residual connection.
Figure 3: State-reader execution before and after optimization. Direct gathering and fused state preparation and feature packing reduce intermediate tensor materialization and redundant data movement. Green blocks indicate optimized operations.
Figure 4: Operator latency before and after optimization. State gathering and state reading achieve latency reductions of 71.8% and 28.1%, respectively.
Qwen3.8-27B Temperature = 0 Draft block size = 7
Task
Benchmark
AR
DSpark
ReaDy (ours)
TPS
TPS
× AR
τ
TPS
× AR
τ
vs. DSpark
Math
GSM8K
96.40
362.25
3.76 ×
5.28
393.60
4.08 ×
5.77
+8.66%
MATH-500
95.99
331.90
3.46 ×
4.93
353.23
3.68 ×
5.27
+6.43%
Code
HumanEval
95.60
287.95
3.01 ×
4.34
310.29
3.25 ×
4.70
+7.76%
MBPP
96.21
296.00
3.08 ×
4.34
317.55
3.30 ×
4.68
+7.28%
Table 1: Decoding performance on a single NVIDIA B200 GPU. TPS excludes prefill; × AR is the speedup over autoregressive decoding. The final column reports ReaDy’s TPS gain over DSpark. Acceptance length τ is the number of generated tokens per verification step, including target-supplied tokens. The average row uses mean TPS and mean τ ; speedups are computed from mean TPS. Best results are bold.
Figure 5: Draft-prefix survival in mathematics, code, and dialogue. At position j , the rate is the fraction of fully observable blocks in which all first j proposals are accepted, pooled within each benchmark. ReaDy improves survival at every position. The curves show cumulative survival, not acceptance conditional on earlier positions.
Method
TPS
τ
Δ TPS (%)
DSpark
280.42
4.1739
0.00
DSpark + FT
286.50
4.2732
2.17
ReaDy
302.78
4.5285
7.97
Table 2: Fine-tuning control. FT denotes fine-tuning without state access; gains are relative to DSpark.
Reader rank
TPS
τ
Δ TPS
Δτ
DSpark
280.42
4.1739
0.00
0.00
8
302.78
4.5285
7.97
8.50
16
302.36
4.5298
7.82
8.53
32
302.21
4.5323
7.77
8.59
128 (full)
302.31
4.5404
7.81
8.78
Table 3: Reader rank. Full rank uses 128-dimensional key and value spaces.
Table 6: Injection position in the final draft layer.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Source corpus
Reported samples
MetaMathQA
395,000
UltraInteract SFT
288,579
UltraChat
207,865
Orca-Math
200,035
UltraFeedback
187,405
Evol-CodeAlpaca
111,272
Appendix
Table 7: Source-corpus counts reported by the Open-PerfectBlend dataset card, before mixture deduplication. These are not per-source counts for ReaDy’s training subsets.
Setting
Value
Optimizer
AdamW
(β1,β2)
(0.9,0.999)
ϵ
10−8
Peak learning rate (stage 1)
1×10−4
Peak learning rate (stage 2)
5×10−5
Weight decay
0
Appendix
Table 8: Optimizer settings for the two adaptation stages.
Figure 6: An Alpaca story-summarization example. Green highlights accepted proposed tokens; red marks the first mismatch. DSpark matches two of the seven proposals before its first mismatch, while ReaDy matches all seven. These counts refer only to accepted draft proposals, excluding target-supplied tokens.
LLM inference acceleration develops algorithms and systems that reduce language-model latency, memory use, and serving cost while preserving output quality.
Linear-attention models replace the growing KV cache with recurrent states, but autoregressive decoding still reads, updates, and writes these states one token at a time. Speculative decoding can reduce this cost by verifying several draft tokens in one target pass, yet existing speculative systems are designed for Transformer KV caches. For stateful linear-attention targets, verification must follow recurrent dependencies across chains and branches, acceptance must update only the accepted state trajectory, and the drafter must avoid submitting candidates that waste stateful verification work. This paper presents SpecLA, a speculative decoding runtime for stateful linear-attention models. SpecLA verifies chains and trees with topology-aware kernels, stores compact factors produced during verification to recover accepted states, and uses confidence pruning plus a target-aligned EAGLE-style drafter to feed useful candidates to the verifier. On an NVIDIA H100 with a public GDN-1.3B target, SpecLA achieves up to 1.70x end-to-end speedup over autoregressive decoding.
Zhibin Wang, Xuying Han, Zhaohua Yang +5
State Key Laboratory of Novel Software Technology Nanjing University Nanjing, China · Alibaba Group Hangzhou, China
Speculative decoding accelerates autoregressive inference by verifying multiple draft tokens in a single target forward pass. However, as the context grows, existing state-of-the-art drafters become increasingly expensive, eroding the very efficiency advantage they are designed to provide. We argue that this scaling is unnecessary. A standalone language model must grow with its prefix because it is solely responsible for every token it produces. A drafter, by contrast, only proposes candidates; the target catches and corrects every error before any token is committed. The drafter's decoding cost can therefore be made entirely independent of the prefix length. We introduce LongSpark, a block-diffusion drafter that achieves this by extracting fixed-size, multiscale views from the target's verification pass, thereby eliminating the need for a growing persistent state. Extensive evaluations demonstrate that LongSpark achieves state-of-the-art end-to-end efficiency across multiple model scales and realistic serving conditions. Notably, it delivers the lowest time-per-output-token on long-context tasks while reducing the drafter's context state by several orders of magnitude.
Autoregressive decoding in Large Language Models (LLMs) generates one token per step, causing high inference latency. Speculative decoding (SD) mitigates this through a guess-and-verify strategy, but existing training-free variants face trade-offs: retrieval-based drafts break when no exact match exists, while logits-based drafts lack structural guidance. We propose RACER (Retrieval-Augmented Contextual Rapid Speculative Decoding), a lightweight and training-free method that integrates retrieved exact patterns with logit-driven future cues. This unification supplies both reliable anchors and flexible extrapolation, yielding richer speculative drafts. Experiments on Spec-Bench, HumanEval, and MGSM-ZH demonstrate that RACER consistently accelerates inference, achieving more than 2× speedup over autoregressive decoding, and outperforms prior training-free methods, offering a scalable, plug-and-play solution for efficient LLM decoding. Our source code is available at https://github.com/hkr04/RACER.
Zihong Zhang, Zuchao Li, Lefei Zhang +2
School of Artificial Intelligence, Wuhan University, Wuhan, China · School of Computer Science, Wuhan University, Wuhan, China · School of Information Management, Wuhan University, Wuhan, China +1