DLite: History-free Speculative Decoding for Agentic AI

Jiaming Xu; Hanzhen Wang; Tianlang Zhao; Jiewen Xiao; Chengze Yuan; Jiayi Pan; Yongkang Zhou; Jiancai Ye

cs.CLOct 16, 2026

DLite: History-free Speculative Decoding for Agentic AI

Authors: Jiaming Xu, Hanzhen Wang, Tianlang Zhao, Jiewen Xiao, Chengze Yuan, Jiayi Pan, Yongkang Zhou, Jiancai Ye,

Organizations: Shanghai Jiao Tong University · Shanghai Innovation Institute · Tsinghua University · Infinigence-AI

Abstract

Speculative decoding has become a mainstream technique for accelerating large language model (LLM) inference by using a lightweight draft model to generate candidate tokens that are subsequently verified by the target model. However, agentic workloads introduce long and continuously evolving contexts and frequent multi-turn interactions. Under such workloads, we observe that existing draft models suffer from up to an 80% degradation in acceptance length, while maintaining their historical KV cache introduces substantial system overhead. In this paper, we identify the growing historical context as a common source of both problems and introduce DLite, a history-free speculative decoding framework for agentic AI. DLite directly reuses contextual hidden states from the target model and performs local speculative prediction without explicitly maintaining the historical context, transforming an increasingly complex modeling problem into local simple prediction problem. A lightweight sequential head further restores intra-block causal dependencies with low overhead. To effectively train this heterogeneous history-free architecture, we introduce a tailored two-stage training strategy. Experiments on general-purpose, long-context and agentic workloads show that DLite achieves 2-6 acceptance length and outperforms the state-of-the-art speculative decoding method, DSpark, up to 70%.

Figures & tables

Figure 1: (a) Agentic workloads characteristics. (b) Speculative decoding performance loss.

Figure 2: Speculative decoding pipeline and comparison of existing draft architectures with DLite .

Figure 3: The inference dataflow of DLite .

Figure 4: (a) The design architecture of the lightweight sequential head. (b) The training loss and acceptance rate curves of direct target-model distillation using KL and CE loss function.

		Math		Code			Chat
Model	Method	GSM8K	MATH-500	HumanEval	MBPP	LCB	MT-Bench	Alpaca
Qwen3-4B	EAGLE3	2.63	2.39	2.47	2.34	2.17	3.07	2.28
	DFlow	5.13	4.92	4.33	4.12	-	3.80	2.90
	DFlash	5.97	5.82	5.25	4.97	4.82	3.39	3.56
	DFlash2	6.14	5.80	5.38	5.14	4.79	4.14	2.50
	DSpark	6.27	6.25	5.62	5.48	5.20	3.81	3.54

Table 1: Acceptance length

\tau

evaluation on general-purpose workloads.

Model	Method	LongBench_v2
		Multi-Doc QA		Long In-ctx Lrn.		Code Repo Ud.		Long-dialog Hist.
		Speedup	$\tau$	Speedup	$\tau$	Speedup	$\tau$	Speedup	$\tau$
Qwen3-4B	Eagle3	1.70 $\times$	2.10	1.76 $\times$	2.28	1.68 $\times$	2.12	2.23 $\times$	2.88
	DFlash	1.60 $\times$	1.90	1.71 $\times$	2.00	1.45 $\times$	1.83	2.62 $\times$	3.04
	DSpark	1.81 $\times$	2.20	2.01 $\times$	2.41	2.02 $\times$	2.44	3.16 $\times$	3.93
	DLite	2.18 $\times$	2.74	2.58 $\times$	3.11	2.66 $\times$	3.14	3.88 $\times$	4.62

Table 2: Speedup and acceptance length (

\tau

) evaluation on long-context workloads.

Metric	Method	AgentMemBench		AgentLongBench		BFCL-v3		SWE-bench
Metric	Method	Speedup	$\tau$	Speedup	$\tau$	Speedup	$\tau$	Speedup	$\tau$
Qwen3-4B	Eagle3	1.66 $\times$	2.09	1.70 $\times$	2.12	1.58 $\times$	2.00	1.48 $\times$	1.88
	DFlash	1.22 $\times$	1.65	0.95 $\times$	1.14	1.44 $\times$	1.80	1.47 $\times$	1.72
	DSpark	1.72 $\times$	2.10	1.45 $\times$	1.84	2.11 $\times$	2.54	1.64 $\times$	1.89
	DLite	2.39 $\times$	2.80	2.07 $\times$	2.51	2.37 $\times$	2.78	2.68 $\times$	3.15
Qwen3-8B	Eagle3	1.77 $\times$	2.19	1.60 $\times$	2.00	1.49 $\times$	1.88	1.33 $\times$	1.71

Table 3: Speedup and acceptance length (

\tau

) evaluation on agentic workloads.

Task	Method	Concurrency (TPS / Speedup)					Avg. $\tau$
Task	Method	1	8	16	32	64	Avg. $\tau$
Qwen3-4B
Alpaca	Baseline	246 / 1.00 $\times$	1798 / 1.00 $\times$	3377 / 1.00 $\times$	5849 / 1.00 $\times$	10808 /1.00 $\times$	–
	DSpark	609 / 2.47 $\times$	4131 / 2.30 $\times$	7018 / 2.08 $\times$	11118 / 1.90 $\times$	13548 / 1.25 $\times$	3.49
	DLite	635 / 2.58 $\times$	4222 / 2.35 $\times$	7293 / 2.16 $\times$	11792 / 2.02 $\times$	14300 / 1.32 $\times$	3.63
MT-Bench	Baseline	245 / 1.00 $\times$	1792 / 1.00 $\times$	3377 / 1.00 $\times$	5883 / 1.00 $\times$	10113 / 1.00 $\times$	–

Table 4: High-concurrency throughput evaluation on Qwen3 models.

Figure 5: Ablation on training strategy and KV cache.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Dimension	Configuration	Alpaca	GSM8K	MBPP	Multi-Doc QA	Avg. $\tau$
Target Hidden States	7	2.34	4.36	3.63	2.22	3.14
	9	2.42	4.40	3.75	2.24	3.20
	14	2.63	4.53	3.76	2.28	3.30
	20	2.62	4.55	3.74	2.26	3.29
	36	2.58	4.51	3.71	2.26	3.27
Hidden Composition	Feature-wise	2.20	4.28	3.50	2.16	3.04

Appendix

Table 5: Design space exploration of DLite on Qwen3-8B. All variants are trained on the same 80K-sample subset, and we report average acceptance length (

\tau

). For the target hidden-state exploration, the first and last target layers are always included in addition to the sampled intermediate layers. Unless otherwise specified, we use 14 target hidden states, sequence-wise hidden-state composition, and a sequential-head rank of 512. Bold indicates the configuration adopted in the final model.

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 293 papers

AgentSpec: Speculative Decoding for Batch Inference of LLM Agents
Aug 25, 2026Xin Wang, Ziming Miao, Yi Zhu +4Speculative Decoding LLM Inference Acceleration
The Ohio State University · Microsoft Research · University of Michigan
AsymSpec: Context-Asymmetric Speculative Decoding for Agentic LLMs
Aug 26, 2026Sheng Liang, Yongyue Zhang, Nathanael Brian +4Speculative Decoding LLM Inference Acceleration
Huawei Technologies Co., Ltd. · University of Science and Technology of China
Hybrid Verified Decoding: Learning to Allocate Verification in Speculative Decoding
May 31, 2026Xin Su, Dawid Majchrowski, Fangyuan Yu +5Speculative Decoding LLM Inference Acceleration
Thoughtworks · Nvidia

Aug 25, 2026J/K move · Enter open · S save

AgentSpec: Speculative Decoding for Batch Inference of LLM Agents

Xin Wang, Ziming Miao, Yi Zhu, Hui Shen+3

The Ohio State University · Microsoft Research · University of Michigan

Speculative Decoding LLM Inference Acceleration LLM Agent Serving

Open paper AskPDF arXiv

Large language model (LLM)-based agent applications often incur high response time. Speculative decoding is a promising solution to improve the inference efficiency of LLM agents without impacting generation quality. However, state-of-the-art speculative decoding algorithms exhibit substantial speed degradation under large batch sizes, limiting their effectiveness to deploy in real-world agent applications. In this work, we first present a systematic analysis of speculative decoding for LLM agents and identify two dominant factors of speedup degradation: high rejection rate of speculative tokens, and under-utilization of dynamic token budgets.B ased on these observations, we propose AgentSpec, a speculative decoding algorithm that addresses the limitations of existing methods for LLM agents. AgentSpec incorporates structure-isolated drafting that constrains speculation to semantically coherent segments of the agent workflow, reducing the drafts of irrelevant semantic paths and achieving an extremely low rejection rate. Moreover, AgentSpec adopts redundancy-aware budget allocation that exploits agent-level information to better utilize the dynamically-free token budget during the agent inference. We implement and evaluate AgentSpec on five different workloads and four different models from four different LLM families in vLLM. Our results demonstrate the superiority of AgentSpec over state-of-the-arts.

Abstract

Speculative Decoding

Explore similar work

AgentSpec: Speculative Decoding for Batch Inference of LLM Agents

AsymSpec: Context-Asymmetric Speculative Decoding for Agentic LLMs

Hybrid Verified Decoding: Learning to Allocate Verification in Speculative Decoding