AgentSpec: Speculative Decoding for Batch Inference of LLM Agents

Xin Wang; Ziming Miao; Yi Zhu; Hui Shen; Zhongwei Wan; Fan Yang; Mi Zhang

cs.CLAug 25, 2026

AgentSpec: Speculative Decoding for Batch Inference of LLM Agents

Authors: Xin Wang, Ziming Miao, Yi Zhu, Hui Shen, Zhongwei Wan, Fan Yang, Mi Zhang

Organizations: The Ohio State University · Microsoft Research · University of Michigan

Abstract

Large language model (LLM)-based agent applications often incur high response time. Speculative decoding is a promising solution to improve the inference efficiency of LLM agents without impacting generation quality. However, state-of-the-art speculative decoding algorithms exhibit substantial speed degradation under large batch sizes, limiting their effectiveness to deploy in real-world agent applications. In this work, we first present a systematic analysis of speculative decoding for LLM agents and identify two dominant factors of speedup degradation: high rejection rate of speculative tokens, and under-utilization of dynamic token budgets.B ased on these observations, we propose AgentSpec, a speculative decoding algorithm that addresses the limitations of existing methods for LLM agents. AgentSpec incorporates structure-isolated drafting that constrains speculation to semantically coherent segments of the agent workflow, reducing the drafts of irrelevant semantic paths and achieving an extremely low rejection rate. Moreover, AgentSpec adopts redundancy-aware budget allocation that exploits agent-level information to better utilize the dynamically-free token budget during the agent inference. We implement and evaluate AgentSpec on five different workloads and four different models from four different LLM families in vLLM. Our results demonstrate the superiority of AgentSpec over state-of-the-arts.

Figures & tables

Figure 1: Speedup in terms of Goodput (number of generated tokens divided by the execution time) ( Liu et al., 2024 ) under various maximum batch size in vLLM for different speculative decoding algorithms on USACO dataset with Code Generation Agent implemented by Reflexion agentic workflow on Qwen-3-8B.

Metric	EAGLE-3	NGram	SuffixDecoding	AgentSpec
Rejection Rate	53.3%	88.7%	65.9%	26.4% ( $\downarrow 50\%$ )

Table 1: Average per-request rejection rate (

\downarrow

) , defined as number of rejected tokens divided by number of draft tokens, for different speculative decoding algorithms on Reflexion agentic workflow.

Figure 2: Execution time comparison for FFN and Attention modules of Qwen-3-8B under various batch size.

Figure 3: Comparison between maximum token budget

M(b)

and average proposed token number for batch inference of speculative decoding on Reflexion workflow.

Figure 5: Repeated token ratio under different minimum repeated span lengths

k

. The repeated ratio is defined as the total length of maximal repeated segments with at least

k

consecutive tokens appearing in historical generations, normalized by the total generation length. Repetitions are categorized by whether they occur under the same or different user queries and within the same or different semantic blocks.

Figure 6: Acceptance ratio (

1-\rho(b)

under different redundancy ratio with different matching pattern length.

Model	Method	Deep Research Agent		Code Generation Agent		GAIA		SWE-Bench
		Tokens/s	Speedup	Tokens/s	Speedup	Tokens/s	Speedup	Tokens/s	Speedup
Qwen-3-8B	Normal	617.41	$\times$ 1.00	636.09	$\times$ 1.00	501.27	$\times$ 1.00	652.44	$\times$ 1.00
	EAGLE-3	605.14	$\times$ 0.98	539.01	$\times$ 0.85	468.22	$\times$ 0.93	523.12	$\times$ 0.80
	NGram	509.62	$\times$ 0.83	422.58	$\times$ 0.66	308.27	$\times$ 0.61	408.28	$\times$ 0.63
	SuffixDecoding	598.74	$\times$ 0.97	579.21	$\times$ 0.91	498.16	$\times$ 0.99	556.64	$\times$ 0.85
	AgentSpec	661.43 ( $\uparrow 9\%$ )	$\times$ 1.07	828.10 ( $\uparrow 43\%$ )	$\times$ 1.30	537.78 ( $\uparrow 8\%$ )	$\times$ 1.07	854.69 ( $\uparrow 42\%$ )	$\times$ 1.31

Table 2: Goodput (tokens/sec) of AgentSpec and baselines on four different agentic workloads with three different models. The best performance is marked in bold. The speedup is measured by comparing the goodput with the normal autoregressive decoding. The relative performance gain compared to the best-performing baseline is marked in green inside bracket.

Figure 7: Speedup in terms of Goodput (token/s) for AgentSpec and baselines compared with normal autoregressive decoding on all subsets in USACO dataset with Code Generation Agent implemented by Reflexion agentic workflow on three different models.

Model	Method	Code Gen.(Bronze)		Code Gen.(Silver)		Code Gen.(Gold)		Code Gen.(Platinum)		Code Gen.(All)
		Tokens/s	Speedup	Tokens/s	Speedup	Tokens/s	Speedup	Tokens/s	Speedup	Tokens/s	Speedup
MiMo-7B	Normal	2768.67	$\times$ 1.00	3093.50	$\times$ 1.00	2532.30	$\times$ 1.00	1141.64	$\times$ 1.00	2572.16	$\times$ 1.00
	NGram	1322.92	$\times$ 0.48	1399.76	$\times$ 0.45	1383.92	$\times$ 0.55	990.87	$\times$ 0.87	1306.92	$\times$ 0.51
	SuffixDecoding	1609.41	$\times$ 0.58	1624.98	$\times$ 0.53	1599.63	$\times$ 0.63	1235.12	$\times$ 1.08	1653.26	$\times$ 0.64
	MTP	1924.46	$\times$ 0.70	2001.28	$\times$ 0.65	2034.30	$\times$ 0.80	1349.10	$\times$ 1.18	1875.18	$\times$ 0.73
	AgentSpec	3324.71 ( $\uparrow 73\%$ )	$\times$ 1.20	3751.44 ( $\uparrow 87\%$ )	$\times$ 1.21	2925.40 ( $\uparrow 44\%$ )	$\times$ 1.16	1527.14 ( $\uparrow 13\%$ )	$\times$ 1.34	3318.99 ( $\uparrow 77\%$ )	$\times$ 1.29

Table 3: Goodput (tokens/sec) of AgentSpec and baselines, including MTP, on MiMo-7B. The best performance is marked in bold. The speedup is measured by comparing the goodput with the normal autoregressive decoding. The relative performance gain compared to the best-performing baseline is marked in green inside bracket.

Model	Method	MT Conversation		Question Answering		Summarization		Translation		Math Reasoning		Retrieval Augmented		All
		Tokens/s	Speedup	Tokens/s	Speedup	Tokens/s	Speedup	Tokens/s	Speedup	Tokens/s	Speedup	Tokens/s	Speedup	Tokens/s	Speedup
Qwen-3-8B	Normal	939.50	$\times$ 1.00	860.97	$\times$ 1.00	1277.50	$\times$ 1.00	2013.02	$\times$ 1.00	1064.78	$\times$ 1.00	191.60	$\times$ 1.00	697.42	$\times$ 1.00
	EAGLE-3	1005.63	$\times$ 1.07	895.25	$\times$ 1.04	1552.62	$\times$ 1.22	2319.01	$\times$ 1.15	1045.28	$\times$ 0.98	168.21	$\times$ 0.88	783.25	$\times$ 1.12
	NGram	581.26	$\times$ 0.62	351.28	$\times$ 0.41	1082.57	$\times$ 0.85	2166.37	$\times$ 1.08	959.38	$\times$ 0.90	263.11	$\times$ 1.37	545.39	$\times$ 0.78
	SuffixDecoding	1007.08	$\times$ 1.07	873.12	$\times$ 1.01	1213.63	$\times$ 0.95	2141.79	$\times$ 1.06	1121.48	$\times$ 1.05	212.19	$\times$ 1.10	712.79	$\times$ 1.02
	AgentSpec	1128.09	$\times$ 1.20	929.85	$\times$ 1.08	1481.90	$\times$ 1.16	2294.12	$\times$ 1.14	1075.43	$\times$ 1.01	267.14	$\times$ 1.40	796.01	$\times$ 1.14

Table 4: Goodput (tokens/sec) of AgentSpec and baselines on Spec-Bench on Qwen-3-8B. The best performance is marked in bold. The speedup is measured by comparing the goodput with the normal autoregressive decoding.

Figure 8: Tailed latency analysis for AgentSpec and baselines on Code Gen agentic workflow.

Time (minutes)	EAGLE-3	NGram	SuffixDecoding	AgentSpec
Draft	5.72	0.68	1.41	1.74
Verification	34.24	39.82	33.12	26.11 ( $\downarrow 21\%$ )
Overall	39.96	40.50	34.53	27.85 ( $\downarrow 19\%$ )

Table 5: Execution breakdown of AgentSpec and baselines on Code Gen agentic Workload. The best performance is marked in bold.

Model	Method	Deep Research Agent		Code Generation Agent
		Tokens/s	Speedup	Tokens/s	Speedup
Qwen-3-8B	Normal	617.41	$\times$ 1.00	636.09	$\times$ 1.00
	EAGLE-3	605.14	$\times$ 0.98	539.01	$\times$ 0.85
	NGram	509.62	$\times$ 0.83	422.58	$\times$ 0.66
	SuffixDecoding	598.74	$\times$ 0.97	579.21	$\times$ 0.91
	AgentSpec (S)	647.12 ( $\uparrow 9\%$ )	$\times$ 1.05	795.54 ( $\uparrow 37\%$ )	$\times$ 1.25

Table 6: Goodput of variants of AgentSpec and baselines on Deep Research Agent and Code Generation Agent workloads. The best performance is marked in bold. The speedup is measured by comparing the goodput with the normal autoregressive decoding.

Figure 9: Speedup in terms of Goodput (tokens/sec) of AgentSpec and baselines on workload of code generation agent under various maximum batch size and different thinking modes.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Speedup	CodeGen	DeepResearch	SWE-Bench	GAIA
1 $\times$ A100 80G	2.02 $\times$	1.48 $\times$	1.69 $\times$	1.19 $\times$
1 $\times$ H100 80G	2.28 $\times$	1.67 $\times$	1.92 $\times$	1.54 $\times$

Appendix

Table 7: Speedup of AgentSpec on different GPU platforms on GPT-OSS-20B across agent workloads.

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 293 papers

Speculate with Memory: Lossless Acceleration for LLM Agents
Jul 14, 2026Yu Li, Qinyuan Ye, Prafulla Kumar Choubey +2Next-Action Prediction LLM Agent Memory
Salesforce Research
LEAP: Learning Efficient Action Proposals For LLM Agents
Oct 2, 2026Zhen Xu, Qizheng Zhang, Gerry Wan +2LLM Agents LLM Agent Training
University of Chicago · Stanford University · Together AI
AsymSpec: Context-Asymmetric Speculative Decoding for Agentic LLMs
Aug 26, 2026Sheng Liang, Yongyue Zhang, Nathanael Brian +4Speculative Decoding LLM Inference Acceleration
Huawei Technologies Co., Ltd. · University of Science and Technology of China

Jul 14, 2026J/K move · Enter open · S save

Speculate with Memory: Lossless Acceleration for LLM Agents

Yu Li, Qinyuan Ye, Prafulla Kumar Choubey, Jiaxin Zhang+1

Salesforce Research

Next-Action Prediction LLM Agent Memory LLM Inference Acceleration Memory-Augmented Agents

Open paper AskPDF arXiv

Speculative execution accelerates LLM agents by using a smaller, cheaper model to predict and pre-launch the next step while the environment is idle. However, existing speculators are stateless and discard all information between tasks, preventing prediction quality from improving with experience. We equip the speculator with three online memory systems that learn from past agent trajectories: a contrastive transition table tracking action-sequence statistics, an episodic memory retrieving contextually similar segments, and a confusion tracker suppressing recurring errors. We evaluate this approach on six benchmarks spanning three speculation types: action prediction, observation prediction, and chained prediction. Memory-augmented speculation yields a 19--39% relative accuracy improvement on action prediction and up to a

2.5\times

increase on observation prediction tasks with repetitive action spaces. These gains grow continuously as memory accumulates and generalize across speculator models of varying cost. All speculation is lossless because it runs during idle time at zero added wall-clock cost, and the actor's trajectory is identical to non-speculative execution.

Abstract

Speculative Decoding

Explore similar work

Speculate with Memory: Lossless Acceleration for LLM Agents

LEAP: Learning Efficient Action Proposals For LLM Agents

AsymSpec: Context-Asymmetric Speculative Decoding for Agentic LLMs