AsymSpec: Context-Asymmetric Speculative Decoding for Agentic LLMs

Sheng Liang; Yongyue Zhang; Nathanael Brian; Hang Lv; Hao Wang; Chen Zhang; Yong Liu

cs.AIAug 26, 2026

AsymSpec: Context-Asymmetric Speculative Decoding for Agentic LLMs

Authors: Sheng Liang, Yongyue Zhang, Nathanael Brian, Hang Lv, Hao Wang, Chen Zhang, Yong Liu

Organizations: Huawei Technologies Co., Ltd. · University of Science and Technology of China

Abstract

Agentic LLM pipelines face escalating inference costs as context accumulates across retrieval, tool use, and multi-turn interactions. To control latency, deployments routinely compress inputs, but this degrades task accuracy. Speculative decoding (SD) accelerates generation losslessly, yet it assumes the drafter and verifier share an identical context, preventing SD from resolving the accuracy-overhead trade-off. We propose AsymSpec, an asymmetric speculative decoding framework that breaks this symmetry: a lightweight drafter reads the full input while the large verifier operates on the compressed view. The drafter steers the verifier via a contrastive $δ$ -fusion of logits, modulated by a divergence-aware acceptance gate that preserves verification stability and high draft acceptance rates. Evaluated across four agentic capabilities and two end-to-end agent benchmarks, AsymSpec reaches $\approx 90\%$ of full-context accuracy on average, delivering $1.3$ -- $1.7\times$ throughput speedups at $0.2$ -- $0.3\times$ the compute cost on isolated text capabilities. These results show that asymmetric context access yields substantial gains precisely when compression discards critical reasoning signals.

Figures & tables

Figure 1: AsymSpec speculation step. (a) Drafting & verification: drafter $S$ reads $x_{\text{full}}$ (logits $a$ , drafts $d_{1:K}$ ) and $x_{\text{comp}}$ (logits $b$ ); verifier $L$ reads $x_{\text{comp}}$ (logits $t$ ). (b) CDA gate & $\delta$ -fusion: $\gamma_{\text{eff}}(i)$ relaxes as full-vs-compressed divergence grows; accepted drafts are committed, otherwise the $\delta$ -fused $\arg\max$ is emitted. Only $S$ consumes $x_{\text{full}}$ .

	Long-context multi-hop (LongBench)			Multi-turn	Tool use	Speedup	FLOPs
Method	hotpotQA	2WikiMQA	MuSiQue	(MultiChal.)	(API-Bank)	over Ceiling	over Ceiling
Floor	49.4	52.8	32.7	23.4	57.7	1.17 $\times$	0.11 $\times$
Ceiling	64.9	76.5	55.0	26.4	66.1	1.00 $\times$	1.00 $\times$
SD	65.5	76.6	55.1	26.7	66.1	1.73 $\times$	1.04 $\times$
SCD ( Yuan et al., 2023 )	46.6	52.7	32.0	22.6	56.7	$1.04\times$	$0.12\times$
RAPID ( Chen et al., 2025 )	63.2	75.3	52.5	25.8	64.3	1.38 $\times$	1.01 $\times$

Table 1: Accuracy and efficiency on isolated agentic capabilities (

K{=}2

). AsymSpec closes

59

–

94\%

of the Floor–Ceiling gap on long-context multi-hop QA and tool use, at

0.23\times

the Ceiling’s compute—a trade-off point inaccessible to symmetric SD or SCD. Speedup and FLOPs are averaged across the three text benchmarks; FLOPs follow the parameters

\times

token-ratio convention. Drafter-alone accuracy across drafter sizes is in Appendix H . Metric definitions in § 4.3 ; per-benchmark FLOPs in Table 5 .

Trunc tokens	Floor	AsymSpec	$\Delta$
500	25.8	52.5	$+26.7$
1500	32.6	53.7	$+21.1$
3000	39.5	55.5	$+16.0$
6000	50.6	59.2	$+8.6$
12000	63.1	63.9	$+0.8$

Table 2: Truncation-budget sweep on LongBench (

K{=}2

\beta{=}1

, 4B drafter; overall F1). Recovery scales monotonically with compression severity and vanishes as the verifier approaches the uncompressed Ceiling (

65.5

F1, full context).

Method	GPS	VQA	FQA	Overall
VL drafter alone	53.4	56.4	67.7	60.5
Floor	62.0	49.1	29.0	44.5
SD	62.0	46.4	28.6	43.8
AsymSpec (ours)	62.1	59.1	45.7	53.9

Table 3: MathVista results under the cross-modal setup. The text-only verifier cannot consume raw images, so the Ceiling is undefined; the VL drafter alone (Qwen3-VL-2B reading the image) is the reference upper bound. Floor is the verifier on the official Bard caption plus EasyOCR text; SD is symmetric speculative decoding with a text drafter on the same caption input.

GAIA
Setting	Method	Acc	Spd	FLOPs
Web-only	Floor	17.3
	Ceiling	18.9
	AsymSpec(4B)	22.0
	AsymSpec(vl-2B)	24.4
File-attach	Floor	26.3

Table 4: End-to-end accuracy and efficiency in live agentic loops, with GAIA per-subset breakdown and SimpleQA. On GAIA, “Ceiling” is the best per-subset reference available under each harness: Qwen3-32B-on-full for the web subset; the vl-2B-drafter-alone reference for the file-attachment subset. Speedup and FLOPs measure LLM-only inference at the aggregate level.

Setting	$L_{\text{full}}$	$L_{\text{comp}}$	ratio	Verifier	Total
GAIA	4650	2434	1.9 $\times$	49%	0.78 $\times$
SimpleQA	3606	2712	1.3 $\times$	74%	0.80 $\times$
LongBench	12437	1532	8.1 $\times$	9%	0.28 $\times$
MultiChallenge	1598	211	7.6 $\times$	13%	0.28 $\times$
API-Bank	6701	909	7.4 $\times$	12%	0.19 $\times$

Table 5: Per-step prefill FLOPs across all benchmarks, computed from the Qwen3 architecture and measured context lengths. Verifier is the

32

B forward on

x_{\text{comp}}

as a fraction of the full-context baseline; Total adds both drafter forwards (conservatively counted as full prefills). Compute reduction ranges from

0.19\times

on heavy-compression benchmarks (API-Bank,

7.4\times

token ratio) to

0.80\times

on light-compression SimpleQA (

1.33\times

ratio), tracking compression headroom monotonically.

Variant	LongBench F1
Floor	45.0
$+$ CDA gate ( $\beta{=}0$ )	52.8
$+$ $\delta$ -fusion, raw-aug ( $a$ only)	56.9
$+$ $\delta$ -fusion, SCD-style ( $t-b$ )	48.0
$+$ $\delta$ -fusion, ours ( $a-b$ )	59.7
Ceiling	65.5

Table 6: Mechanism ablation on LongBench (

K{=}2

, 4B drafter,

\gamma{=}0.5

). The CDA gate and

\delta

-fusion are both necessary; our same-model

\delta

source (

a-b

) outperforms the raw-augmented (

a

) and SCD-style (

t-b

) alternatives by

3

–

12

F1 points. Full hyperparameter and compressor sweeps are in Appendix G .

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	max_new	max_model_len	Harness / prompt
LongBench	1024	24576	QA on passage(s)
MultiChallenge	2000	8192	official dataset prompt
API-Bank (Method A)	256	16384	API specs + dialog $\to$ [Tool(args)]
MathVista	1024	8192	VQA prompt; image via VL drafter
GAIA (web, $n{=}127$ )	1024	32768	smolagents CodeAgent, max_steps =8
GAIA (file, $n{=}38$ )	1024	32768	single-shot file-QA

Appendix

Table 7: Per-benchmark generation parameters and harness bounds. max_new is the per-call output token cap; max_model_len is the vLLM context window. GAIA splits into a web-only ReAct loop and a single-shot file-attachment harness, listed separately.

Method	Drafter ctx	Verifier ctx	$\delta$ source
SD ( Leviathan et al., 2023 )	=	=	—
EAGLE ( Li et al., 2024c )	=	=	—
CD ( Li et al., 2023b )	expert	—	capacity
SCD ( Yuan et al., 2023 )	expert	amateur	capacity
RAPID ( Chen et al., 2025 )	short	long	—
VL-SD ( Huang et al., 2025 ; Huo et al., 2025 ; Kang et al., 2025 )	VL	VL	—

Appendix

Table 8: Design-space placement. “

\delta

source” is what the linear logit combination measures.

Capability	Benchmark	$n$	Compression	Metric
Long-context multi-hop QA	LongBench	600	8.1 $\times$	per-subset F1
Multi-turn instruction following	MultiChallenge	271	7.6 $\times$	llm-judge acc
Tool use	API-Bank	200	7.4 $\times$	API-call exact match
Multimodal reasoning	MathVista	587	cross-modal	accuracy
End-to-end	GAIA L1–3 (full)	165	per-turn / per-file	GAIA exact match
End-to-end	SimpleQA	500	per-turn	llm-judge ( Wei et al., 2024 )

Appendix

Table 9: Capabilities and benchmarks. Compression is the token ratio

|x_{\text{full}}|/|x_{\text{comp}}|

; per-turn LLMLingua-2 for GAIA/SimpleQA ( Appendix B ). LongBench is

200

examples each from hotpotQA, 2WikiMQA, MuSiQue, reported per-subset ( Table 1 ).

Drafter $\rightarrow$ verifier	Floor	Ours	Ceiling	Recovery
Llama-3B $\rightarrow$ Llama-70B	50.6	54.2	66.3	23%
Qwen-4B $\rightarrow$ Llama-70B	50.6	58.4	66.3	50%
Llama-3B $\rightarrow$ Qwen-32B	45.0	47.1	65.5	10%
Qwen-4B $\rightarrow$ Qwen-32B	45.0	59.7	65.5	72%

Appendix

Table 10: LongBench portability across model families (overall mean F1). Floor and Ceiling run each verifier on the compressed and full contexts, respectively; recovery is

(\textsc{AsymSpec}-\text{Floor})/(\text{Ceiling}-\text{Floor})

. The Qwen–Llama rows improve over their respective Floors, with recovery varying across model pairs.

	LongBench	MultiChallenge	API-Bank	MathVista
Method	(4B, K=2)	(4B, K=2)	(1.7B, K=2)	(VL-2B, K=4)
Floor (verifier on compressed)	50.0	52.3	51.2	49.2
Ceiling (verifier on full)	37.5	51.5	49.5	81.7
SD	50.9	77.5	70.0	65.6
fixed- $\gamma$ (no gate)	57.8	68.6	67.2	44.4
$+$ tuned $\lambda$ ( $0.1$ )	88.2	96.0	66.6	46.9

Appendix

Table 11: Throughput (tokens/s) across four benchmarks at our default configurations on a single accelerator; eager mode (no graph capture). MathVista has no text-only Ceiling since Qwen3-32B cannot consume images; the VL drafter alone (Qwen3-VL-2B reading the image) at 81.7 tokens/s is the closest reference upper bound. Eager throughput exhibits non-trivial per-run variance on text benchmarks; the accuracy-equivalent main-table efficiency is consolidated in Table 1 . CDA adds only one scalar division per token over the tuned

\lambda

-variant, so observed gaps reflect measurement variance, not gate overhead. MathVista throughput includes the per-request vision-tower forward, amortized via the cache in Appendix E (patch 2); batched, graph-captured measurement would tighten these further. The final row reports CDA throughput as a ratio over Ceiling—the per-benchmark speedup—with MathVista omitted as it has no text-only Ceiling.

	LongBench F1			MultiChallenge acc			MathVista acc
Method	$K{=}2$	$K{=}4$	$K{=}6$	$K{=}2$	$K{=}4$	$K{=}6$	$K{=}2$	$K{=}4$	$K{=}6$
fixed- $\gamma$ (no gate)	57.0	59.7	57.4	25.5	22.9	20.2	53.0	51.1	50.4
$+$ tuned $\lambda$ ( $0.1$ )	59.0	59.3	57.8	23.2	21.8	17.3	52.5	52.5	50.7
$+$ CDA (ours)	59.7	61.1	58.7	22.5	19.6	18.5	52.6	53.9	52.1

Appendix

Table 12:

K

-sweep at Qwen3-4B drafter (Qwen3-VL-2B for MathVista),

\gamma{=}0.5

\beta{=}1.0

. MultiChallenge declines at

K{=}4

for every method and none recovers its

K{=}2

level — the binding constraint behind the

K{=}2

default (§ 6.1 ); cross-modal MathVista is the lone

K{=}4

beneficiary ( Table 3 ). The

K{=}6

sweep saturates on LongBench (CDA

58.7

, below the

K{=}4

peak of

61.1

) and degrades on MultiChallenge / MathVista, confirming the

K{=}2

default.

Setting	AR	MAL
LongBench, $K{=}2$ , 4B, summary	0.86	2.72
LongBench, $K{=}4$ , 4B, truncate	0.78	4.13
MultiChallenge, $K{=}2$ , 4B, llmlingua	0.80	2.61
API-Bank, $K{=}2$ , 1.7B, signature	0.92	2.84
GAIA, $K{=}2$ , 4B, per-turn llmlingua	0.90	2.81

Appendix

Table 13: Speculative-acceptance diagnostics at our default configurations. AR and MAL remain near vanilla-SD levels under asymmetric context, including in the multi-turn GAIA agent loop.

Benchmark	$D_{i}$ gate	$K{=}2$	$K{=}4$
LongBench	KL (unbounded)	58.6	59.6
LongBench	JSD ( $\leq\ln 2$ , ours)	59.7	61.1
API-Bank	KL (unbounded)	63.5	62.6
API-Bank	JSD ( $\leq\ln 2$ , ours)	63.5	62.6

Appendix

Table 14: JSD vs. KL instantiation of CDA’s divergence

D_{i}

( Equation 3 ;

\gamma{=}0.5

; LongBench overall F1, 4B drafter; API-Bank api-acc, 1.7B drafter). On API-Bank the two gates are identical down to the pass count (339/534 at

K{=}2

, 334/534 at

K{=}4

): API-Bank’s small context divergences leave JSD and KL with the same accept/reject decisions. On LongBench they differ by

\leq 1.5

F1. JSD is chosen for the universal bound (§ 3.3 ), not for accuracy, and is used in all other tables.

$\gamma$	0.4	0.5	0.6	0.7
LongBench F1	56.8	59.7	58.8	59.1
API-Bank acc	63.7	63.5	63.7	63.5
MultiChallenge acc	23.3	23.3	22.1	23.6

Appendix

Table 15: CDA is flat in its only free quantity, the standard speculative-decoding threshold

\gamma

(parameter-free gate

\gamma e^{-D}

K{=}2

; LongBench overall F1, 4B drafter; API-Bank api-acc, 1.7B; MultiChallenge acc, 4B). No cliff on any of the three.

	$K{=}2$			$K{=}4$
$\beta$	F1	AR	MAL	F1	AR	MAL
0.5	50.3	0.853	2.71	57.5	0.781	4.12
1.0	53.7	0.852	2.70	58.0	0.783	4.13
2.0	53.9	0.854	2.71	57.4	0.779	4.12

Appendix

Table 16:

\beta

sweep on LongBench (truncate-1500, 4B drafter,

\gamma{=}0.5

). Within

\beta\in\{1.0,2.0\}

F1 is flat (within

0.2

K{=}2

and

0.6

K{=}4

);

\beta{=}0.5

underperforms by

3

–

3.6

F1 at

K{=}2

, motivating

\beta{=}1.0

as a stable midpoint. Acceptance rate (AR) and mean accepted length (MAL) are flat throughout.

$x_{\text{comp}}$ source	Floor	SCD	AsymSpec	Recovery
Summary	45.0	42.3	58.6	$66\%$
LLMLingua-2	36.1	31.7	54.6	$63\%$
Truncate-1500	32.6	28.8	53.7	$64\%$
Question-only	29.5	25.8	54.8	$70\%$
Ceiling	65.5

Appendix

Table 17: Compressor probe on LongBench (4B drafter,

K{=}2

\gamma{=}0.5

). Recovery is

(\text{AsymSpec}-\text{Floor})/(\text{Ceiling}-\text{Floor})

Drafter	$K$	Drafter alone	fixed- $\gamma$	$+$ tuned $\lambda$	$+$ CDA (ours)
0.6B	2	23.7	50.6	48.4	47.8
0.6B	4	23.7	48.5	46.6	46.4
1.7B	2	38.9	52.9	51.1	50.8
1.7B	4	38.9	48.5	53.6	53.9
4B	2	54.6	57.0	59.0	59.7
4B	4	54.6	59.7	59.3	61.1

Appendix

Table 18: Drafter-size sweep on LongBench F1 (

\gamma{=}0.5

\lambda{=}0.1

for the tuned variant). CDA is the parameter-free JSD gate (§ 3.3 ); the 4B row (our default drafter) is its JSD instantiation of record, the smaller-drafter rows the accuracy-equivalent KL instantiation ( Table 14 ). Drafter alone : the SLM on full passages, no verifier — reference upper bound for what the small model can do unassisted (K-independent).

$\lambda$	MultiChallenge acc
0.05	21.0
0.10 (default tune)	23.2
0.20	23.6
CDA (no scale)	22.5

Appendix

Table 19:

\lambda

sweep of the tuned variant on MultiChallenge under the llm-judge (

n{=}271

, 4B drafter,

\gamma{=}0.5

K{=}2

). The spread is only

2.6

pp (

21.0

–

23.6

). CDA (tuning-free,

22.5

) lies within the same band with no

\lambda

to set.

Drafter	$K$	Drafter alone	fixed- $\gamma$	$+$ tuned $\lambda$	$+$ CDA (ours)
0.6B	2	31.3	59.4	59.6	59.6
0.6B	4	31.3	58.2	57.9	57.9
1.7B	2	61.2	63.7	63.5	63.5
1.7B	4	61.2	62.4	62.5	62.5
4B	2	64.0	60.3	60.7	60.7
4B	4	64.0	63.3	63.7	63.7

Appendix

Table 20: Drafter-size sweep on API-Bank api_acc (

\gamma{=}0.5

\lambda{=}0.1

for the tuned variant). Drafter alone : the SLM alone on full API context, no verifier (K-independent).

Drafter	$K$	Drafter alone	fixed- $\gamma$	$+$ tuned $\lambda$	$+$ CDA (ours)
0.6B	2	13.3	20.3	19.6	21.4
0.6B	4	13.3	21.8	22.9	22.1
1.7B	2	14.0	19.6	20.7	19.6
1.7B	4	14.0	21.8	18.1	21.4
4B	2	25.8	25.5	23.2	22.5
4B	4	25.8	22.9	21.8	19.6

Appendix

Table 21: Drafter-size sweep on MultiChallenge accuracy (

\gamma{=}0.5

, llm-judge,

n{=}271

\lambda{=}0.1

for the tuned variant; single run). Every per-cell gap lies within the

\leq

3 pp headroom on this near-lossless task (§ 6.4 ).

Speculative Decoding

Speculative decoding accelerates generation by drafting multiple tokens and verifying them in parallel with a larger target model.

See all 293 papers

AgentSpec: Speculative Decoding for Batch Inference of LLM Agents
Aug 25, 2026Xin Wang, Ziming Miao, Yi Zhu +4Speculative Decoding LLM Inference Acceleration
The Ohio State University · Microsoft Research · University of Michigan
Beyond the Target: From Imitation to Collaboration in Speculative Decoding
May 24, 2026Jinze Li, Yixing Xu, Guanchen Li +7Speculative Decoding LLM Inference Acceleration
Advanced Micro Devices, Inc., Beijing, China · The University of Hong Kong · University of North Texas
DLite: History-free Speculative Decoding for Agentic AI
Oct 16, 2026Jiaming Xu, Hanzhen Wang, Tianlang Zhao +7Speculative Decoding LLM Inference Acceleration
Shanghai Jiao Tong University · Shanghai Innovation Institute · Tsinghua University +1

Aug 25, 2026J/K move · Enter open · S save

AgentSpec: Speculative Decoding for Batch Inference of LLM Agents

Xin Wang, Ziming Miao, Yi Zhu, Hui Shen+3

The Ohio State University · Microsoft Research · University of Michigan

Speculative Decoding LLM Inference Acceleration LLM Agent Serving

Open paper AskPDF arXiv

Large language model (LLM)-based agent applications often incur high response time. Speculative decoding is a promising solution to improve the inference efficiency of LLM agents without impacting generation quality. However, state-of-the-art speculative decoding algorithms exhibit substantial speed degradation under large batch sizes, limiting their effectiveness to deploy in real-world agent applications. In this work, we first present a systematic analysis of speculative decoding for LLM agents and identify two dominant factors of speedup degradation: high rejection rate of speculative tokens, and under-utilization of dynamic token budgets.B ased on these observations, we propose AgentSpec, a speculative decoding algorithm that addresses the limitations of existing methods for LLM agents. AgentSpec incorporates structure-isolated drafting that constrains speculation to semantically coherent segments of the agent workflow, reducing the drafts of irrelevant semantic paths and achieving an extremely low rejection rate. Moreover, AgentSpec adopts redundancy-aware budget allocation that exploits agent-level information to better utilize the dynamically-free token budget during the agent inference. We implement and evaluate AgentSpec on five different workloads and four different models from four different LLM families in vLLM. Our results demonstrate the superiority of AgentSpec over state-of-the-arts.

Abstract

Speculative Decoding

Explore similar work

AgentSpec: Speculative Decoding for Batch Inference of LLM Agents

Beyond the Target: From Imitation to Collaboration in Speculative Decoding

DLite: History-free Speculative Decoding for Agentic AI