Agentic LLM pipelines face escalating inference costs as context accumulates across retrieval, tool use, and multi-turn interactions. To control latency, deployments routinely compress inputs, but this degrades task accuracy. Speculative decoding (SD) accelerates generation losslessly, yet it assumes the drafter and verifier share an identical context, preventing SD from resolving the accuracy-overhead trade-off. We propose AsymSpec, an asymmetric speculative decoding framework that breaks this symmetry: a lightweight drafter reads the full input while the large verifier operates on the compressed view. The drafter steers the verifier via a contrastive δ-fusion of logits, modulated by a divergence-aware acceptance gate that preserves verification stability and high draft acceptance rates. Evaluated across four agentic capabilities and two end-to-end agent benchmarks, AsymSpec reaches ≈90% of full-context accuracy on average, delivering 1.3--1.7× throughput speedups at 0.2--0.3× the compute cost on isolated text capabilities. These results show that asymmetric context access yields substantial gains precisely when compression discards critical reasoning signals.
Figures & tables
Figure 1: AsymSpec speculation step. (a) Drafting & verification: drafter S reads xfull (logits a , drafts d1:K ) and xcomp (logits b ); verifier L reads xcomp (logits t ). (b) CDA gate & δ -fusion: γeff(i) relaxes as full-vs-compressed divergence grows; accepted drafts are committed, otherwise the δ -fused argmax is emitted. Only S consumes xfull .
Long-context multi-hop (LongBench)
Multi-turn
Tool use
Speedup
FLOPs
Method
hotpotQA
2WikiMQA
MuSiQue
(MultiChal.)
(API-Bank)
over Ceiling
over Ceiling
Floor
49.4
52.8
32.7
23.4
57.7
1.17 ×
0.11 ×
Ceiling
64.9
76.5
55.0
26.4
66.1
1.00 ×
1.00 ×
SD
65.5
76.6
55.1
26.7
66.1
1.73 ×
1.04 ×
SCD ( Yuan et al., 2023 )
46.6
52.7
32.0
22.6
56.7
1.04×
0.12×
RAPID ( Chen et al., 2025 )
63.2
75.3
52.5
25.8
64.3
1.38 ×
1.01 ×
Table 1: Accuracy and efficiency on isolated agentic capabilities ( K=2 ). AsymSpec closes 59 – 94% of the Floor–Ceiling gap on long-context multi-hop QA and tool use, at 0.23× the Ceiling’s compute—a trade-off point inaccessible to symmetric SD or SCD. Speedup and FLOPs are averaged across the three text benchmarks; FLOPs follow the parameters × token-ratio convention. Drafter-alone accuracy across drafter sizes is in Appendix H . Metric definitions in § 4.3 ; per-benchmark FLOPs in Table 5 .
Trunc tokens
Floor
AsymSpec
Δ
500
25.8
52.5
+26.7
1500
32.6
53.7
+21.1
3000
39.5
55.5
+16.0
6000
50.6
59.2
+8.6
12000
63.1
63.9
+0.8
Table 2: Truncation-budget sweep on LongBench ( K=2 , β=1 , 4B drafter; overall F1). Recovery scales monotonically with compression severity and vanishes as the verifier approaches the uncompressed Ceiling ( 65.5 F1, full context).
Method
GPS
VQA
FQA
Overall
VL drafter alone
53.4
56.4
67.7
60.5
Floor
62.0
49.1
29.0
44.5
SD
62.0
46.4
28.6
43.8
AsymSpec (ours)
62.1
59.1
45.7
53.9
Table 3: MathVista results under the cross-modal setup. The text-only verifier cannot consume raw images, so the Ceiling is undefined; the VL drafter alone (Qwen3-VL-2B reading the image) is the reference upper bound. Floor is the verifier on the official Bard caption plus EasyOCR text; SD is symmetric speculative decoding with a text drafter on the same caption input.
Setting
Method
Acc
Spd
FLOPs
GAIA
Web-only
Floor
17.3
Ceiling
18.9
AsymSpec(4B)
22.0
AsymSpec(vl-2B)
24.4
File-attach
Floor
26.3
Table 4: End-to-end accuracy and efficiency in live agentic loops, with GAIA per-subset breakdown and SimpleQA. On GAIA, “Ceiling” is the best per-subset reference available under each harness: Qwen3-32B-on-full for the web subset; the vl-2B-drafter-alone reference for the file-attachment subset. Speedup and FLOPs measure LLM-only inference at the aggregate level.
Setting
Lfull
Lcomp
ratio
Verifier
Total
GAIA
4650
2434
1.9 ×
49%
0.78 ×
SimpleQA
3606
2712
1.3 ×
74%
0.80 ×
LongBench
12437
1532
8.1 ×
9%
0.28 ×
MultiChallenge
1598
211
7.6 ×
13%
0.28 ×
API-Bank
6701
909
7.4 ×
12%
0.19 ×
Table 5: Per-step prefill FLOPs across all benchmarks, computed from the Qwen3 architecture and measured context lengths. Verifier is the 32 B forward on xcomp as a fraction of the full-context baseline; Total adds both drafter forwards (conservatively counted as full prefills). Compute reduction ranges from 0.19× on heavy-compression benchmarks (API-Bank, 7.4× token ratio) to 0.80× on light-compression SimpleQA ( 1.33× ratio), tracking compression headroom monotonically.
Variant
LongBench F1
Floor
45.0
+ CDA gate ( β=0 )
52.8
+δ -fusion, raw-aug ( a only)
56.9
+δ -fusion, SCD-style ( t−b )
48.0
+δ -fusion, ours ( a−b )
59.7
Ceiling
65.5
Table 6: Mechanism ablation on LongBench ( K=2 , 4B drafter, γ=0.5 ). The CDA gate and δ -fusion are both necessary; our same-model δ source ( a−b ) outperforms the raw-augmented ( a ) and SCD-style ( t−b ) alternatives by 3 – 12 F1 points. Full hyperparameter and compressor sweeps are in Appendix G .
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
max_new
max_model_len
Harness / prompt
LongBench
1024
24576
QA on passage(s)
MultiChallenge
2000
8192
official dataset prompt
API-Bank (Method A)
256
16384
API specs + dialog → [Tool(args)]
MathVista
1024
8192
VQA prompt; image via VL drafter
GAIA (web, n=127 )
1024
32768
smolagents CodeAgent, max_steps =8
GAIA (file, n=38 )
1024
32768
single-shot file-QA
Appendix
Table 7: Per-benchmark generation parameters and harness bounds. max_new is the per-call output token cap; max_model_len is the vLLM context window. GAIA splits into a web-only ReAct loop and a single-shot file-attachment harness, listed separately.
Method
Drafter ctx
Verifier ctx
δ source
SD ( Leviathan et al., 2023 )
=
=
—
EAGLE ( Li et al., 2024c )
=
=
—
CD ( Li et al., 2023b )
expert
—
capacity
SCD ( Yuan et al., 2023 )
expert
amateur
capacity
RAPID ( Chen et al., 2025 )
short
long
—
VL-SD ( Huang et al., 2025 ; Huo et al., 2025 ; Kang et al., 2025 )
VL
VL
—
Appendix
Table 8: Design-space placement. “ δ source” is what the linear logit combination measures.
Capability
Benchmark
n
Compression
Metric
Long-context multi-hop QA
LongBench
600
8.1 ×
per-subset F1
Multi-turn instruction following
MultiChallenge
271
7.6 ×
llm-judge acc
Tool use
API-Bank
200
7.4 ×
API-call exact match
Multimodal reasoning
MathVista
587
cross-modal
accuracy
End-to-end
GAIA L1–3 (full)
165
per-turn / per-file
GAIA exact match
End-to-end
SimpleQA
500
per-turn
llm-judge ( Wei et al., 2024 )
Appendix
Table 9: Capabilities and benchmarks. Compression is the token ratio ∣xfull∣/∣xcomp∣ ; per-turn LLMLingua-2 for GAIA/SimpleQA ( Appendix B ). LongBench is 200 examples each from hotpotQA, 2WikiMQA, MuSiQue, reported per-subset ( Table 1 ).
Drafter → verifier
Floor
Ours
Ceiling
Recovery
Llama-3B → Llama-70B
50.6
54.2
66.3
23%
Qwen-4B → Llama-70B
50.6
58.4
66.3
50%
Llama-3B → Qwen-32B
45.0
47.1
65.5
10%
Qwen-4B → Qwen-32B
45.0
59.7
65.5
72%
Appendix
Table 10: LongBench portability across model families (overall mean F1). Floor and Ceiling run each verifier on the compressed and full contexts, respectively; recovery is (\textscAsymSpec−Floor)/(Ceiling−Floor) . The Qwen–Llama rows improve over their respective Floors, with recovery varying across model pairs.
LongBench
MultiChallenge
API-Bank
MathVista
Method
(4B, K=2)
(4B, K=2)
(1.7B, K=2)
(VL-2B, K=4)
Floor (verifier on compressed)
50.0
52.3
51.2
49.2
Ceiling (verifier on full)
37.5
51.5
49.5
81.7
SD
50.9
77.5
70.0
65.6
fixed- γ (no gate)
57.8
68.6
67.2
44.4
+ tuned λ ( 0.1 )
88.2
96.0
66.6
46.9
Appendix
Table 11: Throughput (tokens/s) across four benchmarks at our default configurations on a single accelerator; eager mode (no graph capture). MathVista has no text-only Ceiling since Qwen3-32B cannot consume images; the VL drafter alone (Qwen3-VL-2B reading the image) at 81.7 tokens/s is the closest reference upper bound. Eager throughput exhibits non-trivial per-run variance on text benchmarks; the accuracy-equivalent main-table efficiency is consolidated in Table 1 . CDA adds only one scalar division per token over the tuned λ -variant, so observed gaps reflect measurement variance, not gate overhead. MathVista throughput includes the per-request vision-tower forward, amortized via the cache in Appendix E (patch 2); batched, graph-captured measurement would tighten these further. The final row reports CDA throughput as a ratio over Ceiling—the per-benchmark speedup—with MathVista omitted as it has no text-only Ceiling.
LongBench F1
MultiChallenge acc
MathVista acc
Method
K=2
K=4
K=6
K=2
K=4
K=6
K=2
K=4
K=6
fixed- γ (no gate)
57.0
59.7
57.4
25.5
22.9
20.2
53.0
51.1
50.4
+ tuned λ ( 0.1 )
59.0
59.3
57.8
23.2
21.8
17.3
52.5
52.5
50.7
+ CDA (ours)
59.7
61.1
58.7
22.5
19.6
18.5
52.6
53.9
52.1
Appendix
Table 12: K -sweep at Qwen3-4B drafter (Qwen3-VL-2B for MathVista), γ=0.5 , β=1.0 . MultiChallenge declines at K=4 for every method and none recovers its K=2 level — the binding constraint behind the K=2 default (§ 6.1 ); cross-modal MathVista is the lone K=4 beneficiary ( Table 3 ). The K=6 sweep saturates on LongBench (CDA 58.7 , below the K=4 peak of 61.1 ) and degrades on MultiChallenge / MathVista, confirming the K=2 default.
Setting
AR
MAL
LongBench, K=2 , 4B, summary
0.86
2.72
LongBench, K=4 , 4B, truncate
0.78
4.13
MultiChallenge, K=2 , 4B, llmlingua
0.80
2.61
API-Bank, K=2 , 1.7B, signature
0.92
2.84
GAIA, K=2 , 4B, per-turn llmlingua
0.90
2.81
Appendix
Table 13: Speculative-acceptance diagnostics at our default configurations. AR and MAL remain near vanilla-SD levels under asymmetric context, including in the multi-turn GAIA agent loop.
Benchmark
Di gate
K=2
K=4
LongBench
KL (unbounded)
58.6
59.6
JSD ( ≤ln2 , ours)
59.7
61.1
API-Bank
KL (unbounded)
63.5
62.6
JSD ( ≤ln2 , ours)
63.5
62.6
Appendix
Table 14: JSD vs. KL instantiation of CDA’s divergence Di ( Equation 3 ; γ=0.5 ; LongBench overall F1, 4B drafter; API-Bank api-acc, 1.7B drafter). On API-Bank the two gates are identical down to the pass count (339/534 at K=2 , 334/534 at K=4 ): API-Bank’s small context divergences leave JSD and KL with the same accept/reject decisions. On LongBench they differ by ≤1.5 F1. JSD is chosen for the universal bound (§ 3.3 ), not for accuracy, and is used in all other tables.
γ
0.4
0.5
0.6
0.7
LongBench F1
56.8
59.7
58.8
59.1
API-Bank acc
63.7
63.5
63.7
63.5
MultiChallenge acc
23.3
23.3
22.1
23.6
Appendix
Table 15: CDA is flat in its only free quantity, the standard speculative-decoding threshold γ (parameter-free gate γe−D , K=2 ; LongBench overall F1, 4B drafter; API-Bank api-acc, 1.7B; MultiChallenge acc, 4B). No cliff on any of the three.
K=2
K=4
β
F1
AR
MAL
F1
AR
MAL
0.5
50.3
0.853
2.71
57.5
0.781
4.12
1.0
53.7
0.852
2.70
58.0
0.783
4.13
2.0
53.9
0.854
2.71
57.4
0.779
4.12
Appendix
Table 16: β sweep on LongBench (truncate-1500, 4B drafter, γ=0.5 ). Within β∈{1.0,2.0} F1 is flat (within 0.2 at K=2 and 0.6 at K=4 ); β=0.5 underperforms by 3 – 3.6 F1 at K=2 , motivating β=1.0 as a stable midpoint. Acceptance rate (AR) and mean accepted length (MAL) are flat throughout.
xcomp source
Floor
SCD
AsymSpec
Recovery
Summary
45.0
42.3
58.6
66%
LLMLingua-2
36.1
31.7
54.6
63%
Truncate-1500
32.6
28.8
53.7
64%
Question-only
29.5
25.8
54.8
70%
Ceiling
65.5
Appendix
Table 17: Compressor probe on LongBench (4B drafter, K=2 , γ=0.5 ). Recovery is (AsymSpec−Floor)/(Ceiling−Floor) .
Drafter
K
Drafter alone
fixed- γ
+ tuned λ
+ CDA (ours)
0.6B
2
23.7
50.6
48.4
47.8
0.6B
4
23.7
48.5
46.6
46.4
1.7B
2
38.9
52.9
51.1
50.8
1.7B
4
38.9
48.5
53.6
53.9
4B
2
54.6
57.0
59.0
59.7
4B
4
54.6
59.7
59.3
61.1
Appendix
Table 18: Drafter-size sweep on LongBench F1 ( γ=0.5 , λ=0.1 for the tuned variant). CDA is the parameter-free JSD gate (§ 3.3 ); the 4B row (our default drafter) is its JSD instantiation of record, the smaller-drafter rows the accuracy-equivalent KL instantiation ( Table 14 ). Drafter alone : the SLM on full passages, no verifier — reference upper bound for what the small model can do unassisted (K-independent).
λ
MultiChallenge acc
0.05
21.0
0.10 (default tune)
23.2
0.20
23.6
CDA (no scale)
22.5
Appendix
Table 19: λ sweep of the tuned variant on MultiChallenge under the llm-judge ( n=271 , 4B drafter, γ=0.5 , K=2 ). The spread is only 2.6 pp ( 21.0 – 23.6 ). CDA (tuning-free, 22.5 ) lies within the same band with no λ to set.
Drafter
K
Drafter alone
fixed- γ
+ tuned λ
+ CDA (ours)
0.6B
2
31.3
59.4
59.6
59.6
0.6B
4
31.3
58.2
57.9
57.9
1.7B
2
61.2
63.7
63.5
63.5
1.7B
4
61.2
62.4
62.5
62.5
4B
2
64.0
60.3
60.7
60.7
4B
4
64.0
63.3
63.7
63.7
Appendix
Table 20: Drafter-size sweep on API-Bank api_acc ( γ=0.5 , λ=0.1 for the tuned variant). Drafter alone : the SLM alone on full API context, no verifier (K-independent).
Drafter
K
Drafter alone
fixed- γ
+ tuned λ
+ CDA (ours)
0.6B
2
13.3
20.3
19.6
21.4
0.6B
4
13.3
21.8
22.9
22.1
1.7B
2
14.0
19.6
20.7
19.6
1.7B
4
14.0
21.8
18.1
21.4
4B
2
25.8
25.5
23.2
22.5
4B
4
25.8
22.9
21.8
19.6
Appendix
Table 21: Drafter-size sweep on MultiChallenge accuracy ( γ=0.5 , llm-judge, n=271 , λ=0.1 for the tuned variant; single run). Every per-cell gap lies within the ≤ 3 pp headroom on this near-lossless task (§ 6.4 ).
Large language model (LLM)-based agent applications often incur high response time. Speculative decoding is a promising solution to improve the inference efficiency of LLM agents without impacting generation quality. However, state-of-the-art speculative decoding algorithms exhibit substantial speed degradation under large batch sizes, limiting their effectiveness to deploy in real-world agent applications. In this work, we first present a systematic analysis of speculative decoding for LLM agents and identify two dominant factors of speedup degradation: high rejection rate of speculative tokens, and under-utilization of dynamic token budgets.B ased on these observations, we propose AgentSpec, a speculative decoding algorithm that addresses the limitations of existing methods for LLM agents. AgentSpec incorporates structure-isolated drafting that constrains speculation to semantically coherent segments of the agent workflow, reducing the drafts of irrelevant semantic paths and achieving an extremely low rejection rate. Moreover, AgentSpec adopts redundancy-aware budget allocation that exploits agent-level information to better utilize the dynamically-free token budget during the agent inference. We implement and evaluate AgentSpec on five different workloads and four different models from four different LLM families in vLLM. Our results demonstrate the superiority of AgentSpec over state-of-the-arts.
Xin Wang, Ziming Miao, Yi Zhu +4
The Ohio State University · Microsoft Research · University of Michigan
Speculative decoding (SPD) accelerates large language model (LLM) inference by letting a smaller draft model propose multiple future tokens that are verified in parallel by a larger target model. The dominant SPD paradigm treats the target model as the sole reliable teacher, accepting a draft token only when it exactly matches the target prediction. This design implicitly assumes that the target is always the better choice at every position. In practice, this assumption does not hold. Although the draft is the weaker model overall, it is not uniformly inferior at the token level. In a meaningful fraction of cases where draft and target disagree, the draft's choice is the one that leads to the correct final answer. Inspired by this, we introduce Collaborative Speculative Decoding (CoSpec), a generalization of SPD that no longer treats the target model as the sole token-level authority. CoSpec trains an arbitration policy via reinforcement learning to decide whether to accept tokens from the draft or target model, selectively accepting draft tokens at mismatches when doing so is likely to yield a correct final answer. Experimental results show that CoSpec maintains substantial speedups while surpassing target-only performance. By shifting the emphasis from imitation to collaboration, CoSpec suggests a new perspective on speculative decoding.
Jinze Li, Yixing Xu, Guanchen Li +7
Advanced Micro Devices, Inc., Beijing, China · The University of Hong Kong · University of North Texas
Speculative decoding has become a mainstream technique for accelerating large language model (LLM) inference by using a lightweight draft model to generate candidate tokens that are subsequently verified by the target model. However, agentic workloads introduce long and continuously evolving contexts and frequent multi-turn interactions. Under such workloads, we observe that existing draft models suffer from up to an 80% degradation in acceptance length, while maintaining their historical KV cache introduces substantial system overhead. In this paper, we identify the growing historical context as a common source of both problems and introduce DLite, a history-free speculative decoding framework for agentic AI. DLite directly reuses contextual hidden states from the target model and performs local speculative prediction without explicitly maintaining the historical context, transforming an increasingly complex modeling problem into local simple prediction problem. A lightweight sequential head further restores intra-block causal dependencies with low overhead. To effectively train this heterogeneous history-free architecture, we introduce a tailored two-stage training strategy. Experiments on general-purpose, long-context and agentic workloads show that DLite achieves 2-6 acceptance length and outperforms the state-of-the-art speculative decoding method, DSpark, up to 70%.
Jiaming Xu, Hanzhen Wang, Tianlang Zhao +7
Shanghai Jiao Tong University · Shanghai Innovation Institute · Tsinghua University +1