TraceSpec: Understanding and Improving Speculative Decoding for LLM Agents
Authors: Jihao Zhao, Shuaishuai Zu, Pengfei Li, Biao Qin, Guanghuan Fang, Ben Feng, Baochuan Yang
Organizations: School of Information, Renmin University of China, Beijing, China · Huawei Technologies Co., Ltd., Beijing, China · Huawei Technologies Co., Ltd., Hangzhou, China
LLM agents execute tasks through repeated model calls over evolving histories of reasoning, actions, and tool feedback. Aggregate speculative-decoding metrics merge these distinct decisions, providing limited guidance on where draft supervision is needed. We present TraceSpec, a framework connecting trajectory evaluation with adaptive draft training. It combines dynamic execution with fixed-prefix replay and introduces TraceSpec-Traj, a corpus of 10,000 training states and 2,500 test states spanning browsing and software engineering. Analysis of five speculative baselines reveals that aggregate acceptance can rise with context length while declining within matched late browsing actions. Workload and phase further distinguish these trends, motivating supervision guided by trajectory conditions. TraceSpec trains on intermediate assistant turns and allocates a fixed prediction budget using state-conditioned estimates of prefix-aware acceptance gaps. Across three target sizes and two temperatures, it improves replay throughput by 12.5% to 22.6% over its DSpark base. Qwen3-4B ablations show that adaptive allocation increases accepted length by 3.3% to 10.0% over fixed full-turn supervision, with additional comparisons substantiating the roles of state conditioning and consecutive-token agreement. The resulting evaluation resources, trajectory findings, and training strategy connect the analysis of agent execution to practical draft improvement.
Figures & tables
Figure 1. TraceSpec overview. (a) Trajectory-state construction, coverage, and source-task splits. (b) Dynamic execution and fixed-prefix replay with trajectory diagnosis. (c) Full-turn supervision with state-conditioned allocation and event-normalized sampling under a fixed prediction budget. Three panels connect trajectory data, evaluation, and adaptive draft training. Panel a constructs history and continuation pairs across two workloads, five context strata, and four phases, with source tasks split into training and test partitions. Panel b shows dynamic draft verification by frozen Qwen3, tool feedback updating the history, and five baseline drafters evaluated on identical saved test prefixes. Trajectory diagnostics summarize acceptance, throughput, and episode latency. Panel c expands target-generated training trajectories into assistant segments. Frozen Qwen3 outputs and features supervise a trainable DSpark-based draft. Token overlap contributes to acceptance gaps smoothed by context, phase, workload, and tool state. Base coverage and adaptive priority determine supervision shares, converted to segment sampling probabilities using expected prediction events per visit. Training feedback updates allocation, and the resulting draft returns to evaluation.
Corpus property
Training
Test
Trajectory states
10,000
2,500
Source tasks
5,795
1,380
Web research states
5,000
1,250
Software engineering states
5,000
1,250
Plan
2,098
501
Act
3,151
920
Table 1. Trajectory corpus statistics. Source tasks are disjoint across splits.
Figure 2. Baseline replay acceptance by context length. Qwen3-8B on TraceSpec-Traj, grouped by domain, temperature, and actual input length. All methods receive the same prefixes. Four line charts show accepted length across context intervals for five draft methods in Web research and software engineering at temperatures zero and one.
Figure 3. Baseline acceptance across workloads. Columns show dynamic execution on BrowseComp-Plus and SWE-bench Verified, followed by fixed-prefix replay on TraceSpec-Traj. Groups denote Qwen3-4B, 8B, and 14B; rows denote T=0 and T=1 . Error bars show pointwise 95% bootstrap intervals. Six grouped bar charts in two rows and three columns compare five speculative baselines across three workloads, two temperatures, and three target sizes. Among these baselines, DSpark has the highest overall accepted length in all 18 settings. Accepted length decreases from temperature zero to one for every matched baseline and target.
Workload
Target
DFlash
EAGLE-3
DSpark
Gain (%)
BC
4B
89.17
87.16
82.86
7.1
BC
8B
73.18
77.27
68.14
6.9
BC
14B
147.08
167.77
143.15
2.7
SWE
4B
63.26
63.15
59.96
5.2
SWE
8B
80.12
84.13
75.90
5.3
SWE
14B
95.60
107.77
91.90
3.9
Table 2. Mean episode latency (s), T=0 . BC: BrowseComp-Plus; SWE: SWE-bench Verified. Gain is the latency reduction of DSpark over DFlash.
Figure 4. Online baseline acceptance and throughput by context length. Qwen3-8B at T=0 : upper panels show accepted tokens per round; lower panels show output tokens/s, including prefill. Four line charts compare DFlash, EAGLE-3, and DSpark on two online workloads. Browsing acceptance rises across the five intervals. Software engineering acceptance peaks at 8k to 16k before falling at 16k to 32k. Throughput is lower in the longest than in the shortest interval in both workloads.
Workload
Drafter
L8:16
L16:32
95% CI of ΔL
BC
DFlash
5.661
4.326
[-1.650, -0.941]
BC
EAGLE-3
6.007
4.897
[-1.489, -0.612]
BC
DSpark
6.194
5.322
[-1.205, -0.473]
SWE
DFlash
5.333
4.762
[-0.852, -0.257]
SWE
EAGLE-3
6.149
6.304
[-0.347, 0.716]
SWE
DSpark
6.057
6.106
[-0.238, 0.379]
Table 3. Late-action acceptance: Qwen3-8B, T=0 , step ≥20 . Tasks are paired across 8k to 16k and 16k to 32k contexts; ΔL=L16:32−L8:16 .
Figure 5. Baseline acceptance by domain and phase. Qwen3-8B results on TraceSpec-Traj at T=0 and T=1 , measured in accepted tokens per round. All panels share a common color scale. Four heatmaps compare five baseline drafters across planning, action, reflection, and answer generation. DSpark has the highest accepted length in every displayed phase. At temperature zero, its highest-acceptance phases are Web research actions and software engineering reflections.
Target
Phase
BrowseComp-Plus
SWE-bench Verified
EAGLE-3
DFlash
LongSpark
DLite
DSpark
TraceSpec
EAGLE-3
DFlash
LongSpark
DLite
DSpark
TraceSpec
Qwen3-4B
Plan
3.674
3.581
3.112
2.804
4.031
4.161
4.058
3.972
3.109
2.661
4.407
4.456
Act
4.605
4.072
3.260
3.281
4.617
4.923
5.113
4.491
3.710
2.638
5.116
5.467
Reflect
4.658
4.112
3.064
3.060
4.779
5.123
5.118
4.624
2.956
2.665
5.258
5.402
Answer
3.910
3.463
4.186
3.261
3.980
4.279
3.687
3.356
3.181
2.592
3.901
4.295
Qwen3-8B
Plan
3.946
3.922
3.373
3.035
4.456
4.570
3.271
3.190
2.840
2.614
3.562
3.656
Table 4. Phase-level dynamic acceptance at T=0 (accepted tokens per round, L ). Bold marks the highest value per phase and workload. TraceSpec uses adaptive full-turn supervision.
Target
Phase
TraceSpec-Traj, T=0
TraceSpec-Traj, T=1
EAGLE-3
DFlash
LongSpark
DLite
DSpark
TraceSpec
EAGLE-3
DFlash
LongSpark
DLite
DSpark
TraceSpec
Qwen3-4B
Plan
3.899
3.585
3.611
2.618
4.161
4.856
2.972
2.756
2.863
2.470
3.639
4.218
Act
3.704
3.419
2.788
2.549
3.953
4.909
3.316
2.892
2.824
2.471
3.789
4.695
Reflect
3.777
3.439
3.252
2.651
4.092
4.859
3.303
2.810
3.004
2.499
3.752
4.669
Answer
3.089
2.803
3.013
2.578
3.294
4.126
2.788
2.507
2.881
2.488
3.248
3.993
Qwen3-8B
Plan
3.156
3.345
3.041
2.610
3.808
4.397
2.755
2.925
2.912
2.553
3.620
4.060
Table 5. Replay acceptance on TraceSpec-Traj ( L ). All methods use identical saved prefixes. Column groups separate temperatures; rows and highlighting follow Table 4 .
Training variant
BrowseComp-Plus
SWE-bench Verified
TraceSpec-Traj
T=0
T=1
T=0
T=1
T=0
T=1
TraceSpec
4.803
4.145
5.142
4.859
4.753
4.428
w/o intermediate-turn supervision ( A )
4.307
3.745
4.619
4.169
4.275
3.782
w/o adaptive budget allocation ( B )
4.473
3.904
4.849
4.417
4.602
4.169
w/o state-conditioned estimation ( C1 )
4.420
4.000
4.769
4.449
4.422
4.056
w/o prefix-aware scoring ( C2 )
4.510
4.055
4.855
4.439
4.598
4.220
Table 6. Qwen3-4B component ablations on accepted length L . A and B use fixed allocation; C1 and C2 each modify TraceSpec.
Drafter
BrowseComp-
SWE-bench
TraceSpec-
Plus
Verified
Traj
T=0
T=1
T=0
T=1
T=0
T=1
Verification yield R (%)
DFlash
50.6
40.3
54.9
45.9
42.7
34.8
EAGLE-3
56.9
48.1
61.3
53.0
47.8
40.3
LongSpark
40.5
38.4
43.0
37.0
40.4
36.3
Table 7. Qwen3-4B inference results. Throughput includes prefill and uses native PyTorch methods. Bold marks column maxima within each metric.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Source
Training
Test
DeepResearch-traj ( IPF, 2026 )
5,000
1,250
Nebius SWE-agent ( Nebius, 2024 )
3,806
972
R2E-Gym SFT ( R2E-Gym, 2025a )
1,031
243
R2E-Gym testing agent ( R2E-Gym, 2025b )
163
35
Appendix
Table 8. Source composition of the constructed trajectory corpus.
Target
Phase
BrowseComp-Plus
SWE-bench Verified
EAGLE-3
DFlash
LongSpark
DLite
DSpark
TraceSpec
EAGLE-3
DFlash
LongSpark
DLite
DSpark
TraceSpec
Qwen3-4B
Plan
3.168
2.813
3.072
2.528
3.471
3.614
3.454
3.247
2.800
2.459
3.934
3.987
Act
3.878
3.263
3.074
2.554
4.146
4.228
4.423
3.800
3.069
2.510
4.841
5.246
Reflect
4.032
3.194
2.894
2.532
4.160
4.361
4.592
3.847
2.664
2.540
4.845
5.054
Answer
3.620
3.172
3.736
2.561
4.200
4.289
2.949
2.647
3.034
2.505
3.426
3.713
Qwen3-8B
Plan
3.036
3.021
3.175
2.790
3.639
3.583
2.495
2.556
2.539
2.397
3.016
3.044
Appendix
Table 9. Dynamic acceptance at T=1 . Metric and highlighting follow Table 4 .
Target
Drafter
BrowseComp-Plus
SWE-bench Verified
TraceSpec-Traj
T=0
T=1
T=0
T=1
T=0
T=1
Qwen3-8B
DFlash
48.9
37.1
50.4
37.2
44.1
36.6
EAGLE-3
49.9
38.1
54.0
40.2
42.5
35.9
LongSpark
39.2
36.5
39.6
35.5
39.2
35.9
DLite
36.3
33.1
33.3
31.1
33.2
32.0
DSpark
54.9
45.0
56.9
47.2
50.3
45.9
Appendix
Table 10. Verification yield R (%) for Qwen3-8B and 14B, extending Table 7 . Bold marks column maxima within each target.
Target
Drafter
BrowseComp-Plus
SWE-bench Verified
TraceSpec-Traj
T=0
T=1
T=0
T=1
T=0
T=1
Qwen3-8B
DFlash
42.96
51.35
48.95
49.18
37.02
31.07
EAGLE-3
40.33
41.20
45.98
41.32
32.68
27.81
DLite
34.51
44.86
37.19
35.99
30.69
28.45
DSpark
46.79
54.94
52.63
51.93
40.44
37.06
TraceSpec
48.25
53.92
54.30
55.12
47.14
41.70
Appendix
Table 11. Generation throughput (output tokens/s) for Qwen3-8B and 14B. Timing scope and highlighting follow Table 7 .
Large language model (LLM)-based agent applications often incur high response time. Speculative decoding is a promising solution to improve the inference efficiency of LLM agents without impacting generation quality. However, state-of-the-art speculative decoding algorithms exhibit substantial speed degradation under large batch sizes, limiting their effectiveness to deploy in real-world agent applications. In this work, we first present a systematic analysis of speculative decoding for LLM agents and identify two dominant factors of speedup degradation: high rejection rate of speculative tokens, and under-utilization of dynamic token budgets.B ased on these observations, we propose AgentSpec, a speculative decoding algorithm that addresses the limitations of existing methods for LLM agents. AgentSpec incorporates structure-isolated drafting that constrains speculation to semantically coherent segments of the agent workflow, reducing the drafts of irrelevant semantic paths and achieving an extremely low rejection rate. Moreover, AgentSpec adopts redundancy-aware budget allocation that exploits agent-level information to better utilize the dynamically-free token budget during the agent inference. We implement and evaluate AgentSpec on five different workloads and four different models from four different LLM families in vLLM. Our results demonstrate the superiority of AgentSpec over state-of-the-arts.
Xin Wang, Ziming Miao, Yi Zhu +4
The Ohio State University · Microsoft Research · University of Michigan
Speculative execution accelerates LLM agents by using a smaller, cheaper model to predict and pre-launch the next step while the environment is idle. However, existing speculators are stateless and discard all information between tasks, preventing prediction quality from improving with experience. We equip the speculator with three online memory systems that learn from past agent trajectories: a contrastive transition table tracking action-sequence statistics, an episodic memory retrieving contextually similar segments, and a confusion tracker suppressing recurring errors. We evaluate this approach on six benchmarks spanning three speculation types: action prediction, observation prediction, and chained prediction. Memory-augmented speculation yields a 19--39% relative accuracy improvement on action prediction and up to a 2.5× increase on observation prediction tasks with repetitive action spaces. These gains grow continuously as memory accumulates and generalize across speculator models of varying cost. All speculation is lossless because it runs during idle time at zero added wall-clock cost, and the actor's trajectory is identical to non-speculative execution.
LLM agents are known to be slow in rollouts. An agent completes a task one step at a time. At each step, it reasons and then chooses an action to execute. The next step and action cannot start until the previous one has finished. Speculative decoding accelerates the rollouts at the reason phase by drafting and verifying the inference tokens. Recent works have also started to apply similar ideas at the action phase. These works use off-the-shelf models, usually large, to draft action proposals for target model to verify. Large drafters match the target more often but take longer to propose, while small off-the-shelf models are fast but rarely make the same decision as the target. We ask a more general question: what determines the end-to-end speedup of action speculation? To answer it, we develop a latency framework for the speculative round. The framework compares what a round gains with what it costs. The gain depends on how well the drafter predicts the target and on how many steps the task can take before it ends. The cost comes from drafting, from waiting for target verification and from executing tools. Guided by the framework, we introduce LEAP (Learning Efficient Action Proposals) which keeps the drafter small and makes it accurate by training it on the target actions sequences. With a small 0.6B model, LEAP agrees with the target on most decisions and makes agents up to 60% faster in end-to-end wall clock time, with no systematic change in task success. Across various datasets, target models and draft models, the framework accounts for most of the measured speedups. We also show the draft model can be online trained with no prior trace collection and match the performance of offline training, making LEAP practical to deploy in the real world.
Zhen Xu, Qizheng Zhang, Gerry Wan +2
University of Chicago · Stanford University · Together AI