Trajectory-Level Speculation with Semantic Verification for Tool-Using Agents

Zilin Wang; Yinhong Liu; Chao Chen; Qun Liu; Zhijiang Guo

cs.CLOct 15, 2026

Trajectory-Level Speculation with Semantic Verification for Tool-Using Agents

Authors: Zilin Wang, Yinhong Liu, Chao Chen, Qun Liu, Zhijiang Guo

Organizations: LARK, HKUST (GZ) · Yinwang · University of Cambridge · HKUST

Abstract

Large language model agents solve tasks through long sequences of tool calls, and the prevailing design has a single strong model generate every step, although many steps are routine enough for a much smaller model. Existing ways of sharing this work keep the strong model as the generator: routers and collaborative agents decide which model handles each query or step, and speculative methods check a small model's drafts one step at a time, usually against the strong model's own action. We introduce SpecAgent, a collaboration protocol that turns the strong model from the author of each step into a reviewer of steps drafted by a weak model. The weak model executes read-only tool calls as it drafts, so that every step is grounded in real observations, and holds the first action that would modify the environment. The strong model then verifies the drafted segment in a single prefill pass, commits the longest prefix it judges semantically valid, and repairs the first rejected step. Because verification replaces autoregressive decoding with parallel prefill, the strong model decodes only short verdicts and occasional repairs rather than every step. On $τ^2$ -Bench, SpecAgent comes within one point of the strong model's success rate while roughly halving memory traffic and wall time, and on AutomationBench and VitaBench it recovers 44-75% of the quality gap while lowering both. In an edge-cloud deployment on MCP-Atlas with three cloud models, it closes 53-75% of the weak-strong quality gap while the strong model generates at most half as many tokens. These results suggest that a strong model need not author every step of an agent: reviewing grounded drafts retains most of its quality while cutting both latency and memory traffic.

Figures & tables

Real-World MCP Services	Transactional Interaction	Business Workflows	Life Services
( 3 )	( 4 )	( 30 )	( 16 )
20–30 turns	30–80 turns	40–60 turns	50–100 turns

Table 1: Representative trajectory length per task for frontier agents, in turns (messages from the agent, the user, and tools).

Figure 1: (a) A Strong-only agent generates every step by decoding. (b) SpecAgent lets the Weak model draft ahead, running reads at once and holding the first write. The Strong model reviews the draft in a single prefill pass, keeps the valid steps, and replaces the rejected write; the Weak model then resumes. Bars indicate relative Strong-model memory reads.

Method	Pass@1 (%) $\uparrow$	HBM-read (TB) $\downarrow$	Wall-time (s) $\downarrow$	Est. compute (PFLOP) $\downarrow$	GPU $\cdot$ s $\downarrow$	Calls (W/S) $\downarrow$
$\tau^{2}$ -Bench (Airline)
Weak-only (4B)	80.0	$83.8$	69.7	0.18	66.0	$14.1/0.0$
Strong-only (397B)	86.0	$[333.2,341.6]$	305.2	0.78	4828.4	$0.0/13.8$
AgentCollab	85.3	$[276.6,289.4]$	385.2	1.12	5629.3	$9.0/18.5$
Speculative Actions	72.0	$[429.5,438.2]$	665.6	0.89	9990.6	$4.5/17.4$
SpecAgent	86.0	$\mathbf{[147.6,150.1]}$	126.2	1.01	860.4	$15.5/3.1$

Table 2: Quality and mean per-trajectory resources of the agent models on

\tau^{2}

-Bench, AutomationBench, and VitaBench (Delivery). Compute and HBM-read are estimated from logged tokens and model architecture, assuming prefix reuse within a task. HBM-read of a Strong MoE model is a range, because the number of experts a prefill pass reads is not logged; the dense Weak model has a point estimate. Calls count Weak and Strong model calls per trajectory. Per benchmark, excluding Weak-only, the best value in each column except Calls is in bold and the second best underlined.

Figure 2: Quality against memory traffic and wall time, relative to each domain’s Strong-only run (star). Quadrants, drawn from point estimates, mark win–win, lose–lose, and trade-off regions.

Method	Pass@1 (%) $\uparrow$	HBM-read (TB) $\downarrow$	Strong output tokens $\downarrow$	Wall-time (s) $\downarrow$	Est. compute (PFLOP) $\downarrow$	API cost ( $)$ \downarrow$
Weak-only (4B)	44.6	39.1	–	109	0.67	0.00
Strong: Qwen3.5-397B-A17B
Strong-only (397B)	72.3	$[119.5,128.6]$	3.6k	105	1.72	1.39
SpecAgent	59.4	$[105.7,109.0]$	1.8k	120	2.64	1.12
Strong: GLM-5
Strong-only (GLM-5)	76.2	$[377.6,389.4]$	4.6k	110	10.07	8.91

Table 3: Edge–cloud deployment on MCP-Atlas (101 tasks, real MCP servers, one trajectory per task). The Weak model, Qwen3.5-4B, runs on one local RTX 5090; each Strong model is served by its provider’s cloud API. The Weak-only row is shared by all groups. Resources are per-task means and API cost is the total over all tasks at list price, excluding the local GPU.

	Verify		Repair		Predicted	Observed
	$c_{v}$	$\bar{v}/\bar{d}$	$c_{r}$	$\bar{d}_{r}/\bar{d}$	decoded	HBM	time
Airline	0.145	0.67	0.046	1.27	0.17	0.17	0.16
Retail	0.152	1.35	0.057	1.32	0.30	0.30	0.30
AutomationBench	0.277	0.25	0.204	1.52	0.53	0.54	0.38
VitaBench	0.216	0.53	0.100	1.36	0.28	0.29	0.24

Table 4: Strong decoding relative to Strong-only: predicted from ( 6 ), scaled by the ratio of committed steps per trajectory to that of Strong-only (1.05–1.41), and observed. Time is Strong generation time relative to that of Strong-only.

Figure 3: SpecAgent ’s cost by phase, relative to Strong-only (dashed). Strong verification and repair dominate compute but take a much smaller share of memory traffic.

Figure 4: Design ablations on

\tau^{2}

Retail, varied one at a time from SpecAgent ; SpecAgent and Strong-only (dashed) are main-table runs.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Request family	Newly processed tokens $P_{q}$	Rationale
Weak draft, Strong act	$\max(0,\,I_{q}-M)$	append-only transcript
Strong verification	$c_{q}+\max(0,\,I_{q}-c_{q}-B^{v})$	candidate $c_{q}$ is new text
Strong repair	$300+\max(0,\,I_{q}-300-B^{c})$	rejection note is new text

Appendix

Table 5: Prefix-reuse rules.

M

B^{v}

, and

B^{c}

are running maxima kept per session and updated after each request.

	Qwen3.5-4B (Weak)	Qwen3.5-397B-A17B (Strong)
Hidden size $h$ / vocabulary $V$	2560 / 248,320	4096 / 248,320
Layers: full attention / Gated DeltaNet	8 / 24	15 / 45
Query / KV heads, head dim $d$	16 / 4, 256	32 / 2, 256
DeltaNet value heads $n_{v}$ , $d_{k}{=}d_{v}$	32, 128	64, 128
MLP	dense, $m=9216$	MoE, $E=512$ , $k=10$
Expert / shared width $m_{e}$ / $m_{s}$	–	1024 / 1024

Appendix

Table 6: Architecture parameters used by the estimators (text backbone).

	GLM-5	DeepSeek-V4-Flash-0731
Hidden size $h$ / vocabulary $V$	6144 / 154,880	4096 / 129,280
Layers	78 (first 3 dense)	43
Attention	MLA + DSA	sliding window + compressed
Query / KV heads	64 / 64	64 / 1
Head dim (q/k, v)	256 (192 + 64 rope), 256	512 (64 rope)
Low-rank projection dims	query 2048, KV 512	query 1024, output 1024 (8 groups)

Appendix

Table 7: Architecture parameters of the MCP-Atlas Strong models GLM-5 and DeepSeek-V4-Flash-0731 used by the estimators.

Domain	Method	Pass@1 (%) $\uparrow$	Pass@3 (%) $\uparrow$	Pass 3 (%) $\uparrow$
Airline	Weak-only (4B)	80.0	96.0	64.0
	Strong-only (397B)	86.0	96.0	72.0
Retail	Weak-only (4B)	80.4	95.6	61.4
	Strong-only (397B)	88.6	96.5	76.3

Appendix

Table 8:

\tau^{2}

-Bench single-model baselines. Pass@1 is mean binary reward per attempt; pass@3 requires at least one successful attempt per task, and pass 3 requires all three. Scores are percentages. Airline contains 50 tasks and Retail 114 tasks, each with three attempts.

	$\tau^{2}$ -Bench	AutomationBench	VitaBench	MCP-Atlas
Held ( $E.\textsc{StateChanging}$ )	writes, transfer	writes, final message	writes	writes, final answer
Chunk size $K$	6	8	6	6
Verifier context	full history	full history	tool results cut	old tool results cut

Appendix

Table 9: Adapter bindings that differ across benchmarks.

Tool-Using Agents

Tool-using agents plan and act through external tools, APIs, software interfaces, and environments to complete tasks beyond language generation.

See all 289 papers

Speculate While You Reason: Teaching Agents to Predict Their Next Tool Call via Joint Agent-Speculator RL
Jul 28, 2026Jiabao Ji, Yujian Liu, Li An +4Tool-Using Agents LLM Agent Training
University of California, Santa Barbara · Linkedin Inc
AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents
Mar 15, 2026Shengda Fan, Xuyan Ye, Yupeng Huo +9LLM Agent Evaluation Process-Based Agent Evaluation
Renmin University of China Beijing, China · Beijing Jiaotong University Beijing, China · Shanghai Jiao Tong University Shanghai, China +1
Who Holds the Pen? Let Specifications, Not Agents, Sign Off
Sep 24, 2026Haiqing Li, Xin Ma, Yinhao Wu +7LLM Agent Verification Runtime Enforcement for AI Agents
Department of Computer Science and Engineering, The University of Texas at Arlington, Arlington, TX, USA · Monash University, Melbourne, VIC, Australia · Department of Computer Science, Kent State University, Kent, OH, USA

Jul 28, 2026J/K move · Enter open · S save

Speculate While You Reason: Teaching Agents to Predict Their Next Tool Call via Joint Agent-Speculator RL

Jiabao Ji, Yujian Liu, Li An, Rohit Jain+3

University of California, Santa Barbara · Linkedin Inc

Tool-Using Agents LLM Agent Training RL for Tool Use

Open paper AskPDF arXiv

Large language model agents often spend substantial wall-clock time waiting for tool call results. Tool-call speculation can hide this latency by predicting and pre-executing an agent's next tool call if the prediction matches the agent's eventual tool call, but existing speculators are typically separate draft models or cached traces that are poorly aligned with the deployed agent's own behavior. We identify this speculator-agent gap and show that the target agent itself is a strong next-call speculator. This points to a simpler design: unifying the agent and speculator within the same model. In this paper, we introduce the self-speculating agent, a single model that both solves tasks in agent mode and predicts its next tool call from partial trajectories in speculator mode, fully reusing prefix KV cache. To enable this dual-mode agent without degrading performance, we propose a joint agent-speculator reinforcement learning method, which derives speculation targets from the agent's own rollouts and alternates agent and speculator updates. Across agentic search QA and conversational tool-use agentic tasks, our method improves average next tool-call Hit@1 from 44.1 to 61.2 for Qwen3-4B and from 48.9 to 66.3 for Qwen3.5-4B, while preserving agent task success.

Abstract

Tool-Using Agents

Explore similar work

Speculate While You Reason: Teaching Agents to Predict Their Next Tool Call via Joint Agent-Speculator RL

AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents

Who Holds the Pen? Let Specifications, Not Agents, Sign Off