Trajectory-Level Speculation with Semantic Verification for Tool-Using Agents

Zilin Wang; Yinhong Liu; Chao Chen; Qun Liu; Zhijiang Guo

cs.CLOct 15, 2026

Trajectory-Level Speculation with Semantic Verification for Tool-Using Agents

Authors: Zilin Wang, Yinhong Liu, Chao Chen, Qun Liu, Zhijiang Guo

Organizations: LARK, HKUST (GZ) · Yinwang · University of Cambridge · HKUST

Abstract

Large language model agents solve tasks through long sequences of tool calls, and the prevailing design has a single strong model generate every step, although many steps are routine enough for a much smaller model. Existing ways of sharing this work keep the strong model as the generator: routers and collaborative agents decide which model handles each query or step, and speculative methods check a small model's drafts one step at a time, usually against the strong model's own action. We introduce SpecAgent, a collaboration protocol that turns the strong model from the author of each step into a reviewer of steps drafted by a weak model. The weak model executes read-only tool calls as it drafts, so that every step is grounded in real observations, and holds the first action that would modify the environment. The strong model then verifies the drafted segment in a single prefill pass, commits the longest prefix it judges semantically valid, and repairs the first rejected step. Because verification replaces autoregressive decoding with parallel prefill, the strong model decodes only short verdicts and occasional repairs rather than every step. On $τ^2$ -Bench, SpecAgent comes within one point of the strong model's success rate while roughly halving memory traffic and wall time, and on AutomationBench and VitaBench it recovers 44-75% of the quality gap while lowering both. In an edge-cloud deployment on MCP-Atlas with three cloud models, it closes 53-75% of the weak-strong quality gap while the strong model generates at most half as many tokens. These results suggest that a strong model need not author every step of an agent: reviewing grounded drafts retains most of its quality while cutting both latency and memory traffic.

Figures & tables

Real-World MCP Services	Transactional Interaction	Business Workflows	Life Services
( 3 )	( 4 )	( 30 )	( 16 )
20–30 turns	30–80 turns	40–60 turns	50–100 turns

Table 1: Representative trajectory length per task for frontier agents, in turns (messages from the agent, the user, and tools).

Figure 1: (a) A Strong-only agent generates every step by decoding. (b) SpecAgent lets the Weak model draft ahead, running reads at once and holding the first write. The Strong model reviews the draft in a single prefill pass, keeps the valid steps, and replaces the rejected write; the Weak model then resumes. Bars indicate relative Strong-model memory reads.

Method	Pass@1 (%) $\uparrow$	HBM-read (TB) $\downarrow$	Wall-time (s) $\downarrow$	Est. compute (PFLOP) $\downarrow$	GPU $\cdot$ s $\downarrow$	Calls (W/S) $\downarrow$
$\tau^{2}$ -Bench (Airline)
Weak-only (4B)	80.0	$83.8$	69.7	0.18	66.0	$14.1/0.0$
Strong-only (397B)	86.0	$[333.2,341.6]$	305.2	0.78	4828.4	$0.0/13.8$
AgentCollab	85.3	$[276.6,289.4]$	385.2	1.12	5629.3	$9.0/18.5$
Speculative Actions	72.0	$[429.5,438.2]$	665.6	0.89	9990.6	$4.5/17.4$
SpecAgent	86.0	$\mathbf{[147.6,150.1]}$	126.2	1.01	860.4	$15.5/3.1$

Table 2: Quality and mean per-trajectory resources of the agent models on

\tau^{2}

-Bench, AutomationBench, and VitaBench (Delivery). Compute and HBM-read are estimated from logged tokens and model architecture, assuming prefix reuse within a task. HBM-read of a Strong MoE model is a range, because the number of experts a prefill pass reads is not logged; the dense Weak model has a point estimate. Calls count Weak and Strong model calls per trajectory. Per benchmark, excluding Weak-only, the best value in each column except Calls is in bold and the second best underlined.

Figure 2: Quality against memory traffic and wall time, relative to each domain’s Strong-only run (star). Quadrants, drawn from point estimates, mark win–win, lose–lose, and trade-off regions.

Method	Pass@1 (%) $\uparrow$	HBM-read (TB) $\downarrow$	Strong output tokens $\downarrow$	Wall-time (s) $\downarrow$	Est. compute (PFLOP) $\downarrow$	API cost ( $)$ \downarrow$
Weak-only (4B)	44.6	39.1	–	109	0.67	0.00
Strong: Qwen3.5-397B-A17B
Strong-only (397B)	72.3	$[119.5,128.6]$	3.6k	105	1.72	1.39
SpecAgent	59.4	$[105.7,109.0]$	1.8k	120	2.64	1.12
Strong: GLM-5
Strong-only (GLM-5)	76.2	$[377.6,389.4]$	4.6k	110	10.07	8.91

Table 3: Edge–cloud deployment on MCP-Atlas (101 tasks, real MCP servers, one trajectory per task). The Weak model, Qwen3.5-4B, runs on one local RTX 5090; each Strong model is served by its provider’s cloud API. The Weak-only row is shared by all groups. Resources are per-task means and API cost is the total over all tasks at list price, excluding the local GPU.

	Verify		Repair		Predicted	Observed
	$c_{v}$	$\bar{v}/\bar{d}$	$c_{r}$	$\bar{d}_{r}/\bar{d}$	decoded	HBM	time
Airline	0.145	0.67	0.046	1.27	0.17	0.17	0.16
Retail	0.152	1.35	0.057	1.32	0.30	0.30	0.30
AutomationBench	0.277	0.25	0.204	1.52	0.53	0.54	0.38
VitaBench	0.216	0.53	0.100	1.36	0.28	0.29	0.24

Table 4: Strong decoding relative to Strong-only: predicted from ( 6 ), scaled by the ratio of committed steps per trajectory to that of Strong-only (1.05–1.41), and observed. Time is Strong generation time relative to that of Strong-only.

Figure 3: SpecAgent ’s cost by phase, relative to Strong-only (dashed). Strong verification and repair dominate compute but take a much smaller share of memory traffic.

Figure 4: Design ablations on

\tau^{2}

Retail, varied one at a time from SpecAgent ; SpecAgent and Strong-only (dashed) are main-table runs.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Request family	Newly processed tokens $P_{q}$	Rationale
Weak draft, Strong act	$\max(0,\,I_{q}-M)$	append-only transcript
Strong verification	$c_{q}+\max(0,\,I_{q}-c_{q}-B^{v})$	candidate $c_{q}$ is new text
Strong repair	$300+\max(0,\,I_{q}-300-B^{c})$	rejection note is new text

Appendix

Table 5: Prefix-reuse rules.

M

B^{v}

, and

B^{c}

are running maxima kept per session and updated after each request.

	Qwen3.5-4B (Weak)	Qwen3.5-397B-A17B (Strong)
Hidden size $h$ / vocabulary $V$	2560 / 248,320	4096 / 248,320
Layers: full attention / Gated DeltaNet	8 / 24	15 / 45
Query / KV heads, head dim $d$	16 / 4, 256	32 / 2, 256
DeltaNet value heads $n_{v}$ , $d_{k}{=}d_{v}$	32, 128	64, 128
MLP	dense, $m=9216$	MoE, $E=512$ , $k=10$
Expert / shared width $m_{e}$ / $m_{s}$	–	1024 / 1024

Appendix

Table 6: Architecture parameters used by the estimators (text backbone).

	GLM-5	DeepSeek-V4-Flash-0731
Hidden size $h$ / vocabulary $V$	6144 / 154,880	4096 / 129,280
Layers	78 (first 3 dense)	43
Attention	MLA + DSA	sliding window + compressed
Query / KV heads	64 / 64	64 / 1
Head dim (q/k, v)	256 (192 + 64 rope), 256	512 (64 rope)
Low-rank projection dims	query 2048, KV 512	query 1024, output 1024 (8 groups)

Appendix

Table 7: Architecture parameters of the MCP-Atlas Strong models GLM-5 and DeepSeek-V4-Flash-0731 used by the estimators.

Domain	Method	Pass@1 (%) $\uparrow$	Pass@3 (%) $\uparrow$	Pass 3 (%) $\uparrow$
Airline	Weak-only (4B)	80.0	96.0	64.0
	Strong-only (397B)	86.0	96.0	72.0
Retail	Weak-only (4B)	80.4	95.6	61.4
	Strong-only (397B)	88.6	96.5	76.3

Appendix

Table 8:

\tau^{2}

-Bench single-model baselines. Pass@1 is mean binary reward per attempt; pass@3 requires at least one successful attempt per task, and pass 3 requires all three. Scores are percentages. Airline contains 50 tasks and Retail 114 tasks, each with three attempts.

	$\tau^{2}$ -Bench	AutomationBench	VitaBench	MCP-Atlas
Held ( $E.\textsc{StateChanging}$ )	writes, transfer	writes, final message	writes	writes, final answer
Chunk size $K$	6	8	6	6
Verifier context	full history	full history	tool results cut	old tool results cut

Appendix

Table 9: Adapter bindings that differ across benchmarks.

Tool-Using Agents

Tool-using agents plan and act through external tools, APIs, software interfaces, and environments to complete tasks beyond language generation.

See all 289 papers

Jul 28, 2026

Speculate While You Reason: Teaching Agents to Predict Their Next Tool Call via Joint Agent-Speculator RL

Large language model agents often spend substantial wall-clock time waiting for tool call results. Tool-call speculation can hide this latency by predicting and pre-executing an agent's next tool call if the prediction matches the agent's eventual tool call, but existing speculators are typically separate draft models or cached traces that are poorly aligned with the deployed agent's own behavior. We identify this speculator-agent gap and show that the target agent itself is a strong next-call speculator. This points to a simpler design: unifying the agent and speculator within the same model. In this paper, we introduce the self-speculating agent, a single model that both solves tasks in agent mode and predicts its next tool call from partial trajectories in speculator mode, fully reusing prefix KV cache. To enable this dual-mode agent without degrading performance, we propose a joint agent-speculator reinforcement learning method, which derives speculation targets from the agent's own rollouts and alternates agent and speculator updates. Across agentic search QA and conversational tool-use agentic tasks, our method improves average next tool-call Hit@1 from 44.1 to 61.2 for Qwen3-4B and from 48.9 to 66.3 for Qwen3.5-4B, while preserving agent task success.

Jiabao Ji, Yujian Liu, Li An +4

University of California, Santa Barbara · Linkedin Inc

Tool-Using Agents LLM Agent Training RL for Tool Use

Ask PDF

Mar 15, 2026

AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents

While Large Language Models (LLMs) have evolved into tool-using agents, they remain brittle in long-horizon interactions. Unlike mathematical reasoning where errors are often rectifiable via backtracking, tool-use failures frequently induce irreversible side effects, making accurate step-level verification critical. However, existing process-level benchmarks are predominantly confined to closed-world mathematical domains, failing to capture the dynamic and open-ended nature of tool execution. To bridge this gap, we introduce AgentProcessBench, the first benchmark dedicated to evaluating step-level effectiveness in realistic, tool-augmented trajectories. The benchmark comprises 1,000 diverse trajectories and 8,509 human-labeled step annotations with 89.1% inter-annotator agreement. It features a ternary labeling scheme to capture exploration and an error propagation rule to reduce labeling ambiguity. Extensive experiments reveal key insights: (1) weaker policy models exhibit inflated ratios of correct steps due to early termination; (2) distinguishing neutral and erroneous actions remains a significant challenge for current models; and (3) process-derived signals provide complementary value to outcome supervision, significantly enhancing test-time scaling. We hope AgentProcessBench can foster future research in reward models and pave the way toward general agents. The code and data are available at https://github.com/RUCBM/AgentProcessBench.

Shengda Fan, Xuyan Ye, Yupeng Huo +9

Renmin University of China Beijing, China · Beijing Jiaotong University Beijing, China · Shanghai Jiao Tong University Shanghai, China +1

LLM Agent Evaluation Process-Based Agent Evaluation Tool-Using Agents Agent Benchmarks Long-Horizon Agent Evaluation Process Supervision

Ask PDF

Sep 24, 2026

Who Holds the Pen? Let Specifications, Not Agents, Sign Off

Large language model agents increasingly combine generation, decision-making, execution, and self-evaluation within a single agentic loop. Although they operate under external specifications such as task instructions, guidelines, output schemas, and reusable skills, these specifications typically remain context for the same model that acts and declares completion, leaving no independent specification authority boundary. We identify two resulting gaps. The understanding--execution gap arises when a requirement is understood but not satisfied in execution; the state--authority gap arises when an agent's interpretation or completion claim does not establish the required state. On SkillsBench, using only agent-visible prompts, workspace information, and injected skill specifications, we extract 509 source-grounded task directions. Across seven models, only 79.6%--86.4% are satisfied, while completion-claim rates exceed official evaluator pass rates by 28.7--37.9 percentage points. We therefore separate agent proposals from authoritative state. Agents may plan, act, and request completion, but only admissible evidence from qualified providers may establish specification-governed state. SpecHarness operationalizes this principle by compiling visible specifications into source-linked obligations and governing execution and finalization through versioned obligation state. Verifiable requirements are mediated or validated at runtime, while ambiguous or subjective requirements remain advisory. Experiments on guideline-following and artifact-generation tasks show that specifications can serve not merely as behavioral guidance, but as authority over compliant execution and completion.

Haiqing Li, Xin Ma, Yinhao Wu +7

Department of Computer Science and Engineering, The University of Texas at Arlington, Arlington, TX, USA · Monash University, Melbourne, VIC, Australia · Department of Computer Science, Kent State University, Kent, OH, USA

LLM Agent Verification Runtime Enforcement for AI Agents LLM Agent Harnesses

Ask PDF

Abstract

Tool-Using Agents

Explore similar work

Speculate While You Reason: Teaching Agents to Predict Their Next Tool Call via Joint Agent-Speculator RL

AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents

Who Holds the Pen? Let Specifications, Not Agents, Sign Off