AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents
Authors: Shengda Fan, Xuyan Ye, Yupeng Huo, Zhi-Yuan Chen, Yiju Guo, Shenzhi Yang, Wenkai Yang, Shuqi Ye, Jingwen Chen, Haotian Chen, Xin Cong, Yankai Lin
Organizations: Renmin University of China Beijing, China · Beijing Jiaotong University Beijing, China · Shanghai Jiao Tong University Shanghai, China · Tsinghua University Beijing, China
While Large Language Models (LLMs) have evolved into tool-using agents, they remain brittle in long-horizon interactions. Unlike mathematical reasoning where errors are often rectifiable via backtracking, tool-use failures frequently induce irreversible side effects, making accurate step-level verification critical. However, existing process-level benchmarks are predominantly confined to closed-world mathematical domains, failing to capture the dynamic and open-ended nature of tool execution. To bridge this gap, we introduce AgentProcessBench, the first benchmark dedicated to evaluating step-level effectiveness in realistic, tool-augmented trajectories. The benchmark comprises 1,000 diverse trajectories and 8,509 human-labeled step annotations with 89.1% inter-annotator agreement. It features a ternary labeling scheme to capture exploration and an error propagation rule to reduce labeling ambiguity. Extensive experiments reveal key insights: (1) weaker policy models exhibit inflated ratios of correct steps due to early termination; (2) distinguishing neutral and erroneous actions remains a significant challenge for current models; and (3) process-derived signals provide complementary value to outcome supervision, significantly enhancing test-time scaling. We hope AgentProcessBench can foster future research in reward models and pave the way toward general agents. The code and data are available at https://github.com/RUCBM/AgentProcessBench.
Figures & tables
Figure 1. Comparison of StepAcc across 20 LLMs on AgentProcessBench (%).
Benchmark
Domain
Environment
Step-level
Human Ann.
Evaluation Task
Scale
PRM800K ( Lightman et al., 2023 )
Math
–
✓
✓
Step Effectiveness
75K traj/ 800K steps
MathCheck-GSM ( Zhou et al., 2025 )
Math
–
✓
✗
First-Error Index
516 cases
ProcessBench ( Zheng et al., 2025 )
Math
–
✓
✓
First-Error Index
3.4K cases
PRMBench ( Song et al., 2025 )
Math
–
✓
✗
Step Error Types
6.2K / 83K steps
AgentRewardBench ( Lù et al., 2025 )
Web
Web
✗
✓
Trajectory Rubric
1.3K traj.
Agent-RewardBench ( Men et al., 2025 )
Multi-modal
Multi-modal
Partial
✓
Pair Preference
1.1K pairs
Table 1. Comparison between AgentProcessBench and other reward-oriented benchmarks. AgentProcessBench uniquely provides human-annotated, step-level effectiveness supervision for tool-using agents.
Figure 2. Example of an agent trajectory, corresponding to instance 808 in the benchmark. Each instance in AgentProcessBench consists of a complete tool-using agent trajectory, containing interleaved user messages, assistant responses, and tool calls. During evaluation, the LLM is tasked with annotating each of the assistant’s steps with a label of correct (+1), neutral (0), or incorrect (-1).
Figure 3. An overview of AgentProcessBench. First, we sample trajectories from four representative agent benchmarks generated by five source models. Subsequently, human experts annotate the data via a specialized platform, achieving an inter-annotator agreement of 89.1%. Finally, we utilize the constructed benchmark to evaluate 20 distinct models across various families and parameter scales using the StepAcc and FirstErrAcc metrics.
HotPotQA
GAIA
BFCL
τ2 -Bench
unsuccessful
successful
unsuccessful
successful
unsuccessful
successful
unsuccessful
successful
# Samples
89
161
167
83
148
102
124
126
% Incorrect Step Ratio
64.6%
14.0%
62.6%
12.4%
31.9%
3.8%
48.0%
8.0%
% Neutral Step Ratio
9.9%
3.7%
10.2%
6.7%
3.9%
4.3%
4.7%
2.6%
% Correct Step Ratio
25.5%
82.3%
27.1%
80.9%
64.3%
91.8%
47.3%
89.3%
# Assistant Steps
5.2
2.7
7.8
4.0
9.6
11.5
15.8
12.6
Table 2. Statistics of AgentProcessBench.
Figure 4. Distribution of trajectory-level and step-level labels across models, where both Qwen-series models use the 2507 Instruct version.
Model
HotPotQA
GAIA
BFCL
τ2 -Bench
Average
StepAcc
FirstErrAcc
StepAcc
FirstErrAcc
StepAcc
FirstErrAcc
StepAcc
FirstErrAcc
StepAcc
FirstErrAcc
API-Based Models (Non-Thinking)
GPT-5.2
72.1
69.6
66.3
54.4
71.6
52.8
70.3
56.4
70.1
58.3
GPT-5.2-Chat
71.4
70.0
69.3
58.0
71.6
58.0
80.4
58.4
74.8
61.1
DeepSeek-V3.2
74.4
65.2
61.8
48.0
74.1
50.0
73.3
57.6
71.4
55.2
Gemini-3-Flash-Preview
76.7
70.0
73.8
53.2
77.6
40.8
81.2
56.0
78.3
55.0
Table 3. Evaluation results on AgentProcessBench, reporting StepAcc and FirstErrAcc (%). Best and second-best results for each category and metric are highlighted in bold and underlined , respectively.
Figure 5. Distribution of first error positions.
Figure 6. Row-normalized confusion matrices of step evaluations for Gemini-3-Flash-Preview and Qwen3-30B-A3B-Thinking.
Figure 7. Correlation between step-level evaluation accuracy and trajectory-level final accuracy. It shows high correlation (Pearson r=0.814,p=1.2×10−5 ).
Generator
Final
# Pos
% Pos
Two-Stage
Qwen3-30B-A3B-Instruct-2507
37.7
41.5
47.2
43.4
Qwen3-30B-A3B-Thinking-2507
35.9
45.3
49.1
50.9
DeepSeek-V3.2
47.2
37.7
45.3
49.1
DeepSeek-V3.2-Thinking
56.6
50.9
54.7
64.2
Gemini-3-Flash-Preview
56.6
49.1
54.7
58.5
Table 4. Comparison of Best-of- N ( N=8 ) selection strategies on GAIA. Final selects the first trajectory predicted to be successful at the outcome level. # Pos Step and % Pos Ratio select trajectories based on the number and proportion of positive steps, respectively. The two-stage strategy ( Final → Process ) combines outcome- and process-level criteria by refining outcome-based filtering with step-level signals. As references, majority-voting achieves 49.1% accuracy, while the oracle Pass@ N is 77.4%.
Step Evaluator
Original
w/o EP
Δ
DeepSeek-V3.2
49.1
52.8
+3.7
DeepSeek-V3.2-Thinking
64.2
50.9
−13.2
Gemini-3-Flash-Preview
58.5
52.8
−5.7
Table 5. Ablation of error propagation in step-level evaluation protocols for GAIA Best-of- N ( N=8 ) selection. All scores report using the two-stage ( Final → Process ) selector. Original uses the error-propagation annotation protocol, whereas w/o EP evaluates each step independently without propagating errors. Δ denotes the change from Original to w/o EP .
Figure 8. StepAcc on 800 training samples and 200 validation samples over the course of RL training.
Large Language Models (LLMs) are increasingly deployed as agents that interact with stateful environments over multiple steps: gathering hidden information, composing tool calls, and committing state changes. We refer to this capability as multi-step tool use. Existing benchmarks have advanced tool-use agent evaluation, but often focus on isolated API calls, short trajectories, or settings that are difficult to scale or control. We introduce E-Bench, a fully synthetic benchmark with 323 state-changing tasks across three product domains: Honor of Kings, QQ Music, and Tencent Meeting. E-Bench decouples environment synthesis from task synthesis: graph-guided database filling builds reusable, orphan-free product environments, while generator-solver asymmetry creates tasks with both an information gap and a tool gap, requiring agents to discover hidden data and compose multiple tool calls before changing state. Outcomes are graded deterministically by database-state diffs. Since both environments and tasks are synthetic, E-Bench is controllable at the environment level and scalable at the task level. Benchmarking 11 cutting-edge LLMs shows that multi-step tool use remains challenging: Pass^3 stays below 60% for the strongest models, and even with code execution in the E-Bench-Code extension, reliability (Pass^3) remains below 70%.
Weihuang Zheng, Tianyuan Zou, Eileen Ye +5
Hunyuan Team, Tencent · Institute for AI Industry Research, Tsinghua University · School of Computer Science and Engineering, Southeast University
Existing tool-use benchmarks for LLM agents are overwhelmingly linear: our analysis of six benchmarks shows 55 to 100% of instances are simple chains of 2 to 5 steps. We introduce The Amazing Agent Race (AAR), a benchmark featuring directed acyclic graph (DAG) puzzles (or "legs") with fork-merge tool chains. We release 1,400 instances across two variants: sequential (800 legs) and compositional (600 DAG legs). Agents must navigate Wikipedia, execute multi-step tool chains, and aggregate results into a verifiable answer. Legs are procedurally generated from Wikipedia seeds across four difficulty levels with live-API validation. Three complementary metrics (finish-line accuracy, pit-stop visit rate, and roadblock completion rate) separately diagnose navigation, tool-use, and arithmetic failures. Evaluating three agent frameworks on 1,400 legs, the best achieves only 37.2% accuracy. Navigation errors dominate (27 to 52% of trials) while tool-use errors remain below 17%, and agent architecture matters as much as model scale (Claude Code matches Codex CLI at 37% with 6x fewer tokens). The compositional structure of AAR reveals that agents fail not at calling tools but at navigating to the right pages, a blind spot invisible to linear benchmarks. The project page can be accessed at: https://minnesotanlp.github.io/the-amazing-agent-race
Zae Myung Kim, Dongseok Lee, Jaehyung Kim +2
University of Minnesota Twin Cities · Yonsei University · Grammarly
Tool calling is central to modern language model agents, but aggregate benchmark scores often hide where tool use fails. A model that never calls a needed tool and a model that calls the tool but ignores the result can look similar under final task accuracy. We introduce ToolFailBench, a diagnostic benchmark for measuring tool-use failures across 1,000 tasks in finance, medicine, law, cybersecurity, and real estate. Tool-required tasks return values the model wouldn't guess, forcing it to trust the tool while control tasks attach the same tools but should be answered directly. We label each trace with Tool-Skip, Result-Ignore, Output-Fabrication, and Unnecessary-Tool-Use, using a rule classifier and two LLM judges aggregated by majority vote. Across 19 headline models, the best reaches 86.33% Clean Tool-Use Rate, showing that faithful tool use is not saturated. More importantly, models with similar aggregate scores fail in different ways: most stay disciplined on no-tool controls, while Llama-3.1 models show an Always-Call pattern, and at the same parameter scale Llama-3.1-70B and Qwen2.5-72B differ by 89 percentage points on control-task accuracy. Tool-use evaluation should measure not only whether agents call tools, but whether they use tool outputs correctly and avoid tools when none is needed.