AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents

Shengda Fan; Xuyan Ye; Yupeng Huo; Zhi-Yuan Chen; Yiju Guo; Shenzhi Yang; Wenkai Yang; Shuqi Ye

cs.AIMar 15, 2026

AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents

Authors: Shengda Fan, Xuyan Ye, Yupeng Huo, Zhi-Yuan Chen, Yiju Guo, Shenzhi Yang, Wenkai Yang, Shuqi Ye,

Organizations: Renmin University of China Beijing, China · Beijing Jiaotong University Beijing, China · Shanghai Jiao Tong University Shanghai, China · Tsinghua University Beijing, China

Abstract

While Large Language Models (LLMs) have evolved into tool-using agents, they remain brittle in long-horizon interactions. Unlike mathematical reasoning where errors are often rectifiable via backtracking, tool-use failures frequently induce irreversible side effects, making accurate step-level verification critical. However, existing process-level benchmarks are predominantly confined to closed-world mathematical domains, failing to capture the dynamic and open-ended nature of tool execution. To bridge this gap, we introduce AgentProcessBench, the first benchmark dedicated to evaluating step-level effectiveness in realistic, tool-augmented trajectories. The benchmark comprises 1,000 diverse trajectories and 8,509 human-labeled step annotations with 89.1% inter-annotator agreement. It features a ternary labeling scheme to capture exploration and an error propagation rule to reduce labeling ambiguity. Extensive experiments reveal key insights: (1) weaker policy models exhibit inflated ratios of correct steps due to early termination; (2) distinguishing neutral and erroneous actions remains a significant challenge for current models; and (3) process-derived signals provide complementary value to outcome supervision, significantly enhancing test-time scaling. We hope AgentProcessBench can foster future research in reward models and pave the way toward general agents. The code and data are available at https://github.com/RUCBM/AgentProcessBench.

Figures & tables

Figure 1. Comparison of StepAcc across 20 LLMs on AgentProcessBench (%).

Benchmark	Domain	Environment	Step-level	Human Ann.	Evaluation Task	Scale
PRM800K ( Lightman et al., 2023 )	Math	–	✓	✓	Step Effectiveness	75K traj/ 800K steps
MathCheck-GSM ( Zhou et al., 2025 )	Math	–	✓	✗	First-Error Index	516 cases
ProcessBench ( Zheng et al., 2025 )	Math	–	✓	✓	First-Error Index	3.4K cases
PRMBench ( Song et al., 2025 )	Math	–	✓	✗	Step Error Types	6.2K / 83K steps
AgentRewardBench ( Lù et al., 2025 )	Web	Web	✗	✓	Trajectory Rubric	1.3K traj.
Agent-RewardBench ( Men et al., 2025 )	Multi-modal	Multi-modal	Partial	✓	Pair Preference	1.1K pairs

Table 1. Comparison between AgentProcessBench and other reward-oriented benchmarks. AgentProcessBench uniquely provides human-annotated, step-level effectiveness supervision for tool-using agents.

Figure 2. Example of an agent trajectory, corresponding to instance 808 in the benchmark. Each instance in AgentProcessBench consists of a complete tool-using agent trajectory, containing interleaved user messages, assistant responses, and tool calls. During evaluation, the LLM is tasked with annotating each of the assistant’s steps with a label of correct (+1), neutral (0), or incorrect (-1).

Figure 3. An overview of AgentProcessBench. First, we sample trajectories from four representative agent benchmarks generated by five source models. Subsequently, human experts annotate the data via a specialized platform, achieving an inter-annotator agreement of 89.1%. Finally, we utilize the constructed benchmark to evaluate 20 distinct models across various families and parameter scales using the StepAcc and FirstErrAcc metrics.

	HotPotQA		GAIA		BFCL		$\tau^{2}$ -Bench
	unsuccessful	successful	unsuccessful	successful	unsuccessful	successful	unsuccessful	successful
# Samples	89	161	167	83	148	102	124	126
% Incorrect Step Ratio	64.6%	14.0%	62.6%	12.4%	31.9%	3.8%	48.0%	8.0%
% Neutral Step Ratio	9.9%	3.7%	10.2%	6.7%	3.9%	4.3%	4.7%	2.6%
% Correct Step Ratio	25.5%	82.3%	27.1%	80.9%	64.3%	91.8%	47.3%	89.3%
# Assistant Steps	5.2	2.7	7.8	4.0	9.6	11.5	15.8	12.6

Table 2. Statistics of AgentProcessBench.

Figure 4. Distribution of trajectory-level and step-level labels across models, where both Qwen-series models use the 2507 Instruct version.

Model	HotPotQA		GAIA		BFCL		$\tau^{2}$ -Bench		Average
	StepAcc	FirstErrAcc	StepAcc	FirstErrAcc	StepAcc	FirstErrAcc	StepAcc	FirstErrAcc	StepAcc	FirstErrAcc
API-Based Models (Non-Thinking)
GPT-5.2	72.1	69.6	66.3	54.4	71.6	52.8	70.3	56.4	70.1	58.3
GPT-5.2-Chat	71.4	70.0	69.3	58.0	71.6	58.0	80.4	58.4	74.8	61.1
DeepSeek-V3.2	74.4	65.2	61.8	48.0	74.1	50.0	73.3	57.6	71.4	55.2
Gemini-3-Flash-Preview	76.7	70.0	73.8	53.2	77.6	40.8	81.2	56.0	78.3	55.0

Table 3. Evaluation results on AgentProcessBench, reporting StepAcc and FirstErrAcc (%). Best and second-best results for each category and metric are highlighted in bold and underlined , respectively.

Figure 5. Distribution of first error positions.

Panel of Figure 6 — Figure 6. Row-normalized confusion matrices of step evaluations for Gemini-3-Flash-Preview and Qwen3-30B-A3B-Thinking.

Figure 7. Correlation between step-level evaluation accuracy and trajectory-level final accuracy. It shows high correlation (Pearson

r=0.814,p=1.2\times 10^{-5}

Generator	Final	# Pos	% Pos	Two-Stage
Qwen3-30B-A3B-Instruct-2507	37.7	41.5	47.2	43.4
Qwen3-30B-A3B-Thinking-2507	35.9	45.3	49.1	50.9
DeepSeek-V3.2	47.2	37.7	45.3	49.1
DeepSeek-V3.2-Thinking	56.6	50.9	54.7	64.2
Gemini-3-Flash-Preview	56.6	49.1	54.7	58.5

Table 4. Comparison of Best-of-

N

(

N{=}8

) selection strategies on GAIA. Final selects the first trajectory predicted to be successful at the outcome level. # Pos Step and % Pos Ratio select trajectories based on the number and proportion of positive steps, respectively. The two-stage strategy ( Final

\rightarrow

Process ) combines outcome- and process-level criteria by refining outcome-based filtering with step-level signals. As references, majority-voting achieves 49.1% accuracy, while the oracle Pass@

N

is 77.4%.

Step Evaluator	Original	w/o EP	$\Delta$
DeepSeek-V3.2	49.1	52.8	$+3.7$
DeepSeek-V3.2-Thinking	64.2	50.9	$-13.2$
Gemini-3-Flash-Preview	58.5	52.8	$-5.7$

Table 5. Ablation of error propagation in step-level evaluation protocols for GAIA Best-of-

N

(

N{=}8

) selection. All scores report using the two-stage ( Final

\rightarrow

Process ) selector. Original uses the error-propagation annotation protocol, whereas w/o EP evaluates each step independently without propagating errors.

\Delta

denotes the change from Original to w/o EP .

Figure 8. StepAcc on 800 training samples and 200 validation samples over the course of RL training.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 9 — Figure 9. PRM Evaluation Prompt Template

LLM Agent Evaluation

Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.

See all 1,132 papers

E-Bench: Benchmarking Multi-Step Tool-Use Agents in Real-World Product Scenarios
Jul 26, 2026Weihuang Zheng, Tianyuan Zou, Eileen Ye +5Computer-Use Agent Benchmarks Tool-Use Evaluation
Hunyuan Team, Tencent · Institute for AI Industry Research, Tsinghua University · School of Computer Science and Engineering, Southeast University
The Amazing Agent Race: Strong Tool Users, Weak Navigators
Apr 11, 2026Zae Myung Kim, Dongseok Lee, Jaehyung Kim +2LLM Agent Evaluation AI Agent Benchmarks
University of Minnesota Twin Cities · Yonsei University · Grammarly
ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents
Jul 6, 2026Harsh SoniAI Agent Reliability Tool-Use Evaluation
UC Berkeley

Jul 26, 2026J/K move · Enter open · S save

E-Bench: Benchmarking Multi-Step Tool-Use Agents in Real-World Product Scenarios

Weihuang Zheng, Tianyuan Zou, Eileen Ye, Alphet Liu+4

Hunyuan Team, Tencent · Institute for AI Industry Research, Tsinghua University · School of Computer Science and Engineering, Southeast University

Computer-Use Agent Benchmarks Tool-Use Evaluation Tool-Augmented Language Model Agents AI Agent Benchmarks Synthetic Benchmark Generation Tool Use

Open paper AskPDF arXiv

Large Language Models (LLMs) are increasingly deployed as agents that interact with stateful environments over multiple steps: gathering hidden information, composing tool calls, and committing state changes. We refer to this capability as multi-step tool use. Existing benchmarks have advanced tool-use agent evaluation, but often focus on isolated API calls, short trajectories, or settings that are difficult to scale or control. We introduce E-Bench, a fully synthetic benchmark with 323 state-changing tasks across three product domains: Honor of Kings, QQ Music, and Tencent Meeting. E-Bench decouples environment synthesis from task synthesis: graph-guided database filling builds reusable, orphan-free product environments, while generator-solver asymmetry creates tasks with both an information gap and a tool gap, requiring agents to discover hidden data and compose multiple tool calls before changing state. Outcomes are graded deterministically by database-state diffs. Since both environments and tasks are synthetic, E-Bench is controllable at the environment level and scalable at the task level. Benchmarking 11 cutting-edge LLMs shows that multi-step tool use remains challenging: Pass^3 stays below 60% for the strongest models, and even with code execution in the E-Bench-Code extension, reliability (Pass^3) remains below 70%.

Abstract

LLM Agent Evaluation

Explore similar work

E-Bench: Benchmarking Multi-Step Tool-Use Agents in Real-World Product Scenarios

The Amazing Agent Race: Strong Tool Users, Weak Navigators

ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents