Aligning Agents via Planning: A Benchmark for Trajectory-Level Reward Modeling

Jiaxuan Wang; Yulan Hu; Wenjin Yang; Zheng Pan; Xin Li; Lan-Zhe Guo

cs.AIApr 9, 2026

Aligning Agents via Planning: A Benchmark for Trajectory-Level Reward Modeling

Authors: Jiaxuan Wang, Yulan Hu, Wenjin Yang, Zheng Pan, Xin Li, Lan-Zhe Guo

Organizations: State Key Laboratory of Novel Software Technology, Nanjing University · School of Intelligence Science and Technology, Nanjing University · AMAP, Alibaba Group

Abstract

In classical Reinforcement Learning from Human Feedback (RLHF), Reward Models (RMs) serve as the fundamental signal provider for model alignment. As Large Language Models evolve into agentic systems capable of autonomous tool invocation and complex reasoning, the paradigm of reward modeling faces unprecedented challenges -- most notably, the lack of benchmarks specifically designed to assess RM capabilities within tool-integrated environments. To address this gap, we present Plan-RewardBench, a trajectory-level preference benchmark designed to evaluate how well judges distinguish preferred versus distractor agent trajectories in complex tool-using scenarios. Plan-RewardBench covers four representative task families -- (i) Safety Refusal, (ii) Tool-Irrelevance / Unavailability, (iii) Complex Planning, and (iv) Robust Error Recovery -- comprising validated positive trajectories and confusable hard negatives constructed via multi-model natural rollouts, rule-based perturbations, and minimal-edit LLM perturbations. We benchmark representative RMs (generative, discriminative, and LLM-as-Judge) under a unified pairwise protocol, reporting accuracy trends across varying trajectory lengths and task categories. Furthermore, we provide diagnostic analyses of prevalent failure modes. Our results reveal that all three evaluator families face substantial challenges, with performance degrading sharply on long-horizon trajectories, underscoring the necessity for specialized training in agentic, trajectory-level reward modeling. Ultimately, Plan-RewardBench aims to serve as both a practical evaluation suite and a reusable blueprint for constructing agentic planning preference data.

Figures & tables

Benchmark	Unit	MT	Tools	Exec	Plan	Rec	Safety
RewardBench2	response	✗	✗	✗	✗	✗	✓
RM-Bench	response	✗	✗	✗	✗	✗	✓
Long-RewardBench	response	✗	✗	✗	✗	✗	mixed
FC-RewardBench	tool-call	✗	✗	✗	✗	✗	✗
TRBENCH	ctx $\rightarrow$ resp	✓	✓	partial	partial	✗	✗
Agent-RewardBench	step (mm)	partial	✗	partial	✓	✗	✓

Table 1: High-level comparison of RM/agent-evaluator benchmarks. MT = multi-turn context; Tools = multiple tools available; Exec = executed tool/environment feedback included; Plan = planning/process quality; Rec = error recovery; Safety = safety/refusal/tool-misuse handling. ctx

\rightarrow

resp denotes step-level evaluation on a single assistant response conditioned on its dialogue/tool context (often obtained by slicing longer trajectories). Plan-RewardBench targets trajectory-level preference judgment in complex tool-augmented settings.

Figure 1: Plan-RewardBench overview. (Left) Evaluation scope. RewardBench evaluates only the final response, while FC-RewardBench focuses on tool-call name/arguments; both can miss tool-grounded contradictions (e.g., a fluent answer that contradicts the tool output). (Right) Construction pipeline. We source tasks and tool environments from Toucan/MCP, expand candidates via natural rollouts and perturbations, perform multi-judge scoring with meta-review and human audit, and assemble difficulty-/bias-controlled preference pairs for trajectory-level evaluation and training.

Figure 2: Trajectory source distribution. Most trajectories come from diverse natural rollouts (70%), supplemented with perturbations and rule-based injections to broaden failure-mode coverage.

Figure 3: Failure modes in rejected trajectories. Dominant errors by family: (a) Safety—unsafe compliance; (b) Irrelevance—tool hallucination; (c) Planning—redundant calls / partial completion; (d) Robustness—blind retries.

Scenario family	#Audited	Cohen’s $\kappa$
Safety Refusal	51	0.86
Tool-Irrelevance / Unavail.	80	0.81
Robust Error Recovery	80	0.75
Complex Planning	80	0.71

Table 2: Human audit agreement. Two annotators independently choose the preferred trajectory on a stratified subset of pairs following our bucket-specific rubrics. The results indicate substantial to almost perfect agreement (

\kappa>0.7

) across all families.

Bucket	#Pairs	Avg Tool	Avg Tokens		Max Tokens
Bucket	#Pairs	Tokens	Chosen	Rejected	(C / R)
Tool-Irrelevance / Unavailable	275	1,003	1,363	1,358	5,381 / 5,010
Planning-Multi (Easy)	109	1,662	3,405	3,431	5,645 / 5,671
Planning-Multi (Hard)	73	1,394	6,523	6,554	17,088/ 17,059
Robust Recovery	361	1,298	4,545	4,462	17,216 / 29,622
Planning-Single (Easy)	144	701	1,186	1,247	1,383 / 1,527

Table 3: Statistics of Plan-RewardBench pairs. We report Avg Tool Tokens (shared environment context) separately. The Avg Tokens and Max Tokens columns represent the total input length, encompassing the tool registry, user query, and the full agent trajectory. Note that within each pair, the tool registry is fixed; length differences between Chosen and Rejected stem solely from the trajectory content.

Discriminative Reward Models (Pointwise $\rightarrow$ Scalar)
In	Out	Model	Complex Planning				Specific Scenarios			Avg
In	Out	Model	Multi-E	Multi-H	Sngl-E	Sngl-H	Robust	Safety	Irrel.	Avg
	r	Inf-ORM-Llama3.1-70B	70.31	65.03	79.86	74.05	69.78	58.53	66.91	69.21
	r	InternLM2-7B-Reward	63.76	66.23	79.86	74.03	67.05	54.90	62.91	66.96
	r	Skywork-Reward-V2 (Qwen3-8B)	70.64	61.44	64.93	70.57	75.35	56.86	65.45	66.46
	r	Skywork-Reward-V2 (Llama-3.1-8B)	73.85	61.44	69.79	72.15	65.10	53.92	64.91	65.88

Table 4: Main results on Plan-RewardBench (pairwise accuracy, %). Multi-E/H and Sngl-E/H denote Multi-turn and Single-turn Planning (Easy/Hard). Specific Scenarios: Robust (Error Recovery), Safety (Refusal), and Irrel. (Tool Irrelevance/Unavailability). Avg represents the macro-average across all 7 evaluation columns. Results are sorted by Avg. In: pairwise; pointwise. Out: ✓ choice;

\diamondsuit

critique; r scalar reward.

Figure 4: Macro-accuracy vs. input length. (a) Pairwise LLM judges show sharp degradation and collapse in the extreme tail (

>32

k tokens). (b) Pointwise RMs exhibit a more linear decay. The dashed line indicates random chance (0.5).

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Model Name	Organization	Paper
Trajectory Generation & Minimal-edit Perturbations
qwen-plus-latest	Alibaba	Vendor docs
qwen-max	Alibaba	Vendor docs
moonshotai/Kimi-K2-Instruct	Moonshot AI	Team et al. (2025)
deepseek-ai/DeepSeek-R1	DeepSeek	Guo et al. (2025a)
Qwen/Qwen3-32B	Alibaba	Yang et al. (2025)

Appendix

Table 5: Model details across all stages: generation/perturbation, judge panel, and evaluated models.

Selection rule	Type	PRB Acc.	BFCL Overall	Agentic	Multi-Turn
First@1 (baseline)	–	–	48.71	21.53	47.87
Random@1	–	–	49.32	22.09	48.46
Pass@4 (oracle)	–	–	61.47	33.82	59.38
Qwen-Plus	LLM Judge	69.96	55.14	28.12	53.81
Qwen-Max	LLM Judge	68.92	54.19	27.18	52.93
RRM-32B	Generative RM	66.45	52.35	25.60	51.10

Appendix

Table 7: Best-of-

N

reranking on BFCL v4 (

N=4

; base agent: Qwen3-32B-Instruct in function-calling mode).

Reward Model Evaluation

Reward model evaluation measures how reliably models rank, score, and guide the selection or improvement of AI-generated responses and actions.

See all 46 papers

May 3, 2026

RMGAP: Benchmarking the Generalization of Reward Models across Diverse Preferences

Reinforcement Learning from Human Feedback has become the standard paradigm for language model alignment, where reward models directly determine alignment effectiveness. In this work, we focus on how to evaluate the generalizability of reward models. By "generalizability", we mean the ability of RMs to correctly rank responses to align with diverse user preferences. However, existing reward model benchmarks are typically designed around a universal preference, failing to assess this generalization. To address this critical gap, we introduce RMGAP, a benchmark comprising 1,097 instances across Chat, Writing, Reasoning, and Safety domains. Since different users exhibit diverse preferences for the same task, we first generate four distinct responses with different linguistic profiles for each collected prompt. However, the original prompt set lacks the specificity to convey different preferences. We therefore construct tailored prompts by contrasting these candidates and designing scenarios in which one response becomes the uniquely appropriate choice. Moreover, we observe that users often express the same preference using different phrasings, and thus extend each prompt with two paraphrased variants. Our evaluation of 24 state-of-the-art RMs reveals their substantial limitations: even the best RM achieves only 49.27% Best-of-N accuracy, highlighting considerable room for improvement in reward model generalization. Related data and code are available at https://github.com/nanzhi84/RMGAP.

Yangyang Zhou, Yi-Chen Li

Beijing University of Posts and Telecommunications · National Key Laboratory for Novel Software Technology, Nanjing University, China · School of Artificial Intelligence, Nanjing University, China

Reward Modeling LLM Evaluation Preference Modeling Reward Model Evaluation

Ask PDF

Oct 28, 2025

OpenReward: Learning to Reward Long-form Agentic Tasks via Reinforcement Learning

Reward models (RMs) have become essential for aligning large language models (LLMs), serving as scalable proxies for human evaluation in both training and inference. However, existing RMs struggle on knowledge-intensive and long-form tasks, where evaluating correctness requires grounding beyond the model's internal knowledge. This limitation hinders them from reliably discriminating subtle quality differences, especially when external evidence is necessary. To address this, we introduce OpenRM, a tool-augmented long-form reward model that systematically judges open-ended responses by invoking external tools to gather relevant evidence. We train OpenRM with Group Relative Policy Optimization (GRPO) on over 27K synthesized pairwise examples generated through a controllable data synthesis framework. The training objective jointly supervises intermediate tool usage and final outcome accuracy, incentivizing our reward model to learn effective evidence-based judgment strategies. Extensive experiments on three newly-collected datasets and two widely-used benchmarks demonstrate that OpenRM substantially outperforms existing reward modeling approaches. As a further step, we integrate OpenRM into both inference-time response selection and training-time data selection. This yields consistent gains in downstream LLM alignment tasks, highlighting the potential of tool-augmented reward models for scaling reliable long-form evaluation.

Ziyou Hu, Zhengliang Shi, Minghang Zhu +5

Leiden University, Leiden, Netherland · Shandong University, Qingdao, China · Tsinghua University, Beijing, China

Reward Modeling RL for Language Models LLM Alignment

Ask PDF

Jan 7, 2026

SCRIBE: Structured Mid-Level Supervision for Tool-Using Language Models

Training reliable tool-augmented agents remains a significant challenge, largely due to the difficulty of credit assignment in multi-step reasoning. While process-level reward models offer a promising direction, existing LLM-based judges often produce noisy and inconsistent signals because they lack fine-grained, task-specific rubrics to distinguish high-level planning from low-level execution. In this work, we introduce SCRIBE (Skill-Conditioned Reward with Intermediate Behavioral Evaluation), a reinforcement learning framework that intervenes at a novel mid-level abstraction. SCRIBE grounds reward modeling in a curated library of skill prototypes, transforming open-ended LLM evaluation into a constrained verification problem. By routing each subgoal to a corresponding prototype, the reward model is equipped with precise, structured rubrics that substantially reduce reward variance. Experimental results show that SCRIBE achieves state-of-the-art performance across a range of reasoning and tool-use benchmarks. In particular, it improves the AIME25 accuracy of a Qwen3-4B model from 43.3% to 63.3%, and significantly increases success rates in complex multi-turn tool interactions. Further analysis of training dynamics reveals a co-evolution across abstraction levels, where mastery of mid-level skills consistently precedes the emergence of effective high-level planning behaviors. Finally, we demonstrate that SCRIBE is additive to low-level tool optimizations, providing a scalable and complementary pathway toward more autonomous and reliable tool-using agents.

Yuxuan Jiang, Francis Ferraro

University of Maryland, Baltimore County

Process Reward Models RL for Language Model Reasoning LLM Agent Skill Learning LLM Agent Training RL for Tool Use

Ask PDF

Abstract

Reward Model Evaluation

Explore similar work

RMGAP: Benchmarking the Generalization of Reward Models across Diverse Preferences

OpenReward: Learning to Reward Long-form Agentic Tasks via Reinforcement Learning

SCRIBE: Structured Mid-Level Supervision for Tool-Using Language Models