Reason-guided Human Reward Modeling for Preference-based Reinforcement Learning

Anshul Nayak; Shahil Shaik; Jonathan Smereka; Yue Wang

cs.ROOct 16, 2026

Reason-guided Human Reward Modeling for Preference-based Reinforcement Learning

Authors: Anshul Nayak, Shahil Shaik, Jonathan Smereka, Yue Wang

Organizations: Clemson University, Clemson, SC 29631 USA · U.S. Army DEVCOM Ground Vehicle Systems Center, Warren, MI 48092 USA

Abstract

Reinforcement learning (RL) policies optimized for task rewards may produce behaviors that are efficient but misaligned with human expectations, safety preferences, or intent. This paper presents a reason-guided preference-based reinforcement learning (PbRL) framework that learns a human reward model from trajectory-level feedback. In addition to preference labels, the framework incorporates reason tags explaining why a behavior is preferred, enabling the learned reward to capture both human choices and their underlying rationale. Compared with preference-only reward learning, the reason-guided formulation provides additional supervision for learning human-aligned rewards and improves the interpretability of the resulting policy. The learned human reward is combined with the environment reward during policy optimization, allowing the agent to improve task performance while producing more interpretable, human- aligned, and preference-sensitive behavior

Figures & tables

Figure 1: Offline Human Preference Learning: Human observes a pair of clips and records preference as well as reason labels. NN model is trained offline to predict preference index $f^{rew}$ using cross entropy (CE) and reason tags $f^{reason}$ using binary cross entropy (BCE). Policy Optimization: Learned human reward and task reward jointly perform policy optimization that is both task-efficient and aligned with interpretable human feedback.

Parameter	Value
Hidden layer dim	256
Optimizer	SGD
Learning Rate	$2\times 10^{-4}$
Effective Batch Size	8
epochs	20
$\lambda_{H}$	$[0,0.5]$

Table 1: Hyperparameters

Panel of Figure 2 — Figure 2: Policy Evaluation Comparison: successful rollout comparisons between the environment-only and the combined environment and human policy under similar environment conditions.

Policy	Path length	Success (%)
$R_{\mathrm{env}}$	$11.02\pm 0.96$	$67.5\pm 4.2$
$R_{\mathrm{env}}+\lambda_{H}R_{\mathrm{human}}$	$10.92\pm 0.19$	$77.5\pm 5.7$

Table 2: Reward metrics for rollout evaluation at

\lambda_{H}=0.2

$\lambda_{H}$	Training objective	Obstacle clearance	Path length	Success rate (%)	subjective Evaluation (%)
0.1	$\mathcal{L}_{\mathrm{pref}}$	$1.82\pm 0.20$	$11.44\pm 1.46$	$66.7\pm 4.7$	$33.1\pm 6.4$
0.1	$\mathcal{L}_{\mathrm{pref}}+\alpha_{\mathrm{reason}}\mathcal{L}_{\mathrm{reason}}$	$1.90\pm 0.27$	$11.25\pm 0.52$	$73.3\pm 0.0$	$65.4\pm 7.2$
0.1	$\mathcal{L}_{\mathrm{pref}}+\mathcal{L}_{\mathrm{reg}}$	$1.80\pm 0.23$	$11.16\pm 0.36$	$75.0\pm 2.4$	$68.3\pm 8.1$
0.1	$\mathcal{L}_{\mathrm{pref}}+\alpha_{\mathrm{reason}}\mathcal{L}_{\mathrm{reason}}+\mathcal{L}_{\mathrm{reg}}$	$\textbf{1.99}\pm 0.30$	$11.29\pm 0.87$	$\textbf{78.3}\pm 2.4$	$67.2\pm 5.1$
0.2	$\mathcal{L}_{\mathrm{pref}}$	$1.78\pm 0.14$	$11.72\pm 1.33$	$65.0\pm 2.4$	$23.1\pm 4.9$
0.2	$\mathcal{L}_{\mathrm{pref}}+\alpha_{\mathrm{reason}}\mathcal{L}_{\mathrm{reason}}$	$2.01\pm 0.14$	$11.65\pm 0.52$	$68.3\pm 2.4$	$66.6\pm 4.7$

Table 3: Ablation study comparing reward-model training objective. Preference-only baseline

\mathcal{L}_{\mathrm{pref}}

, while environment+human rows use the corresponding ablation objective.

Panel of Figure 3 — Figure 3: Reward-model validation and policy-training curves for ablation variants.

Preference-Based RL

Preference-based reinforcement learning learns policies from comparisons or rankings of behaviors instead of requiring hand-designed numerical rewards.

See all 118 papers

Similarity as Reward Alignment: Robust and Versatile Preference-based Reinforcement Learning
Jun 14, 2025Sara Rajaram, R. James Cotton, Fabian H. SinzOffline RL Robust RL
Institute for Computer Science and Campus Institute for Data Science, University of Göttingen, Göttingen, Germany · University Hospital Tübingen, Tübingen, Germany · International Max Planck Research School for Intelligent Systems, Tübingen, Germany +2
CoRe: Combined Rewards with Vision-Language Model Feedback for Preference-Aligned Reinforcement Learning
Jul 2, 2026Hexian Ni, Tao Lu, Yinghao CaiRobotic RL Multimodal Reward Modeling
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China.
Causally Robust Reward Learning from Reason-Augmented Preference Feedback
Mar 5, 2026Minjune Hwang, Yigit Korkmaz, Daniel Seita +1Reward Modeling Preference-Based RL
Thomas Lord Department of Computer Science, University of Southern California

Jun 14, 2025J/K move · Enter open · S save

Similarity as Reward Alignment: Robust and Versatile Preference-based Reinforcement Learning

Sara Rajaram, R. James Cotton, Fabian H. Sinz

Institute for Computer Science and Campus Institute for Data Science, University of Göttingen, Göttingen, Germany · University Hospital Tübingen, Tübingen, Germany · International Max Planck Research School for Intelligent Systems, Tübingen, Germany+2

Offline RL Robust RL Preference-Based RL

Open paper AskPDF arXiv

Preference-based Reinforcement Learning (PbRL) entails a variety of approaches for aligning models with human intent to alleviate the burden of reward engineering. However, most previous PbRL work has not investigated the robustness to labeler errors, inevitable with labelers who are non-experts or operate under time constraints. We introduce Similarity as Reward Alignment (SARA), a simple contrastive framework that is both resilient to noisy labels and adaptable to diverse feedback formats. SARA learns a latent representation of preferred samples and computes rewards as similarities to the learned latent. On preference data with varying realistic noise rates, we demonstrate competitive and more stable performance on continuous control offline RL benchmarks, with statistically significant improvements over baselines (Wilcoxon signed-rank, p < 0.01). We also compute correlation to the environment rewards as a proxy for measuring alignment to the underlying preference criteria. We show that the SARA computed rewards display higher correlation across noise rates compared to baselines.

Abstract

Preference-Based RL

Explore similar work

Similarity as Reward Alignment: Robust and Versatile Preference-based Reinforcement Learning

CoRe: Combined Rewards with Vision-Language Model Feedback for Preference-Aligned Reinforcement Learning

Causally Robust Reward Learning from Reason-Augmented Preference Feedback