Causally Robust Reward Learning from Reason-Augmented Preference Feedback

Minjune Hwang; Yigit Korkmaz; Daniel Seita; Erdem Bıyık

cs.AIMar 5, 2026

Causally Robust Reward Learning from Reason-Augmented Preference Feedback

Authors: Minjune Hwang, Yigit Korkmaz, Daniel Seita, Erdem Bıyık

Organizations: Thomas Lord Department of Computer Science, University of Southern California

Abstract

Preference-based reward learning is widely used for shaping agent behavior to match a user's preference, yet its sparse binary feedback makes it especially vulnerable to causal confusion. The learned reward often latches onto spurious features that merely co-occur with preferred trajectories during training, collapsing when those correlations disappear or reverse at test time. We introduce ReCouPLe, a lightweight framework that uses natural language rationales to provide the missing causal signal. Each rationale is treated as a guiding projection axis in an embedding space, training the model to score trajectories based on features aligned with that axis while de-emphasizing context that is unrelated to the stated reason. Because the same rationales (e.g., "avoids collisions", "completes the task faster") can appear across multiple tasks, ReCouPLe naturally reuses the same causal direction whenever tasks share semantics, and transfers preference knowledge to novel tasks without extra data or language-model fine-tuning. Our learned reward model can ground preferences on the articulated reason, aligning better with user intent and generalizing beyond spurious features. ReCouPLe outperforms baselines by up to 1.5x in reward accuracy under distribution shifts, and 2x in downstream policy performance in novel tasks. We have released our code at https://github.com/mj-hwang/ReCouPLe

Figures & tables

Figure 1: Preference learning can be susceptible to causal confusion, especially with the presence of non-causal distractor features that merely co-occur with preferred trajectories. In the example above, the reward model struggles to identify the exact feature of a trajectory that determined the user’s preference. By providing a reason, the agent can identify the causal feature.

Figure 2: ReCouPLe decomposes the task reward by orthogonally projecting the trajectory representation to the reason language embedding and decomposing the representation into reason-aligned and reason-orthogonal components. This allows the reward model to isolate the causal feature specified in the rationale to explain the user’s preference.

	2-task				4-task
	In Distribution		Color Swapped		In Distribution				Color Swapped
Model	Pick	Place	Pick	Place	Pick	Push	Place	Pull	Pick	Push	Place	Pull
Single Task
BT	0.980	1.000	0.540	0.830	0.980	1.000	1.000	1.000	0.540	0.987	0.830	0.867
Multi-Task
BT-Multi	0.953	1.000	0.600	0.820	0.987	1.000	1.000	1.000	0.707	1.000	0.840	0.907

Table 1: Reward accuracy comparison for ManiSkill ( RQ1 ), averaged over 3 seeds. Left block: 2-task setting; right block: 4-task setting. Methods with highest accuracies in each OOD task are bold-faced.

Figure 3: ManiSkill policy learning results, averaged over 3 seeds (mean

\pm

std).

Model	Training Tasks			Novel Task
	Push	Push-Wall	Pick-Place-Wall	Pick-Place
Multi-Task
BT-Multi	0.873	0.893	0.577	0.547
Multi-Task w/ Reasons
RFP	0.870	0.900	0.647	0.553
ReCouPLe-EC	0.863	0.843	0.650	0.663

Table 2: Reward accuracy comparison for Meta-World ( RQ2 ), averaged over 3 seeds.

Figure 4: Meta-World policy earning on the held-out task, averaged over 3 seeds (mean

\pm

std). Both ReCouPLe variants outperform the baselines, showing task transfer capability.

Model	2-task		4-task
	ID	OOD	ID	OOD
ReCouPLe	0.995	0.872	1.000	0.878
ReCouPLe-no-consistency	0.980	0.726	0.977	0.745
ReCouPLe-no-consistency-no-ratio	0.987	0.727	0.990	0.730

Table 3: Mean reward prediction accuracy over ManiSkill manipulation tasks, averaged over 3 seeds. ID and OOD refer to in-distribution and out-of-distribution tasks, respectively.

	2-task				4-task
	In Distribution		Color Swapped		In Distribution				Color Swapped
Model	Pick	Push	Pick	Push	Pick	Push	Place	Pull	Pick	Push	Place	Pull
Single Task
BT	0.833	1.000	0.167	0.610	0.833	1.000	0.980	1.000	0.167	0.610	0.460	0.053
Multi-Task
BT-Multi	0.870	0.999	0.167	0.673	0.867	1.000	0.987	1.000	0.533	0.867	0.833	0.587

Table 4: Reward accuracy comparison for in image-based ManiSkill ( RQ1 ) environment, averaged over 3 seeds. Methods with highest accuracies in each OOD task are bold-faced.

Model	Push (Visual OOD)	Push-Wall (Visual OOD)
BT (Baseline)	0.233	0.113
BT-Multi	0.367	0.227
ReCouPLe-EC	0.713	0.760
ReCouPLe-IC	0.793	0.900

Table 5: Reward accuracy comparison for custom visual causal confusion tasks in Meta-World. ReCouPLe successfully avoids overfitting to spurious background colors under diverse reasoning.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Task Description	Larger Cube Color	Smaller Cube Color
MS-Pick-Larger	“pick up larger cube to target sphere”	Red	Blue
MS-Place-Larger	“place larger cube in target bin”	Blue	Red
MS-Push-Larger	“push larger cube toward target line”	Blue	Red
MS-Pull-Larger	“pull larger cube toward green line”	Red	Blue

Appendix

Table 6: Task descriptions and settings for ManiSkill tasks.

Panel of Figure 5 — Figure 5: Terminal states for custom ManiSkill tasks. Each column represents a specific task, with the top row showing preferred trajectories (manipulating the larger cube) and the bottom row showing non-preferred trajectories (manipulating the smaller cube). Tasks and their respective color confounder are defined in Table 6 .

Task	Task Description
Push	“make contact and push puck to goal”
Push-Wall	“make contact, bypass wall via waypoint, and push puck to goal”
Pick-Place	“pick up puck, lift it, and place it on goal”
Pick-Place-Wall	“pick up puck, use waypoint to bypass wall, and place it on goal”

Appendix

Table 7: Natural language task descriptions for Meta-World tasks.

Code	Reason Text
Push	“pushes more decisively after making contact”
Push	“pushes puck closer to goal”
Push-Wall	“pushes puck toward waypoint”
Push-Wall	“guides puck past wall”
Push-Wall	“pushes puck toward goal after clearing wall”
Push & Push-Wall	“maintains firm grip on puck”

Appendix

Table 8: Reason codes used per task.

Panel of Figure 6 — Figure 6: Visualizations of terminal states from the custom visual causal confusion environments. During training, background color is perfectly correlated with trajectory preference. Robustness is evaluated on an OOD distribution where this mapping is inverted, requiring the reward model to generalize beyond spurious visual features.

Model	Pick (OOD)	Place (OOD)
BT (Baseline)	0.540	0.830
ReCouPLe-EC (Original)	0.820	0.940
ReCouPLe-EC (Diversified)	0.867	0.967
ReCouPLe-IC (Original)	0.633	0.807
ReCouPLe-IC (Diversified)	0.620	0.820

Appendix

Table 9: Robustness to Rationale Diversity (ManiSkill OOD Reward Accuracies)

Model	Pick (OOD)	Place (OOD)
BT (Baseline)	0.540	0.830
ReCouPLe-EC (25% Reasons)	0.783	0.940
ReCouPLe-EC (50% Reasons)	0.767	0.947
ReCouPLe-IC (25% Reasons)	0.673	0.807
ReCouPLe-IC (50% Reasons)	0.660	0.847

Appendix

Table 10: Scalability to Sparse Explanations (ManiSkill OOD Reward Accuracies)

Preference Queries	200	500	1000	2000
Task: Pick (OOD)
BT-Multi	0.573	0.627	0.600	0.693
RFP	0.540	0.613	0.620	0.760
ReCouPLe-EC	0.733	0.820	0.820	0.913
ReCouPLe-IC	0.547	0.580	0.633	0.793
Task: Place (OOD)

Appendix

Table 11: Scaling Analysis on ManiSkill (2-Task Setup). The 1000-query column corresponds to the original dataset size used in our main experiments in Section 5.1

Reward Modeling

Reward modeling learns scores or rankings that capture human goals and task success, guiding AI selection, reinforcement learning, and alignment.

See all 313 papers

Reason-guided Human Reward Modeling for Preference-based Reinforcement Learning
Oct 16, 2026Anshul Nayak, Shahil Shaik, Jonathan Smereka +1Preference-Based RL Reward Modeling
Clemson University, Clemson, SC 29631 USA · U.S. Army DEVCOM Ground Vehicle Systems Center, Warren, MI 48092 USA
CoRe: Combined Rewards with Vision-Language Model Feedback for Preference-Aligned Reinforcement Learning
Jul 2, 2026Hexian Ni, Tao Lu, Yinghao CaiRobotic RL Multimodal Reward Modeling
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China.
PrefMoE: Robust Preference Modeling with Mixture-of-Experts Reward Learning
May 1, 2026Ziqin Yuan, Ruiqi Wang, Dezhong Zhao +2Reward Modeling Mixture of Experts
1Purdue University, West Lafayette, IN, USA. · 2Beijing University of Chemical Technology, Beijing, China. · 3Indiana University Bloomington, Bloomington, IN, USA.

Oct 16, 2026J/K move · Enter open · S save

Reason-guided Human Reward Modeling for Preference-based Reinforcement Learning

Anshul Nayak, Shahil Shaik, Jonathan Smereka, Yue Wang

Clemson University, Clemson, SC 29631 USA · U.S. Army DEVCOM Ground Vehicle Systems Center, Warren, MI 48092 USA

Preference-Based RL Reward Modeling Human Preference Modeling RL from Human Feedback Interpretability in RL

Open paper AskPDF arXiv

Reinforcement learning (RL) policies optimized for task rewards may produce behaviors that are efficient but misaligned with human expectations, safety preferences, or intent. This paper presents a reason-guided preference-based reinforcement learning (PbRL) framework that learns a human reward model from trajectory-level feedback. In addition to preference labels, the framework incorporates reason tags explaining why a behavior is preferred, enabling the learned reward to capture both human choices and their underlying rationale. Compared with preference-only reward learning, the reason-guided formulation provides additional supervision for learning human-aligned rewards and improves the interpretability of the resulting policy. The learned human reward is combined with the environment reward during policy optimization, allowing the agent to improve task performance while producing more interpretable, human- aligned, and preference-sensitive behavior

Abstract

Reward Modeling

Explore similar work

Reason-guided Human Reward Modeling for Preference-based Reinforcement Learning

CoRe: Combined Rewards with Vision-Language Model Feedback for Preference-Aligned Reinforcement Learning

PrefMoE: Robust Preference Modeling with Mixture-of-Experts Reward Learning