Preference Based Reward

Preference-based reward learning aims to infer reward functions from human preferences, bypassing the difficulty of manually specifying complex reward structures for AI agents. Current research focuses on improving the efficiency and robustness of these methods, exploring techniques like graph neural networks to model reference-dependent choices and incorporating feature-level preferences for more accurate reward learning. Challenges include mitigating issues like reward misidentification, ensuring fairness across multiple objectives, and improving sample efficiency through techniques such as dynamics-aware reward models. These advancements are crucial for developing more reliable and human-aligned AI systems across various applications, from recommender systems to robotics.

Papers

October 8, 2024

Trajectory Improvement and Reward Learning from Comparative Language Feedback
Zhaojing Yang, Miru Jun, Jeremy Tien, Stuart J. Russell, Anca Dragan, Erdem Bıyık
Latent Space Reward Learning Robot Trajectory Language Feedback Trajectory Design Preference Based Reward

August 21, 2024

Modeling Reference-dependent Choices with Graph Neural Networks
Liang Zhang, Guannan Liu, Junjie Wu, Yong Tan
Graph Neural Network Decision Making Choice Model Price Sensitivity Utility Learning Preference Based Reward Individual Preference Stability

June 11, 2024

3D-Properties: Identifying Challenges in DPO and Charting a Path Forward
Yuzi Yan, Yibo Miao, Jialian Li, Yipin Zhang, Jian Xie, Zhijie Deng, Dong Yan
Large Language Model Technical Challenge Direct Preference Optimization 3D Feature Way Forward Preference Based Reward

May 23, 2024

Pragmatic Feature Preferences: Learning Reward-Relevant Preferences from Human Input
Andi Peng, Yuying Sun, Tianmin Shu, David Abel
Reward Model Linear Bandit Reward Learning Natural Language Input Social Learning Preference Based Reward Feature Preference

February 28, 2024

Sample-Efficient Preference-based Reinforcement Learning with Dynamics Aware Rewards
Katherine Metcalf, Miguel Sarabia, Natalie Mackraz, Barry-John Theobald
Reinforcement Learning Preference Based Reinforcement Learning Preference Based Reward

November 24, 2023

Directly Attention Loss Adjusted Prioritized Experience Replay
Zhuoying Chen, Huiping Li, Zhaoxu Wang
Reinforcement Learning Multi Agent Reinforcement Learning Policy Gradient Experience Replay Attention Loss Preference Based Reward

June 16, 2023

Fairness in Preference-based Reinforcement Learning
Umer Siddique, Abhinav Sinha, Yongcan Cao
Procedural Fairness Policy Learning Vector Valued Preference Based Reinforcement Learning Preference Based Reward

January 9, 2023

On The Fragility of Learned Reward Functions
Lev McKinney, Yawen Duan, David Krueger, Adam Gleave
Reward Function Reward Learning Preference Based Reward

November 12, 2022

Rewards Encoding Environment Dynamics Improves Preference-based Reinforcement Learning
Katherine Metcalf, Miguel Sarabia, Barry-John Theobald
Reinforcement Learning Reward Function Preference Based Reinforcement Learning Preference Based Reward

April 13, 2022

Causal Confusion and Reward Misidentification in Preference-Based Reward Learning
Jeremy Tien, Jerry Zhi-Yang He, Zackory Erickson, Anca D. Dragan, Daniel S. Brown
Reinforcement Learning Learning Policy Preference Based Reward Reward Misspecification