Reward Learning

Reward learning focuses on automatically inferring or shaping reward functions for reinforcement learning agents, enabling them to learn complex tasks without explicit programming. Current research emphasizes efficient reward function design using large language models, inverse reinforcement learning from human demonstrations (including suboptimal or preference-based feedback), and novel algorithms to improve sample efficiency and robustness. These advancements are crucial for deploying reinforcement learning in real-world applications, such as robotics, personalized education, and the responsible development of AI systems, by enabling more effective and human-aligned learning.

Papers

March 28, 2024

Fine-Tuning Language Models with Reward Learning on Policy
Hao Lang, Fei Huang, Yongbin Li
Reinforcement Learning Reward Model Reward Learning Health Policy Recommendation Language Model Fine Tuning Synthetic Preference

March 21, 2024

DreamReward: Text-to-3D Generation with Human Preference
Junliang Ye, Fangfu Liu, Qixiu Li, Zhengyi Wang, Yikai Wang, Xinzhou Wang, Yueqi Duan, Jun Zhu
Human Preference Text to 3D Generation Text to 3D Reward Learning Preference Reward

March 18, 2024

TARN-VIST: Topic Aware Reinforcement Network for Visual Storytelling
Weiran Chen, Xin Li, Jiaqi Su, Guiqian Zhu, Ying Li, Yi Ji, Chunping Liu
Reward Learning Visual Storytelling Adaptive Network Cross Modal Task

March 9, 2024

A Generalized Acquisition Function for Preference-based Reward Learning
Evan Ellis, Gaurav R. Ghosal, Stuart J. Russell, Anca Dragan, Erdem Bıyık
Reward Function Reward Learning Information Gain Equivalence Class Reward Function Parameter

February 9, 2024

High-Precision Geosteering via Reinforcement Learning and Particle Filters
Ressi Bonti Muhammad, Apoorv Srivastava, Sergey Alyaev, Reidar Brumer Bratvold, Daniel M. Tartakovsky
Reinforcement Learning Particle Filter Reward Learning Decision Optimization

January 17, 2024

Crowd-PrefRL: Preference-Based Reward Learning from Crowds
David Chhan, Ellen Novoseller, Vernon J. Lawhern
Reinforcement Learning Preference Feedback Crowded Environment Reward Learning Pairwise Preference

December 14, 2023

Auto MC-Reward: Automated Dense Reward Design with Large Language Models for Minecraft
Hao Li, Xue Yang, Zhaokai Wang, Xizhou Zhu, Jie Zhou, Yu Qiao, Xiaogang Wang, Hongsheng Li, Lewei Lu, Jifeng Dai
Sparse Reward Reward Learning Reward Structure Dense Reward

October 21, 2023

Learning Reward for Physical Skills using Large Language Model
Yuwei Zeng, Yiqing Xu
Large Language Model Reward Function Reward Learning Animal Level Agility Physical Task

October 13, 2023

Distance-rank Aware Sequential Reward Learning for Inverse Reinforcement Learning with Sub-optimal Demonstrations
Lu Li, Yuxin Pan, Ruobing Chen, Jie Liu, Zilin Wang, Yu Liu, Zhiheng Li
Inverse Reinforcement Learning Reward Learning Reward Maximization Suboptimal Demonstration Reward Prediction Reward Ambiguity

October 5, 2023

Aligning Text-to-Image Diffusion Models with Reward Backpropagation
Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, Katerina Fragkiadaki
Diffusion Model Text to Image Diffusion Model Text to Image Model Reward Learning Text to Image Diffusion Reward Gradient

September 28, 2023

Infer and Adapt: Bipedal Locomotion Reward Learning from Demonstrations via Inverse Reinforcement Learning
Feiyang Wu, Zhaoyuan Gu, Hanran Wu, Anqi Wu, Ye Zhao
Imitation Learning Robot Learning Inverse Reinforcement Learning Noisy Demonstration Reward Learning Explanatory Inference Push Recovery

September 26, 2023

STARC: A General Framework For Quantifying Differences Between Reward Functions
Joar Skalse, Lucy Farnik, Sumeet Ramesh Motwani, Erik Jenner, Adam Gleave, Alessandro Abate
Reinforcement Learning Reward Function General Framework Reward Learning Reward Value Unraveling Learning Difference

July 19, 2023

STRAPPER: Preference-based Reinforcement Learning via Self-training Augmentation and Peer Regularization
Yachen Kang, Li He, Jinxin Liu, Zifeng Zhuang, Donglin Wang
Reinforcement Learning Consistency Regularization Reward Learning Preference Based Reinforcement Learning Peer Learning Augmentation Free Self Supervised Learning

May 25, 2023

Reward-Machine-Guided, Self-Paced Reinforcement Learning
Cevahir Koprulu, Ufuk Topcu
Reward Learning RL Algorithm

May 19, 2023

Shattering the Agent-Environment Interface for Fine-Tuning Inclusive Language Models
Wanqiao Xu, Shi Dong, Dilip Arumugam, Benjamin Van Roy
Language Model Reinforcement Learning Autoregressive Language Model Reward Learning Language Model Fine Tuning Agent Environment Interaction

May 16, 2023

Reward Learning with Intractable Normalizing Functions
Joshua Hoegerman, Dylan P. Losey
Non Humanoid Robot Reward Function Reward Learning Unified Normalization Reward Function Parameter Bayesian Reward Model

April 13, 2023

RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, Tong Zhang
Generative Model Reward Learning Generative Foundation Model Reward Fine Tuning

March 7, 2023

A Multiplicative Value Function for Safe and Efficient Reinforcement Learning
Nick Bührer, Zhejun Zhang, Alexander Liniger, Fisher Yu, Luc Van Gool
Proximal Policy Optimization Safe Reinforcement Learning Value Function Efficient Reinforcement Learning Reward Learning

February 23, 2023

Reward Learning as Doubly Nonparametric Bandits: Optimal Design and Scaling Laws
Kush Bhatia, Wenshuo Guo, Jacob Steinhardt
Reward Function Scaling Law Reward Learning Optimal Design

February 20, 2023

Fantastic Rewards and How to Tame Them: A Case Study on Reward Learning for Task-oriented Dialogue Systems
Yihao Feng, Shentao Yang, Shujian Zhang, Jianguo Zhang, Caiming Xiong, Mingyuan Zhou, Huan Wang
Reinforcement Learning Case Study Reward Function Task Oriented Task Oriented Dialogue System Reward Report Reward Learning End to End Response Generation