Reward Collapse

Reward collapse describes the undesirable phenomenon where reward models in reinforcement learning, particularly those trained on human feedback or learned from limited data, converge to a limited range of reward values, hindering the learning of diverse and high-quality policies. Current research focuses on mitigating this issue through improved reward model architectures and training algorithms, such as incorporating uncertainty quantification, prompt-aware optimization, and data smoothing techniques to better represent the true reward landscape. Addressing reward collapse is crucial for advancing reinforcement learning applications, especially in areas like large language model alignment and robotics, where effective reward functions are essential for achieving desired behaviors.

Papers

August 23, 2024

SAMBO-RL: Shifts-aware Model-based Offline Reinforcement Learning
Wang Luo, Haoran Li, Zicheng Zhang, Congying Han, Jiayu Lv, Tiande Guo
Policy OpTimization Policy Learning Model Based Offline Reinforcement Learning Reward Collapse

June 1, 2024

Combining Experimental and Historical Data for Policy Evaluation
Ting Li, Chengchun Shi, Qianglin Wen, Yang Sui, Yongli Qin, Chunbo Lai, Hongtu Zhu
Policy Evaluation Robust Estimation Sequential Decision Making Historical Data Reward Collapse

February 23, 2024

Fine-Tuning of Continuous-Time Diffusion Models as Entropy-Regularized Control
Masatoshi Uehara, Yulai Zhao, Kevin Black, Ehsan Hajiramezanali, Gabriele Scalia, Nathaniel Lee Diamant, Alex M Tseng, Tommaso Biancalani, Sergey Levine
Diffusion Model Fine Tuning Policy Entropy Reward Collapse Entropy Reward

January 29, 2024

Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF
Banghua Zhu, Michael I. Jordan, Jiantao Jiao
Reinforcement Learning Reinforcement Learning From Human Feedback Iterative Denoising Reward Overoptimization Reward Collapse

May 31, 2023

Offline Meta Reinforcement Learning with In-Distribution Online Adaptation
Jianhao Wang, Jin Zhang, Haozhe Jiang, Junyu Zhang, Liwei Wang, Chongjie Zhang
Online Adaptation Offline Meta Reinforcement Learning Offline Adaptation Reward Collapse

May 28, 2023

Reward Collapse in Aligning Large Language Models
Ziang Song, Tianle Cai, Jason D. Lee, Weijie J. Su
Reward Model Reward Collapse

January 18, 2023

DIRECT: Learning from Sparse and Shifting Rewards using Discriminative Reward Co-Training
Philipp Altmann, Thomy Phan, Fabian Ritz, Thomas Gabor, Claudia Linnhoff-Popien
Reward Function Many Sparse Policy OpTimization Reward Collapse