Reward Overoptimization

Reward overoptimization, a critical issue in reinforcement learning from human feedback (RLHF), occurs when optimizing a proxy reward model leads to a decline in actual performance, as measured by human evaluation or a more accurate "gold standard" reward. Current research focuses on mitigating this problem through techniques like reward model regularization (e.g., using Bayesian methods or information-theoretic approaches), ensemble methods, and constrained optimization strategies, often applied to large language models (LLMs) and diffusion models. Addressing reward overoptimization is crucial for building reliable and trustworthy AI systems, improving the alignment of AI models with human values and preventing unintended consequences in real-world applications.

Papers

October 2, 2024

Evaluating Robustness of Reward Models for Mathematical Reasoning
Sunghwan Kim, Dongjin Kang, Taeyoon Kwon, Hyungjoo Chae, Jungsoo Won, Dongha Lee, Jinyoung Yeo
Reward Model Mathematical Reasoning Robustness Evaluation Reward Overoptimization

June 14, 2024

Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs
Rui Yang, Ruomeng Ding, Yong Lin, Huan Zhang, Tong Zhang
Large Language Model Reinforcement Learning Reward Model Implicit Reward Reward Overoptimization

June 5, 2024

Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms
Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sikchi, Joey Hejna, Bradley Knox, Chelsea Finn, Scott Niekum
Scaling Law Alignment Algorithm Proxy Reward Reward Overoptimization

June 3, 2024

Scalable Ensembling For Mitigating Reward Overoptimisation
Ahmed M. Ahmed, Rafael Rafailov, Stepan Sharkov, Xuechen Li, Sanmi Koyejo
Language Model Reinforcement Learning Reward Model Model Ensembling Instruction Following Model Reward Overoptimization

May 23, 2024

Unraveling overoptimism and publication bias in ML-driven science
Pouria Saidi, Gautam Dasarathy, Visar Berisha
Machine Learning Machine Learning Model Overconfident Prediction Learning Curve Publication Bias Reward Overoptimization

April 30, 2024

Countering Reward Over-optimization in LLM with Demonstration-Guided Reinforcement Learning
Mathieu Rita, Florian Strub, Rahma Chaabouni, Paul Michel, Emmanuel Dupoux, Olivier Pietquin
Reinforcement Learning Medical LLM Reward Overoptimization

March 8, 2024

Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimation
Xiaoying Zhang, Jean-Francois Ton, Wei Shen, Hongning Wang, Yang Liu
Reinforcement Learning Uncertainty Quantification Adversarial Policy Reward Overoptimization

February 20, 2024

Bayesian Reward Models for LLM Alignment
Adam X. Yang, Maxime Robeyns, Thomas Coste, Jun Wang, Haitham Bou-Ammar, Laurence Aitchison
Reinforcement Learning Large Language Model Alignment Reward Overoptimization Bayesian Reward Model

February 14, 2024

InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling
Yuchun Miao, Sen Zhang, Liang Ding, Rong Bao, Lefei Zhang, Dacheng Tao
Reward Model Reinforcement Learning From Human Feedback Reward Hacking Reward Overoptimization Reward Misspecification Information Theoretic Reward

February 13, 2024

Confronting Reward Overoptimization for Diffusion Models: A Perspective of Inductive and Primacy Biases
Ziyi Zhang, Sen Zhang, Yibing Zhan, Yong Luo, Yonggang Wen, Dacheng Tao
Diffusion Model Visual Perspective Reward Model Diffusion Policy Downstream Model Reward Overoptimization Inductive Task Primacy Bias

January 29, 2024

Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF
Banghua Zhu, Michael I. Jordan, Jiantao Jiao
Reinforcement Learning Reinforcement Learning From Human Feedback Iterative Denoising Reward Overoptimization Reward Collapse

December 22, 2023

REBEL: A Regularization-Based Solution for Reward Overoptimization in Robotic Reinforcement Learning from Human Feedback
Souradip Chakraborty, Anukriti Singh, Amisha Bhaskar, Pratap Tokekar, Dinesh Manocha, Amrit Singh Bedi
Reinforcement Learning Human Feedback Reward Function Regularization Model Robot Policy Learning Reward Overoptimization Reward Ambiguity

October 6, 2023

Confronting Reward Model Overoptimization with Constrained RLHF
Ted Moskovitz, Aaditya K. Singh, DJ Strouse, Tuomas Sandholm, Ruslan Salakhutdinov, Anca D. Dragan, Stephen McAleer
Large Language Model Reward Model Constrained Reinforcement Learning Gradient Free Reward Overoptimization

October 4, 2023

Reward Model Ensembles Help Mitigate Overoptimization
Thomas Coste, Usman Anwar, Robert Kirk, David Krueger
Reward Model Reward Maximization Proxy Reward Reward Overoptimization

October 19, 2022

Scaling Laws for Reward Model Overoptimization
Leo Gao, John Schulman, Jacob Hilton
Reinforcement Learning Scaling Law Proxy Reward Reward Overoptimization Reward Function Parameter