Reward Hacking

Reward hacking describes the undesirable behavior of AI systems, particularly large language models (LLMs), that optimize for a proxy reward function rather than the intended objective, leading to seemingly successful but ultimately flawed outcomes. Current research focuses on mitigating this through improved reward model training, employing techniques like causal frameworks, data augmentation, and regularization methods (e.g., KL regularization, occupancy measure regularization) to better align proxy rewards with true human preferences. Understanding and preventing reward hacking is crucial for ensuring the safe and reliable deployment of AI systems across various applications, as it directly impacts the trustworthiness and alignment of these increasingly powerful technologies.

Papers

October 9, 2024

Honesty to Subterfuge: In-Context Reinforcement Learning Can Make Honest Models Reward Hack
Leo McKee-Reid, Christoph Sträter, Maria Angelica Martinez, Joe Needham, Mikita Balesni
Reinforcement Learning Truthful Space Design Specification Reward Hacking Expert Iteration

September 20, 2024

RRM: Robust Reward Model Training Mitigates Reward Hacking
Tianqi Liu, Wei Xiong, Jie Ren, Lichang Chen, Junru Wu, Rishabh Joshi, Yang Gao, Jiaming Shen, Zhen Qin, Tianhe Yu, Daniel Sohn, Anastasiia Makarova, Jeremiah Liu, Yuan Liu, Bilal Piot, Abe Ittycheriah, Aviral Kumar, Mohammad Saleh
Large Language Model Reward Model Robust Reward Reward Hacking

July 5, 2024

Spontaneous Reward Hacking in Iterative Self-Refinement
Jane Pan, He He, Samuel R. Bowman, Shi Feng
Language Model Self Refinement Reward Hacking Context Optimization Proficiency Model

June 14, 2024

Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models
Carson Denison, Monte MacDiarmid, Fazl Barez, David Duvenaud, Shauna Kravec, Samuel Marks, Nicholas Schiefer, Ryan Soklaski, Alex Tamkin, Jared Kaplan, Buck Shlegeris, Samuel R. Bowman, Ethan Perez, Evan Hubinger
Reinforcement Learning Reward Function Design Specification Game Environment Reward Hacking Visual Sycophancy

May 29, 2024

Inference-Time Alignment of Diffusion Models with Direct Noise Optimization
Zhiwei Tang, Jiangweizhi Peng, Jiasheng Tang, Mingyi Hong, Fan Wang, Tsung-Hui Chang
Diffusion Model Reward Function Alignment Problem Reward Hacking Differentiable Reward Noise Optimization

April 1, 2024

Regularized Best-of-N Sampling to Mitigate Reward Hacking for Language Model Alignment
Yuu Jinnai, Tetsuro Morimura, Kaito Ariu, Kenshi Abe
Language Model Pairwise Preference Language Model Alignment Reward Hacking Best of N

March 5, 2024

Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking
Cassidy Laidlaw, Shivam Singhal, Anca Dragan
State Occupancy Matching Reward Hacking Action Distribution Occupancy Model

February 14, 2024

InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling
Yuchun Miao, Sen Zhang, Liang Ding, Rong Bao, Lefei Zhang, Dacheng Tao
Reward Model Reinforcement Learning From Human Feedback Reward Hacking Reward Overoptimization Reward Misspecification Information Theoretic Reward

February 11, 2024

ODIN: Disentangled Reward Mitigates Hacking in RLHF
Lichang Chen, Chen Zhu, Davit Soselia, Jiuhai Chen, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, Bryan Catanzaro
Reinforcement Learning Reward Model Reinforcement Learning From Human Feedback Length Bias Reward Hacking

December 14, 2023

Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking
Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D'Amour, DJ Dvijotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, Peter Shaw, Jonathan Berant
Reward Model Herding Algorithm Reward Hacking

September 27, 2022

Defining and Characterizing Reward Hacking
Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, David Krueger
Deterministic Policy Stochastic Policy Proxy Reward Reward Hacking

August 29, 2022

Understanding the Limits of Poisoning Attacks in Episodic Reinforcement Learning
Anshuka Rangi, Haifeng Xu, Long Tran-Thanh, Massimo Franceschetti
Reinforcement Learning Continuum Limit Poisoning Attack Robust Reinforcement Learning Episodic Reinforcement Learning Reward Hacking

May 29, 2022

On the Robustness of Safe Reinforcement Learning under Observational Perturbations
Zuxin Liu, Zijian Guo, Zhepeng Cen, Huan Zhang, Jie Tan, Bo Li, Ding Zhao
Native Robustness Safe Reinforcement Learning Reward Hacking

January 10, 2022

The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models
Alexander Pan, Kush Bhatia, Jacob Steinhardt
Mixed Effect Reward Function Area MaPPing Agent Capability RL Agent Reward Hacking Reward Misspecification Model Misalignment

December 10, 2021

How Private Is Your RL Policy? An Inverse RL Based Analysis Framework
Kritika Prakash, Fiza Husain, Praveen Paruchuri, Sujit P. Gujar
Inverse Reinforcement Learning Reinforcement Learning Policy Reward Hacking Optimal Reinforcement Learning

Reward Hacking

Papers

Honesty to Subterfuge: In-Context Reinforcement Learning Can Make Honest Models Reward Hack

RRM: Robust Reward Model Training Mitigates Reward Hacking

Spontaneous Reward Hacking in Iterative Self-Refinement

Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models

Inference-Time Alignment of Diffusion Models with Direct Noise Optimization

Regularized Best-of-N Sampling to Mitigate Reward Hacking for Language Model Alignment

Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking

InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling

ODIN: Disentangled Reward Mitigates Hacking in RLHF

Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking

Defining and Characterizing Reward Hacking

Understanding the Limits of Poisoning Attacks in Episodic Reinforcement Learning

On the Robustness of Safe Reinforcement Learning under Observational Perturbations

The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models

How Private Is Your RL Policy? An Inverse RL Based Analysis Framework