Reward Over-Optimization

Follow this topic Ask about this topic

Latest papers

14 papers4 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Rent-Seeking Language Models: Reward Models Set the Prices, Frequent Audits Curb the Rent
Oct 14, 2026Zixiao Wang, Xingbo Du, Duzhen Zhang +1Reward Over-Optimization RL for Language Models
Mohamed bin Zayed University of Artificial Intelligence
CriticHack: Evaluating Visual Rewards Under Robot Policy Optimization
Oct 1, 2026Jiaxuan Luo, Xingguo Xu, Shanshan Wang +2RL for Robotics Reward Hacking
Johns Hopkins University · Dalian University of Technology · Rutgers University +2
Revisiting scaling laws for reward optimization
Sep 29, 2026Ali Aouad, Aymane El Gadarri, Vivek F. FariasReward Modeling Reward Over-Optimization
MIT
Optimizing the Score, Losing Sight of the Task: Reward Hacking Across Weights, Selection, and Prompts
Sep 22, 2026Vansh WahiReward Hacking LLM Evaluation
Proof-Carrying Cognition: Closing the Verification Gap with Reality-Settled Reward
Sep 9, 2026Eshwar Reddy M, Sourav KarmakarReward Modeling Reward Hacking
AI Engineer, Testsigma University of San Diego · Senior AI Scientist, Intuit India
Mitigating Over-Optimization in PRM-Guided Search in Mathematical Reasoning by Optimizing the Guide
Aug 30, 2026Taejong Joo, Diego KlabjanReward Over-Optimization Inference-Time Search
Department of Industrial Engineering & Management Sciences Northwestern University
Pessimism's Paradox: Conservative Offline Training Amplifies Reward Hacking During Online Adaptation in Reasoning Models
Jun 29, 2026Subramanyam Sahoo, Aman Chadha, Vinija Jain +1Offline-To-Online RL Reward Hacking
1Horizon Research · 2Apple · 3Meta +1
Self-Improvement Can Self-Regress: The Rise-and-Collapse Failure Mode of LLM Self-Training
Jun 17, 2026Jianzhe LinContinual RL RL for Code Generation
MetaAI · *Work done at Meta
EvalStop: Using World Feedback to Detect and Correct Reward Overoptimization in Multi-Tenant RLHF Platforms
Jun 2, 2026Guilin Zhang, Chuanyi Sun, Kai Zhao +3Reward Over-Optimization Early Stopping
Workday AI Research · The George Washington University, Washington, DC, USA.
Reward Bias Substitution: Single-Axis Bias Mitigations Redirect Optimization Pressure
May 27, 2026Max Lamparth, Daniel Fein, Andreas Haupt +2Reward Modeling Reward Over-Optimization
Stanford University · University of Pennsylvania
On the Implicit Reward Overfitting and the Low-rank Dynamics in RLVR
May 7, 2026Hao Ye, Jisheng Dang, Junfeng Fang +7Reward Over-Optimization Reinforcement Learning
Lanzhou University · National University of Singapore · Beijing Institute of Technology
Wasserstein Distributionally Robust Regret Optimization for Reinforcement Learning from Human Feedback
Apr 30, 2026Yikai Wang, Shang Liu, Jose BlanchetReward Over-Optimization Regret Minimization in RL
Department of Statistics and Operations Research, University of North Carolina · Imperial Business School, Imperial College London · Department of Management Science and Engineering, Stanford University
From Curiosity to Caution: Mitigating Reward Hacking for Best-of-N with Pessimism
Apr 6, 2026Zhuohao Yu, Zhiwei Steven Wu, Adam BlockReward Hacking Reward Over-Optimization
Carnegie Mellon University · Columbia University
Best-of-Tails: Bridging Optimism and Pessimism in Inference-Time Alignment
Mar 6, 2026Hsiang Hsu, Eric Lei, Chun-Fu ChenReward Over-Optimization Inference-Time Language Model Alignment
JPMorganChase Global Technology Applied Research

Oct 14, 2026J/K move · Enter open · S save

Rent-Seeking Language Models: Reward Models Set the Prices, Frequent Audits Curb the Rent

Zixiao Wang, Xingbo Du, Duzhen Zhang, Le Song

Mohamed bin Zayed University of Artificial Intelligence

Reward Over-Optimization RL for Language Models Reward Hacking Reward Modeling LLM Auditing

Open paper AskPDF arXiv

Reward model overoptimization in RLHF is usually described as reward hacking. We recast it as rent-seeking: the policy, a firm, earns reward from a regulator, the proxy reward model, that true quality does not back. The lens turns overoptimization into measurable quantities, namely rent, a capture index and deadweight loss, and into testable predictions from the economics of regulation. In a fixed, version-controlled testbed with three model families, accuracy-matched proxy reward models and two gold reward models, RL turns 41% to 66% of its proxy gain into rent while gold reward plateaus. Rent follows each regulator's prices. Neither proxy verifies facts, but one pays for confident specifics and the other for displays of diligence, so RL policies learn to fabricate details about nonexistent entities (97% to 100% of probes) or to deflect factual questions (67% to 98%). Kinship between policy and regulator adds a fixed premium of about 0.1 standard deviations but gives the policy no extra leverage under optimization. At a fixed expected fine, frequent mild audits beat rare severe ones, and random audits are as efficient per audit as targeted ones. Bringing the bunching estimator of tax economics to RL training, we also find a first sign that policies learn to stay just below a targeted audit's threshold.