Reward Over-Optimization

Follow this topic Ask about this topic

Latest papers

15 papers2 papers in the last four weeks, down 50% on the four weeks before. 0.0% of all new papers.

Whether or Where? Tail Calibration Signals Finite-Budget Reward Hacking in Best-of-N Selection
Oct 25, 2026Cheng Gui, Yihua Shao, Lei Li +2Best-of-N Sampling Reward Over-Optimization
City University of Hong Kong · University of Science and Technology Beijing · University of Washington +1
Rent-Seeking Language Models: Reward Models Set the Prices, Frequent Audits Curb the Rent
Oct 14, 2026Zixiao Wang, Xingbo Du, Duzhen Zhang +1Reward Over-Optimization RL for Language Models
Mohamed bin Zayed University of Artificial Intelligence
CriticHack: Evaluating Visual Rewards Under Robot Policy Optimization
Oct 1, 2026Jiaxuan Luo, Xingguo Xu, Shanshan Wang +2RL for Robotics Reward Hacking
Johns Hopkins University · Dalian University of Technology · Rutgers University +2
Revisiting scaling laws for reward optimization
Sep 29, 2026Ali Aouad, Aymane El Gadarri, Vivek F. FariasReward Modeling Reward Over-Optimization
MIT
Optimizing the Score, Losing Sight of the Task: Reward Hacking Across Weights, Selection, and Prompts
Sep 22, 2026Vansh WahiReward Hacking LLM Evaluation
Proof-Carrying Cognition: Closing the Verification Gap with Reality-Settled Reward
Sep 9, 2026Eshwar Reddy M, Sourav KarmakarReward Modeling Reward Hacking
AI Engineer, Testsigma University of San Diego · Senior AI Scientist, Intuit India
Mitigating Over-Optimization in PRM-Guided Search in Mathematical Reasoning by Optimizing the Guide
Aug 30, 2026Taejong Joo, Diego KlabjanReward Over-Optimization Inference-Time Search
Department of Industrial Engineering & Management Sciences Northwestern University
Pessimism's Paradox: Conservative Offline Training Amplifies Reward Hacking During Online Adaptation in Reasoning Models
Jun 29, 2026Subramanyam Sahoo, Aman Chadha, Vinija Jain +1Offline-To-Online RL Reward Hacking
1Horizon Research · 2Apple · 3Meta +1
Self-Improvement Can Self-Regress: The Rise-and-Collapse Failure Mode of LLM Self-Training
Jun 17, 2026Jianzhe LinContinual RL RL for Code Generation
MetaAI · *Work done at Meta
EvalStop: Using World Feedback to Detect and Correct Reward Overoptimization in Multi-Tenant RLHF Platforms
Jun 2, 2026Guilin Zhang, Chuanyi Sun, Kai Zhao +3Reward Over-Optimization Early Stopping
Workday AI Research · The George Washington University, Washington, DC, USA.
Reward Bias Substitution: Single-Axis Bias Mitigations Redirect Optimization Pressure
May 27, 2026Max Lamparth, Daniel Fein, Andreas Haupt +2Reward Modeling Reward Over-Optimization
Stanford University · University of Pennsylvania
On the Implicit Reward Overfitting and the Low-rank Dynamics in RLVR
May 7, 2026Hao Ye, Jisheng Dang, Junfeng Fang +7Reward Over-Optimization Reinforcement Learning
Lanzhou University · National University of Singapore · Beijing Institute of Technology
Wasserstein Distributionally Robust Regret Optimization for Reinforcement Learning from Human Feedback
Apr 30, 2026Yikai Wang, Shang Liu, Jose BlanchetReward Over-Optimization Regret Minimization in RL
Department of Statistics and Operations Research, University of North Carolina · Imperial Business School, Imperial College London · Department of Management Science and Engineering, Stanford University
From Curiosity to Caution: Mitigating Reward Hacking for Best-of-N with Pessimism
Apr 6, 2026Zhuohao Yu, Zhiwei Steven Wu, Adam BlockReward Hacking Reward Over-Optimization
Carnegie Mellon University · Columbia University
Best-of-Tails: Bridging Optimism and Pessimism in Inference-Time Alignment
Mar 6, 2026Hsiang Hsu, Eric Lei, Chun-Fu ChenReward Over-Optimization Inference-Time Language Model Alignment
JPMorganChase Global Technology Applied Research

Oct 25, 2026J/K move · Enter open · S save

Whether or Where? Tail Calibration Signals Finite-Budget Reward Hacking in Best-of-N Selection

Cheng Gui, Yihua Shao, Lei Li, Jingcai Guo+1

City University of Hong Kong · University of Science and Technology Beijing · University of Washington+1

Best-of-N Sampling Reward Over-Optimization Model Calibration Probability Calibration

Open paper AskPDF arXiv

Best-of-N (BoN) inference returns the candidate with the highest proxy reward, so a larger

N

tests the proxy's extreme tail more than its average ranking quality. We show that the exact BoN value is a Beta-weighted integral of conditional true reward over within-prompt score quantiles. It follows that BoN depends on scores only through their ranks. It also follows that BoN can decline only if the calibration curve in the tail turns downward, and the budget where the decline appears depends on where that turn lies. Under a bounded additive model, the error tail sets the limit: Gaussian errors keep improving, Laplace errors reach a plateau, and heavy-tailed errors return to the prior mean. When error variance is largest at low true reward, even Gaussian noise can push BoN below random selection. Synthetic phase maps agree with this taxonomy. On real candidate pools spanning several verifiable domains, base policies, and learned reward models, tail failures are rare and all occur in code generation. A held-out tail calibration statistic reliably predicts where a cross-fitted budget cap helps. It does about as well as rank accuracy, and the cap is competitive with a Soft-BoN baseline. Tail calibration therefore does not replace ranking metrics. It is an audit that is unchanged by any monotone rescaling of the reward scores and shows which budgets are at risk.