Reward Hacking

Follow this topic Ask about this topic

Reward hacking occurs when a system optimizes a proxy—such as a reward model, judge, rubric, or benchmark verifier—without achieving the intended goal. Research studies this in settings including reinforcement learning from human feedback, verifiable-reward language-model training, tool-using agents, and image or robot generation. Common approaches include building controlled tests and adversarial audits, hardening verifiers, using uncertainty-aware or multiple reward signals, and monitoring behavior through outputs or internal activations. A central challenge is that optimization can expose new weaknesses or adapt around defenses, so high scores and low detector readings do not necessarily establish genuine task success or safety.

Latest papers

182 papers43 papers in the last four weeks, up 105% on the four weeks before. 0.2% of all new papers.

Hack-Verifiable Environments: Towards Evaluating Reward Hacking at Scale
May 20, 2026Amit Roth, Ankur Samanta, Matan Halevy +2Reward Hacking AI Agent Benchmarks
1Tel Aviv University · 2Columbia University · 3Taso Labs
Imperfect World Models are Exploitable
May 15, 2026Logan Mondal Bhamidipaty, Esmeralda S. Whitammer, David Abel +2Reward Hacking World Model-Based Planning
University of Edinburgh · Stanford University
Temper and Tilt Lead to SLOP: Reward Hacking Mitigation with Inference-Time Alignment
May 13, 2026Ye Wang, Jing Liu, Toshiaki Koike-AkinoReward Hacking LLM Alignment
Mitsubishi Electric Research Laboratories (MERL) · Broadway, Cambridge, MA 02139, USA
Pareto-Guided Optimal Transport for Multi-Reward Alignment
May 13, 2026Ying Ba, Tianyu Zhang, Mohan Zhou +5Diffusion Model Alignment Reward Hacking
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · Beijing Key Laboratory of Research on Large Models and Intelligent Governance · Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE 4Tomorrow Advancing Life +2
Effective Harness Engineering for Algorithm Discovery with Coding Agents
May 13, 2026Yoichi Ishibashi, Taro Yano, Masafumi OyamadaReward Hacking Automated Algorithm Discovery
NEC Corporation.
Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJack
May 12, 2026Hao Wang, Hanchen Li, Qiuyang Mang +3Computer-Use Agent Benchmarks Reward Hacking
UC Berkeley
Reward Hacking in Rubric-Based Reinforcement Learning
May 12, 2026Anas Mahmoud, MohammadHossein Rezaei, Zihao Wang +3Reward Hacking Reinforcement Learning
Scale AI
Semantic Reward Collapse and the Preservation of Epistemic Integrity in Adaptive AI Systems
May 12, 2026William ParrisReward Modeling Reward Hacking
BDB Labs / BagelTech · San Diego, California, USA
Power Reinforcement Post-Training of Text-to-Image Models with Super-Linear Advantage Shaping
May 11, 2026Haoyuan Sun, Jing Wang, Yuxin Song +9Reward Hacking Preference Optimization
Nanyang Technological University · Baidu Inc. · Zhejiang University +3
Test-Time Personalization: A Diagnostic Framework and Probabilistic Fix for Scaling Failures
May 9, 2026Linhai Zhang, Yulan HeReward Modeling Reward Hacking
King’s College London · The Alan Turing Institute
TMPO: Trajectory Matching Policy Optimization for Diverse and Efficient Diffusion Alignment
May 9, 2026Jiaming Li, Chenyu Zhu, Nanxi Yi +9Diffusion Model Alignment Reward Hacking
1MAIR Lab, Huazhong University of Science and Technology · 2Kuaishou Technology · 3Nanyang Technological University +2
Theoretical Limits of Language Model Alignment
May 8, 2026Lucas Monteiro Paes, Natalie Mackraz, Barry-John Theobald +1Reward Hacking LLM Alignment
Apple
Market-Alignment Risk in Pricing Agents: Trace Diagnostics and Trace-Prior RL under Hidden Competitor State
May 7, 2026Peiying Zhu, Sidi ChangReward Hacking Reinforcement Learning
Blossom AI · Blossom AI Labs
Gaming the Metric, Not the Harm: Certifying Safety Audits against Strategic Platform Manipulation
May 7, 2026Florian A. D. Burnat, Brittany I. DavidsonReward Hacking Algorithmic Auditing
University of Bath
Explaining and Preventing Alignment Collapse in Iterative RLHF
May 5, 2026Etienne Gauthier, Francis Bach, Michael I. JordanReward Hacking Policy Gradient
Inria, Ecole Normale Sup´erieure, PSL Research University · Ecole Normale Sup´erieure, PSL Research University · PSL Research University +1
Towards Understanding Specification Gaming in Reasoning Models
May 4, 2026Kei Nishimura-Gasparian, Robert McCarthy, David LindnerReward Hacking RL for Language Model Reasoning
1MATS · University College London, London, UK · 3Google Deep- Mind.
Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use
May 3, 2026Kunvar ThamanAI Agent Reliability Reward Hacking
Independent Researcher
What Makes a Good Terminal-Agent Benchmark Task: A Guideline for Adversarial, Difficult, and Legible Evaluation Design
Apr 30, 2026Ivan BercovichComputer-Use Agent Benchmarks Reward Hacking
Uncertainty-Aware Reward Discounting for Mitigating Reward Hacking
Apr 29, 2026Disha SinghaReward Hacking Reinforcement Learning
Distill-Belief: Closed-Loop Inverse Source Localization and Characterization in Physical Fields
Apr 28, 2026Yiwei Shi, Zixing Song, Mengyue Yang +2Belief-Space Planning Reward Hacking
SEMT, University of Bristol, Bristol, UK · LUCAS, Loughborough University, Loughborough, UK
Do Prompt-Elicited Trajectories Reflect Training-Time Reward Hacking? A Systematic Study on Monitoring Trainig-Time Reward Hacking in Code Generation
Apr 26, 2026Lichen Li, Hengguang Zhou, Yijun Liang +2RL for Code Generation Reward Hacking
Peking University · University of California, Los Angeles · University of Maryland +2
Emergent Strategic Reasoning Risks in AI: A Taxonomy-Driven Evaluation Framework
Apr 23, 2026Tharindu Kumarage, Lisa Bauer, Yao Ma +7Reward Hacking Language Model Safety Evaluation
Amazon Nova Responsible AI
Terminal Wrench: A Dataset of 331 Reward-Hackable Environments and 3,632 Exploit Trajectories
Apr 19, 2026Ivan Bercovich, Ivgeni Segal, Kexun Zhang +3Reward Hacking AI Agent Security Benchmarks
1Fewshot Corp · *Independent Researcher · 2Carnegie Mellon University
Beyond Static Snapshots: A Grounded Evaluation Framework for Language Models at the Agentic Frontier
Apr 19, 2026Jazmia HenryReward Hacking RL for Language Models
University of Oxford
Debate as Reward: A Multi-Agent Reward System for Scientific Ideation via RL Post-Training
Apr 17, 2026Moein Salimi, Babak Hosseini Mohtasham, Amin Aghakasiri +6Reward Hacking RL for Language Models
Sharif University of Technology · University of Tehran · Sharif University of Technology[cs.AI]
Detecting and Suppressing Reward Hacking with Gradient Fingerprints
Apr 17, 2026Songtao Wang, Quang Hieu Pham, Fangcong Yin +4Reward Hacking Reinforcement Learning with Verifiable Rewards
AUniversity of Alberta · N New York University · LLMU Munich +1
LLMs Gaming Verifiers: RLVR can Lead to Reward Hacking
Apr 16, 2026Lukas Helff, Quentin Delfosse, David Steinmann +6Reward Hacking Logical Reasoning
TU Darmstadt · DFKI · Intrinsic +4
Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challenges
Apr 15, 2026Xiaohua Wang, Muzhao Tian, Yuqi Zeng +20Reward Hacking LLM Alignment
Fudan NLP Group
Safety Training Modulates Harmful Misalignment Under On-Policy RL, But Direction Depends on Environment Design
Apr 14, 2026Leon Eshuijs, Shihan Wang, Antske FokkensReward Hacking RL for Language Models
Vrije Universiteit Amsterdam · Utrecht University
Robust Optimization for Mitigating Reward Hacking with Correlated Proxies
Apr 13, 2026Zixuan Liu, Xiaolin Sun, Zizhan ZhengReward Hacking Robust RL
Department of Computer Science Tulane University New Orleans, LA 70118, USA

May 20, 2026J/K move · Enter open · S save

Hack-Verifiable Environments: Towards Evaluating Reward Hacking at Scale

Amit Roth, Ankur Samanta, Matan Halevy, Yoav Levine+1

1Tel Aviv University · 2Columbia University · 3Taso Labs

Reward Hacking AI Agent Benchmarks Automated Evaluation

Open paper AskPDF arXiv

Aligning autonomous agents with human intent remains a central challenge in modern AI. A key manifestation of this challenge is reward hacking, whereby agents appear successful under the evaluation signal while violating the intended objective. Reward hacking has been observed across a wide range of settings, yet methods for reliably measuring it at scale remain lacking. In this work, we introduce a new evaluation paradigm for measuring reward hacking. Whereas prior studies have primarily analyzed it post hoc by inspecting agent trajectories, we instead embed detectable reward hacking opportunities directly into environments. This makes their exploitation verifiable by design, enabling deterministic and automated measurement of whether and how agents exploit such vulnerabilities. We instantiate this approach in

\textit{TextArena}

and release

\textit{Hack-Verifiable TextArena}

, a testbed in which reward hacking can be measured reliably. Using this benchmark, we analyze reward hacking behavior across language models in diverse environments and settings. We open source the code at https://github.com/MajoRoth/hack-verifiable-environments/.