Reward Hacking

Ask about this topic

Overview

Reward hacking occurs when a system optimizes a proxy—such as a reward model, judge, rubric, or benchmark verifier—without achieving the intended goal. Research studies this in settings including reinforcement learning from human feedback, verifiable-reward language-model training, tool-using agents, and image or robot generation. Common approaches include building controlled tests and adversarial audits, hardening verifiers, using uncertainty-aware or multiple reward signals, and monitoring behavior through outputs or internal activations. A central challenge is that optimization can expose new weaknesses or adapt around defenses, so high scores and low detector readings do not necessarily establish genuine task success or safety.

Latest papers

CardsList

182 papers43 papers in the last four weeks, up 105% on the four weeks before. 0.2% of all new papers.

  1. Calibration Collapse Under Sycophancy Fine-Tuning: How Reward Hacking Breaks Uncertainty Quantification in LLMs

    Apr 12, 2026Subramanyam SahooLanguage Model CalibrationReward Hacking

    Cambridge AI Safety Hub (CAISH)

  2. From Curiosity to Caution: Mitigating Reward Hacking for Best-of-N with Pessimism

    Apr 6, 2026Zhuohao Yu, Zhiwei Steven Wu, Adam BlockReward HackingReward Over-Optimization

    Carnegie Mellon University · Columbia University

  3. MC-CPO: Mastery-Conditioned Constrained Policy Optimization for Pedagogically Safe Intelligent Tutoring Systems

    Apr 5, 2026Oluseyi Olukola, Nick RahimiIntelligent Tutoring SystemsPolicy Learning

    School of Computing Sciences and Computer Engineering University of Southern Mississippi Hattiesburg MS 39406 USA

  4. Pedagogical Safety in Educational Reinforcement Learning: Formalizing and Detecting Reward Hacking in AI Tutoring Systems

    Apr 5, 2026Oluseyi Olukola, Nick RahimiReward HackingSafe RL

    School of Computing Sciences and Computer Engineering, Hattiesburg, MS, USA

  5. Mitigating Reward Hacking in RLHF via Advantage Sign Robustness

    Apr 3, 2026Shinnosuke Ono, Johannes Ackermann, Soichiro Nishimori +2Reward HackingRL from Human Feedback

    The University of Tokyo · RIKEN AIP

  6. From Rebound to Remedy: Understanding and Mitigating Reward Hacking via Representation Engineering

    Apr 1, 2026Rui Wu, Ruixiang TangReward HackingRL Fine-Tuning

    Department of Computer Science Rutgers University

  7. Extending MONA in Camera Dropbox: Reproduction, Learned Approval, and Design Implications for Reward-Hacking Mitigation

    Mar 31, 2026Nathan HeathReward HackingReward Modeling

    Independent Researcher

  8. SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot Reinforcement Learning

    Mar 30, 2026Philip Schroeder, Thomas Weng, Karl Schmeckpeper +3RL for RoboticsMultimodal Reward Modeling

    MIT · RAI Institute

  9. Reward Hacking as Equilibrium under Finite Evaluation

    Mar 30, 2026Jiacheng Wang, Jinbin HuangReward HackingAI Agent Evaluation

  10. Reward Sharpness-Aware Fine-Tuning for Diffusion Models

    Mar 22, 2026Kwanyoung Kim, Byeongsu SimDiffusion Model AlignmentRL for Diffusion Models

    Department of AI Convergence, GIST · Samsung Research

  11. From the Inside Out: Progressive Distribution Refinement for Confidence Calibration

    Mar 17, 2026Xizhong Yang, Yinan Xia, Huiming Wang +1Test-Time RLReinforcement Learning

    Southeast University · Kuaishou Technology · SUTD

  12. Diffusion Reinforcement Learning via Centered Reward Distillation

    Mar 14, 2026Yuanzhi Zhu, Xi Wang, Stéphane Lathuilière +1RL for Diffusion ModelsDiffusion Model Alignment

    LIX, École Polytechnique, CNRS, IPP · Inria at Univ. Grenoble Alpes, CNRS, LJK

  13. Mending the Holes: Mitigating Reward Hacking in Reinforcement Learning for Multilingual Translation

    Mar 13, 2026Yifeng Liu, Siqi Ouyang, Yatish Hosmane Revanasiddappa +1Low-Resource MTMultilingual Language Models

    Carnegie Mellon University

  14. Rethinking Multiple-Choice Questions for RLVR: Unlocking Potential via Distractor Design

    Mar 13, 2026Xu Guo, Qiming Ge, Jian Tong +8RL for Language Model ReasoningMultiple-Choice Question Answering

    Shanghai AI Laboratory · Shanghai Innovation Institute · Fudan University

  15. Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Training

    Mar 12, 2026Yixin Liu, Yue Yu, DiJia Su +7LLM-as-a-JudgeLLM Alignment

    Meta Superintelligence Labs · Yale Univeristy

  16. RewardHackingAgents: Benchmarking Evaluation Integrity for LLM ML-Engineering Agents

    Mar 11, 2026Yonas Atinafu, Robin CohenLLM Agent EvaluationAI Agent Security Benchmarks

    David R. Cheriton School of Computer Science, University of Waterloo, Waterloo, Canada

  17. Countdown-Code: A Testbed for Studying The Emergence and Generalization of Reward Hacking in RLVR

    Mar 7, 2026Muhammad Khalifa, Zohaib Khan, Omer Tafveez +2RL BenchmarksReward Hacking

    NVIDIA · University of Michigan · University of Illinois Urbana-Champaign

  18. Revisiting the (Sub)Optimality of Best-of-N for Inference-Time Alignment

    Mar 5, 2026Ved Sriraman, Adam BlockBest-of-N SamplingInference-Time Language Model Alignment

    Department of Computer Science, Columbia University · Department of Electrical Engineering, Columbia University

  19. Monitoring Emergent Reward Hacking During Generation via Internal Activations

    Mar 4, 2026Patrick Wilhelm, Thorsten Wittkopp, Odej KaoReward HackingEmergent Misalignment in Language Models

    Technical University of Berlin, Germany · BIFOLD - Berlin Institute for the Foundations of Learning and Data, Germany

  20. ContextRL: Enhancing MLLM's Knowledge Discovery Efficiency with Context-Augmented RL

    Feb 26, 2026Xingyu Lu, Jinpeng Wang, YiFan Zhang +12RL for VLMsReinforcement Learning with Verifiable Rewards

    Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China · Harbin Institute of Technology, Shenzhen, Shenzhen, China · Chinese Academy of Sciences, Beijing, China +2

  21. IR3^3: Contrastive Inverse Reinforcement Learning for Interpretable Detection and Mitigation of Reward Hacking

    Feb 23, 2026Mohammad Beigi, Ming Jin, Junshan Zhang +3Reward HackingReinforcement Learning

    UC Davis · Virginia Tech · Salesforce +1

  22. Reward Under Attack: Analyzing the Robustness and Hackability of Process Reward Models

    Feb 20, 2026Rishabh Tiwari, Aditya Tomar, Udbhav Bamba +5Process Reward ModelsReward Hacking

    UC Berkeley · Transmute AI · ICSI +1

  23. Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards

    Feb 20, 2026Johannes Ackermann, Michael Noukhovitch, Takashi Ishida +1Reward HackingReinforcement Learning with Verifiable Rewards

    The University of Tokyo · Mila · RIKEN AIP

  24. Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation

    Feb 10, 2026Pei-Chi Pan, Yingbin Liang, Sen LinReward ModelingReward Hacking

    Department of Computer Science University of Houston · The Ohio State University · University of Houston

  25. Adversarial Reward Auditing for Active Detection and Mitigation of Reward Hacking

    Feb 2, 2026Mohammad Beigi, Ming Jin, Junshan Zhang +2Adversarial TrainingReward Hacking

    Department of Computer Science, University of California, Davis, USA · Virginia Tech, Blacksburg, USA · Meta AI Research.

  26. Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMs

    Jan 16, 2026Lecheng Yan, Ruizhe Li, Guanhua Chen +5Reward HackingMemorization in Language Models

    Southern University of Science and Technology · University of Science and Technology of China · University of Aberdeen +5

  27. Understanding Reward Hacking in Text-to-Image Reinforcement Learning

    Jan 6, 2026Yunqi Hong, Kuei-Chun Kao, Hengguang Zhou +1Reward HackingMultimodal Reward Modeling

    Department of Computer Science, University of California, Los Angeles

  28. Training Language Models to Use Prolog as a Tool

    Dec 8, 2025Niklas Mellgren, Peter Schneider-Kamp, Lukas Galke PoechReward HackingLogical Reasoning

    University of Southern Denmark

  29. UMM-RM: An Upcycle-and-Merge MoE Reward Model for Mitigating Reward Hacking

    Nov 30, 2025Lingling Fu, Yongfu XueReward ModelingReward Hacking

    Guangxi University

  30. Test-Time Alignment of Text-to-Image Diffusion Models via Null-Text Embedding Optimisation

    Nov 25, 2025Taehoon Kim, Henry Gouk, Timothy HospedalesDiffusion Model AlignmentReward Hacking

    School of Informatics, University of Edinburgh · 2Samsung AI Center, Cambridge