Reward Hacking

Follow this topic Ask about this topic

Momentum

20 papers in the last four weeks, up 100% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 112

All topics

From Reward-Hack Activations to Agentic Risk States: Context-Calibrated Mechanistic Monitoring in LLM Agents
Jun 4, 2026Patrick Wilhelm, Odej KaoReward Hacking AI Agent Safety
Distributed Operating Systems, Technische Universität Berlin, Berlin, Germany · BIFOLD - Berlin Institute for the Foundations of Learning and Data, Berlin, Germany
Self-Commitment Latency: A Reward-Free Probe for Prompted Implicit Hacking
Jun 4, 2026Bonan Shen, Youting Wang, Dingyan Shang +1Reward Hacking Language Model Probing
Independent Researcher · Northeastern University · Syracuse University
Reproducing, Analyzing, and Detecting Reward Hacking in Rubric-Based Reinforcement Learning
Jun 3, 2026Xuekang Wang, Zhuoyuan Hao, Shuo Hou +3Reward Hacking RL for Language Models
Tsinghua University · Harbin Institute of Technology, Shenzhen · Xi’an Jiaotong University
Large Language Models Hack Rewards, and Society
Jun 2, 2026Wei Liu, Xinyi Mou, Hanqi Yan +2Reward Hacking RL for Language Models
King’s College London · Fudan University · Shanghai Innovation Institute +1
When RLHF Fails: A Mechanistic Taxonomy of Reward Hacking, Collapse, and Evaluator Gaming
Jun 2, 2026Zelalem Abahana, David Evans, Satish Mahadevan Srinivasan +1Reward Hacking LLM Evaluation
College of Engineering, The Pennsylvania State University, University Park, PA, USA · Department of Computer Science, University of Virginia, Charlottesville, VA, USA · 3Penn State Great Valley, The Pennsylvania State University, Malvern, PA, USA +1
Follow-Your-Preference++: Rethinking Preference Alignment for Image Inpainting
Jun 2, 2026Junkun Yuan, Yutao Shen, Toru Aonishi +2Pairwise Preference Learning Reward Modeling
Zhejiang University · The University of Tokyo · Tsinghua University
HARVE: Hacking-Aware Reward-Head Vector Editing for Robust Reward Models
Jun 2, 2026Shuang Liu, Yuxuan Bo, Qiuyang Zhao +4Reward Modeling Reward Hacking
Carnegie Mellon University · University of Virginia · Harvard University +4
Are we really tilting? The mechanics of reward guidance in flow and diffusion models
Jun 1, 2026Sanjit Dandapanthula, Nicholas M. BoffiReward Hacking Diffusion Model Guidance
Carnegie Mellon University
Before the Model Learns the Bug:Fuzzing RLVR Verifiers
May 31, 2026Jaideep RayReward Hacking Automated Software Testing
What Makes a Medical Checker Trainable? Diagnosing Signal Collapse and Reward Hacking in Checker-Guided RAG for Biomedical QA
May 25, 2026Yuelyu Ji, Min Gu Kwak, Hang Zhang +3Reward Hacking Retrieval-Augmented Generation
University of Pittsburgh, Pittsburgh, PA, USA
Hide to Guide: Learning via Semantic Masking
May 24, 2026Ruitao Liu, Qinghao Hu, Alex Hu +6Reward Hacking RL for Language Model Reasoning
1MIT · 2NVIDIA
Directional Alignment Mitigates Reward Hacking in Reinforcement Learning for Language Models
May 24, 2026Wenlong Deng, Jiaji Huang, Kaan Ozkara +4Reward Hacking RL for Language Models
University of British Columbia · Vector Institute · Amazon
Hack-Verifiable Environments: Towards Evaluating Reward Hacking at Scale
May 20, 2026Amit Roth, Ankur Samanta, Matan Halevy +2Reward Hacking AI Agent Benchmarks
1Tel Aviv University · 2Columbia University · 3Taso Labs
Imperfect World Models are Exploitable
May 15, 2026Logan Mondal Bhamidipaty, Esmeralda S. Whitammer, David Abel +2Reward Hacking World Model-Based Planning
University of Edinburgh · Stanford University
Temper and Tilt Lead to SLOP: Reward Hacking Mitigation with Inference-Time Alignment
May 13, 2026Ye Wang, Jing Liu, Toshiaki Koike-AkinoReward Hacking LLM Alignment
Mitsubishi Electric Research Laboratories (MERL) · Broadway, Cambridge, MA 02139, USA
Pareto-Guided Optimal Transport for Multi-Reward Alignment
May 13, 2026Ying Ba, Tianyu Zhang, Mohan Zhou +5Diffusion Model Alignment Reward Hacking
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · Beijing Key Laboratory of Research on Large Models and Intelligent Governance · Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE 4Tomorrow Advancing Life +2
Effective Harness Engineering for Algorithm Discovery with Coding Agents
May 13, 2026Yoichi Ishibashi, Taro Yano, Masafumi OyamadaReward Hacking Automated Algorithm Discovery
NEC Corporation.
Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJack
May 12, 2026Hao Wang, Hanchen Li, Qiuyang Mang +3Computer-Use Agent Benchmarks Reward Hacking
UC Berkeley
Reward Hacking in Rubric-Based Reinforcement Learning
May 12, 2026Anas Mahmoud, MohammadHossein Rezaei, Zihao Wang +3Reward Hacking Reinforcement Learning
Scale AI
Semantic Reward Collapse and the Preservation of Epistemic Integrity in Adaptive AI Systems
May 12, 2026William ParrisReward Modeling Reward Hacking
BDB Labs / BagelTech · San Diego, California, USA
Power Reinforcement Post-Training of Text-to-Image Models with Super-Linear Advantage Shaping
May 11, 2026Haoyuan Sun, Jing Wang, Yuxin Song +9Reward Hacking Preference Optimization
Nanyang Technological University · Baidu Inc. · Zhejiang University +3
Test-Time Personalization: A Diagnostic Framework and Probabilistic Fix for Scaling Failures
May 9, 2026Linhai Zhang, Yulan HeReward Modeling Reward Hacking
King’s College London · The Alan Turing Institute
TMPO: Trajectory Matching Policy Optimization for Diverse and Efficient Diffusion Alignment
May 9, 2026Jiaming Li, Chenyu Zhu, Nanxi Yi +9Diffusion Model Alignment Reward Hacking
1MAIR Lab, Huazhong University of Science and Technology · 2Kuaishou Technology · 3Nanyang Technological University +2
Theoretical Limits of Language Model Alignment
May 8, 2026Lucas Monteiro Paes, Natalie Mackraz, Barry-John Theobald +1Reward Hacking LLM Alignment
Apple
Market-Alignment Risk in Pricing Agents: Trace Diagnostics and Trace-Prior RL under Hidden Competitor State
May 7, 2026Peiying Zhu, Sidi ChangReward Hacking Reinforcement Learning
Blossom AI · Blossom AI Labs
Gaming the Metric, Not the Harm: Certifying Safety Audits against Strategic Platform Manipulation
May 7, 2026Florian A. D. Burnat, Brittany I. DavidsonReward Hacking Algorithmic Auditing
University of Bath
Explaining and Preventing Alignment Collapse in Iterative RLHF
May 5, 2026Etienne Gauthier, Francis Bach, Michael I. JordanReward Hacking Policy Gradient
Inria, Ecole Normale Sup´erieure, PSL Research University · Ecole Normale Sup´erieure, PSL Research University · PSL Research University +1
Towards Understanding Specification Gaming in Reasoning Models
May 4, 2026Kei Nishimura-Gasparian, Robert McCarthy, David LindnerReward Hacking RL for Language Model Reasoning
1MATS · University College London, London, UK · 3Google Deep- Mind.
Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use
May 3, 2026Kunvar ThamanAI Agent Reliability Reward Hacking
Independent Researcher
What Makes a Good Terminal-Agent Benchmark Task: A Guideline for Adversarial, Difficult, and Legible Evaluation Design
Apr 30, 2026Ivan BercovichComputer-Use Agent Benchmarks Reward Hacking

Jun 4, 2026, cs.AIJ/K move · Enter open · S save

From Reward-Hack Activations to Agentic Risk States: Context-Calibrated Mechanistic Monitoring in LLM Agents

Patrick Wilhelm, Odej Kao

Distributed Operating Systems, Technische Universität Berlin, Berlin, Germany · BIFOLD - Berlin Institute for the Foundations of Learning and Data, Berlin, Germany

Reward Hacking AI Agent Safety Mechanistic Interpretability AI Agent Monitoring

Open paper AskPDF arXiv

Language-model agents act through repeated cycles of observation, reasoning, and action selection, making safety monitoring depend on both internal model state and environment context. We study reward-hacking monitors in ReAct-style agents acting in Gameable ALFWorld and WebShop. Agents are instrumented with activation-based reward-hack scores, token-level entropy, and decision-context features. We find that adapters fine-tuned on \textit{School-of-Reward-Hacks} dataset can transfer reward-hack tendencies into agentic action selection, especially when the environment exposes proxy-reward affordances. However, mitigating such behavior cannot rely on activation dynamics alone. High reward-hack activation identifies a latent policy state, but does not necessarily imply an immediate exploit action. Across next-step prediction tasks, entropy and context-calibrated internal features improve risk estimation over reward-hack activation alone. Activation-direction steering further reduces proxy-exploit behavior in selected mixed-adapter regimes. Overall, our results support context-calibrated internal monitoring for agents: reward-hack activation identifies a latent policy state, while entropy and decision context help determine when that state becomes risky action.

Momentum

Related topics

Latest papers 112