Reward Hacking

Follow this topic Ask about this topic

Momentum

20 papers in the last four weeks, up 100% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 112

All topics

SpecGuard: Proving a Task Is Broken Before the Agent Cheats
Oct 6, 2026Param Biyani, Krishnamurthy DvijothamAI Agent Safety Software Engineering Agents
MATS · Google DeepMind
Better Call Reward: Reward Hacking as Strategic Abstention in Legal Reasoning Models
Oct 5, 2026Subramanyam Sahoo, Justin ShenkReward Hacking LLM Auditing
Horizon Research.
RubricArmor: Adversarial Evolution Improves LLM-Based Rubric Generation
Oct 4, 2026Haocheng Yang, Yuchao Zhang, Licheng Pan +8Reward Hacking Rubric-Based RL
National University of Singapore · Beijing University of Chemical Technology · Zhejiang University +7
CATCH: A Controllable Analysis Testbed for Reward Hacking in Coding RL
Sep 30, 2026Shouli Wang, Yanfeng Jia, Zhihao Ou +6RL Benchmarks RL for Code Generation
Tsinghua University · Beihang University · Southern University of Science and Technology +2
Improving Predicted MOS Scores, Not Perceived Quality: Multi-Predictor Test-Time Optimization of Enhanced Speech
Sep 30, 2026Tsubasa Ochiai, Marc Delcroix, Nahomi Kusunoki +5Reward Hacking Test-Time Optimization
NTT, Inc., Japan · Waseda University, Japan
Scoring Higher, Answering Worse: Mitigating Reward Hacking in Rubric-Based RL via Protocol-Level Rubrics
Sep 30, 2026Maoqi Liu, Junwei He, Bowen Zhang +5Reward Hacking Rubric-Based RL
Beijing University of Posts and Telecommunications · ByteDance
STAR-GRPO: Canonical Anchoring and Reliability-First Advantages against Representation-Dependent Reward Hacking
Sep 29, 2026Wan Tian, Zhongyi Li, Xiang Xu +3Reward Hacking Group Relative Policy Optimization
Peking University · Beihang University · Nanjing University
CheatBench: Measuring Reward Gaming in AI Agents
Sep 28, 2026Long Phan, Stephen K. Yang, Jason J. Lim +10Reward Hacking AI Agent Benchmarks
Center for AI Safety · Work done while at the Center for AI Safety.
CoRe: Co-Evolving Reward Models for Mitigating Latent Reward Hacking in Video Diffusion Models
Sep 28, 2026Zhaolong Su, Yujin Han, Feng Wang +3Diffusion Model Alignment Reward Modeling
Cornell University · The University of Hong Kong · Johns Hopkins University
Verifier Errors in RLVR: Reward Hacking, Limits of Feedback, and Selective Control
Sep 28, 2026Christian Moya, Elliott Thornley, Guang LinReward Hacking Reinforcement Learning
Purdue University · National University of Singapore
When the Score Becomes the Target: Rethinking Metric Validity in Autonomous Driving
Sep 28, 2026Morui Zhu, Deyuan Qu, Qi Chen +2Autonomous Driving Benchmarks Reward Hacking
University of North Texas · Toyota InfoTech Labs
Maintaining Benchmarks Against Increasingly Capable Agents: Detection and Remediation of Unearned Passes
Sep 28, 2026Weijun Luo, Kelvin Luu, Xinyi Liu +6Reward Hacking Benchmark Auditing
Scale AI
Reward Hacking Challenges Oversight of Autonomous Research Agents
Sep 23, 2026Yue Huang, Zhangchen Xu, Yuchen Ma +12Reward Hacking AI Agent Auditing
Bake AI · University of Notre Dame · University of Washington +6
Optimizing the Score, Losing Sight of the Task: Reward Hacking Across Weights, Selection, and Prompts
Sep 22, 2026Vansh WahiReward Hacking LLM Evaluation
Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations
Sep 16, 2026Leon Bergen, Usha Bhalla, Andrew Lee +15Reward Hacking LLM Evaluation
Core contributors · Equal senior contribution
ImpossibleRubrics: Stress-Testing Generated Rubrics as Reward Signals
Sep 15, 2026Bowen Qin, Yi Xie, Yesheng Liu +1Reward Hacking Language Model Generation Evaluation
National University of Singapore · Peking University · Institute of Automation, Chinese Academy of Sciences
Shallow Beliefs: Synthetic document finetuning does not inoculate against emergent misalignment from reward hacking
Sep 14, 2026Arun Jose, Julian StastnyReward Hacking Fine-Tuning
Astra Fellowship · Redwood Research
EvoRS: On-Policy Self-Evolution of Reward Systems for Open-Ended Reinforcement Learning
Sep 14, 2026Weiyuan Li, Aili Chen, Xintao Wang +11Reward Hacking Reinforcement Learning
School of Data Science, Fudan University · Shanghai Key Laboratory of Data Science · College of Computer Science and Artificial Intelligence, Fudan University +3
BenchShield: Formal Model-Backed Instrumentation for Reward Integrity in LLM-Agent Evaluation Infrastructure
Sep 12, 2026Shenghan Zheng, Zonglin Di, Yimin Liu +19Reward Hacking LLM Agent Evaluation
Dartmouth College · Independent · Ohio State University +10
Proof-Carrying Cognition: Closing the Verification Gap with Reality-Settled Reward
Sep 9, 2026Eshwar Reddy M, Sourav KarmakarReward Modeling Reward Hacking
AI Engineer, Testsigma University of San Diego · Senior AI Scientist, Intuit India
SWE-Bench Pro Verified: A Reliable Benchmark for Software Engineering Agents
Sep 8, 2026Pujun Zheng, Zixin Shang, Shufan Jiang +5Reward Hacking Software Engineering Agents
East China Normal University · Shanghai Artificial Intelligence Laboratory · Fudan University
Harness-agnostic detection and immunization of reward hacking in self-evolving language models
Sep 7, 2026Rongxin Yang, Yang Liu, Shang Luo +10Reward Hacking Language Model Self-Improvement
1Fullive-AI · 2Peking University · 4Nanyang Technological University +2
LLM-as-a-Judge Is Not an Oracle: Why Self-Improving Agents Need Deterministic Guardrails
Sep 2, 2026Vansh WahiLLM-as-a-Judge Self-Improving Agents
AI Research Engineer
CARE: Contrastive Anchor-based Rubric Evolution for Large Language Model Post-Training
Sep 1, 2026Siyuan Li, Xinxin Song, Chen Ruinian +5Reward Hacking Rubric-Based RL
Department of Automation, Tsinghua University · Meituan
Uncovering and Mitigating Aggregation-Induced Reward Hacking in Multi-Reward Reinforcement Learning
Aug 31, 2026Yu Yuan, Yaoyou Fan, Lili Zhao +5Multi-Objective Reinforcement Learning Reward Hacking
University of Science and Technology of China · 2State Key Laboratory of Cognitive Intelligence · 3Peking University +1
What Emerges and What Breaks in Self-Play Driving
Aug 31, 2026Laur Sisask, Ardi Tampuu, Tambet MatiisenReward Hacking Self-Play RL
University of Tartu, Tartu, Estonia
BAITBENCH: Measuring Agent Reward Hacking with Optional Shortcuts Planted in ML Tasks
Aug 31, 2026Pradyumna Shyama Prasad, Meiri Anto, Leon Eshuijs +3Reward Hacking AI Agent Evaluation
National University of Singapore · MIT · Vrije Universiteit Amsterdam +3
Beneath the Diff: Diagnosing and Mitigating Algorithmic Mode Collapse in Code-Level Autonomous Research Loops
Aug 31, 2026Bowei He, Weixu Zhang, Yili Jin +1Reward Hacking Automated Algorithm Discovery
MBZUAI · McGill University · MirrorSpace Technology +1
TACS: Trajectory-Aware Candidate Selection for LLM Jailbreak Suffix Optimization
Aug 30, 2026Shiliang XiaoReward Hacking LLM Jailbreak Attacks
Can escalation channels redirect reward hacking toward defect disclosure?
Aug 29, 2026Francesca GomezReward Hacking Coding Agents
Wiser Human

Oct 6, 2026, cs.AIJ/K move · Enter open · S save

SpecGuard: Proving a Task Is Broken Before the Agent Cheats

Param Biyani, Krishnamurthy Dvijotham

MATS · Google DeepMind

AI Agent Safety Software Engineering Agents Formal Verification Reward Hacking

Open paper AskPDF arXiv

As autonomous coding agents get increasingly deployed, the risk that accidental or adversarially injected misspecifications in tasks lead to dangerous agent behavior is critical to address. Prior work has shown that agents given such tasks rarely flag the conflict and instead cheat, editing tests or hard-coding expected outputs, and the actions taken to cheat can cause real damage, such as deleting a security defense to make a corrupted test pass. It remains unclear whether such conflicts can be established with independently verifiable evidence before the agent acts. We present SpecGuard, which detects and formally certifies these conflicts between task intent and tests. Given only the task description and codebase, SpecGuard autoformalizes the intended behaviour into a Lean 4 specification. The tests are formalized independently, and the Lean kernel checks whether any implementation could satisfy both formalizations, producing a machine-checked certificate when none can. On conflicted SWE-bench tasks, SpecGuard detects up to 72.8% of conflicts and formally certifies up to 51.1%, with a nearly five-fold lower conflict miss rate than model-based judgment. SpecGuard provides a pre-execution safety check that identifies reward-hacking opportunities through formal certification of task-level conflicts, before any agent behavior is observed. Our code is available at https://github.com/prmbiy/specguard.

Momentum

Related topics

Latest papers 112