When RLHF Fails: A Mechanistic Taxonomy of Reward Hacking, Collapse, and Evaluator Gaming

Zelalem Abahana; David Evans; Satish Mahadevan Srinivasan; Matjaz Gams

cs.LGJun 2, 2026

When RLHF Fails: A Mechanistic Taxonomy of Reward Hacking, Collapse, and Evaluator Gaming

Authors: Zelalem Abahana, David Evans, Satish Mahadevan Srinivasan, Matjaz Gams

Organizations: College of Engineering, The Pennsylvania State University, University Park, PA, USA · Department of Computer Science, University of Virginia, Charlottesville, VA, USA · 3Penn State Great Valley, The Pennsylvania State University, Malvern, PA, USA · Department of Intelligent Systems, Joˇzef Stefan Institute, Ljubljana, Slovenia

Abstract

RLHF evaluation should track how failures emerge, where they localize, and which warning signals appear before external quality degrades. We study this problem with a compact RLHF pipeline built for this paper, including PPO, DPO, uncertainty-penalized PPO (UP-PPO), reward-model uncertainty, approximate policy drift, diversity and repetition diagnostics, and two external LLM judges. Rather than treating reward hacking as a single terminal event, we classify matched checkpoint and prompt-level transitions by the directions of learned reward R_phi, judge scores R_dag and R2_dag, and their average R_dag. The main empirical findings are that aggressive PPO produces the clearest localized reward-hacking signal, UP-PPO reduces but does not eliminate that signal, row-level diagnostics reveal failures hidden by checkpoint averages, and pre-transition features partially anticipate future localized reward hacking. The central conclusion is methodological: RLHF failures are training dynamics that can be classified, localized, and partially anticipated, not only final-model pathologies. The repository is available at github.com/zabahana/rlhf-failure-modes-diagnostics. The pipeline is also deployed as a live interactive web demo for model comparison and diagnostic views at rlhf-failures.zelalem.ai.

Explore similar work

Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use
May 3, 2026Kunvar ThamanAgentic Benchmarks Large Language Model Agents
Reproducing, Analyzing, and Detecting Reward Hacking in Rubric-Based Reinforcement Learning
Jun 3, 2026Xuekang Wang, Zhuoyuan Hao, Shuo Hou +3Rubric-Based Reinforcement Learning Implicit Bias
A Unifying Lens on Reward Uncertainty in RLHF
Jun 8, 2026Ely Hahami, Yoel Zimmermann, Ray Zhou +1Reinforcement Learning From Human Feedback External Reward Models

May 3, 2026 · cs.LGJ/K move · Enter open · S save

Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use

Kunvar Thaman

Independent Researcher

Reinforcement learning (RL) trained language model agents with tool access are increasingly deployed in coding assistants, research tools, and autonomous systems. We introduce the Reward Hacking Benchmark (RHB), a suite of multi-step tasks requiring sequential tool operations with naturalistic shortcut opportunities such as skipping verification steps, inferring answers from task-adjacent metadata, or tampering with evaluation-relevant functions. RHB supports independent and chained task regimes, where chain length acts as a proxy for longer-horizon agent behavior. We evaluate 13 frontier models from OpenAI, Anthropic, Google, and DeepSeek. Exploit rates range from 0% (Claude Sonnet 4.5) to 13.9% (DeepSeek-R1-Zero), varying sharply by post-training style. A controlled sibling comparison (DeepSeek-V3 vs. DeepSeek-R1-Zero) shows RL post-training is associated with substantially higher reward hacking (0.6% vs. 13.9%), with consistent gaps across all four task families. We identify six exploit categories and find that 72% of reward hacking episodes include explicit chain-of-thought rationale, suggesting models often frame exploits as legitimate problem-solving. Simple environmental hardening reduces exploit rates by 5.7 percentage points (87.7% relative) without degrading task success. Models with near-zero exploit rates on standard tasks show elevated rates on harder variants, suggesting that production-aligned post-training appears to suppress reward hacking only below a complexity threshold where honest solutions remain tractable.

Agentic Benchmarks Large Language Model Agents Exploitation Tool Deepseek

Open paper Ask about it PDF

Abstract

Explore similar work

Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use

Reproducing, Analyzing, and Detecting Reward Hacking in Rubric-Based Reinforcement Learning

A Unifying Lens on Reward Uncertainty in RLHF