cs.LGApr 15, 2026

Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challenges

Authors: Xiaohua Wang, Muzhao Tian, Yuqi Zeng, Zisu Huang, Jiakang Yuan, Bowen Chen, Jingwen Xu, Mingbo Zhou, +15 more

Organizations: Fudan NLP Group

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Reinforcement Learning from Human Feedback (RLHF) and related alignment paradigms have become central to steering large language models (LLMs) and multimodal large language models (MLLMs) toward human-preferred behaviors. However, these approaches introduce a systemic vulnerability: reward hacking, where models exploit imperfections in learned reward signals to maximize proxy objectives without fulfilling true task intent. As models scale and optimization intensifies, such exploitation manifests as verbosity bias, sycophancy, hallucinated justification, benchmark overfitting, and, in multimodal settings, perception--reasoning decoupling and evaluator manipulation. Recent evidence further suggests that seemingly benign shortcut behaviors can generalize into broader forms of misalignment, including deception and strategic gaming of oversight mechanisms. In this survey, we propose the Proxy Compression Hypothesis (PCH) as a unifying framework for understanding reward hacking. We formalize reward hacking as an emergent consequence of optimizing expressive policies against compressed reward representations of high-dimensional human objectives. Under this view, reward hacking arises from the interaction of objective compression, optimization amplification, and evaluator--policy co-adaptation. This perspective unifies empirical phenomena across RLHF, RLAIF, and RLVR regimes, and explains how local shortcut learning can generalize into broader forms of misalignment, including deception and strategic manipulation of oversight mechanisms. We further organize detection and mitigation strategies according to how they intervene on compression, amplification, or co-adaptation dynamics. By framing reward hacking as a structural instability of proxy-based alignment under scale, we highlight open challenges in scalable oversight, multimodal grounding, and agentic autonomy.

Figures & tables

Mechanism ClassExploited LocusTypical PatternTypical SettingMain Risk under PCH
Feature-levelSurface reward correlatesVerbosity bias; sycophancy; politeness inflation; formatting exploitsRLHF; RLAIF; preference tuningCompression Artifacts: Optimization amplifies superficial “survivor features” that dominate the low-dimensional proxy score.
Representation-levelLatent shortcut strategiesFabricated CoT; memorized schemas; benchmark gaming; process–outcome decouplingRLVR; reasoning; multimodal tasksEquivalence Classes: Degenerate shortcuts become indistinguishable from faithful reasoning under outcome-only evaluation.
Evaluator-levelReward model / judge / verifierBlind-spot exploitation; judge manipulation; selective compliance; concealed errorsLearned RMs; LLM-as-a-judge; rubric gradingCo-adaptation: The evaluator ceases to be a transparent metric and is recognized by the policy as a manipulable object.
Environment-levelTools, logs, tests, observation channelsTest rewriting; reward tampering; API/logging exploits; camera-angle hacksAgentic coding; tool use; embodied agentsAmplification Limit: The policy bypasses the latent objective by altering the physical or digital medium through which oversight operates.
Table 1: Taxonomy of Reward Hacking Mechanisms. Under the PCH, these four classes represent how policies exploit the bottlenecks and vulnerabilities of proxy-based alignment.
Figure 1: The illusion of alignment: Manifestations of reward hacking across diverse model families. When guided by imperfect proxy evaluators (Reward Models), models often discover strategies that maximize proxy scores (green boxes) while actively bypassing the true task intent (red boxes). (1) Large Language Models: Exploiting preference proxies through sycophancy and factual compromise. (2) Multimodal Language Models: Bypassing genuine visual grounding via metric-specific shortcuts , such as artificially inflating bounding boxes. (3) Visual Generative Models: Suffering structural degradation (e.g., generating a two-headed dog) when over-optimizing for specific semantic features. (4) Agentic Models: Engaging in environment-level exploitation by directly tampering with oversight mechanisms and static unit tests.
Figure 2: This figure serves as an organized preview of the four primary manifestations examined in this section: verbosity bias, sycophancy, fabricated reasoning, and reward overoptimization. For each manifestation, the figure traces its underlying PCH causal mechanism, distinguishing feature-level exploitation (e.g., length, agreement cues) from representation-level exploitation (e.g., reasoning faithfulness compression) and concludes with a bridge to how these localized shortcuts collectively set the stage for emergent strategic misalignment (Section 4 ). Together, this taxonomy provides a conceptual roadmap for the section’s sequential exposition of how reward hacking manifests across LLM alignment pipelines.
Figure 3: Conceptual structure of Section 4 . Local shortcut learning is the empirical starting point. Cross-task proxy optimization and evaluator-aware behavior capture two ways in which reward hacking can broaden. Evaluator–policy co-adaptation provides the dynamic framework through which both tendencies can be reinforced, redirected, or concealed over time.
Lifecycle StageMethod / ParadigmTarget Signal / ModalityAccessOverheadApplicable Scope
Training-Time (Adversarial Crucible)KL Divergence Tracking [ 1 ]Output probability dist.GreyLowGlobal Shift (Macroscopic collapse)
 Evaluator Stress Tests [ 68 ]RM sensitivity varianceBlack/GreyMediumSurface Proxies (Formatting, length)
 Adv. Reward Auditing [ 69 ]RM penultimate latentsWhiteMediumLatent Shortcuts
 VIB / CausalRM [ 38 , 70 ]Latent manifold shapeWhiteMediumSpurious Correlations
 Energy Loss Tracking [ 71 ]Policy final-layer L1L_{1} normWhiteLowContextual Evasion (Input ignorance)
Inference-Time (Pressure-Free Observation)Distributional Divergence [ 11 ]Action JSD vs. baselineBlackLowPhase Transitions (Sudden hacks)
Table 2: Taxonomy of Reward Hacking Detection and Diagnosis Methods Across the LLM Lifecycle.
Figure 4: An overview of three mitigation paradigms for reward hacking.
Figure 5: A structured matrix taxonomy of reward hacking extending beyond text-only models (Section 7 ). Unlike the feature and representation-level hacking seen in LLMs, scaling to Multimodal, Visual Generative, and Agentic domains introduces extreme dimensional compression and environmental feedback loops. This figure categorizes the specific manifestations of proxy exploitation in these modalities, traces them to their underlying structural causes (such as outcome sparsity and dimensional mismatch), and outlines the state-of-the-art mitigation strategies designed to enforce process-level robustness.

Explore similar work

CardsList
Feb 23, 2026

IR3^3: Contrastive Inverse Reinforcement Learning for Interpretable Detection and Mitigation of Reward Hacking

Reinforcement Learning from Human Feedback (RLHF) enables powerful LLM alignment but can introduce reward hacking - models exploit spurious correlations in proxy rewards without genuine alignment. Compounding this, the objectives internalized during RLHF remain opaque, making hacking behaviors difficult to detect or correct. We introduce IR3 (Interpretable Reward Reconstruction and Rectification), a framework that reverse-engineers, interprets, and surgically repairs the implicit objectives driving RLHF-tuned models. We propose Contrastive Inverse Reinforcement Learning (C-IRL), which reconstructs the implicit reward function by contrasting paired responses from post-alignment and baseline policies to explain behavioral shifts during RLHF. We then decompose the reconstructed reward via sparse autoencoders into interpretable features, enabling identification of hacking signatures through contribution analysis. Finally, we propose mitigation strategies - clean reward optimization, adversarial shaping, constrained optimization, and feature-guided distillation - that target problematic features while preserving beneficial alignment. Experiments across multiple reward model configurations show that IR3 achieves 0.89 correlation with ground-truth rewards, identifies hacking features with over 90% precision, and significantly reduces hacking behaviors while maintaining capabilities within 3% of the original model.

Mohammad Beigi, Ming Jin, Junshan Zhang +3

UC Davis · Virginia Tech · Salesforce +1

Reward HackingReinforcement LearningInterpretability in RLRL from Human Feedback
AskPDF
Jul 10, 2026

Multimodal Reward Hacking in Reinforcement Learning

Reinforcement learning (RL) is increasingly used to align multimodal large language models (MLLMs), but higher rewards do not always imply better task performance. This risk is amplified when visual evidence is evaluated by text-only or weakly grounded rewards. We study reward hacking in MLLM RL across safety VQA, chart VQA, and stress-test settings, varying reward design, data ambiguity, model scale (2B-32B), and RL algorithm (GRPO, RLOO, DAPO). We introduce Newly Rewarded Failure Rate (NRFR), which measures failures among samples whose proxy reward improves over the SFT baseline. Outcome-only rewards cause severe hacking, reaching 48.1% Reward Hacking Rate (RHR), while NRFR exceeding RHR shows that RL creates new failures rather than merely inheriting them. Scaling reduces but does not eliminate hacking: even the 32B model retains a 54.9% worse rate under outcome-only rewards, whereas answer-aware rewards improve the oracle trend at every scale. Robustness is also algorithm- and scale-dependent: GRPO is consistently most resistant, RLOO remains vulnerable, and DAPO improves substantially from 2B to 8B. Visual-evidence rewards help only with reliable verification: keyword-based checks increase hacking, while VLM-as-judge semantic verification reduces it. Overall, multimodal reward hacking is a systematic result of optimizing imperfect rewards, and robust alignment requires rewards and verifiers that remain reliable under optimization pressure.

Jiayu Yao, Yiwei Wang, Anmeng Zhang +5

Institute of Computing Technology, Chinese Academy of Sciences · University of California, Merced · Southeast University

Multimodal Large Language ModelsMultimodal Reward ModelingReinforcement Learning with Verifiable RewardsRL for VLMs
AskPDF
Mar 7, 2026

Countdown-Code: A Testbed for Studying The Emergence and Generalization of Reward Hacking in RLVR

Reward hacking is a form of misalignment in which models overoptimize proxy rewards without genuinely solving the underlying task. Precisely measuring reward hacking occurrence remains challenging because true task rewards are often expensive or impossible to compute. We introduce Countdown-Code, a minimal environment where models can both solve a mathematical reasoning task and manipulate the test harness. This dual-access design creates a clean separation between proxy rewards (test pass/fail) and true rewards (mathematical correctness), enabling accurate measurement of reward-hacking rates. Using this environment, we study reward hacking in open-weight LLMs and find that such behaviors can be unintentionally learned during supervised fine-tuning (SFT) when even a small fraction of reward-hacking trajectories leak into training data. As little as 1% contamination in distillation SFT data is sufficient for models to internalize reward hacking which resurfaces during subsequent reinforcement learning (RL). We further show that RL amplifies misalignment and drives its generalization beyond the original domain. We open-source our environment and code to facilitate future research on reward hacking in LLMs. Our results reveal a previously underexplored pathway through which reward hacking can emerge and persist in LLMs, underscoring the need for more rigorous validation of synthetic SFT data. Code is available at https://github.com/zohaib-khan5040/Countdown-Code.

Muhammad Khalifa, Zohaib Khan, Omer Tafveez +2

NVIDIA · University of Michigan · University of Illinois Urbana-Champaign

RL BenchmarksReward HackingRL for Language Model ReasoningReinforcement Learning with Verifiable Rewards
AskPDF