Monitoring Emergent Reward Hacking During Generation via Internal Activations

Patrick Wilhelm; Thorsten Wittkopp; Odej Kao

cs.CLMar 4, 2026

Monitoring Emergent Reward Hacking During Generation via Internal Activations

Authors: Patrick Wilhelm, Thorsten Wittkopp, Odej Kao

Organizations: Technical University of Berlin, Germany · BIFOLD - Berlin Institute for the Foundations of Learning and Data, Germany

Abstract

Fine-tuned large language models can exhibit reward-hacking behavior arising from emergent misalignment, which is difficult to detect from final outputs alone. While prior work has studied reward hacking at the level of completed responses, it remains unclear whether such behavior can be identified during generation. We propose an activation-based monitoring approach that detects reward-hacking signals from internal representations as a model generates its response. Our method trains sparse autoencoders on residual stream activations and applies lightweight linear classifiers to produce token-level estimates of reward-hacking activity. Across multiple model families and fine-tuning mixtures, we find that internal activation patterns reliably distinguish reward-hacking from benign behavior, generalize to unseen mixed-policy adapters, and exhibit model-dependent temporal structure during chain-of-thought reasoning. Notably, reward-hacking signals often emerge early, persist throughout reasoning, and can be amplified by increased test-time compute in the form of chain-of-thought prompting under weakly specified reward objectives. These results suggest that internal activation monitoring provides a complementary and earlier signal of emergent misalignment than output-based evaluation, supporting more robust post-deployment safety monitoring for fine-tuned language models.

Figures & tables

Figure 1: Activation-based reward-hacking monitoring pipeline. Token-level internal activations are transformed into a generation-level reward-hacking prediction.

Figure 3: Temporal structure of reward-hacking activation during chain-of-thought generation. Monitor probabilities

p(\mathrm{hack})

are shown as a function of generation progress, aggregated into fixed bins of 64 tokens to allow comparison across CoT lengths (64, 128, 256, and 512 tokens). Rows correspond to model families and colored curves indicate different proportions of reward-hacking data in the fine-tuning corpus. Distinct temporal dynamics emerge across model families, early decrease for Llama , late-stage amplification for Qwen , and mixture-dependent behavior for Falcon . This model specific behavior is observable consistent across different CoT regimes and output lengths.

Figure 4: Relative change in hack-associated activation under chain-of-thought prompting. Bars show the percentage change in mean monitor probability relative to direct answering, for mixed-policy adapters and different chain-of-thought lengths. Amplification is strongest for partially misaligned adapters (5%, 10%), varies across model families, and diminishes as misalignment saturates (50%, 90%)

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Layer-wise logistic-regression accuracy for Falcon using the SAE $\rightarrow$ standardization $\rightarrow$ PCA $\rightarrow$ LR pipeline. Curves compare different PCA output dimensions ( $d\in\{2,64,256\}$ ), showing improved separability with larger $d$ across most layers.

Figure 6: Layer-wise logistic-regression accuracy for Llama using the SAE $\rightarrow$ standardization $\rightarrow$ PCA $\rightarrow$ LR pipeline. Curves compare different PCA output dimensions ( $d\in\{2,64,256\}$ ), showing improved separability with larger $d$ across most layers.

Figure 7: Layer-wise logistic-regression accuracy for Qwen using the SAE $\rightarrow$ standardization $\rightarrow$ PCA $\rightarrow$ LR pipeline. Curves compare different PCA output dimensions ( $d\in\{2,64,256\}$ ), showing improved separability with larger $d$ across most layers.

Figure 8: Adapter-conditioned late-stage slopes of reward-hack activation. Each panel corresponds to a fine-tuning mixture, plotting the late-stage slope

\beta_{\mathrm{late}}

as a function of CoT length for different model families. Positive slopes indicate increasing reward-hack activation near completion, while negative slopes indicate late-stage attenuation. Across mixtures, model families exhibit consistent slope sign, with magnitude modulated by the proportion of reward-hacking data.

Figure 9: Dose–response of late-stage change

\Delta_{\mathrm{late}}

with respect to the proportion of reward-hacking data in fine-tuning. Each panel corresponds to a CoT length, plotting

\Delta_{\mathrm{late}}

as a function of mixture percentage for different model families. Positive values indicate late-stage concentration of reward-hack activation, while negative values indicate early-stage concentration. Qualitative trends are consistent across CoT regimes, with mixture proportion primarily affecting magnitude rather than direction.

Figure 10: F1 Scores of the proposed monitoring system compared to the ChatGPT-4o labeling

Reward Hacking

Reward hacking studies how AI systems exploit imperfect objectives and how to detect, measure, and prevent behavior that scores well but misses intent.

See all 181 papers

Oct 20, 2026

Reward Misspecification Degrades Chain-of-Thought Monitorability

Monitoring the chain-of-thought (CoT) of language models (LMs) is a key strategy for detecting and preventing misbehaviors for AI safety. As such, it is critical to identify reinforcement learning (RL) practices that lead to unmonitorable chain-of-thought. One such practice may be unavoidable: reward misspecification. As RL environments become more complex, it is infeasible to have perfect verification, and models are rewarded for behaviors that are different from the true task intent. We show that reward misspecification alone already degrades monitorability. We create a testbed with pairs of true reward aligned with task intent and misspecified rewards that favor a different behavior, so that reward hacking can be induced by training and labeled verifiably. Across three models and three tasks, CoT monitorability remains low as models' hack rates increase: the CoTs keep reasoning about the task intent while the outputs violate the intent and get rewarded. To mitigate unmonitorability without prior knowledge of the true reward or the hack, we collect rollouts with diverse rewards across training steps and use a LM to infer a reward description. We then inject the description into the input, which teaches the model to verbalize the reward. This reward description training improves monitorability by an average of 24% or 42% absolute, depending on the metric, across all models and tasks, and yields CoTs that reason about the misspecified reward explicitly. Code is available at https://github.com/Lorenayannnnn/reward-misspecification-degrades-cot-monitorability.

Tianyi Lorena Yan, John Hewitt

Columbia University

LLM Safety Monitoring CoT Monitoring Reinforcement Learning RL for Language Models Reward Hacking

Ask PDF

Sep 16, 2026

Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations

As models scale, reward hacking becomes more frequent, more sophisticated, and more consequential. Does it leave a telltale signature in model representations? This work analyzes how reward hacking is represented internally in frontier open source LLMs, and how those representations can be used to understand and discover the range of hacking behaviors a model displays. In particular, we find that simple difference of means vectors coherently represent reward hacking in Kimi K3, GLM 5.2, and Qwen 3.8 Max across a variety of behaviors in common evaluations. Despite their simplicity, these vectors are both generalizable and interpretable, and we can use them to reliably detect reward hacking. We first evaluate reward hacking in commonly reported benchmarks like DeepSWE and SWE-bench, finding that models reward hack excessively in these environments; GLM 5.2 hacks in 57.2% of rollouts on DeepSWE and in 73% of rollouts on SWE-bench. Catching these requires monitors; LLM monitors are effective, but expensive detectors. We show that DoM vectors are similarly effective but virtually free, catching 3.1% more hacks in Kimi K3 and 7.9% fewer hacks in GLM 5.2 on DeepSWE at a monitor matched false positive rate. DoM vectors run on the chain-of-thought also predict reward hacks in the model's subsequent actions, meaning we can run them online and catch potential hacks before they occur. Finally, we analyze probe-hits that LLM monitors do not catch and discover other undesirable behaviors, as well as show transfer to finding hacks in non-SWE evaluations. Together, these results provide evidence that simple, white-box methods can be used to scalably study and monitor reward hacking behaviors in frontier open source models

Leon Bergen, Usha Bhalla, Andrew Lee +15

Core contributors · Equal senior contribution

Reward Hacking LLM Evaluation LLM Auditing

Ask PDF

Oct 7, 2026

RH-Detect: A Unified Benchmark for Reward Hacking Detection

Reward hacking, where a model exploits an evaluation signal without completing the intended task, threatens the reliability of deployed language model systems. Existing datasets use different labels, response formats, and metadata conventions, making detector results difficult to compare. We present RH-Detect, a benchmark that combines reward-hacking-relevant subsets from eleven public datasets, comprising 92,761 rows and six behavior categories, into a common schema. On 5,021 open-ended evaluation units, each comprising a task prompt and a free-form model continuation, including multi-turn tool-use trajectories, we evaluate six off-the-shelf language models from five families as reward hacking detectors without additional training. The best model achieves a pooled AUROC of 0.962, with accuracy above 93%. For the four strongest models, however, accuracy on the two multi-turn tool-use datasets, MALT and TRACE, is 10.7-15.9 percentage points lower than on the other sources at a common decision threshold, highlighting a key gap for deployment-time monitoring. We find that different input formats have different effects across models. Removing thinking raises Qwen3.5-4B AUROC from 0.779 to 0.849, but lowers Qwen Flash from 0.977 to 0.950. We also evaluate the benchmark as a training dataset for detectors. Holding out each source in turn, single-token SFT improves average AUROC on five of six held-out sources. A GRPO follow-up on that failure case yields a slight improvement in detection performance. Our results show that a single pooled score can conceal variation across data sources, detector inputs, and training procedures.

Junwei Quan, Evgenii Opryshko, Rohan Subramani +1

University of Toronto · Vector Institute · Trajectory Labs +1

Reward Hacking LLM Evaluation LLM Auditing

Ask PDF

Abstract

Reward Hacking

Explore similar work

Reward Misspecification Degrades Chain-of-Thought Monitorability

Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations

RH-Detect: A Unified Benchmark for Reward Hacking Detection