Fine-tuned large language models can exhibit reward-hacking behavior arising from emergent misalignment, which is difficult to detect from final outputs alone. While prior work has studied reward hacking at the level of completed responses, it remains unclear whether such behavior can be identified during generation. We propose an activation-based monitoring approach that detects reward-hacking signals from internal representations as a model generates its response. Our method trains sparse autoencoders on residual stream activations and applies lightweight linear classifiers to produce token-level estimates of reward-hacking activity. Across multiple model families and fine-tuning mixtures, we find that internal activation patterns reliably distinguish reward-hacking from benign behavior, generalize to unseen mixed-policy adapters, and exhibit model-dependent temporal structure during chain-of-thought reasoning. Notably, reward-hacking signals often emerge early, persist throughout reasoning, and can be amplified by increased test-time compute in the form of chain-of-thought prompting under weakly specified reward objectives. These results suggest that internal activation monitoring provides a complementary and earlier signal of emergent misalignment than output-based evaluation, supporting more robust post-deployment safety monitoring for fine-tuned language models.
Figures & tables
Figure 1: Activation-based reward-hacking monitoring pipeline. Token-level internal activations are transformed into a generation-level reward-hacking prediction.
Figure 3: Temporal structure of reward-hacking activation during chain-of-thought generation. Monitor probabilities p(hack) are shown as a function of generation progress, aggregated into fixed bins of 64 tokens to allow comparison across CoT lengths (64, 128, 256, and 512 tokens). Rows correspond to model families and colored curves indicate different proportions of reward-hacking data in the fine-tuning corpus. Distinct temporal dynamics emerge across model families, early decrease for Llama , late-stage amplification for Qwen , and mixture-dependent behavior for Falcon . This model specific behavior is observable consistent across different CoT regimes and output lengths.
Figure 4: Relative change in hack-associated activation under chain-of-thought prompting. Bars show the percentage change in mean monitor probability relative to direct answering, for mixed-policy adapters and different chain-of-thought lengths. Amplification is strongest for partially misaligned adapters (5%, 10%), varies across model families, and diminishes as misalignment saturates (50%, 90%)
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Layer-wise logistic-regression accuracy for Falcon using the SAE → standardization → PCA → LR pipeline. Curves compare different PCA output dimensions ( d∈{2,64,256} ), showing improved separability with larger d across most layers.
Figure 6: Layer-wise logistic-regression accuracy for Llama using the SAE → standardization → PCA → LR pipeline. Curves compare different PCA output dimensions ( d∈{2,64,256} ), showing improved separability with larger d across most layers.
Figure 7: Layer-wise logistic-regression accuracy for Qwen using the SAE → standardization → PCA → LR pipeline. Curves compare different PCA output dimensions ( d∈{2,64,256} ), showing improved separability with larger d across most layers.
Figure 8: Adapter-conditioned late-stage slopes of reward-hack activation. Each panel corresponds to a fine-tuning mixture, plotting the late-stage slope βlate as a function of CoT length for different model families. Positive slopes indicate increasing reward-hack activation near completion, while negative slopes indicate late-stage attenuation. Across mixtures, model families exhibit consistent slope sign, with magnitude modulated by the proportion of reward-hacking data.
Figure 9: Dose–response of late-stage change Δlate with respect to the proportion of reward-hacking data in fine-tuning. Each panel corresponds to a CoT length, plotting Δlate as a function of mixture percentage for different model families. Positive values indicate late-stage concentration of reward-hack activation, while negative values indicate early-stage concentration. Qualitative trends are consistent across CoT regimes, with mixture proportion primarily affecting magnitude rather than direction.
Figure 10: F1 Scores of the proposed monitoring system compared to the ChatGPT-4o labeling
Reward hacking studies how AI systems exploit imperfect objectives and how to detect, measure, and prevent behavior that scores well but misses intent.