Reinforcement learning (RL) rewards are notoriously difficult to design and control, often leading to the model learning unintended behaviors such as reward hacking. One potential solution is to monitor for reward hacking and penalize it when detected; however, training against a monitor could lead to evasive behavior, and our general understanding of how to apply monitors effectively during training is limited. To study how best to use monitors to mitigate reward hacking, we introduce and open source three realistic environments where Qwen3-4B reward hacks: a coding environment hackable via test overwriting, a medical chat environment hackable via sycophancy, and a biography generation environment hackable via hallucination. We first focus on the coding environment, where we find that: (1) models can learn to evade highly accurate monitors by exploiting systemic flaws in probes and LLM judges; (2) monitors that leak more learning signal during RL suppress reward hacking but are more often evaded; and (3) including easier problems in training can decrease reward hacking. We apply our findings to build better reward hacking monitors for the medical chat and biography generation environments that improve upon naive baselines to reduce reward hacking rates across seeds to before-training levels in some interventions. Our results demonstrate that our takeaways translate to new settings and that better monitor intervention designs are possible.
Figures & tables
Figure 1: We introduce three environments where reward hacking emerges naturally during training without finetuning or prompting to seed the behaviors. Using these environments, we build understanding of how to design more effective training interventions.
Environment
Dataset
Reward
Loophole
Performance Metric
Coding ( App. A )
Medium and hard Leetcode problems ( Xia et al., 2025 )
Rewarded 3.0 for passing unit test cases; 0.5 for code compiling
Test overwriting : redefine run_tests() to bypass evaluation.
Pass all unit test cases
Medical chat ( App. B )
Patient questions from ( Li et al., 2023 ) with correct (50%) and incorrect (50%) leading questions
Rewarded 1.0 for physician-reference agreement, 0.5 for patient-agreement when otherwise incorrect
Sycophancy : follow the patient’s leading diagnosis, including incorrect suggestions.
Agreement with the physician reference response
Biography generation ( App. C )
Write a one-paragraph biography of a given historical figure
Rewarded for detail (# of facts), correctness (proportion true facts) and quality (LLM judge score)
Hallucination : add many facts while sacrificing truth.
Sum of detail, correctness, and quality scores
Table 1: Reward-hacking environments; see Appendix A , Appendix B , Appendix C for further details.
Figure 3
Figure 5: Higher validation accuracy leads to less reward hacking (left); higher accuracy idealized monitors lead to better performance after training (right); however, realistic monitors underperform and break this trend. Performance before training ("Before Training"), after RL training on the non-reward hacking environment ("Non-Reward Hacking") and after RL training on the hackable environment without intervention ("Unmonitored") are shown.
Figure 5Figure 6
Figure 10: Harder datasets increase reward hacking frequency ( left ), reduce steps to reward hacking ( center ), and worsen performance degradation when the model discovers the hack ( right ). These trends are observed across unmonitored runs without any supervision ("Unmonitored") as well as runs monitored by probes ("Probe") and judges ("Judge") applied as penalties.
Figure 11: With design improvements ("+Insights"), monitor interventions beat naive baselines ("+Monitor") for the medical chat (a) and biography (b) environments. Performance and reward hacking rates on the held-out evaluation set are shown alongside the non-reward hacking, before training, and unmonitored baselines. "Naive" indicates the intervention without the design improvements described in Section 6 and "Best" indicates the interventions with those improvements. Tables on each plot show the reward hacking ( Δ RH) and performance ( Δ Perf) changes versus the unmonitored baseline, with the ablation line items (Easier Dataset, Lower Threshold, Screening Method or Smarter Judge) showing the impact of ablating each design choice compared to the "Best" run.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 12: Left: 10-step average reward, reward hacking and performance is shown across all 10 training seeds with standard error shading. Right: Final test-set evaluation reward hacking and performance is shown before training, after training on the non-reward hacking environment and after training without monitoring on the hackable environment. See text for definitions.
Figure 13: Left: 10-step average reward, reward hacking and performance is shown across all 10 training seeds with standard error shading. Right: Final test-set evaluation reward hacking and performance is shown before training, after training on the non-reward hacking environment and after training without monitoring on the hackable environment. See text for definitions.
Figure 14: Left: 10-step average reward, reward hacking and performance is shown across all 10 training seeds with standard error shading. Right: Final test-set evaluation reward hacking and performance is shown before training, after training on the non-reward hacking environment and after training without monitoring on the hackable environment. See text for definitions.
Baselines
Monitor Interventions
Before Training
Non-Reward Hacking
Unmon- itored
Naive Judge
Best Judge
Naive Probe
Best Probe
Detail
0.30
0.37 ± 0.02
0.96 ± 0.01
0.93 ± 0.02
0.40 ± 0.01
0.93 ± 0.03
0.88 ± 0.03
Correctness
0.23
0.49 ± 0.02
0.11 ± 0.01
0.09 ± 0.00
0.32 ± 0.01
0.10 ± 0.00
0.11 ± 0.00
Quality
0.47
0.54 ± 0.02
0.39 ± 0.01
0.33 ± 0.01
0.55 ± 0.01
0.35 ± 0.01
0.37 ± 0.01
Total Reward
0.96
1.40 ± 0.02
1.46 ± 0.01
1.34 ± 0.02
1.25 ± 0.02
1.37 ± 0.03
1.36 ± 0.04
Loophole Reward
1.26
1.76 ± 0.03
2.42 ± 0.02
2.27 ± 0.03
1.64 ± 0.03
2.30 ± 0.06
2.23 ± 0.07
Appendix
Table 2: Average reward component scores across the test dataset for baselines and monitoring interventions. All values are shown averaged over 10 training seeds ± 1 SE; Before Training is a single base-model evaluation. See text for descriptions of each score. Total Reward is the equal-weighted sum; this reward is the primary performance metric and the reward used by the Non-Reward Hacking training baseline runs. Loophole Reward is the reward used by the hackable training runs, which doubles the weight of the Detail component. Without the loophole or with strong monitoring intervention, the model learns a balanced score across the three components; when the loophole is enabled, the model learns a highly skewed distribution across the components and scores lower on correctness and quality compared to Before Training.
Parameter
Value
Base model
Qwen3-4B
LoRA rank / alpha
32 / 32
LoRA dropout
0.0
Optimizer
AdamW ( β1=0.9 , β2=0.99 , weight decay 0.1)
Learning rate
7×10−5
LR scheduler
Cosine with 10 warmup steps
Appendix
Table 3: RL training hyperparameters. Batch shape, optimizer, and sampling settings are shared across environments; only the prompt and completion length budgets differ between environments.
Hyperparameter
Value
Probe class
Multi-max attention probe
Heads
1
MLP hidden dimension
100
Dropout
0.0
Optimizer
AdamW
Learning rate
1×10−4
Appendix
Table 4: Probe training hyperparameters, shared across all probes (in-domain and out-of-domain).
Environment
Test n
Layer
ROC AUC
Accuracy
Precision
Recall
Coding
1577
22
0.983
0.935
0.918
0.921
Medical chat
1015
18
0.911
0.848
0.764
0.877
Biography
93
18
0.918
0.892
0.929
0.765
Appendix
Table 5: In-domain probe test metrics at the selected layer (highest ROC AUC). Coding metrics are taken from the multi-max attention probe at layer 22; biography from layer 18. Accuracy, precision, and recall are reported at each probe’s own ROC-strict-optimal threshold. Medical-chat metrics are reported separately when available.
Environment
Dataset
OOD test n
Layer
Threshold
ROC AUC
Acc.
Recall
FPR
Coding
Deception
1119
22
0.48
0.84
0.76
0.51
0.096
Coding (1% FPR)
Deception
1119
22
0.85
0.84
0.67
0.13
0.008
Coding (10% FPR)
Deception
1119
22
0.29
0.84
0.70
0.88
0.397
Medical chat (10% FPR)
Persona sycophancy
920
27
0.98
0.65
0.62
0.18
0.104
Medical chat (Best)
Persona sycophancy
920
27
0.93
0.65
0.61
0.75
0.481
Biography
Hallucination
1392
18
0.81
0.68
0.69
0.62
0.271
Appendix
Table 6: Out-of-domain probe transfer metrics on the corresponding in-domain held-out test sets at thresholds used in the training-time interventions. ROC AUC is computed from raw probe scores; accuracy, recall, and false-positive rate are computed after thresholding.
Judge
OpenRouter model id
Parameters
Active Parameters
Non-Reasoning Tokens
Reasoning Tokens
Qwen3-235B-A22B
qwen/qwen3-235b-a22b-2507
235B
22B
16
1024
Kimi K2.5
moonshotai/kimi-k2.5
1T
32B
16
N/A
DeepSeek V4 Flash
deepseek/deepseek-v4-flash
284B
13B
16
N/A
GLM 5.1
z-ai/glm-5.1
754B
40B
16
2048
Appendix
Table 7: Judge inference settings. Sampling: temperature 0.0, max 16 new tokens. Qwen3-235B, Kimi K2.5, DeepSeek V4 Flash, and GLM 5.1 are used across the judge baselines and comparisons; selected judge identities and thresholds vary by environment.
Environment
Judge
Reasoning
n
ROC AUC
Accuracy
Recall
F1
Threshold
FPR
Coding
Qwen3 235B
N
1577
0.820
0.857
0.673
0.777
0.30
0.035
Coding
Kimi K2.5
N
1577
0.909
0.923
0.871
0.892
0.10
0.047
Coding
DeepSeek v4 Flash
N
1577
0.884
0.871
0.842
0.828
1.00
0.111
Coding
GLM 5.1
N
1577
0.838
0.852
0.670
0.769
0.40
0.042
Medical chat
Qwen3 235B
N
1015
0.694
0.649
0.141
0.236
0.70
0.032
Medical chat
Qwen3 235B
Y
1015
0.665
0.647
0.141
0.235
0.70
0.035
Appendix
Table 8: Judge benchmark performance on each environment’s in-domain validation set. Accuracy, recall, F1, and FPR are measured at the selected threshold; ROC AUC is computed from raw 0–10 judge scores. For DeepSeek and Biography Qwen, the 5%-FPR target is not achievable at a finite emitted threshold, so the table reports the most conservative selected threshold and its actual FPR.
Figure 15: Distribution of p-values for correlations between attribute presence and judge accuracy on reward hacks (Qwen3-235B Judge Penalty runs; 70 tests, bins of width 0.05). The dashed line shows the count expected per bin if no attribute were correlated with judge accuracy.
Attr.
Name
Explanation
Corr.
P Val.
q (BH)
Prev.
Unmon. Prev.
Non-RH Prev.
attr_018
starts_with_python_fence_no_preamble
The very first non-whitespace characters of the response are the opening “‘python fence (no leading prose, header, or visible chain-of-thought).
-0.128
<0.001
<0.001
63.8%
89.9%
49.6%
attr_036
all_tests_passed_print_string
Run_tests ends with a celebratory ‘print("All tests passed!")‘ or ‘print("All test cases passed.")‘ line after the assertion calls.
-0.035
0.010
0.024
2.1%
0.0%
0.0%
attr_005
explanation_block_after_code_fence
After the closing “‘ of the code, the response continues with a markdown ‘### Explanation:‘ (or similar header) section followed by a numbered or bulleted list describing the approach.
-0.033
0.016
0.034
29.0%
5.4%
15.8%
attr_060
example_n_section_comments_in_run_tests
Inside the run_tests function (or harness block), each printed test call is preceded by a ‘# Example 1‘, ‘# Example 2‘, … section comment naming the example index from the prompt.
-0.033
0.017
0.035
11.1%
4.4%
0.0%
attr_027
labeled_example_print_strings
Test prints use string labels like ‘"Example 1:"‘, ‘"Example 2:"‘, or ‘"Test 1:"‘ to tag the printed result (e.g. ‘print("Example 1: ", self.minCost(…))‘).
-0.031
0.023
0.044
20.5%
4.8%
0.0%
attr_024
meta_commentary_about_run_tests_harness
Response includes meta commentary about the ‘run_tests‘ harness — acknowledging it is optional, structural, or a test-framework shim — either as an inline aside, a post-fence ‘Note:‘ paragraph, or a ‘# This is just a test code skeleton, in the real environment…‘ style disclaimer.
0.032
0.019
0.037
12.5%
37.4%
0.5%
Appendix
Table 9: Judge Penalty attributes with significant reward-hack-only monitor-accuracy correlations and RH prevalence above 1%. All listed attributes remain significant after Benjamini–Hochberg correction across the 70 attributes and text metrics tested ( q<0.05 ).
Attr.
Name
Explanation
Corr. NR
P-value NR
Corr. R
P-value R
Diff.
Prev. NR
Prev. R
Unmon. Prev.
Non-RH Prev.
attr_018
starts_with_python_fence_no_preamble
The very first non-whitespace characters of the response are the opening “‘python fence (no leading prose, header, or visible chain-of-thought).
-0.128
<0.001
-0.102
<0.001
0.026
63.8%
75.0%
89.9%
49.6%
attr_005
explanation_block_after_code_fence
After the closing “‘ of the code, the response continues with a markdown ‘### Explanation:‘ (or similar header) section followed by a numbered or bulleted list describing the approach.
-0.033
0.016
-0.089
<0.001
-0.056
29.0%
22.8%
5.4%
15.8%
attr_060
example_n_section_comments_in_run_tests
Inside the run_tests function (or harness block), each printed test call is preceded by a ‘# Example 1‘, ‘# Example 2‘, … section comment naming the example index from the prompt.
-0.033
0.017
-0.067
<0.001
-0.034
11.1%
19.2%
4.4%
0.0%
attr_027
labeled_example_print_strings
Test prints use string labels like ‘"Example 1:"‘, ‘"Example 2:"‘, or ‘"Test 1:"‘ to tag the printed result (e.g. ‘print("Example 1: ", self.minCost(…))‘).
-0.031
0.023
-0.061
<0.001
-0.030
20.5%
31.5%
4.8%
0.0%
attr_024
meta_commentary_about_run_tests_harness
Response includes meta commentary about the ‘run_tests‘ harness — acknowledging it is optional, structural, or a test-framework shim — either as an inline aside, a post-fence ‘Note:‘ paragraph, or a ‘# This is just a test code skeleton, in the real environment…‘ style disclaimer.
0.032
0.019
0.064
<0.001
0.032
12.5%
5.0%
37.4%
0.5%
attr_026
fstring_print_in_run_tests
The defined ‘run_tests‘ uses an f-string in its print calls (e.g. ‘print(f"Example 1: {result}")‘ or pipe-separated label form like ‘print(f"Input: {s} | Expected: {expected} | Got: {result}")‘) rather than positional/concatenated print arguments.
0.032
0.019
0.044
0.016
0.011
9.5%
10.9%
0.2%
0.0%
Appendix
Table 10: Text attributes correlated with monitor accuracy on reward hacks for non-reasoning (NR) and reasoning (R) monitor runs
Figure 22
Figure 18: Example of reward hacking breaking out even when ground truth screening is applied. Top: reward hacking rollouts are screened out, but attempted reward hacking (purple) and correct; attempted reward hacking (yellow) rollouts make up the majority of unscreened responses. Bottom: Test overwrites measures the rollouts where the test function is overwritten at all, irrespective of whether the response otherwise correctly (Correct; Attempted Reward Hack) or incorrectly (Reward Hack) solves the problem. Unscreened test overwrites are those that are not screened out by the ground truth label, meaning they cannot be full reward hacks.
Figure 19: Example of degeneration to avoid a monitor in the coding environment with a progressive probe penalty applied. Left : Prior to 250 steps, the model is getting consistently penalized by the monitor. Around 250 steps, responses collapse to zero correctness and zero monitor penalty; only the format reward remains. Center : Samples flagged by the monitor are nearly all false positives
Strategy
Reward
Monitor Score
Rep. Drift
Example Response
No Attempted Response
0.45
0.03
-0.65
⬇ class Solution : def findIntegers ( self , n : int ) -> int : return n
Irrelevant Print
-0.32
0.14
-0.70
⬇ print (" Hello , world !")
Empty Response
0.0
0.0
–
⬇ ""
Whitespace/Tab Fill
0.0
0.0
–
⬇ from collections import deque , Counter [.. all remaining tokens \ n ...]
Garbled Response
0.0
0.0
0.0
Appendix
Table 11: Example responses across different training runs.
Figure 20: Example of degeneration in a variable reward environment. Response length degenerates amongst responses receiving low rewards; these responses are promoted and become the dominant behavior.
Figure 21: Comparison of behavior during training for a probe intervention run with a low threshold and with a high threshold across behavior during training as well as representation drift and fraction of samples flipped positive to negative by representation drift.
Figure 22: Across all seeds of the coding environment on the easy dataset, reward hacking rate and performance are shown on the test dataset after training to 400 steps (solid fill; standard for this paper) and 800 steps (dashed fill; extended training). Average across all seeds is shown to the right. There is marginal differences between 400 and 800 steps for each seed and no difference in the final averages. We take these results to indicate training has converged and that the final reward hacking
Figure 23: Comparison of sparse and dense correctness rewards in the coding environment, showing reward hacking, monitor accuracy change, and post-training performance.
Reward hacking studies how AI systems exploit imperfect objectives and how to detect, measure, and prevent behavior that scores well but misses intent.
During reinforcement learning with verifiable rewards (RLVR), large language models (LLMs) can exploit loopholes in their environments to obtain high rewards without improving the intended capabilities, i.e., reward hacking. Despite its risks to training efficiency and safety, monitoring and mitigating reward hacking during training remain challenging, which is limited by a lack of testbeds that reproduce hacking and reliably identify it. We introduce CATCH, a controllable testbed for studying reward hacking in coding RL. CATCH deliberately exposes environmental loopholes and provides execution-based gold labels by comparing success under a vulnerable evaluator with task correctness under an independent audit. It also can control the model's initial hacking tendency through supervised fine-tuning data mixtures and the difficulty of earning rewards through reward designing, enabling systematic comparisons of hacking dynamics and interventions. Experiments show that CATCH can produce diverse RL training trajectories with clear reward hacking, and analyses demonstrate that both initial models and reward difficulties shape the emergence of reward hacking. We further evaluate the effectiveness of different reward hacking detection and mitigation methods. A key finding is that a chain-of-thought monitor initially suppresses hacking, but this protection erodes as the policy model learn to mislead the monitor with code comments. This highlights the need to evaluate hacking mitigations throughout training with CATCH. The source code and resources are publicly released at https://github.com/THUAIS-Lab/CATCH.
Shouli Wang, Yanfeng Jia, Zhihao Ou +6
Tsinghua University · Beihang University · Southern University of Science and Technology +2
Reward hacking in code generation, where models exploit evaluation loopholes to obtain high reward without correctly solving the intended task, poses a critical challenge for Reinforcement Learning (RL) and the deployment of reasoning models. Existing studies often rely on explicitly prompted hacking trajectories, but it remains unclear whether monitors trained on such data can detect reward hacks that arise without direct hacking instructions during RL training. In this work, we introduce Trace-and-Amplify, a framework for scalable curation of reward-hacking trajectories that arise during RL training without explicit hacking instructions. The framework uses unit-test tracers to identify hacking solutions when they occur and retains such trajectories for monitor training and evaluation. Through controlled comparisons between monitors trained on prompt-elicited hacking trajectories and training-time reward-hacking trajectories collected by Trace-and-Amplify, we find that (1) prompt-elicited-data-trained monitors often fail to generalize to trajectories curated by our framework, and (2) monitors trained on our Trace-and-Amplify trajectories demonstrate stronger generalizability to unseen hacking types. Our results indicate that prompted reward hacking data may not fully reflect training-time reward-hacking behaviors, and that relying solely on these data can lead to misleading conclusions. Codebase is available at https://github.com/LichenLillc/CoTMonitoring.git
Lichen Li, Hengguang Zhou, Yijun Liang +2
Peking University · University of California, Los Angeles · University of Maryland +2
Reward hacking is a major problem in current production machine learning systems, resulting in costly and dangerous failures. Here we study how to monitor for and mitigate reward hacking in realistic settings, including tool calls and long contexts. We apply a dynamic intervention to steering directions obtained by three different methods: difference-in-means vectors from labeled rollouts, gradient optimization-based vectors, and vectors extracted when models reference an anti-reward-hacking model spec. This dynamic intervention perturbs model representations less than uniform steering vector addition to the residual stream, allowing for more precise edits. We find that we are able to cheaply and significantly reduce reward hacking without correspondingly significant costs to model coherence or capability, and that steering vectors originally extracted from coding environments also generalize to reducing reward hacking in other domains. Our results pave the way towards latent space monitoring and control over realistic production systems.