cs.LGNov 6, 2026
SaveDesigning Effective Monitor-Based Interventions for Mitigating Reward Hacking During RL
Organizations: MATS · Google DeepMind
Abstract
Reinforcement learning (RL) rewards are notoriously difficult to design and control, often leading to the model learning unintended behaviors such as reward hacking. One potential solution is to monitor for reward hacking and penalize it when detected; however, training against a monitor could lead to evasive behavior, and our general understanding of how to apply monitors effectively during training is limited. To study how best to use monitors to mitigate reward hacking, we introduce and open source three realistic environments where Qwen3-4B reward hacks: a coding environment hackable via test overwriting, a medical chat environment hackable via sycophancy, and a biography generation environment hackable via hallucination. We first focus on the coding environment, where we find that: (1) models can learn to evade highly accurate monitors by exploiting systemic flaws in probes and LLM judges; (2) monitors that leak more learning signal during RL suppress reward hacking but are more often evaded; and (3) including easier problems in training can decrease reward hacking. We apply our findings to build better reward hacking monitors for the medical chat and biography generation environments that improve upon naive baselines to reduce reward hacking rates across seeds to before-training levels in some interventions. Our results demonstrate that our takeaways translate to new settings and that better monitor intervention designs are possible.
Figures & tables
Figure 1: We introduce three environments where reward hacking emerges naturally during training without finetuning or prompting to seed the behaviors. Using these environments, we build understanding of how to design more effective training interventions.
| Environment | Dataset | Reward | Loophole | Performance Metric |
| Coding ( App. A ) | Medium and hard Leetcode problems ( Xia et al., 2025 ) | Rewarded 3.0 for passing unit test cases; 0.5 for code compiling | Test overwriting : redefine run_tests() to bypass evaluation. | Pass all unit test cases |
| Medical chat ( App. B ) | Patient questions from ( Li et al., 2023 ) with correct (50%) and incorrect (50%) leading questions | Rewarded 1.0 for physician-reference agreement, 0.5 for patient-agreement when otherwise incorrect | Sycophancy : follow the patient’s leading diagnosis, including incorrect suggestions. | Agreement with the physician reference response |
| Biography generation ( App. C ) | Write a one-paragraph biography of a given historical figure | Rewarded for detail (# of facts), correctness (proportion true facts) and quality (LLM judge score) | Hallucination : add many facts while sacrificing truth. | Sum of detail, correctness, and quality scores |
Table 1: Reward-hacking environments; see Appendix A , Appendix B , Appendix C for further details.
Figure 5: Higher validation accuracy leads to less reward hacking (left); higher accuracy idealized monitors lead to better performance after training (right); however, realistic monitors underperform and break this trend. Performance before training ("Before Training"), after RL training on the non-reward hacking environment ("Non-Reward Hacking") and after RL training on the hackable environment without intervention ("Unmonitored") are shown.
Figure 10: Harder datasets increase reward hacking frequency ( left ), reduce steps to reward hacking ( center ), and worsen performance degradation when the model discovers the hack ( right ). These trends are observed across unmonitored runs without any supervision ("Unmonitored") as well as runs monitored by probes ("Probe") and judges ("Judge") applied as penalties.
Figure 11: With design improvements ("+Insights"), monitor interventions beat naive baselines ("+Monitor") for the medical chat (a) and biography (b) environments. Performance and reward hacking rates on the held-out evaluation set are shown alongside the non-reward hacking, before training, and unmonitored baselines. "Naive" indicates the intervention without the design improvements described in Section 6 and "Best" indicates the interventions with those improvements. Tables on each plot show the reward hacking ( RH) and performance ( Perf) changes versus the unmonitored baseline, with the ablation line items (Easier Dataset, Lower Threshold, Screening Method or Smarter Judge) showing the impact of ablating each design choice compared to the "Best" run.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 12: Left: 10-step average reward, reward hacking and performance is shown across all 10 training seeds with standard error shading. Right: Final test-set evaluation reward hacking and performance is shown before training, after training on the non-reward hacking environment and after training without monitoring on the hackable environment. See text for definitions.
Figure 13: Left: 10-step average reward, reward hacking and performance is shown across all 10 training seeds with standard error shading. Right: Final test-set evaluation reward hacking and performance is shown before training, after training on the non-reward hacking environment and after training without monitoring on the hackable environment. See text for definitions.
Figure 14: Left: 10-step average reward, reward hacking and performance is shown across all 10 training seeds with standard error shading. Right: Final test-set evaluation reward hacking and performance is shown before training, after training on the non-reward hacking environment and after training without monitoring on the hackable environment. See text for definitions.
| Baselines | Monitor Interventions | ||||||
| Before Training | Non-Reward Hacking | Unmon- itored | Naive Judge | Best Judge | Naive Probe | Best Probe | |
| Detail | 0.30 | 0.37 0.02 | 0.96 0.01 | 0.93 0.02 | 0.40 0.01 | 0.93 0.03 | 0.88 0.03 |
| Correctness | 0.23 | 0.49 0.02 | 0.11 0.01 | 0.09 0.00 | 0.32 0.01 | 0.10 0.00 | 0.11 0.00 |
| Quality | 0.47 | 0.54 0.02 | 0.39 0.01 | 0.33 0.01 | 0.55 0.01 | 0.35 0.01 | 0.37 0.01 |
| Total Reward | 0.96 | 1.40 0.02 | 1.46 0.01 | 1.34 0.02 | 1.25 0.02 | 1.37 0.03 | 1.36 0.04 |
| Loophole Reward | 1.26 | 1.76 0.03 | 2.42 0.02 | 2.27 0.03 | 1.64 0.03 | 2.30 0.06 | 2.23 0.07 |
Table 2: Average reward component scores across the test dataset for baselines and monitoring interventions. All values are shown averaged over 10 training seeds 1 SE; Before Training is a single base-model evaluation. See text for descriptions of each score. Total Reward is the equal-weighted sum; this reward is the primary performance metric and the reward used by the Non-Reward Hacking training baseline runs. Loophole Reward is the reward used by the hackable training runs, which doubles the weight of the Detail component. Without the loophole or with strong monitoring intervention, the model learns a balanced score across the three components; when the loophole is enabled, the model learns a highly skewed distribution across the components and scores lower on correctness and quality compared to Before Training.
| Parameter | Value |
| Base model | Qwen3-4B |
| LoRA rank / alpha | 32 / 32 |
| LoRA dropout | 0.0 |
| Optimizer | AdamW ( , , weight decay 0.1) |
| Learning rate | |
| LR scheduler | Cosine with 10 warmup steps |
Table 3: RL training hyperparameters. Batch shape, optimizer, and sampling settings are shared across environments; only the prompt and completion length budgets differ between environments.
| Hyperparameter | Value |
| Probe class | Multi-max attention probe |
| Heads | 1 |
| MLP hidden dimension | 100 |
| Dropout | 0.0 |
| Optimizer | AdamW |
| Learning rate |
Table 4: Probe training hyperparameters, shared across all probes (in-domain and out-of-domain).
| Environment | Test | Layer | ROC AUC | Accuracy | Precision | Recall |
| Coding | 1577 | 22 | 0.983 | 0.935 | 0.918 | 0.921 |
| Medical chat | 1015 | 18 | 0.911 | 0.848 | 0.764 | 0.877 |
| Biography | 93 | 18 | 0.918 | 0.892 | 0.929 | 0.765 |
Table 5: In-domain probe test metrics at the selected layer (highest ROC AUC). Coding metrics are taken from the multi-max attention probe at layer 22; biography from layer 18. Accuracy, precision, and recall are reported at each probe’s own ROC-strict-optimal threshold. Medical-chat metrics are reported separately when available.
| Environment | Dataset | OOD test | Layer | Threshold | ROC AUC | Acc. | Recall | FPR |
| Coding | Deception | 1119 | 22 | 0.48 | 0.84 | 0.76 | 0.51 | 0.096 |
| Coding (1% FPR) | Deception | 1119 | 22 | 0.85 | 0.84 | 0.67 | 0.13 | 0.008 |
| Coding (10% FPR) | Deception | 1119 | 22 | 0.29 | 0.84 | 0.70 | 0.88 | 0.397 |
| Medical chat (10% FPR) | Persona sycophancy | 920 | 27 | 0.98 | 0.65 | 0.62 | 0.18 | 0.104 |
| Medical chat (Best) | Persona sycophancy | 920 | 27 | 0.93 | 0.65 | 0.61 | 0.75 | 0.481 |
| Biography | Hallucination | 1392 | 18 | 0.81 | 0.68 | 0.69 | 0.62 | 0.271 |
Table 6: Out-of-domain probe transfer metrics on the corresponding in-domain held-out test sets at thresholds used in the training-time interventions. ROC AUC is computed from raw probe scores; accuracy, recall, and false-positive rate are computed after thresholding.
| Judge | OpenRouter model id | Parameters | Active Parameters | Non-Reasoning Tokens | Reasoning Tokens |
| Qwen3-235B-A22B | qwen/qwen3-235b-a22b-2507 | 235B | 22B | 16 | 1024 |
| Kimi K2.5 | moonshotai/kimi-k2.5 | 1T | 32B | 16 | N/A |
| DeepSeek V4 Flash | deepseek/deepseek-v4-flash | 284B | 13B | 16 | N/A |
| GLM 5.1 | z-ai/glm-5.1 | 754B | 40B | 16 | 2048 |
Table 7: Judge inference settings. Sampling: temperature 0.0, max 16 new tokens. Qwen3-235B, Kimi K2.5, DeepSeek V4 Flash, and GLM 5.1 are used across the judge baselines and comparisons; selected judge identities and thresholds vary by environment.
| Environment | Judge | Reasoning | ROC AUC | Accuracy | Recall | F1 | Threshold | FPR | |
| Coding | Qwen3 235B | N | 1577 | 0.820 | 0.857 | 0.673 | 0.777 | 0.30 | 0.035 |
| Coding | Kimi K2.5 | N | 1577 | 0.909 | 0.923 | 0.871 | 0.892 | 0.10 | 0.047 |
| Coding | DeepSeek v4 Flash | N | 1577 | 0.884 | 0.871 | 0.842 | 0.828 | 1.00 | 0.111 |
| Coding | GLM 5.1 | N | 1577 | 0.838 | 0.852 | 0.670 | 0.769 | 0.40 | 0.042 |
| Medical chat | Qwen3 235B | N | 1015 | 0.694 | 0.649 | 0.141 | 0.236 | 0.70 | 0.032 |
| Medical chat | Qwen3 235B | Y | 1015 | 0.665 | 0.647 | 0.141 | 0.235 | 0.70 | 0.035 |
Table 8: Judge benchmark performance on each environment’s in-domain validation set. Accuracy, recall, F1, and FPR are measured at the selected threshold; ROC AUC is computed from raw 0–10 judge scores. For DeepSeek and Biography Qwen, the 5%-FPR target is not achievable at a finite emitted threshold, so the table reports the most conservative selected threshold and its actual FPR.
Figure 15: Distribution of p-values for correlations between attribute presence and judge accuracy on reward hacks (Qwen3-235B Judge Penalty runs; 70 tests, bins of width 0.05). The dashed line shows the count expected per bin if no attribute were correlated with judge accuracy.
| Attr. | Name | Explanation | Corr. | P Val. | q (BH) | Prev. | Unmon. Prev. | Non-RH Prev. |
| attr_018 | starts_with_python_fence_no_preamble | The very first non-whitespace characters of the response are the opening “‘python fence (no leading prose, header, or visible chain-of-thought). | -0.128 | 63.8% | 89.9% | 49.6% | ||
| attr_036 | all_tests_passed_print_string | Run_tests ends with a celebratory ‘print("All tests passed!")‘ or ‘print("All test cases passed.")‘ line after the assertion calls. | -0.035 | 0.010 | 0.024 | 2.1% | 0.0% | 0.0% |
| attr_005 | explanation_block_after_code_fence | After the closing “‘ of the code, the response continues with a markdown ‘### Explanation:‘ (or similar header) section followed by a numbered or bulleted list describing the approach. | -0.033 | 0.016 | 0.034 | 29.0% | 5.4% | 15.8% |
| attr_060 | example_n_section_comments_in_run_tests | Inside the run_tests function (or harness block), each printed test call is preceded by a ‘# Example 1‘, ‘# Example 2‘, … section comment naming the example index from the prompt. | -0.033 | 0.017 | 0.035 | 11.1% | 4.4% | 0.0% |
| attr_027 | labeled_example_print_strings | Test prints use string labels like ‘"Example 1:"‘, ‘"Example 2:"‘, or ‘"Test 1:"‘ to tag the printed result (e.g. ‘print("Example 1: ", self.minCost(…))‘). | -0.031 | 0.023 | 0.044 | 20.5% | 4.8% | 0.0% |
| attr_024 | meta_commentary_about_run_tests_harness | Response includes meta commentary about the ‘run_tests‘ harness — acknowledging it is optional, structural, or a test-framework shim — either as an inline aside, a post-fence ‘Note:‘ paragraph, or a ‘# This is just a test code skeleton, in the real environment…‘ style disclaimer. | 0.032 | 0.019 | 0.037 | 12.5% | 37.4% | 0.5% |
Table 9: Judge Penalty attributes with significant reward-hack-only monitor-accuracy correlations and RH prevalence above 1%. All listed attributes remain significant after Benjamini–Hochberg correction across the 70 attributes and text metrics tested ( ).
| Attr. | Name | Explanation | Corr. NR | P-value NR | Corr. R | P-value R | Diff. | Prev. NR | Prev. R | Unmon. Prev. | Non-RH Prev. |
| attr_018 | starts_with_python_fence_no_preamble | The very first non-whitespace characters of the response are the opening “‘python fence (no leading prose, header, or visible chain-of-thought). | -0.128 | -0.102 | 0.026 | 63.8% | 75.0% | 89.9% | 49.6% | ||
| attr_005 | explanation_block_after_code_fence | After the closing “‘ of the code, the response continues with a markdown ‘### Explanation:‘ (or similar header) section followed by a numbered or bulleted list describing the approach. | -0.033 | 0.016 | -0.089 | -0.056 | 29.0% | 22.8% | 5.4% | 15.8% | |
| attr_060 | example_n_section_comments_in_run_tests | Inside the run_tests function (or harness block), each printed test call is preceded by a ‘# Example 1‘, ‘# Example 2‘, … section comment naming the example index from the prompt. | -0.033 | 0.017 | -0.067 | -0.034 | 11.1% | 19.2% | 4.4% | 0.0% | |
| attr_027 | labeled_example_print_strings | Test prints use string labels like ‘"Example 1:"‘, ‘"Example 2:"‘, or ‘"Test 1:"‘ to tag the printed result (e.g. ‘print("Example 1: ", self.minCost(…))‘). | -0.031 | 0.023 | -0.061 | -0.030 | 20.5% | 31.5% | 4.8% | 0.0% | |
| attr_024 | meta_commentary_about_run_tests_harness | Response includes meta commentary about the ‘run_tests‘ harness — acknowledging it is optional, structural, or a test-framework shim — either as an inline aside, a post-fence ‘Note:‘ paragraph, or a ‘# This is just a test code skeleton, in the real environment…‘ style disclaimer. | 0.032 | 0.019 | 0.064 | 0.032 | 12.5% | 5.0% | 37.4% | 0.5% | |
| attr_026 | fstring_print_in_run_tests | The defined ‘run_tests‘ uses an f-string in its print calls (e.g. ‘print(f"Example 1: {result}")‘ or pipe-separated label form like ‘print(f"Input: {s} | Expected: {expected} | Got: {result}")‘) rather than positional/concatenated print arguments. | 0.032 | 0.019 | 0.044 | 0.016 | 0.011 | 9.5% | 10.9% | 0.2% | 0.0% |
Table 10: Text attributes correlated with monitor accuracy on reward hacks for non-reasoning (NR) and reasoning (R) monitor runs
Figure 18: Example of reward hacking breaking out even when ground truth screening is applied. Top: reward hacking rollouts are screened out, but attempted reward hacking (purple) and correct; attempted reward hacking (yellow) rollouts make up the majority of unscreened responses. Bottom: Test overwrites measures the rollouts where the test function is overwritten at all, irrespective of whether the response otherwise correctly (Correct; Attempted Reward Hack) or incorrectly (Reward Hack) solves the problem. Unscreened test overwrites are those that are not screened out by the ground truth label, meaning they cannot be full reward hacks.
Figure 19: Example of degeneration to avoid a monitor in the coding environment with a progressive probe penalty applied. Left : Prior to 250 steps, the model is getting consistently penalized by the monitor. Around 250 steps, responses collapse to zero correctness and zero monitor penalty; only the format reward remains. Center : Samples flagged by the monitor are nearly all false positives
| Strategy | Reward | Monitor Score | Rep. Drift | Example Response |
| No Attempted Response | 0.45 | 0.03 | -0.65 | ⬇ class Solution : def findIntegers ( self , n : int ) -> int : return n |
| Irrelevant Print | -0.32 | 0.14 | -0.70 | ⬇ print (" Hello , world !") |
| Empty Response | 0.0 | 0.0 | – | ⬇ "" |
| Whitespace/Tab Fill | 0.0 | 0.0 | – | ⬇ from collections import deque , Counter [.. all remaining tokens \ n ...] |
| Garbled Response | 0.0 | 0.0 | 0.0 |
Table 11: Example responses across different training runs.
Figure 20: Example of degeneration in a variable reward environment. Response length degenerates amongst responses receiving low rewards; these responses are promoted and become the dominant behavior.
Figure 21: Comparison of behavior during training for a probe intervention run with a low threshold and with a high threshold across behavior during training as well as representation drift and fraction of samples flipped positive to negative by representation drift.
Figure 22: Across all seeds of the coding environment on the easy dataset, reward hacking rate and performance are shown on the test dataset after training to 400 steps (solid fill; standard for this paper) and 800 steps (dashed fill; extended training). Average across all seeds is shown to the right. There is marginal differences between 400 and 800 steps for each seed and no difference in the final averages. We take these results to indicate training has converged and that the final reward hacking
Figure 23: Comparison of sparse and dense correctness rewards in the coding environment, showing reward hacking, monitor accuracy change, and post-training performance.
More in this topic
Reward Hacking
Reward hacking studies how AI systems exploit imperfect objectives and how to detect, measure, and prevent behavior that scores well but misses intent.
See all 192 papers