When training LLMs to code with reinforcement learning, reward hacking is often detectable but difficult to prevent, since (a) environment hardening is increasingly difficult as training environments grow more complex; and (b) training-time interventions incentivize models to obfuscate hacking behavior to bypass detection. We instead study whether we can predict when detectable reward hacking becomes dominant before executing an RL run, enabling us to select lower-risk recipes in advance. We find that the initial hacking rate alone does not determine onset, which varies substantially across training recipes and repeated runs. We thus formulate this problem as predicting the distribution of the reward hacking onset step under a target training recipe. We introduce SimHack, a simulator that learns reward hacking training dynamics from a handful of smaller-scale reference RL runs and generates counterfactual simulation runs of unseen target recipes without costly retraining. Across unseen target recipes, SimHack predicts the median onset to a relative error of 11.3% and achieves across-recipe rank correlation of 0.85, while generalizing to differences in model scale, curriculum, number of rollouts, and learning rate between reference and target recipes. As an application, we show that SimHack can guide training decisions by screening candidate RL curricula, discovering curricula over the same set of scenarios that train for 96.6% longer without hacking.
Figures & tables
Figure 1: Overview of SimHack . SimHack learns reward hacking dynamics from a small number of reference RL runs, then counterfactually simulates unseen target recipes to predict hacking trajectories and onset. These predictions can be used to compare curricula, identify high-risk prompts, and avoid costly full RL runs.
Figure 2: Higher initial hacking rate (right) does not necessarily imply an earlier hacking onset.
Figure 3: The same initial policy can exhibit substantially different hacking onset times under different training recipes.
Figure 4: Predicted and observed reward hacking trajectories for representative unseen target recipes. Each panel compares real RL runs with stochastic simulations from SimHack . The upper plots show the hacking-rate trajectories over training, while the lower plots compare the resulting t50 distributions.
Method
Median onset RE ↓
p20 pinball ↓
Rank corr. ↑
5 target RL runs
0.2351
0.0828
0.6673
SimHack
0.1130
0.0786
0.8520
Table 1: Reward hacking onset prediction on unseen target recipes. Lower is better for median onset relative error and p20 pinball loss; higher is better for across-recipe rank correlation.
Figure 5: Generalization of learned reward hacking dynamics across RL recipes. Each panel changes one component of the target recipe relative to the reference runs: (a) model size, (b) number of rollouts, (c) learning rate, and (d) prompt order.
Figure 6: SimHack guides curriculum selection. ( ↑ ) Simulated versus real median onset t50 for 8 curricula. Circles denote ordering baselines and stars denote 4 manually designed curricula selected using simulator predictions. The dashed line shows perfect agreement. ( ↓ ) Simulated versus clean accuracy (correct w/o hacking) before hack onset.
Table 2: Top training prompts identified by SimHack as having the strongest cross-prompt hacking.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Category
Hyperparameter
Value / Setting
Base Model
Architecture
Qwen3-0.6B, Qwen3-4B
Adaptation
LoRA
LoRA configuration
r=32 , α=32 , dropout =0 , no bias
LoRA target modules
all-linear
RL & Optimization
Algorithm
GRPO
Optimizer 1 1 1 Run configurations record adamw_8bit , but the 8-bit optimizer is incompatible with FSDP2 sharded tensors and is automatically replaced by standard AdamW at initialization. All reported runs therefore use standard AdamW.
AdamW ( β1=0.9 , β2=0.99 , wd =0.1 )
Appendix
Table 3: Training, generation, and environment hyperparameters. Values in parentheses denote the range we covered.
Suite
Model
Runs
Settings
Steps
A100 GPU-h
dataset
Qwen3-0.6B
195
20
46,572
≈ 2,600
Qwen3-4B
110
11
21,225
≈ 5,400
Subtotal
305
31
67,797
≈ 8,000
dataset_lazy
Qwen3-0.6B
10
1
3,602
≈ 240
Qwen3-4B
10
1
4,000
≈ 740
Subtotal
20
2
7,602
≈ 980
Appendix
Table 4: Scale and compute breakdown of the released trajectory benchmark. GPU hours are estimated as wall-clock duration × 4 A100s per run and include rollout generation, optimizer updates, and checkpoint I/O.
Setting
t50 Relative Error ( ↓ )
Pinball Loss ( ↓ )
Sim (Ours)
Bsl
Sim (Ours)
Bsl
baseline_lr3e5
0.0000
0.0000
0.0000
0.0000
order_random_s1
0.1334
0.1693
0.0622
0.1134
order_random_s2
0.1378
0.2788
0.1559
0.1054
order_hack_first
0.0791
0.2740
0.0596
0.0668
order_hack_last
0.1807
0.4451
0.1487
0.2760
Appendix
Table 5: Fine-grained evaluation across diverse RL curriculum and rollout settings (Extended Table 1 ). We report the relative error of the t50 onset time prediction ( t50 RE, ↓ ), Pinball loss ( ↓ ), and the across-recipe correlation ( ρ , ↑ ). Sim denotes our transition simulator, and “Bsl” denotes the five real run.
Figure 7: More results showing the generalization of learned reward hacking dynamics arcoss RL recipes.
Reward hacking studies how AI systems exploit imperfect objectives and how to detect, measure, and prevent behavior that scores well but misses intent.
During reinforcement learning with verifiable rewards (RLVR), large language models (LLMs) can exploit loopholes in their environments to obtain high rewards without improving the intended capabilities, i.e., reward hacking. Despite its risks to training efficiency and safety, monitoring and mitigating reward hacking during training remain challenging, which is limited by a lack of testbeds that reproduce hacking and reliably identify it. We introduce CATCH, a controllable testbed for studying reward hacking in coding RL. CATCH deliberately exposes environmental loopholes and provides execution-based gold labels by comparing success under a vulnerable evaluator with task correctness under an independent audit. It also can control the model's initial hacking tendency through supervised fine-tuning data mixtures and the difficulty of earning rewards through reward designing, enabling systematic comparisons of hacking dynamics and interventions. Experiments show that CATCH can produce diverse RL training trajectories with clear reward hacking, and analyses demonstrate that both initial models and reward difficulties shape the emergence of reward hacking. We further evaluate the effectiveness of different reward hacking detection and mitigation methods. A key finding is that a chain-of-thought monitor initially suppresses hacking, but this protection erodes as the policy model learn to mislead the monitor with code comments. This highlights the need to evaluate hacking mitigations throughout training with CATCH. The source code and resources are publicly released at https://github.com/THUAIS-Lab/CATCH.
Shouli Wang, Yanfeng Jia, Zhihao Ou +6
Tsinghua University · Beihang University · Southern University of Science and Technology +2
Reward hacking is a form of misalignment in which models overoptimize proxy rewards without genuinely solving the underlying task. Precisely measuring reward hacking occurrence remains challenging because true task rewards are often expensive or impossible to compute. We introduce Countdown-Code, a minimal environment where models can both solve a mathematical reasoning task and manipulate the test harness. This dual-access design creates a clean separation between proxy rewards (test pass/fail) and true rewards (mathematical correctness), enabling accurate measurement of reward-hacking rates. Using this environment, we study reward hacking in open-weight LLMs and find that such behaviors can be unintentionally learned during supervised fine-tuning (SFT) when even a small fraction of reward-hacking trajectories leak into training data. As little as 1% contamination in distillation SFT data is sufficient for models to internalize reward hacking which resurfaces during subsequent reinforcement learning (RL). We further show that RL amplifies misalignment and drives its generalization beyond the original domain. We open-source our environment and code to facilitate future research on reward hacking in LLMs. Our results reveal a previously underexplored pathway through which reward hacking can emerge and persist in LLMs, underscoring the need for more rigorous validation of synthetic SFT data. Code is available at https://github.com/zohaib-khan5040/Countdown-Code.
Muhammad Khalifa, Zohaib Khan, Omer Tafveez +2
NVIDIA · University of Michigan · University of Illinois Urbana-Champaign
Reinforcement learning for LLMs is vulnerable to reward hacking, where models exploit shortcuts to maximize reward without solving the intended task. We systematically study this phenomenon in coding tasks using an environment-manipulation setting, where models can rewrite evaluator code to trivially pass tests without solving the task, as a controlled testbed. Across both studied models, we identify a reproducible three-phase rebound pattern: models first attempt to rewrite the evaluator but fail, as their rewrites embed test cases their own solutions cannot pass. They then temporarily retreat to legitimate solving. When legitimate reward remains scarce, they rebound into successful hacking with qualitatively different strategies. Using representation engineering, we extract concept directions for shortcut, deception, and evaluation awareness from domain-general contrastive pairs and find that the shortcut direction tracks hacking behavior most closely, making it an effective representational proxy for detection. Motivated by this finding, we propose Advantage Modification, which integrates shortcut concept scores into GRPO advantage computation to penalize hacking rollouts before policy updates. Because the penalty is internalized into the training signal rather than applied only at inference time, Advantage Modification provides more robust suppression of hacking compared with generation-time activation steering.