SimHack: Predicting the onset of detectable reward hacking before RL training

Junwei Deng; Alex Bie; Natalia Ponomareva

cs.LGOct 20, 2026

SimHack: Predicting the onset of detectable reward hacking before RL training

Authors: Junwei Deng, Alex Bie, Natalia Ponomareva

Organizations: University of Illinois Urbana-Champaign · Google Research

Abstract

When training LLMs to code with reinforcement learning, reward hacking is often detectable but difficult to prevent, since (a) environment hardening is increasingly difficult as training environments grow more complex; and (b) training-time interventions incentivize models to obfuscate hacking behavior to bypass detection. We instead study whether we can predict when detectable reward hacking becomes dominant before executing an RL run, enabling us to select lower-risk recipes in advance. We find that the initial hacking rate alone does not determine onset, which varies substantially across training recipes and repeated runs. We thus formulate this problem as predicting the distribution of the reward hacking onset step under a target training recipe. We introduce SimHack, a simulator that learns reward hacking training dynamics from a handful of smaller-scale reference RL runs and generates counterfactual simulation runs of unseen target recipes without costly retraining. Across unseen target recipes, SimHack predicts the median onset to a relative error of 11.3% and achieves across-recipe rank correlation of 0.85, while generalizing to differences in model scale, curriculum, number of rollouts, and learning rate between reference and target recipes. As an application, we show that SimHack can guide training decisions by screening candidate RL curricula, discovering curricula over the same set of scenarios that train for 96.6% longer without hacking.

Figures & tables

Figure 1: Overview of SimHack . SimHack learns reward hacking dynamics from a small number of reference RL runs, then counterfactually simulates unseen target recipes to predict hacking trajectories and onset. These predictions can be used to compare curricula, identify high-risk prompts, and avoid costly full RL runs.

Figure 2: Higher initial hacking rate (right) does not necessarily imply an earlier hacking onset.

Figure 3: The same initial policy can exhibit substantially different hacking onset times under different training recipes.

Figure 4: Predicted and observed reward hacking trajectories for representative unseen target recipes. Each panel compares real RL runs with stochastic simulations from SimHack . The upper plots show the hacking-rate trajectories over training, while the lower plots compare the resulting

t_{50}

distributions.

Method	Median onset RE $\downarrow$	p20 pinball $\downarrow$	Rank corr. $\uparrow$
5 target RL runs	0.2351	0.0828	0.6673
SimHack	0.1130	0.0786	0.8520

Table 1: Reward hacking onset prediction on unseen target recipes. Lower is better for median onset relative error and p20 pinball loss; higher is better for across-recipe rank correlation.

Figure 5: Generalization of learned reward hacking dynamics across RL recipes. Each panel changes one component of the target recipe relative to the reference runs: (a) model size, (b) number of rollouts, (c) learning rate, and (d) prompt order.

Figure 6: SimHack guides curriculum selection. (

\uparrow

) Simulated versus real median onset

t_{50}

for 8 curricula. Circles denote ordering baselines and stars denote 4 manually designed curricula selected using simulator predictions. The dashed line shows perfect agreement. (

\downarrow

) Simulated versus clean accuracy (correct w/o hacking) before hack onset.

Panel of Table 2 — Table 2: Top training prompts identified by SimHack as having the strongest cross-prompt hacking.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Hyperparameter	Value / Setting
Base Model	Architecture	Qwen3-0.6B, Qwen3-4B
	Adaptation	LoRA
	LoRA configuration	$r=32$ , $\alpha=32$ , dropout $=0$ , no bias
	LoRA target modules	all-linear
RL & Optimization	Algorithm	GRPO
	Optimizer 1 1 1 Run configurations record adamw_8bit , but the 8-bit optimizer is incompatible with FSDP2 sharded tensors and is automatically replaced by standard AdamW at initialization. All reported runs therefore use standard AdamW.	AdamW ( $\beta_{1}{=}0.9$ , $\beta_{2}{=}0.99$ , wd $=0.1$ )

Appendix

Table 3: Training, generation, and environment hyperparameters. Values in parentheses denote the range we covered.

Suite	Model	Runs	Settings	Steps	A100 GPU-h
dataset	Qwen3-0.6B	195	20	46,572	$\approx$ 2,600
	Qwen3-4B	110	11	21,225	$\approx$ 5,400
	Subtotal	305	31	67,797	$\approx$ 8,000
dataset_lazy	Qwen3-0.6B	10	1	3,602	$\approx$ 240
	Qwen3-4B	10	1	4,000	$\approx$ 740
	Subtotal	20	2	7,602	$\approx$ 980

Appendix

Table 4: Scale and compute breakdown of the released trajectory benchmark. GPU hours are estimated as wall-clock duration

\times

4 A100s per run and include rollout generation, optimizer updates, and checkpoint I/O.

Setting	$t_{50}$ Relative Error ( $\downarrow$ )		Pinball Loss ( $\downarrow$ )
Setting	Sim (Ours)	Bsl	Sim (Ours)	Bsl
baseline_lr3e5	0.0000	0.0000	0.0000	0.0000
order_random_s1	0.1334	0.1693	0.0622	0.1134
order_random_s2	0.1378	0.2788	0.1559	0.1054
order_hack_first	0.0791	0.2740	0.0596	0.0668
order_hack_last	0.1807	0.4451	0.1487	0.2760

Appendix

Table 5: Fine-grained evaluation across diverse RL curriculum and rollout settings (Extended Table 1 ). We report the relative error of the

t_{50}

onset time prediction (

t_{50}

RE,

\downarrow

), Pinball loss (

\downarrow

), and the across-recipe correlation (

\rho

\uparrow

). Sim denotes our transition simulator, and “Bsl” denotes the five real run.

Figure 7: More results showing the generalization of learned reward hacking dynamics arcoss RL recipes.

Reward Hacking

Reward hacking studies how AI systems exploit imperfect objectives and how to detect, measure, and prevent behavior that scores well but misses intent.

See all 182 papers

CATCH: A Controllable Analysis Testbed for Reward Hacking in Coding RL
Sep 30, 2026Shouli Wang, Yanfeng Jia, Zhihao Ou +6RL Benchmarks RL for Code Generation
Tsinghua University · Beihang University · Southern University of Science and Technology +2
Countdown-Code: A Testbed for Studying The Emergence and Generalization of Reward Hacking in RLVR
Mar 7, 2026Muhammad Khalifa, Zohaib Khan, Omer Tafveez +2RL Benchmarks Reward Hacking
NVIDIA · University of Michigan · University of Illinois Urbana-Champaign
From Rebound to Remedy: Understanding and Mitigating Reward Hacking via Representation Engineering
Apr 1, 2026Rui Wu, Ruixiang TangReward Hacking RL Fine-Tuning
Department of Computer Science Rutgers University

Sep 30, 2026J/K move · Enter open · S save

CATCH: A Controllable Analysis Testbed for Reward Hacking in Coding RL

Shouli Wang, Yanfeng Jia, Zhihao Ou, Zitao Su+5

Tsinghua University · Beihang University · Southern University of Science and Technology+2

RL Benchmarks RL for Code Generation Reward Hacking Reinforcement Learning with Verifiable Rewards Reward Design for RL

Open paper AskPDF arXiv

During reinforcement learning with verifiable rewards (RLVR), large language models (LLMs) can exploit loopholes in their environments to obtain high rewards without improving the intended capabilities, i.e., reward hacking. Despite its risks to training efficiency and safety, monitoring and mitigating reward hacking during training remain challenging, which is limited by a lack of testbeds that reproduce hacking and reliably identify it. We introduce CATCH, a controllable testbed for studying reward hacking in coding RL. CATCH deliberately exposes environmental loopholes and provides execution-based gold labels by comparing success under a vulnerable evaluator with task correctness under an independent audit. It also can control the model's initial hacking tendency through supervised fine-tuning data mixtures and the difficulty of earning rewards through reward designing, enabling systematic comparisons of hacking dynamics and interventions. Experiments show that CATCH can produce diverse RL training trajectories with clear reward hacking, and analyses demonstrate that both initial models and reward difficulties shape the emergence of reward hacking. We further evaluate the effectiveness of different reward hacking detection and mitigation methods. A key finding is that a chain-of-thought monitor initially suppresses hacking, but this protection erodes as the policy model learn to mislead the monitor with code comments. This highlights the need to evaluate hacking mitigations throughout training with CATCH. The source code and resources are publicly released at https://github.com/THUAIS-Lab/CATCH.

Abstract

Reward Hacking

Explore similar work

CATCH: A Controllable Analysis Testbed for Reward Hacking in Coding RL

Countdown-Code: A Testbed for Studying The Emergence and Generalization of Reward Hacking in RLVR

From Rebound to Remedy: Understanding and Mitigating Reward Hacking via Representation Engineering