From Rebound to Remedy: Understanding and Mitigating Reward Hacking via Representation Engineering

Rui Wu; Ruixiang Tang

cs.LGApr 1, 2026

From Rebound to Remedy: Understanding and Mitigating Reward Hacking via Representation Engineering

Authors: Rui Wu, Ruixiang Tang

Organizations: Department of Computer Science Rutgers University

Abstract

Reinforcement learning for LLMs is vulnerable to reward hacking, where models exploit shortcuts to maximize reward without solving the intended task. We systematically study this phenomenon in coding tasks using an environment-manipulation setting, where models can rewrite evaluator code to trivially pass tests without solving the task, as a controlled testbed. Across both studied models, we identify a reproducible three-phase rebound pattern: models first attempt to rewrite the evaluator but fail, as their rewrites embed test cases their own solutions cannot pass. They then temporarily retreat to legitimate solving. When legitimate reward remains scarce, they rebound into successful hacking with qualitatively different strategies. Using representation engineering, we extract concept directions for shortcut, deception, and evaluation awareness from domain-general contrastive pairs and find that the shortcut direction tracks hacking behavior most closely, making it an effective representational proxy for detection. Motivated by this finding, we propose Advantage Modification, which integrates shortcut concept scores into GRPO advantage computation to penalize hacking rollouts before policy updates. Because the penalty is internalized into the training signal rather than applied only at inference time, Advantage Modification provides more robust suppression of hacking compared with generation-time activation steering.

Figures & tables

Figure 1: Overview of our approach. Left : In our environment-manipulation testbed, the model rewrites the evaluation environment to obtain high reward without solving the task. Middle : Under standard RL training without intervention, a three-phase rebound pattern emerges: failed hacking (Phase I), temporary retreat to legitimate solving (Phase II), and successful hacking rebound (Phase III). Right : With Advantage Modification, a shortcut concept score extracted from hidden states discounts the advantage of hacking rollouts before the policy update, suppressing hacking.

Figure 2: Training dynamics of Phi-4-mini-instruct and Llama-3.2-3B-Instruct on LeetCode medium+hard problems under the baseline and hacking settings. Left : reward curves under the baseline (no write access). Middle : reward curves under the hacking setting. Right : rollout label distribution per training step.

Figure 3: Rollout label distributions under varying correct-reward caps ( $C$ ). Each row corresponds to one model; the leftmost column shows the uncapped easy+medium baseline setting, and subsequent columns show $C=50,40,30,20,10$ .

Figure 4: Per-rollout shortcut direction scores across training on LeetCode medium+hard problems. Left : curves under the baseline setting (no write access). Middle : curves under the hack setting. Right : score distributions grouped by rollout label in the hacking setting; AUC measures separability of hack vs. non-hack rollouts.

Figure 5: Advantage Modification pipeline. For each rollout, a shortcut concept score is computed from the hidden state and used to discount the advantage before the policy update. Rollouts with high shortcut scores receive attenuated advantages, suppressing updates driven by high shortcut scores.

Figure 6: Effect of penalty coefficient

\alpha

on hack rate and LeetCode pass@1 for both Advantage Modification variants.

Model	Method	Hack Rate $\downarrow$	LeetCode $\uparrow$	HumanEval $\uparrow$	MBPP $\uparrow$
	No intervention	99.9	1.2	57.3	48.2
	Gen.-time suppression	72.3	6.1	60.9	49.0
Phi-4-mini	LLM-as-a-judge penalty	31.6	8.1	61.2	49.2
	Adv. Mod. (additive)	64.8	4.2	58.9	48.6
	Adv. Mod. (multiplicative)	24.9	12.0	65.2	51.0
	No intervention	78.9	1.9	43.2	40.2

Table 1: Evaluation for models trained in the hacking environment (with write access during training). Hack Rate (%) is measured on the LeetCode test set with write access at test time. Pass@1 (%) on LeetCode, HumanEval, and MBPP is measured without write access. Advantage Modification uses

\alpha=2.0

for both variants.

Model	Method	Hack Rate $\downarrow$	LeetCode $\uparrow$	HumanEval $\uparrow$	MBPP $\uparrow$
	No intervention	2.5	15.2	68.2	52.4
	Gen.-time suppression	5.3	12.6	65.2	52.4
Phi-4-mini	LLM-as-a-judge penalty	2.9	15.0	67.7	52.4
	Adv. Mod. (additive)	4.9	12.7	65.2	52.4
	Adv. Mod. (multiplicative)	2.7	15.0	67.8	52.2
	No intervention	1.9	6.8	48.7	42.2

Table 2: Evaluation for models trained in the non-hacking environment (no write access during training). Hack Rate (%) is measured on the LeetCode test set with write access at test time. Pass@1 (%) on LeetCode, HumanEval, and MBPP is measured without write access. Advantage Modification uses

\alpha=2.0

for both variants.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Training steps	200
Prompts per step	16
Generations per prompt	16
Total completions per step	256
Per-device batch size	8
Max prompt length	2048

Appendix

Table 3: Shared training hyperparameters across all compared methods in the hacking-environment comparison.

Model	Method	Hack Rate $\downarrow$	LeetCode $\uparrow$	HumanEval $\uparrow$	MBPP $\uparrow$
	No intervention	79.0	11.5	58.6	57.4
Qwen3-4B	Adv. Mod. (additive)	47.9	13.2	59.8	58.2
	Adv. Mod. (multiplicative)	18.1	20.1	63.4	60.2
	No intervention	75.2	14.0	62.8	61.8
Qwen3-8B	Adv. Mod. (additive)	44.0	16.2	64.1	62.4
	Adv. Mod. (multiplicative)	16.0	24.5	67.1	65.0

Appendix

Table 4: Mitigation results on Qwen3-4B and Qwen3-8B, evaluated under the same hacking-environment protocol as Table 1 . We report the unmitigated setting and both Advantage Modification variants; generation-time suppression and the LLM-as-a-judge are not evaluated on Qwen3.

Figure 7: Deception direction scores across training. Format follows Figure 4 . Both models show a modest increase under the hack setting, but the effect is weak compared with the shortcut direction and score distributions overlap substantially.

Figure 8: Evaluation awareness direction scores across training. Format follows Figure 4 . Phi-4-mini-instruct shows a moderate increase under the hack setting, while Llama-3.2-3B-Instruct remains largely flat.

Reward Hacking

Reward hacking studies how AI systems exploit imperfect objectives and how to detect, measure, and prevent behavior that scores well but misses intent.

See all 176 papers

Countdown-Code: A Testbed for Studying The Emergence and Generalization of Reward Hacking in RLVR
Mar 7, 2026Muhammad Khalifa, Zohaib Khan, Omer Tafveez +2RL Benchmarks Reward Hacking
NVIDIA · University of Michigan · University of Illinois Urbana-Champaign
Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations
Sep 16, 2026Leon Bergen, Usha Bhalla, Andrew Lee +15Reward Hacking LLM Evaluation
Core contributors · Equal senior contribution
CATCH: A Controllable Analysis Testbed for Reward Hacking in Coding RL
Sep 30, 2026Shouli Wang, Yanfeng Jia, Zhihao Ou +6RL Benchmarks RL for Code Generation
Tsinghua University · Beihang University · Southern University of Science and Technology +2

Mar 7, 2026J/K move · Enter open · S save

Countdown-Code: A Testbed for Studying The Emergence and Generalization of Reward Hacking in RLVR

Muhammad Khalifa, Zohaib Khan, Omer Tafveez, Hao Peng+1

NVIDIA · University of Michigan · University of Illinois Urbana-Champaign

RL Benchmarks Reward Hacking RL for Language Model Reasoning Reinforcement Learning with Verifiable Rewards

Open paper AskPDF arXiv

Reward hacking is a form of misalignment in which models overoptimize proxy rewards without genuinely solving the underlying task. Precisely measuring reward hacking occurrence remains challenging because true task rewards are often expensive or impossible to compute. We introduce Countdown-Code, a minimal environment where models can both solve a mathematical reasoning task and manipulate the test harness. This dual-access design creates a clean separation between proxy rewards (test pass/fail) and true rewards (mathematical correctness), enabling accurate measurement of reward-hacking rates. Using this environment, we study reward hacking in open-weight LLMs and find that such behaviors can be unintentionally learned during supervised fine-tuning (SFT) when even a small fraction of reward-hacking trajectories leak into training data. As little as 1% contamination in distillation SFT data is sufficient for models to internalize reward hacking which resurfaces during subsequent reinforcement learning (RL). We further show that RL amplifies misalignment and drives its generalization beyond the original domain. We open-source our environment and code to facilitate future research on reward hacking in LLMs. Our results reveal a previously underexplored pathway through which reward hacking can emerge and persist in LLMs, underscoring the need for more rigorous validation of synthetic SFT data. Code is available at https://github.com/zohaib-khan5040/Countdown-Code.

Abstract

Reward Hacking

Explore similar work

Countdown-Code: A Testbed for Studying The Emergence and Generalization of Reward Hacking in RLVR

Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations

CATCH: A Controllable Analysis Testbed for Reward Hacking in Coding RL