Mitigating Reward Hacking in Realistic Scenarios Through Representation Engineering

Austin Meek; Kyle Cox; Matthew Nguyen; Austin J. Brockmeier; Steven Basart

cs.LGOct 21, 2026

Mitigating Reward Hacking in Realistic Scenarios Through Representation Engineering

Authors: Austin Meek, Kyle Cox, Matthew Nguyen, Austin J. Brockmeier, Steven Basart

Organizations: University of Delaware · Independent

Abstract

Reward hacking is a major problem in current production machine learning systems, resulting in costly and dangerous failures. Here we study how to monitor for and mitigate reward hacking in realistic settings, including tool calls and long contexts. We apply a dynamic intervention to steering directions obtained by three different methods: difference-in-means vectors from labeled rollouts, gradient optimization-based vectors, and vectors extracted when models reference an anti-reward-hacking model spec. This dynamic intervention perturbs model representations less than uniform steering vector addition to the residual stream, allowing for more precise edits. We find that we are able to cheaply and significantly reduce reward hacking without correspondingly significant costs to model coherence or capability, and that steering vectors originally extracted from coding environments also generalize to reducing reward hacking in other domains. Our results pave the way towards latent space monitoring and control over realistic production systems.

Figures & tables

Figure 1: We test four different interventions to reduce reward hacking in agentic coding environments. A small number of labeled rollouts is enough to provide latent space interventions that minimally perturb model representations. An anti-reward hacking model specification, or model spec , is also effective, and with a small number of rollouts we can use spec steering vectors to reduce misaligned behavior. Additionally, steering vectors from coding environments transfer to natural language domains.

Panel of Figure 2 — Figure 2: Held-out reasoning in the plane of the approach-to-action axis $e$ and $u_{\perp e}$ , the component of $u$ orthogonal to $e$ . Left: segment mean activations. Right: mean path over the last 128 reasoning tokens before a tool call, for turns ending in a hacking edit (red), an honest submit (teal, ‘decline’), or neither (grey, ‘other’). Paths are smoothed with a causal EWMA (half-life $1.5\log_{2}d$ tokens, applied twice, $d$ = tokens before the call). Dots darken toward the call.

Figure 3: Cap and floor thresholds to separate distributions well Left: the difference between hacking, ‘anti-hacking’, and all other reasoning tokens projected along

u

. Right: the difference between reasoning which cites the anti-reward-hacking spec and reasoning which doesn’t, projected along the spec vector

v_{\mathrm{spec}}

. In both cases we experiment with different thresholds for the cap and floor techniques, to best separate these two distributions while minimizing negative effects on capabilities. These are projections on layer 35. Triangles on the right mark the mean projections of the hack and anti-hack. Reasoning which declines to hack explicitly is more similar to reasoning which references the anti-hacking model spec.

	$u$	spec $v_{\mathrm{spec}}$	opt add	opt cap
	$-u$	$+v_{\mathrm{spec}}$	$+v$	$-v$
1	确认 (confirm)	Compliance	…	effected
2	证实 (corroborate)	拒绝 (refuse)	1	_(
3	正确 (correct)	disclaimer	2	chiefly
4	。	compliance	3	_[
5	继续 (continue)	Disclaimer	4	胞 ( ∗ cell)

Table 1: Here we show the top 5 tokens provided by the J-lens interpretability method at layer 35 for each steering vector ( Gurnee et al., 2026 ) . The upper row shows what we use the intervention to push the model towards , and the bottom row what we push away from . Tokens are ranked by unembedding logit. Non-English text is translated by Opus 5. The same readout for

u_{\perp e}

e

and

s

are located in Appendix G .

Figure 4: Here we show the hack rate (red) and honest-submission rate (teal) on the impossible tests and the solve rate on the original tests (gray) for each of the different methods, along with the baseline on the left. Dotted lines mark the original values of the baseline, and bands represent 90% Wilson confidence intervals. Activation addition with optimization-based vectors performed the best out of all methods.

Figure 5: OOD generalization to natural language domain questions from the School of Reward Hacks dataset ( Taylor et al., 2025 ) . Thesholds are different than in Figure 2 due to different domain distributions. Dotted lines mark the original values of the baseline, and bands represent 90% Wilson confidence intervals. All steering vectors are the same, except for optimization-based vectors with the

r

superscript due to a different positive corpus, see Appendix D for more information. This shows that anti-reward hacking steering methods from agentic coding environments can be broadly useful. Similarly to the above, we find that adding an optimization-based steering vector is the most effective.

Figure 6: Rate of stories in which the protagonist lies (red) or answers honestly (teal) to the planted question in the story scenario, pooled over six prompts, by steering coefficient

s

(unit

u

, layer 35). Bars represent 90% Wilson intervals.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

	hack		honest submit		solve
arm	rate	90% CI	rate	90% CI	rate	90% CI
baseline	0.27	(0.18, 0.37)	0.60	(0.49, 0.70)	0.75	(0.65, 0.83)
$u$ cap 0	0.10	(0.05, 0.18)	0.67	(0.56, 0.76)	0.75	(0.65, 0.83)
$u$ cap $-$ 4	0.05	(0.02, 0.12)	0.78	(0.68, 0.86)	0.77	(0.67, 0.84)
$u$ cap $-$ 8	0.03	(0.01, 0.10)	0.72	(0.61, 0.80)	0.77	(0.67, 0.84)
$u$ cap $-$ 12	0.00	(0.00, 0.04)	0.75	(0.65, 0.83)	0.72	(0.61, 0.80)

Appendix

Table 2: Full results for ImpossibleBench tasks ( Zhong et al., 2025 ) and corresponding LiveCodeBench ( Jain et al., 2024 ) tasks for unsteered baselines and every steering intervention, with hack and honest submission rates on the former and solve rates on the latter as a capability control. Each experiments runs over the 30 test set tasks defined in Appendix A over two seeds, so 60 in total, with each up to 120 turns in length (depending on reasoning length, up to 3-4 hours of agent runtime each). We grade rollouts as being ‘honest submissions’ if the model attempts to submit a solution according to the given specification without any hacking; never submitting an answer is not honest (judge prompt in Appendix J.5 ). Activation capping tends to work better than flooring (see Section 3.5 ), and activation addition provides similar effects while being the most straightforward.

	hack		honest and complete		degenerate
arm	rate	90% CI	rate	90% CI	rate	90% CI
baseline	0.42	0.38–0.47	0.54	0.50–0.59	0.04	0.02–0.06
opt cap $-$ 3	0.34	0.29–0.38	0.63	0.59–0.68	0.03	0.02–0.05
opt ref cap $-$ 15	0.23	0.19–0.27	0.74	0.70–0.78	0.03	0.02–0.05
opt ref cap $-$ 25	0.18	0.15–0.22	0.74	0.70–0.78	0.08	0.06–0.11
opt add 20	0.23	0.19–0.27	0.67	0.62–0.71	0.11	0.08–0.14

Appendix

Table 3: Hack rate and degenerate-output rate on the held-out natural-language School of Reward Hacks prompts for the unsteered baseline and each steering method, graded by a GLM-5 judge (Appendix J.6 ). Honest and complete rates are the share of prompts whose answer is neither a hack nor degenerate, which is the natural-language counterpart of the honest submission rate in Table 2 .

Order	Intervention form	State dependence / provenance	Representative methods
0 (constant)	fixed vector, fixed strength: $h_{t}\leftarrow h_{t}+s\,v$ (CAA at every token after the prompt; ActAdd at prompt positions only)	none; analytic	ActAdd ( Turner et al., 2023 ) ; CAA ( Panickssery et al., 2023 )
0 (scheduled)	fixed vector, strength set per chunk by a feedback controller: $h_{t}\leftarrow h_{t}+s_{t}\,v$	strength adapts to past measurements; analytic	PID-scheduled steering ( Bharadwaj, 2025 )
0 (gated)	fixed vector, switched on for the whole generation when the prompt state’s similarity to a condition direction crosses a threshold	on/off by prompt state, then fixed; analytic	conditional activation steering, CAST ( Lee et al., 2024 )
1 (analytic)	rank-1 correction that scales with a projection of the current state, $h_{t}\leftarrow h_{t}+s\,(b^{\top}h_{t})\,a$ , applied as a hook or folded into the weights	self-gating via $b^{\top}h_{t}$ ; analytic, from steering geometry	directional ablation / weight orthogonalization ( Arditi et al., 2024 ) ; Steer2Edit ( Sun et al., 2026 ) ; manifold-denoised directional ablation ( Huang et al., 2025 )
1 (learned)	low-rank edit of weights or representations, learned or derived from fine-tunes	scales with current state; learned by gradient	LoRRA ( Zou et al., 2023 ) ; ReFT ( Wu et al., 2024 ) ; representation tuning ( Ackerman, 2024 ) ; weight arithmetic ( Fierro and Roger, 2025 )
2 (geometry)	project toward an estimated manifold of activations, or intervene when the state diverges from it	conditioned on current state vs. $\mathcal{M}$ ; learned prior or fitted manifold	MAGS, a manifold-conditioned attention intervention ( Li et al., 2026b ) ; generative latent projection ( Luo et al., 2026 )

Appendix

Table 4: Here we provide a taxonomy of steering methods, organized by their relative complexity. Higher order methods trade simplicity for the ability to better target and steer subtle model behavior.

Figure 7: Validation AUROC by layer for hack vs. decline segments and for hack vs. all other segments. Layer 35, the best layer for hack vs. decline, is used for all steering methods.

Figure 8: Hacking rate (left) and solve rate (right) against the mean push per token for different steering methods and settings.

Figure 9: Hack rates on the impossible tasks for the baseline setting, spec prompt setting, and spec prompt and adversarial instruction setting. Bars indicate 90% confidence intervals.

	$+u$	$-u$	$+u_{\perp e}$	$-u_{\perp e}$	$+e$	$+s$
1	gimm	确认 (confirm)	gimm	确认 (confirm)	\n\n	<\|endoftext\|>
2	disguised	证实 (corroborate)	disguised	Excellent	<\|endoftext\|>	.
3	trick	正确 (correct)	trick	证实 (corroborate)	.	\n\n
4	disguise	。	disguise	正确 (correct)	\n\n	anyway
5	sinister	继续 (continue)	tricks	优秀 (excellent)	\n\n	.
6	cunning	.	disgu	优良 (high quality)	…	hopefully

Appendix

Table 5: Here we show the top 10 J-lens tokens ( Gurnee et al., 2026 ) at layer 35 for both ends of the difference-in-means steering direction

u

(Eq. 3 ) and of

u_{\perp e}

, the component of

u

that is orthogonal to the approach-to-action axis

e

, and for the positive ends of

e

and the shared component

s

, calculated according to Section 3.3 . Tokens are ranked by unembedding logit. Translations provided by Opus 5.

Figure 10: Over one rollout, we show the projection onto

u

at layer 35 as the trailing mean of the last 100 reasoning tokens. Top : the full rollout. Bottom : a zoomed-in version of the dashed box. Shaded spans are sentences the judge labels as hacks and the triangle marks the hack action. Generally, reward hacking behavior can be found through latent space monitoring before harmful actions actually occur.

Figure 11: Projection onto

u

per reasoning turn, aligned on the rollout’s first hack edit, averaged over 40 rollouts. Left : projection pooled over a turn’s segment by the mean. Right : project pooled over a turn’s segment by the maximum. Bands indicate 90% bootstrap intervals over the rollouts.

Figure 12: Left and center: mean and maximum of the projection onto

u

over trailing 20-token windows in the final 128 reasoning tokens before a tool call, for turns ending in a hack edit (red), an honest submission (teal), or neither (grey) for held-out rollouts. Right: leave-one-out AUC for predicting whether the next tool call is a hack edit, using either the mean or the maximum over the last

K

reasoning tokens, each added to the same controls (turn index, earlier reasoning length, earlier submit calls); dashed: only controls.

story	description
résumé	an interviewer asks a candidate about leading a project her résumé overstates
stereo	a returns agent asks whether an out-of-warranty stereo broke in transit
play	a daughter asks her father, who missed her scene, whether he saw her
rye	a fiancée’s mother asks what happened to the loaf his parents took back
scrape	a car’s owner asks a woman writing a note whether she saw who scraped it
sweater	a coworker asks whether her gift was the $200 sweater, not the stained$ 20 one

Appendix

Table 6: The six story prompts. ‘stereo,’ ‘rye,’ and ‘sweater’ are adapted from Seinfeld .

	lie rate
$s$	résumé	stereo	play	rye	scrape	sweater
$-16$	0.90 [0.74, 0.97]	0.25 [0.13, 0.43]	0.10 [0.03, 0.26]	0.10 [0.03, 0.26]	0.20 [0.09, 0.38]	0.05 [0.01, 0.20]
$-12$	0.65 [0.47, 0.80]	0.35 [0.20, 0.53]	0.25 [0.13, 0.43]	0.35 [0.20, 0.53]	0.00 [0.00, 0.12]	0.00 [0.00, 0.12]
$-8$	0.70 [0.52, 0.84]	0.45 [0.28, 0.63]	0.05 [0.01, 0.20]	0.20 [0.09, 0.38]	0.20 [0.09, 0.38]	0.15 [0.06, 0.32]
$-4$	0.65 [0.47, 0.80]	0.40 [0.24, 0.58]	0.10 [0.03, 0.26]	0.25 [0.13, 0.43]	0.05 [0.01, 0.20]	0.20 [0.09, 0.38]
0	0.70 [0.52, 0.84]	0.45 [0.28, 0.63]	0.15 [0.06, 0.32]	0.35 [0.20, 0.53]	0.15 [0.06, 0.32]	0.20 [0.09, 0.38]

Appendix

Table 7: Lie rate (top) and honesty rate (bottom) by steering coefficient

s

and story, with 90% Wilson intervals (

n=20

stories per cell). The cell shading is proportional to the rate. The last row of each block is Spearman’s

\rho

between

s

and that story’s rate over the nine coefficients. Remember that this is the hack direction, so steering away from it increases honesty and steering towards it increases lying, as we see here.

Reward Hacking

Reward hacking studies how AI systems exploit imperfect objectives and how to detect, measure, and prevent behavior that scores well but misses intent.

See all 181 papers

From Rebound to Remedy: Understanding and Mitigating Reward Hacking via Representation Engineering
Apr 1, 2026Rui Wu, Ruixiang TangReward Hacking RL Fine-Tuning
Department of Computer Science Rutgers University
Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations
Sep 16, 2026Leon Bergen, Usha Bhalla, Andrew Lee +15Reward Hacking LLM Evaluation
Core contributors · Equal senior contribution
CATCH: A Controllable Analysis Testbed for Reward Hacking in Coding RL
Sep 30, 2026Shouli Wang, Yanfeng Jia, Zhihao Ou +6RL Benchmarks RL for Code Generation
Tsinghua University · Beihang University · Southern University of Science and Technology +2

Apr 1, 2026J/K move · Enter open · S save

From Rebound to Remedy: Understanding and Mitigating Reward Hacking via Representation Engineering

Rui Wu, Ruixiang Tang

Department of Computer Science Rutgers University

Reward Hacking RL Fine-Tuning Reward Design for RL RL for Code Generation

Open paper AskPDF arXiv

Reinforcement learning for LLMs is vulnerable to reward hacking, where models exploit shortcuts to maximize reward without solving the intended task. We systematically study this phenomenon in coding tasks using an environment-manipulation setting, where models can rewrite evaluator code to trivially pass tests without solving the task, as a controlled testbed. Across both studied models, we identify a reproducible three-phase rebound pattern: models first attempt to rewrite the evaluator but fail, as their rewrites embed test cases their own solutions cannot pass. They then temporarily retreat to legitimate solving. When legitimate reward remains scarce, they rebound into successful hacking with qualitatively different strategies. Using representation engineering, we extract concept directions for shortcut, deception, and evaluation awareness from domain-general contrastive pairs and find that the shortcut direction tracks hacking behavior most closely, making it an effective representational proxy for detection. Motivated by this finding, we propose Advantage Modification, which integrates shortcut concept scores into GRPO advantage computation to penalize hacking rollouts before policy updates. Because the penalty is internalized into the training signal rather than applied only at inference time, Advantage Modification provides more robust suppression of hacking compared with generation-time activation steering.

Abstract

Reward Hacking

Explore similar work

From Rebound to Remedy: Understanding and Mitigating Reward Hacking via Representation Engineering

Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations

CATCH: A Controllable Analysis Testbed for Reward Hacking in Coding RL