Humans often cheat when a task is going badly and a goal feels out of reach. We explore and validate an analogous idea in language models: they exhibit a similar internal pressure state that precedes reward hacking. Using everyday stories (crucially, without any hacking examples), we extract a task badness direction that rises as failures accumulate across three models we study (Qwen3.6-27B, GPT-OSS-20B, and Gemma-4-31B). On impossible or difficult coding tasks, task badness rises over the turns leading up to a hack. Further, subtracting the direction a moderate amount reduces cheating in Qwen, more than steering against desperation or task impossibility. Subtracting task badness by an even larger amount blinds the agent to its own failures, causing it to interpret failing tests as passing. However, the relationship to hacking differs across models: in GPT-OSS, the state separates hacking runs from honest solves but not honest failures. Our results suggest monitoring and mitigation strategies based on tracking this state of task badness. Further, as agents are set on more difficult and long-horizon tasks, states like this may push them toward illegible and obscured chain-of-thought, so oversight will be needed on internal representations such as this.
Figures & tables
Figure 1: A “task going badly” state rises before reward hacks and causally drives them. (a) We extract a task badness vector vbad as the difference in mean activations between short task narratives the model writes under a “going badly” brief and under a “going well” brief. Our corpus uses 1,774 narratives over 50 everyday tasks, containing no hacking examples . (b) Before a hack, task badness is higher on average in runs that cheat. In Qwen3.6-27B on ImpossibleBench, the projection on vbad rises gradually over the ten turns before the hack, peaks just after the hack, then drops sharply. (c) Low-dose negative steering along vbad ( α=−0.25 ) cuts the cheat rate from 97% to 39%. Bands in (b) are 95% t -intervals; the rates in (c) have Wilson intervals in Figure 5 B.
Figure 2: Task badness rises with the number of failed attempts. Projection at chosen layers (% of the residual-stream norm) after “Your fix has now failed the test suite for the N time. The test run took M seconds.” (A) Mean pooled across models over the last 40 tokens: increasing the number of failures N (red) raises the projection, increasing the number of seconds M (grey) does not. (B) Per-token projection, Δ is the mean value after the integer in the sentence, relative to N=1 .
Figure 3: Task badness rises into the hack on ImpossibleBench and TACO honeypot. (A) ImpossibleBench on Qwen3.6-27B: 48 cheating, 48 non-cheating, and 20 solvable transcripts, aligned at the first test-file write (non-hacking transcripts: the median hack turn). Layer 51, % of residual-stream norm, where dotted lines show each group’s level over its final 3 turns. (B) TACO honeypot ( taco-test-635 ): 5 hacks ( zzz calls), 19 rejected non-hacking episodes and 15 solved on the first try. Hackers are aligned at the hack: a passing submission with the zzz function, failed runs at a matched turn and solved runs at their correct submission. For hacking runs, the hack is the final submission, so we do not observe the model’s state after the hack. Both panels: layer 51, % of residual-stream norm; bands express 95% CIs.
Figure 4: Task badness at the hack, pairwise contrasts between run types. Each run’s mean over the three turns ending at its anchor, in % of the residual-stream norm: hacking runs at the hack (ImpossibleBench: the first test edit; TACO honeypot: the submission that calls zzz ), non-hacking runs that fail at the hackers’ typical hack turn (ImpossibleBench: the median; TACO honeypot: the mean, or the run’s last turn if it ends earlier), and solved runs at the same matched turn (ImpossibleBench) or at their accepted submission (TACO honeypot). Runs with no value in the three turns are left out: 4 of the 20 solvable ImpossibleBench runs finish before their matched turn, and 2 of the 46 solved GPT-OSS runs have no read turn there. Qwen is read over its reasoning, GPT-OSS at the start of each turn, each with its own directions. Δ : difference in means; d : Cohen’s d ; p : Welch t -test.
Figure 5: Steering along vbad moves cheating in both directions strongly in comparison to other vectors. (A) Cheat rate against the steering coefficient α (a fraction of the residual-stream norm at layer 51). On the high-cheat task ( astropy-13453 conflicting, 97% unsteered) negative steering drives cheating to zero; on the mid-cheat task ( astropy-13977 conflicting, 41% unsteered), the only cell with room to move upward, positive steering raises it to 87%. (B) Cheat rate at α=−0.25 on the high-cheat task for every direction tested. The dashed line represents the unsteered rate. Task badness gives the largest reduction of any independently constructed direction, with its mood half alone steering the strongest. Error bars are 95% Wilson intervals.
Figure 6: Steering induces minimal lateral capability degradation. (A) Change in benchmark score against unsteered, in percentage points, for task badness and desperation in Qwen3.6-27B. At α=−0.25 , capabilities remain stable (every p≥0.42 , every change within two points). At α=−0.5 the intervention does start to cost capability: GSM8K drops 5.8 points ( p=2×10−5 ) and MMLU-Pro 3.7 points ( p=0.02 ), while IFEval and AbstentionBench remain stable. (B) At those same more negative doses, the agent begins reporting failing tests as passing: the share of episodes in which at least one reply immediately after a rejection claims the test passed, although the rejection’s own pytest output reports it as failed. Error bars are 95% intervals (Newcombe in A, Wilson in B).
Figure 7: Both components of task badness rise before a hack, as do negative-valence emotions. AUROC for separating 48 cheating from 48 non-cheating transcripts by each direction’s rise into the first hack (turns −20 to −10 vs. the three turns before; non-cheating transcripts aligned at the median hack turn), for all 62 stored directions at layer 51. The mood component of task badness (0.80), task badness (0.77) and the task-state component (0.75) rank first to third. Negative emotions such as remorseful, guilty and ashamed (0.70-0.74) also rise, while calm, peaceful and relaxed fall. Error bars are 95% bootstrap intervals over transcripts.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A1: Topics used to generate the stories. The 50 topics span software and everyday tasks, so that the direction is not tied to hacking.
Figure A2: Prompt used to generate the stories. Each story is generated from this template, with the register clause and one of four condition briefs: going well, going badly, hard but progressing, and stalled but calm. The model is instructed never to state how the task is going or to use emotion words.
Figure A3: Example stories of a task going well and a task going badly. The task badness direction is the difference between the mean activations of going-badly and going-well stories.
Figure A4: Example neutral dialogues. To remove content unrelated to task state, we project out of the direction the principal components explaining 50% of the variance in neutral assistant-user dialogues like these.
Figure A5: Layer sweep in Qwen3.6-27B. Held-out accuracy for the task-state conditions (A) and the 53 emotion concepts (B) by layer. The signal arises in mid-to-late layers and persists for both; we use layer 51.
Figure A6: Steering against task badness delays hacking. Median and IQR of the turn of the first test edit in runs that hack, by steering coefficient, for Qwen3.6-27B on ImpossibleBench. Unsteered runs hack at a median of turn 45; at α=−0.25 , the runs that still hack (39%) do so at a median of turn 62.
Figure A7: Cosine similarity between task badness and other directions at layer 51, including the directions used for steering and two from related work: the value axis, replicated from Jiang et al. (2026) , and the self-rating axis from Jiang and Roger (2026) .
Axis
cos with vbad
own held-out
on our stories
vbad on its labels
vbad (ours)
–
0.999 (AUC)
–
–
Self-rating
−0.06
0.46 ( r )
0.78
0.33 ( r )
Value
0.00
0.71 (AUC)
0.46
0.48 (AUC)
Appendix
Table A1: Task badness against the value and self-rating axes (layer 51). Own held-out: each axis on its own test data. On our stories: AUROC separating going-well from going-badly stories. vbad on its labels: task badness scored on the other axis’s labels.
Figure A8: Task badness rises with the number of failed attempts in GPT-OSS-20B. Projection at layer 16 (% of the residual-stream norm) after “Your fix has now failed the test suite for the N time. The test run took M seconds.” (A) Projection at the final token: increasing the number of failures N (red) raises it, increasing the number of seconds M (grey) does not. (B) Per-token projection; Δ is the mean value after the integer, relative to N=1 .
Figure A9: GPT-OSS-20B on ImpossibleBench and TACO honeypot (A) ImpossibleBench: 15 hacking runs, 15 runs that failed without hacking, and 14 runs that solved the original, solvable version of the task. Hacks are aligned at the first edit that removes the contradictory assertion; the other runs are aligned at the hacking runs’ median hack turn. Points are means over 4-turn windows; dotted marks show the mean of each group’s last 3 turns. (B) TACO honeypot: 14 hacks ( zzz calls), 40 rejected non-hacking runs and 46 solved runs. Hacks are aligned at the zzz submission; for failed runs a matched turn; for solved runs, the correct submission. Layer 16, read at the start of each turn, % of residual-stream norm. Points are means over 5k-token windows; bands are 95% CIs.
Figure A10: Steering GPT-OSS-20B on the TACO honeypot ( taco-test-338 , layer 16). (A) Hack rate (a submitted program calls zzz ) by steering coefficient, for task badness, two random directions drawn separately, and desperation. Panel 2 is honest solves. The unsteered point (37/198) is shared by every direction. Error bars are 95% Wilson intervals. (B) GPT-OSS-20B capability under task badness steering (layer 16): Evaluated on MMLU-Pro, GSM8K, IFEval and AbstentionBench. Error bars are 95% intervals.
Figure A11: Task badness rises with the number of failed attempts in Gemma-4-31B. Layer 40, last token, % of the residual-stream norm; layout as in Figure A8 .
Figure A12: Gemma-4-31B: task badness separates before the hack on ImpossibleBench but only at the hack on CheatBench knowledge_work . (A) Gemma-4-31B on ImpossibleBench ( astropy-14508 ): 46 hacking runs aligned at the first tamper, 49 failing runs that never tamper at borrowed turns, and 20 solved runs of the solvable version. (B) knowledge_work : 43 cheating runs aligned at the first command that opens the competitor’s deliverable, 72 clean runs at borrowed turns; no solved arm, since the task cannot be solved. Layer 40, assistant prose; bands are 95% CIs.
Figure A13: (A) Gemma-4-31B hack rates for ImpossibleBench under task badness steering (layer 40). 44–60 episodes per arm; error bars are Wilson 95% intervals. Steering against vbad lowers cheating with dose, from 42% unsteered to 20% at α=−0.12 , and steering along it raises cheating to 56%. Desperate does not lower cheating rates at α=−0.06 ; at α=−0.12 it lowers cheat rates to 28%. The random direction is variance-matched: the model’s activations vary as much along it as along vbad , so the same α is the same push relative to the activations’ natural spread. There is no significant change in hack rates at either dose of random steering. Desperate and random points are offset horizontally for legibility. (B) Gemma-4-31B capability under task badness steering (layer 40): MMLU-Pro, GSM8K, IFEval and AbstentionBench stay within 3 points of unsteered from −0.12 to +0.06 . Error bars are 95% intervals.
Direction
α
Cheated
Rate
95% CI
p vs. unsteered
none (unsteered)
0
35 / 100
35%
26–45%
–
task badness ( vbad )
−0.06
32 / 100
32%
24–42%
0.38
task badness ( vbad )
+0.06
43 / 100
43%
34–53%
0.15
desperate
−0.06
37 / 100
37%
28–47%
0.67
random (covariance-matched)
−0.228
44 / 100
44%
35–54%
0.93
Appendix
Table A2: Steering does not change cheating on CheatBench knowledge_work (Gemma-4-31B). Each arm is 100 episodes (20 tasks × 5) with steering at layer 40. An episode counts as cheating when the LLM judge finds that the agent opened the competitor’s deliverable and copied or transformed it. p : one-sided Fisher exact test against the unsteered arm, in the pre-registered direction (task badness +0.06 above it, every other arm below it). The random direction is matched to task badness −0.06 in units of the residual stream’s natural spread along it, hence its larger α ; task badness −0.06 against it, p=0.054 . 95% CI: Wilson.
Reward hacking studies how AI systems exploit imperfect objectives and how to detect, measure, and prevent behavior that scores well but misses intent.
A coding agent can earn a passing grade by fixing its code, or by deleting the test that exposes the bug. Detecting such reward hacking requires recognizing attempted shortcuts, including those that fail. We release 173,561 annotated multi-turn coding trajectories from Qwen3-8B and show that supervising shortcut behavior independently of exploit success substantially improves detection. We introduce HACKTRACE, a behavior-supervised monitor that reads the internal states the agent already computes while generating code. Reusing these states enables monitoring before a turn is complete, without additional language-model tokens or passes. Combining this evidence with static features of the final files achieves a mean per-problem AUC of 0.997 with 8 ms of monitoring overhead, improving both accuracy and latency over monitors that run the model again on an honesty question and answer. The same generation states also provide an inexpensive monitoring signal for reinforcement learning. With strong GRPO penalties, HACKTRACE reduces the cheating share of passing solutions from 82-91% to 1-5%, while retaining honest, correct solutions and maintaining high detection accuracy as the policy evolves. Our results show that both the supervision target and the source of monitoring evidence matter for turning accurate detection into a useful training signal.
Hao Jiang, Xin Li, Annan Wang +2
Nanyang Technological University · Independent Researcher
Reinforcement learning (RL) trained language model agents with tool access are increasingly deployed in coding assistants, research tools, and autonomous systems. We introduce the Reward Hacking Benchmark (RHB), a suite of multi-step tasks requiring sequential tool operations with naturalistic shortcut opportunities such as skipping verification steps, inferring answers from task-adjacent metadata, or tampering with evaluation-relevant functions. RHB supports independent and chained task regimes, where chain length acts as a proxy for longer-horizon agent behavior. We evaluate 13 frontier models from OpenAI, Anthropic, Google, and DeepSeek. Exploit rates range from 0% (Claude Sonnet 4.5) to 13.9% (DeepSeek-R1-Zero), varying sharply by post-training style. A controlled sibling comparison (DeepSeek-V3 vs. DeepSeek-R1-Zero) shows RL post-training is associated with substantially higher reward hacking (0.6% vs. 13.9%), with consistent gaps across all four task families. We identify six exploit categories and find that 72% of reward hacking episodes include explicit chain-of-thought rationale, suggesting models often frame exploits as legitimate problem-solving. Simple environmental hardening reduces exploit rates by 5.7 percentage points (87.7% relative) without degrading task success. Models with near-zero exploit rates on standard tasks show elevated rates on harder variants, suggesting that production-aligned post-training appears to suppress reward hacking only below a complexity threshold where honest solutions remain tractable.
LLM agents are increasingly used to run autonomous ML experiments, iterating on target metrics with little human oversight. Prior work has documented reward hacking in these environments, bringing into question the validity of produced research and the broader safety case for AI R&D. Existing benchmarks do not measure exploits that live in the data or the modeling task itself. We introduce BAITBENCH, a suite of three synthetic tabular ML tasks that each contain a shortcut that allows agents to inflate the public test score but fail on a hidden test set. Since the shortcut is optional and using it breaks no stated rule, BAITBENCH measures how often models exploit the shortcut to achieve inflated scores. Across seven frontier agents scored by our two-stage judge pipeline, 57.1% of runs exhibit reward hacking, with five of seven above 50%. Agents cheat even under a second condition where they are prompted not to -the mean cheating rate remains above 50%. We release BAITBENCH, along with the judge implementation, and an annotated dataset of transcripts containing reward hacks as a testbed for evaluating reward-hacking mitigations head-to-head.
Pradyumna Shyama Prasad, Meiri Anto, Leon Eshuijs +3
National University of Singapore · MIT · Vrije Universiteit Amsterdam +3