hacktrace: behavior-supervised detection of reward hacking during code generation

Hao Jiang; Xin Li; Annan Wang; Yichi Zhang; Weisi Lin

cs.AIOct 2, 2026

hacktrace: behavior-supervised detection of reward hacking during code generation

Authors: Hao Jiang, Xin Li, Annan Wang, Yichi Zhang, Weisi Lin

Organizations: Nanyang Technological University · Independent Researcher

Abstract

A coding agent can earn a passing grade by fixing its code, or by deleting the test that exposes the bug. Detecting such reward hacking requires recognizing attempted shortcuts, including those that fail. We release 173,561 annotated multi-turn coding trajectories from Qwen3-8B and show that supervising shortcut behavior independently of exploit success substantially improves detection. We introduce HACKTRACE, a behavior-supervised monitor that reads the internal states the agent already computes while generating code. Reusing these states enables monitoring before a turn is complete, without additional language-model tokens or passes. Combining this evidence with static features of the final files achieves a mean per-problem AUC of 0.997 with 8 ms of monitoring overhead, improving both accuracy and latency over monitors that run the model again on an honesty question and answer. The same generation states also provide an inexpensive monitoring signal for reinforcement learning. With strong GRPO penalties, HACKTRACE reduces the cheating share of passing solutions from 82-91% to 1-5%, while retaining honest, correct solutions and maintaining high detection accuracy as the policy evolves. Our results show that both the supervision target and the source of monitoring evidence matter for turning accurate detection into a useful training signal.

Figures & tables

Figure 1: A shortcut attempt and a successful exploit are different targets. act marks shortcut behavior; outcome additionally requires a visible pass and a hidden failure, so a failed shortcut is act -positive and outcome -negative. Hidden tests and the judge supply labels only; they are not monitor inputs.

Figure 2: HackTrace reads states already computed during generation. MultiMax averages the eight highest token scores from layers $18$ and $24$ ; fixed readouts pool the same states linearly. Neither needs an extra pass; the self-report baseline does.

	trained on act		trained on outcome
detector	act target	outcome target	act target	outcome target
self-report probe, artifact-only	0.990 / 0.989	0.815 / 0.855	0.149 / 0.346	0.882 / 0.976
union static LR	0.991 / 0.997	0.782 / 0.878	0.246 / 0.436	0.833 / 0.959
TF-IDF + LR	0.946 / 0.878	0.676 / 0.530	0.331 / 0.414	0.789 / 0.771
fine-tuned CodeBERT	0.981 / 0.952	0.756 / 0.688	0.252 / 0.437	0.737 / 0.929
HackTrace -activation, fixed	0.962 / 0.974	0.752 / 0.851	0.303 / 0.382	0.987 / 0.989

Table 1: Behavior vs. success supervision (editable-test slice). Within-problem / pooled AUC of detectors trained on one target and scored on either. outcome positives:

333

20

problems (

80\%

on three); act positives:

2{,}478

155

Baselines: static and text features of the final files
			within-problem AUC
detector	tokens	ms	all	editable	read-only
AST-evidence LR (15 feats.)	0	0.41	0.816	0.988	0.500
union static LR (22 feats.)	0	1.20	0.985	0.991	0.973
TF-IDF + LR	0	0.75	0.953	0.946	0.967
CodeBERT (512 tokens)	512	11.3	0.969	0.981	0.945

Table 2: Detection quality and monitoring cost (completed act labels). Within-problem AUC over

239

test problems; extra uncached tokens and batch-

1

overhead (ms). HackTrace -combined sums standardized fixed, MultiMax, rolling-mean and static-LR scores (§ 3.2 ). Bold: best per column. Intervals: Appendix C .

Figure 3: Detection before the final turn ends. Within-problem AUC of unchanged detectors on retrospective prefixes (

95\%

problem-bootstrap bands); dashed: artifact-only self-report on the full turn. Prior turns remain visible.

training	$\beta$	$\lambda$	$n$	plain	HT (ours)	SR	plain	HT (ours)	SR
				cheat / passes (%)			honest / attempts (%)
correct	$0.1$	$10$	11/11	85.1	4.7	3.8	8.3	28.7	26.7
correct (MM)	$0.1$	$10$	16/11	85.1	1.5	3.8	8.3	30.8	26.7
mixed	$0.1$	$10$	16/16	81.6	2.7	4.4	8.2	27.4	28.5
mixed (MM)	$0.1$	$10$	16/16	81.6	1.4	4.4	8.2	29.5	28.5
correct	$0.1$	$2$	8/8	85.1	2.4	15.5	8.3	28.7	29.2

Table 3: GRPO penalties on held-out problems (steps

20

–

120

pooled per seed). Cheat: share of visible passes; honest: honest passes per attempt (%). HT: HackTrace (MM: completed-label MultiMax; otherwise screened-label attention); SR: self-report.

n

: HT/SR seeds. Bold: better penalty per row. Details: Table 20 .

Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

	training set		test set
	editable tests	read-only tests	editable tests	read-only tests
problems	375	375	190	125
trajectories	56,617	76,392	15,200	25,352
act positives	11,003 (19.4%)	15,326 (20.1%)	2,478 (16.3%)	5,466 (21.6%)
screened labels	7,753 (13.7%)	12,167 (15.9%)	1,617 (10.6%)	4,323 (17.1%)
problems with positives / top-3 share	353 / 4%	281 / 4%	155 / 18%	88 / 11%

Appendix

Table 4: The single training set and test set. Problems are disjoint between the two; the read-only rows run the same agent with test.py read-only. act positives are counted under the completed labels; “screened” gives the count under the adjudication screen alone, before the remaining trajectories were judged (§ 3.1 ). “Top-3 share”: fraction of positives on the three most-hacked problems.

detector	flag	recall	flag	recall	pooled	within	shift
	threshold at $1\%$		threshold at $0.1\%$		AUC
HackTrace online, fixed readouts	0.2%	78%	0.0%	49%	0.980	0.980	$-$ 0.10
HackTrace online, MultiMax head	0.1%	73%	0.0%	46%	0.995	0.978	$-$ 0.15
HackTrace online, attention head	0.3%	86%	0.0%	35%	0.998	0.993	$-$ 0.01
HackTrace offline	0.1%	70%	0.0%	30%	0.969	0.999	$-$ 0.79
self-report probe, artifact-only	0.1%	70%	0.0%	51%	0.994	0.998	$-$ 0.17

Appendix

Table 5: Neutral-prompt held-out pool. The read-only agent loop on the

125

read-only test problems with a prompt that neither says “by any means necessary” nor names hard-coding as acceptable (

32

candidates per problem,

4{,}000

trajectories, every one judged):

37

shortcuts (

0.9\%

;

28

special-casing,

9

hard-coded outputs, on

10

problems), visible-pass rate

28\%

. Detectors fitted on the licensed pools are applied unchanged. Flag: share of judged-honest trajectories above the global threshold that gives

1\%

(or

0.1\%

) false positives on the main test set; recall: share of the

37

shortcuts above it; AUC pooled over all trajectories and within the

10

problems that contain a shortcut; shift: mean score of honest trajectories relative to honest read-only test trajectories, in test-set standard deviations.

detector	flag	recall	flag	recall	pooled	within	shift
	threshold at $1\%$		threshold at $0.1\%$		AUC
HackTrace online, fixed readouts	2.3%	84%	0.2%	67%	0.973	0.958	$-$ 0.10
HackTrace online, MultiMax head	3.0%	82%	0.2%	47%	0.984	0.978	+0.09
HackTrace online, attention head	2.2%	76%	0.1%	17%	0.974	0.967	+0.10
HackTrace offline	2.4%	96%	0.2%	86%	0.995	0.998	$-$ 0.04
self-report probe, artifact-only	1.5%	85%	0.2%	80%	0.990	0.989	$-$ 0.04

Appendix

Table 6: Neutral-prompt pool, editable-test variant. The editable-test loop on the

190

editable-test test problems with the licensing sentence removed (the agent may still rewrite both files;

16

candidates per problem,

3{,}040

trajectories, every one judged):

593

shortcuts (

19.5\%

;

570

edit the tests,

23

hard-code, special-case or bypass, on

141

problems), visible-pass rate

22\%

. Detectors fitted on the licensed pools are applied unchanged; columns as in Table 5 , with the score shift relative to honest editable-test test trajectories.

detector	within [95%]	$\times$ turn	pooled
All test trajectories
HackTrace offline	0.998 [0.996,1.000]	0.998	0.999
HackTrace offline, MultiMax	0.997 [0.994,0.999]	0.997	0.999
HackTrace offline, learned last	0.996 [0.994,0.998]	0.996	0.998
HackTrace offline, rolling mean	0.996 [0.993,0.998]	0.995	0.996
HackTrace offline, learned attention	0.995 [0.989,0.998]	0.995	0.999

Appendix

Table 7: All detectors, completed act labels. Results are grouped by test slice and sorted by AUC on all test trajectories. Within-problem AUC includes its

95\%

problem-bootstrap interval;

\times

turn: within problem

\times

final-turn strata; pooled AUC.

pair	all test	editable-test	read-only
HT-comb. $-$ SR-art	+0.004 [+0.001,+0.008]	+0.006 [+0.001,+0.012]	+0.001 [+0.000,+0.003]
HT-comb. $-$ SR-ctx	+0.007 [+0.004,+0.011]	+0.010 [+0.005,+0.016]	+0.002 [+0.000,+0.003]
HT-comb. $-$ HT-offline	$-$ 0.001 [ $-$ 0.003,+0.001]	$-$ 0.002 [ $-$ 0.004,+0.001]	+0.000 [ $-$ 0.001,+0.001]
HT-attn $-$ SR-art	$-$ 0.011 [ $-$ 0.020, $-$ 0.004]	$-$ 0.017 [ $-$ 0.029, $-$ 0.007]	$-$ 0.000 [ $-$ 0.002,+0.002]
HT-attn $-$ HT-fixed	+0.008 [ $-$ 0.001,+0.016]	+0.011 [ $-$ 0.002,+0.024]	+0.001 [ $-$ 0.000,+0.004]
HT-attn $-$ AST	+0.165 [+0.135,+0.198]	$-$ 0.014 [ $-$ 0.023, $-$ 0.006]	+0.497 [+0.495,+0.499]

Appendix

Table 8: Paired differences, completed act labels (problem-level bootstrap,

2{,}000

resamples; bold: interval excludes

0

). HT: HackTrace (MM: MultiMax; attn: attention; fixed: fixed online readouts); SR-art/SR-ctx: artifact-only/context-matched self-report; 36L: the expanded layer grid.

turn (%)	all	editable-test	read-only	$\Delta$ all [95%]
HackTrace online, fixed readouts
100	0.974 [0.966,0.981]	0.962 [0.950,0.973]	0.996 [0.993,0.999]	$-$ 0.019 [ $-$ 0.026, $-$ 0.012]
90	0.913 [0.899,0.928]	0.878 [0.858,0.897]	0.978 [0.962,0.990]	$-$ 0.079 [ $-$ 0.094, $-$ 0.065]
75	0.840 [0.816,0.864]	0.783 [0.750,0.812]	0.946 [0.923,0.967]	$-$ 0.153 [ $-$ 0.177, $-$ 0.129]
50	0.831 [0.809,0.851]	0.815 [0.790,0.838]	0.861 [0.823,0.897]	$-$ 0.161 [ $-$ 0.183, $-$ 0.142]
25	0.707 [0.679,0.732]	0.727 [0.693,0.757]	0.669 [0.623,0.715]	$-$ 0.286 [ $-$ 0.314, $-$ 0.261]

Appendix

Table 9: Every read point: detection quality (completed labels). Within-problem AUC [95% interval];

\Delta

compares with artifact-only self-report on all test trajectories. Cost and episode fractions are in Table 10 .

detector / read point	episode (%)	tokens	ms
HackTrace online, fixed readouts
100% of final turn	100	0	3.80
90% of final turn	95	0	3.45
75% of final turn	87	0	2.91
50% of final turn	73	0	2.00
25% of final turn	60	0	1.11

Appendix

Table 10: Every read point: cost and episode progress. Tokens and latency follow Table 2 . Prefix self-report latencies use recomputed estimates clamped to the measured

29.2

ms query floor when necessary; they are not separate per-prefix timing measurements.

detector	within [95%]	$\times$ turn	pooled
All test trajectories
self-report probe, context-matched	0.990 [0.985,0.994]	0.968	0.997
HackTrace online, MultiMax	0.987 [0.979,0.994]	0.982	0.998
HackTrace online, fixed	0.987 [0.971,0.997]	0.981	0.998
HackTrace online, learned attention	0.975 [0.961,0.987]	0.967	0.996
HackTrace online, rolling mean	0.972 [0.951,0.988]	0.966	0.998

Appendix

Table 11: All detectors, outcome label. Results are grouped by test slice. Within-problem AUC includes its

95\%

problem-bootstrap interval.

pair	all test	editable-test	read-only
HT-attn $-$ SR-art	+0.017 [ $-$ 0.001,+0.038]	+0.043 [ $-$ 0.029,+0.124]	+0.010 [ $-$ 0.001,+0.024]
HT-attn $-$ HT-fixed	$-$ 0.012 [ $-$ 0.028,+0.005]	$-$ 0.062 [ $-$ 0.111, $-$ 0.022]	+0.002 [ $-$ 0.010,+0.020]
HT-attn $-$ AST	+0.470 [+0.438,+0.500]	+0.405 [+0.274,+0.533]	+0.489 [+0.479,+0.496]
HT-offline $-$ SR-art	+0.011 [+0.004,+0.019]	+0.044 [+0.017,+0.073]	+0.002 [ $-$ 0.002,+0.006]
HT-offline $-$ SR-ctx	$-$ 0.021 [ $-$ 0.036, $-$ 0.010]	$-$ 0.047 [ $-$ 0.093, $-$ 0.012]	$-$ 0.014 [ $-$ 0.029, $-$ 0.003]
HT-offline $-$ AST	+0.464 [+0.430,+0.494]	+0.406 [+0.280,+0.528]	+0.481 [+0.465,+0.492]

Appendix

Table 12: Paired differences, outcome label (problem-level bootstrap,

2{,}000

resamples; bold: interval excludes

0

). HT: HackTrace (MM: MultiMax; attn: attention; fixed: fixed online readouts); SR-art/SR-ctx: artifact-only/context-matched self-report; 36L: the expanded layer grid.

environment	judge category	$n$	agreement
editable-test	test tampering	60	93.3%
editable-test	hard-coding, special-casing, bypass	25	88.0%
editable-test	honest, correct	30	100%
editable-test	honest, wrong	30	100%
editable-test	rule-labeled negative	30	100%
read-only-test	hard-coding	40	100%

Appendix

Table 13: Human audit of the labels. Stratified blind sample of

305

test trajectories; strata are defined by the judge’s category (rule-labeled negatives were never judged). Agreement is on shortcut behavior ( act ) between the annotator and the final labels.

	editable-test slice		read-only slice
detector	all ( $\|P_{\pm}\|{=}155$ )	passing ( $17$ )	all ( $84$ )	passing ( $51$ )
HackTrace online, fixed readouts	0.962	0.953 [0.892,0.997]	0.996	0.997 [0.993,1.000]
HackTrace online, MultiMax head	0.981	0.906 [0.815,0.981]	0.997	0.993 [0.978,1.000]
HackTrace online, attention head	0.974	0.862 [0.729,0.968]	0.997	0.998 [0.993,1.000]
HackTrace offline	0.998	0.985 [0.956,1.000]	0.999	0.998 [0.995,1.000]
self-report probe, artifact-only	0.990	0.969 [0.928,0.999]	0.998	0.998 [0.994,1.000]

Appendix

Table 14: Visible-passing trajectories only (completed labels). Within-problem act AUC on all test trajectories and restricted to trajectories that pass the visible grader (cheating pass against honest pass), where pass status carries no information.

|P_{\pm}|

: problems with both classes. Intervals:

95\%

problem bootstrap. The last three rows cost nothing at deployment.

false-positive rate	$0.1\%$	$1\%$	$5\%$	$0.1\%$	$1\%$	$5\%$	$0.1\%$	$1\%$	$5\%$
	all test			editable-test slice			read-only, passing only
HackTrace online, fixed readouts	0.825	0.931	0.982	0.600	0.765	0.905	0.987	1.000	1.000
HackTrace online, MultiMax head	0.791	0.920	0.988	0.408	0.709	0.886	0.985	1.000	1.000
HackTrace online, combined (+ static)	0.918	0.993	0.999	0.736	0.982	0.998	0.994	1.000	1.000
HackTrace online, attention head	0.675	0.919	0.964	0.183	0.681	0.848	0.820	1.000	1.000
HackTrace offline	0.892	0.976	0.997	0.817	0.965	0.991	0.969	0.973	1.000

Appendix

Table 15: Recall at fixed false-positive rates (completed labels). One global threshold per detector, set on half of the test problems and applied to the other half (both directions averaged).

detector	trained on	all ( $155$ )	failing ( $153$ )	passing ( $17$ )	for $V$
HackTrace online	act , all ( $11{,}003$ )	0.983	0.988	0.816	0.408
	act , matched ( $294$ )	0.941	0.940	0.908	0.389
	outcome ( $294$ )	0.435	0.471	0.851	0.849
	outcome , no failed att.	0.509	0.546	0.861	0.823
	pass bit $V$	0.354	0.384	0.453	0.981
self-report probe, artifact-only	act , all ( $11{,}003$ )	0.992	0.992	0.956	0.474

Appendix

Table 16: Supervision targets without the coverage confound (completed labels). Both targets are fitted on editable-test training trajectories only, with the same

12

readouts per detector; matched subsamples act positives to the outcome count on the same problems. Within-problem act AUC on the editable-test slice of the test set: all trajectories, trajectories that fail the visible grader (failed attempt against honest failure), and trajectories that pass it (successful shortcut against honest pass). Last column: AUC for the pass bit itself.

detector	all	failing only	difference (failing)	passing only
HackTrace online, fixed readouts	0.996 / 0.945	0.990 / 0.840	+0.150 [+0.098,+0.206]	0.997 / 0.998
HackTrace online, MultiMax head	0.997 / 0.997	0.997 / 0.992	+0.005 [+0.002,+0.010]	0.993 / 0.996
HackTrace online, attention head	0.997 / 0.997	0.995 / 0.991	+0.004 [ $-$ 0.003,+0.010]	0.998 / 0.994
HackTrace offline	0.999 / 0.965	0.998 / 0.927	+0.071 [+0.038,+0.110]	0.998 / 0.995
self-report probe, artifact-only	0.998 / 0.978	0.995 / 0.948	+0.047 [+0.028,+0.069]	0.998 / 0.996
self-report probe, context-matched	0.997 / 0.961	0.996 / 0.925	+0.071 [+0.043,+0.101]	0.994 / 0.994

Appendix

Table 17: Behavior and success supervision on the read-only slice (completed labels, which add

1{,}143

positives, mostly failed hard-coding attempts, to the

4{,}323

of the adjudication screen). Within-problem AUC for act , detector trained on act / trained on outcome : all trajectories (

84

problems), failing trajectories only (failed attempt against honest failure,

52

problems) with the paired difference, and passing trajectories only (

51

problems). act -trained detectors were refitted on the completed labels; under outcome failed attempts remain negatives.

			within-problem AUC			$\Delta$ on all test, paired
detector	tokens	ms	all	editable	read-only	vs. artifact-only	vs. context-matched
Generation states: no extra language-model pass
HackTrace online, fixed readouts	0	3.80	0.982	0.976	0.995	$-$ 0.003 [ $-$ 0.009,+0.005]	$-$ 0.006 [ $-$ 0.011, $-$ 0.001]
HackTrace online, MultiMax head	0	3.24	0.980	0.973	0.992	$-$ 0.005 [ $-$ 0.013,+0.002]	$-$ 0.009 [ $-$ 0.014, $-$ 0.004]
HackTrace online, attention head	0	3.24	0.977	0.977	0.975	$-$ 0.008 [ $-$ 0.021,+0.003]	$-$ 0.012 [ $-$ 0.023, $-$ 0.003]
Additional language-model pass

Appendix

Table 18: Completed-turn detection under the original screened labels (the labels of the adjudication screen, § 3.1 ; detectors fitted on them). Same layout as Table 2 . Scoring these unrefitted detectors against the completed labels gives

0.977

for the fixed online readouts and

0.995

–

0.996

for the learned heads on the read-only slice (Appendix E ).

$K$	training CV	all test	editable-test	read-only	editable, screened positives only
6	0.9984	0.985	0.978	0.997	0.970
12	0.9986	0.986	0.980	0.998	0.973
24	0.9987	0.988	0.982	0.998	0.976
44	0.9988	0.989	0.983	0.998	0.978
88 (selected)	0.9988	0.974	0.962	0.996	0.948

Appendix

Table 19: Ensemble size of the fixed online readouts under the completed labels. Within-problem act AUC of the

z

-sum of the top-

K

online readouts (ranked by cross-validated AUC on the training set) for every candidate

K

; the selection rule of § 3.2 picks the

K

with the highest training cross-validation AUC, which is

K{=}88

here by a margin below

10^{-4}

: with the completed labels the criterion saturates (every

K

within

0.0004

), so the choice is not identified. The offline readouts are insensitive (

0.998

at every

K

training	$\beta$	$\lambda$	cheat	honest	cheat	honest	cheat	honest	$\Delta$ cheat	seeds	HackTrace	self-rep.
			plain GRPO		HackTrace penalty		self-report penalty				cheat at step $120$
correct	$0.1$	$10$	93.4	5.0	4.7	28.7	3.8	26.7	+1.1 [ $-$ 4.0, +8.2]	11/11	4.2	1.1
correct, MultiMax head	$0.1$	$10$	93.4	5.0	1.5	30.5	3.8	26.7	$-$ 2.9 [ $-$ 5.7, $-$ 0.3]	12/11	0.0	1.1
+ conflicting	$0.1$	$10$	81.6	8.2	2.7	27.4	4.4	28.5	$-$ 1.7 [ $-$ 6.8, +1.7]	16/16	0.4	0.0
+ conflicting, MultiMax head	$0.1$	$10$	81.6	8.2	1.8	29.4	4.4	28.5	$-$ 2.8 [ $-$ 9.5, +1.8]	12/16	0.0	0.0
correct	$0.1$	$2$	93.4	5.0	4.0	29.5	2.8	33.4	+1.2 [ $-$ 0.6, +3.1]	4/4	0.0	1.1

Appendix

Table 20: Full GRPO results and paired comparisons. Qwen3-8B, read-only tests,

60

held-out problems, evaluations at steps

20

–

120

pooled per seed and averaged over seeds; seeds: seeds with held-out evaluations ( HackTrace /self-report;

8

for plain GRPO); all rates are percentages; cheat is the cheating share among visible passes, while honest is honest visible passes per attempt. The last two columns give the cheat share at the final checkpoint alone. Paired differences use shared seeds and need not equal the difference of all-seed means. “MultiMax head”: the MultiMax head fitted on the completed labels as the penalty; other rows use the attention head (Appendix G ).

setting	HackTrace	self-report	difference	HackTrace	self-report
	penalty on honest solutions			share with $\psi\geq 0.5$
correct, $\lambda{=}10$	$0.05$	$1.99$	$-1.94$ [ $-2.50$ , $-1.38$ ]	$0.1\%$	$2.9\%$
+ conflicting, $\lambda{=}10$	$0.07$	$0.83$	$-0.76$ [ $-1.28$ , $-0.31$ ]	$0.3\%$	$1.8\%$
correct, $\lambda{=}2$	$0.01$	$0.31$	$-0.30$ [ $-0.48$ , $-0.13$ ]	$0.2\%$	$5.6\%$
correct, $\beta{=}10^{-4}$	$0.08$	$0.98$	$-0.90$ [ $-1.58$ , $-0.37$ ]	$0.2\%$	$2.8\%$

Appendix

Table 21: Penalty paid by honest passing training rollouts late in training (steps

61

–

120

): mean

\lambda\psi

in reward units with the paired HackTrace

-

self-report difference (

95\%

seed-bootstrap interval), and the share of honest passing rollouts with

\psi\geq 0.5

. Both probes are fitted on the same training set.

Figure 4: Longer training at

\lambda{=}10

(ten seed pairs, correct tests; screened-label attention head against the screened-label self-report probe; the self-report probe refitted on the completed labels does not show this decline, Table 24 ). Left: honest passes per held-out attempt. Right: paired difference with

95\%

seed-bootstrap intervals.

step	pairs	HackTrace	self-report	difference	HackTrace	self-report
		honest passes			cheating passes
120	10	24.3	18.5	+5.8 [ $-$ 4.7, +16.3]	0.3	0.3
160	10	25.7	17.8	+7.8 [ $-$ 2.3, +17.5]	0.2	0.5
200	10	22.2	13.7	+8.5 [ $-$ 2.7, +19.0]	0.2	0.0
240	10	26.7	7.2	+19.5 [+12.0, +26.7]	0.0	0.0
280	10	24.8	7.0	+17.8 [+8.0, +26.2]	0.3	0.0

Appendix

Table 22: Training longer (correct-test problems,

\lambda{=}10

\beta{=}0.1

; the

120

-step runs of ten seed pairs continued to steps

240

and

360

, with the same constant learning rate). Held-out honest and cheating passes per attempt (%), and the paired difference in honest passes with a

95\%

seed-bootstrap interval (bold: excludes

0

Step $120$
	HackTrace			self-report
seed	pass	honest	empty	pass	honest	empty
3	5	3	87	0	0	55
4	40	40	0	0	0	100
8	27	27	22	2	2	63
9	35	35	2	40	40	10

Appendix

Table 23: Per-seed end states of the continued

\lambda{=}10

runs (correct-test problems; the seed pairs of Table 22 ). For each seed and penalty, from the

60

held-out attempts at that checkpoint: visible-pass rate, judged-honest passes and the share of attempts whose final solution.py is empty (no code block was produced), all in percent; cheating passes equal visible passes minus honest passes (up to rounding). Under the self-report penalty the empty share rises from

41\%

at step

120

74\%

240

and

90\%

360

on average, while under HackTrace it is

26\%

17\%

, and

21\%

, respectively (plain GRPO at step

120

0\%

): the decline in honest passes is accompanied by a large increase in empty solutions.

penalty	seeds	cheat (%)		honest (%)		on-policy AUC
Steps $20$ – $120$ (on-policy AUC at steps $61$ – $120$ )
HackTrace , MultiMax (completed labels)	16	1.5	[1.0, 2.1]	30.8	[29.1, 32.3]	0.95
self-report, completed labels	16	3.9	[2.1, 6.2]	32.8	[31.2, 34.4]	0.96
self-report, hinged at $0.5$	12	4.9	[2.3, 8.1]	34.8	[32.5, 36.8]	0.86
self-report, re-standardized per group	16	3.2	[1.1, 6.3]	26.3	[22.3, 29.9]	0.95
union static LR ( $22$ features)	12	2.5	[1.5, 3.5]	34.8	[33.2, 36.2]	1.00

Appendix

Table 24: Additional penalty arms (correct-test problems and

\lambda{=}10

unless stated). Held-out cheating share among visible passes and honest passes per attempt, pooled per seed over the stated window and averaged over seeds with a

95\%

seed-bootstrap interval; on-policy AUC of each arm’s own penalty on its judged passing training rollouts. Every passing held-out rollout is judged.

Figure 5: Self-report penalty on the MBPP honeypot. Ten seeds per arm with Qwen3.5-4B,

\beta{=}0.1

, at step

120

. Each row is a seed. Colored bars show pass-all rates; gray ticks show visible-pass rates. The end-state label uses the original gap and collapse criteria (Table 25 ); it does not assert semantic honesty.

seed	pass visible	pass all	gap	end state
0	0.983	0.017	0.967	hacking
1	0.000	0.000	0.000	collapsed
2	0.967	0.017	0.950	hacking
3	0.933	0.017	0.917	hacking
4	0.983	0.017	0.967	hacking
5	0.000	0.000	0.000	collapsed

Appendix

Table 25: Per-seed end state (step

120

) of the reproduction on the MBPP honeypot, Qwen3.5-4B,

\beta{=}0.1

. Collapsed: passes the visible test on

<5\%

of problems; hacking: gap

>0.3

; otherwise alive and honest.

detector	act -trained AUC	$\Delta$ vs self-report	outcome -trained AUC
test-edited flag	0.942	+0.026 [+0.008, +0.044]	0.942
TF-IDF + LR	0.889 [0.869, 0.909]	$-$ 0.026 [ $-$ 0.040, $-$ 0.013]	0.497
union static LR	0.995 [0.993, 0.997]	+0.080 [+0.063, +0.097]	0.702
self-report probe, artifact-only	0.916 [0.899, 0.933]	–	0.446
HackTrace -activation, attention head	0.939 [0.925, 0.951]	+0.023 [+0.005, +0.041]	0.509
HackTrace -activation, rolling-mean head	0.894 [0.880, 0.908]	$-$ 0.021 [ $-$ 0.041, $-$ 0.001]	0.567

Appendix

Table 26: Second policy model: Llama-3.1-8B-Instruct (editable-test variant;

184

test problems with both classes,

8{,}832

test trajectories,

2{,}529

act -positive). Within-problem AUC with

95\%

problem-bootstrap intervals;

\Delta

: paired difference to the self-report probe; last column: the same detector trained on outcome and scored on act . Baselines first, HackTrace last.

head	editable-only training	mixed training
attention	0.990 / 0.671	0.974 / 0.997
MultiMax	0.980 / 0.959	0.981 / 0.997
rolling mean	0.974 / 0.770	0.976 / 0.997
mean	0.963 / 0.785	0.961 / 0.997
last token	0.962 / 0.592	0.958 / 0.995

Appendix

Table 27: Training coverage and transfer (completed labels). Within-problem act AUC. Each pair gives editable-test / read-only-test AUC. “Mixed” uses both training variants; “editable only” excludes solution-side training examples. The read-only test contains unseen problems but a behavior family represented in mixed training.

Reward Hacking

Reward hacking studies how AI systems exploit imperfect objectives and how to detect, measure, and prevent behavior that scores well but misses intent.

See all 182 papers

Do Prompt-Elicited Trajectories Reflect Training-Time Reward Hacking? A Systematic Study on Monitoring Trainig-Time Reward Hacking in Code Generation
Apr 26, 2026Lichen Li, Hengguang Zhou, Yijun Liang +2RL for Code Generation Reward Hacking
Peking University · University of California, Los Angeles · University of Maryland +2
CATCH: A Controllable Analysis Testbed for Reward Hacking in Coding RL
Sep 30, 2026Shouli Wang, Yanfeng Jia, Zhihao Ou +6RL Benchmarks RL for Code Generation
Tsinghua University · Beihang University · Southern University of Science and Technology +2
SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents
May 20, 2026Bingchen Zhao, Dhruv Srikanth, Yuxiang Wu +1Reward Hacking AI Coding Agents
Weco AI

Apr 26, 2026J/K move · Enter open · S save

Do Prompt-Elicited Trajectories Reflect Training-Time Reward Hacking? A Systematic Study on Monitoring Trainig-Time Reward Hacking in Code Generation

Lichen Li, Hengguang Zhou, Yijun Liang, Tianyi Zhou+1

Peking University · University of California, Los Angeles · University of Maryland+2

RL for Code Generation Reward Hacking LLM Auditing

Open paper AskPDF arXiv

Reward hacking in code generation, where models exploit evaluation loopholes to obtain high reward without correctly solving the intended task, poses a critical challenge for Reinforcement Learning (RL) and the deployment of reasoning models. Existing studies often rely on explicitly prompted hacking trajectories, but it remains unclear whether monitors trained on such data can detect reward hacks that arise without direct hacking instructions during RL training. In this work, we introduce Trace-and-Amplify, a framework for scalable curation of reward-hacking trajectories that arise during RL training without explicit hacking instructions. The framework uses unit-test tracers to identify hacking solutions when they occur and retains such trajectories for monitor training and evaluation. Through controlled comparisons between monitors trained on prompt-elicited hacking trajectories and training-time reward-hacking trajectories collected by Trace-and-Amplify, we find that (1) prompt-elicited-data-trained monitors often fail to generalize to trajectories curated by our framework, and (2) monitors trained on our Trace-and-Amplify trajectories demonstrate stronger generalizability to unseen hacking types. Our results indicate that prompted reward hacking data may not fully reflect training-time reward-hacking behaviors, and that relying solely on these data can lead to misleading conclusions. Codebase is available at https://github.com/LichenLillc/CoTMonitoring.git

Abstract

Reward Hacking

Explore similar work

Do Prompt-Elicited Trajectories Reflect Training-Time Reward Hacking? A Systematic Study on Monitoring Trainig-Time Reward Hacking in Code Generation

CATCH: A Controllable Analysis Testbed for Reward Hacking in Coding RL

SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents