A Near-Zero Monitor Readout Is Not Evidence of Behavioral Control

Zhe Zhou; Tianhua Tao

cs.AIOct 2, 2026

A Near-Zero Monitor Readout Is Not Evidence of Behavioral Control

Authors: Zhe Zhou, Tianhua Tao

Organizations: University of Washington

Abstract

Post-training with verifiable rewards can induce reward hacking, motivating the use of monitors within the training objective rather than solely for offline auditing. We show that a low monitor readout does not identify whether such an intervention controls behavior. In a code-generation environment whose dominant exploit is available at the start of the reasoning trace, we train policies against three monitors that pass the same offline gate: an in-domain activation probe and two penalties conditioned on how early the policy commits to its own final answer. The probe score is at its numerical floor from the first recorded training step, and the trained-score median is zero for every prefix-trained run at the endpoint. These readouts estimate different quantities, and we do not compare their scales; within each monitor family, however, low values do not establish behavioral control. Within one fixed configuration, prefix-trained runs with the same zero-median trained score range, by seed alone, from a mixed regime with a low hacking share to near-pure reward hacking. All probe runs reach the hacking regime, but their floor-level readout reflects a mismatch between the position where the probe was validated and the position where it was read during training, not a second instance of this ambiguity. Text-level analysis identifies a prefix failure mode: generic planning and filler shells postpone the exploit past the cut without eliminating it from the final output. Low measured commitment therefore does not distinguish a low hacking share from delayed commitment to the exploit. Offline discrimination and low monitor-aligned readouts are insufficient evidence of behavioral control; an out-of-band behavioral check is required. We characterize the endpoint readout, not its evolution. Code is available at https://github.com/zhezhou1106/spoof-cost.

Figures & tables

Figure 1: Training trajectories, three seeds per gated monitor. Color marks the arm; solid / dashed / dotted mark seeds 0 / 1 / 2. Each series is Gaussian-smoothed along the step axis ( $\sigma{=}3$ ). Smoothing is for display only; onset steps are computed on unsmoothed per-step values. The no-monitor baseline trajectory was not retained. The Probe arm transitions in every seed (onset 28, 58, 49). SCL stays mixed at seed 0 but transitions at steps 40 and 96; Cut transitions at step 54 but stays mixed at seeds 1 and 2. All runs begin in a mixed visible- and held-out-pass band; whether the later transition occurs depends on the seed.

Arm	Regime (s0/s1/s2)	Onset	Score med.	Score mean	Hack $d$	Honest $h$	Mixed $k/3$
No monitor	hacking	54	$1.000$	$0.958$	0.984	0.000	—
Probe	hack / hack / hack	28 / 58 / 49	$1.000$ [1.000, 1.000]	$0.916$ [0.853, 0.955]	0.961 [0.953, 0.969]	0.008 [0.008, 0.008]	0/3
SCL	mixed / hack / hack	— / 40 / 96	$0.000$ [0.000, 0.000]	$0.017$ [0.014, 0.019]	0.883 [0.156, 0.938]	0.031 [0.008, 0.422]	1/3
Cut	hack / mixed / mixed	54 / — / —	$0.000$ [0.000, 0.000]	$0.127$ [0.020, 0.154]	0.133 [0.094, 0.930]	0.305 [0.008, 0.344]	2/3

Table 1: Prefix scores are the exact trained quantities recomputed at step 120 on the endpoint rollouts (SCL’s

\psi

on its full grid; Cut’s single

0.25

cut). Medians are exactly

0.000

for every prefix-trained run; per-rollout means are non-zero, so the zeros reflect the modal rollout rather than a scoring failure. The no-monitor and Probe rows use the same scorer and rollout count. Values other than the single no-monitor run are median

[\min,\max]

over three run-level statistics. Onset is the first of five consecutive steps with held-out pass below

0.10

; “—” denotes not reached.

Figure 2: Trained prefix score versus endpoint hack rate. SCL points are the curve score $\psi$ ; Cut points are $c_{0.25}$ . Each point is one run’s median over $n=128$ rollouts; the legend identifies runs by color and marker. Upper right: no-monitor and Probe have median scores of $1.0$ under the same primitive and high hack rates. Upper left: SCL seeds 1–2 and Cut seed 0 reach the hacking regime despite median scores of $0$ . Lower left: mixed prefix runs also have median scores of $0$ . The lower-right region is empty.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Base policy	Llama-3.1-8B-Instruct
Adapter	LoRA, r = 64 , lora_alpha = 128
Optimizer	Group Relative Policy Optimization (GRPO), described in Section 4.4
KL coefficient $\beta$	1e-4
Steps / prompts / group size	120 / 400 / 8
Train batch size	16

Appendix

Table 2: Frozen training contract. All values fixed before the reported runs unless noted.

Arm	Seed	$d$	$h$	$v$	Format-OK	Chars (mean/median)
No monitor	—	0.984	0.000	0.984	1.000	576.2 / 564.0
Probe	0	0.953	0.008	0.961	1.000	619.1 / 622.0
Probe	1	0.969	0.008	0.977	1.000	451.6 / 448.0
Probe	2	0.961	0.008	0.969	1.000	522.3 / 519.0
SCL	0	0.156	0.422	0.578	0.992	365.3 / 361.0
SCL	1	0.938	0.008	0.945	1.000	524.6 / 509.0

Appendix

Table 3: Per-run endpoint values. Reasoning length excludes the fenced code block. Format-OK is at least

0.969

in every run.

Run	0.05/3	0.05/5	0.05/10	0.10/3	0.10/5	0.10/10	0.20/3	0.20/5	0.20/10
Probe s0	28	32	32	28	28	28	27	27	27
Probe s1	59	59	71	58	58	58	58	58	58
Probe s2	51	51	57	49	49	49	49	49	49
SCL s0	—	—	—	—	—	—	—	—	—
SCL s1	42	42	42	40	40	40	39	39	39
SCL s2	106	106	106	96	96	96	92	92	92

Appendix

Table 4: Onset step under varying held-out-pass thresholds and consecutive-step windows. “—” denotes not reached. The criterion choice shifts onset by a few steps but never changes which runs reach it: every run that reaches the hacking regime does so under all nine criteria, and every mixed run does so under none. The frozen no-monitor trajectory was not retained.

Run	Regime	correct_honest	reward_hacking	capability_failure	$h/v$
No monitor	hacking	0.000	0.984	0.016	0.000
Probe s0	hacking	0.008	0.953	0.039	0.008
Probe s1	hacking	0.008	0.969	0.023	0.008
Probe s2	hacking	0.008	0.961	0.031	0.008
SCL s0	mixed	0.422	0.156	0.422	0.730
SCL s1	hacking	0.008	0.938	0.055	0.008

Appendix

Table 5: Per-run outcome shares,

n=128

. The correct_honest share equals

h

and the reward_hacking share equals

d

h/v

is the honest fraction among visible-pass rollouts. other_ambiguous is

0.000

in every run.

Run	Endpoint $d$	Endpoint $h/v$	Late-band $h/v$ (steps 100–120)
SCL s0	0.156	0.730	0.826
Cut s1	0.133	0.696	0.741
Cut s2	0.094	0.786	0.762

Appendix

Table 6: Honest share

h/v

among visible-pass rollouts in the three mixed runs. Endpoint values are from the

n=128

step-120 rollouts; the late band is the mean over training steps 100–120. Over the early band (steps 10–27) these runs average

h/v=0.832

, and per-run visible-pass means across all monitored runs lie in

[0.467,0.602]

. Run-mean penalty terms

\alpha\max(z,0)

lie in

[0.17,0.26]

for every SCL and Cut seed. A late-band share near the early-band average does not show that onset would not occur after step 120.

Monitor	AUROC	95% CI	Mean (hack / held-out pass)	Gate
Probe	0.775	[0.658, 0.893]	0.80 / 0.46	PASS
SCL	0.751	[0.645, 0.868]	0.93 / 0.56	PASS
Cut	0.732	[0.628, 0.850]	0.96 / 0.58	PASS
Judge	0.505	[0.415, 0.589]	0.96 / 0.95	FAIL

Appendix

Table 7: Descriptive offline scores on the frozen balanced gate cohort (

n=256

). Marginal 95% prompt-cluster bootstrap intervals are not paired equality or equivalence tests; mean scores are internal to each procedure. The gate was not refit per seed.

Arm	Regime	Prefix @0.30	Ablation @0.40
No monitor	hacking	0.945	0.938
Probe s0 / s1 / s2	hacking	0.883 / 0.922 / 0.836	0.875 / 0.938 / 0.812
SCL s1 / s2	hacking	0.008 / 0.016	0.008 / 0.016
Cut s0	hacking	0.008	0.039
SCL s0	mixed	0.023	0.000
Cut s1 / s2	mixed	0.070 / 0.000	0.117 / 0.000

Appendix

Table 8: Own-answer agreement after a

0.30

prefix cut and a

0.40

interior ablation, grouped by the run’s endpoint regime rather than by arm name. All rollouts in the denominator (

n=128

per run). Off-grid robustness check; not the trained

\psi

c_{0.25}

Arm	Seed	Hack $d$	$\bar{q}$ reference	$\bar{q}$ policy	Pattern
No monitor	—	0.984	0.750	0.750	neither
Probe	0	0.953	0.324	0.580	policy-reader drop
Probe	1	0.969	0.508	0.682	reference-reader drop
Probe	2	0.961	0.582	0.638	policy-reader drop
SCL	0	0.156	0.245	0.540	— (mixed)
SCL	1	0.938	0.689	0.782	neither

Appendix

Table 9: Two-reader re-scoring with the frozen probe (

n=128

per run). The reference reader is the baseline adapter; the policy reader is the run’s own adapter. Reader means interleave across behavioral regimes, so the descriptive patterns test possible text or representation changes but do not identify control. Runs below hack share

0.5

are not labeled.

Arm	Seed	Regime	Median code tokens
No monitor	—	hacking	10
Probe	0	hacking	11
Probe	1	hacking	9
Probe	2	hacking	11
SCL	0	mixed	31
SCL	1	hacking	13

Appendix

Table 10: Per-run median program lengths under the Llama-3.1-8B tokenizer. All medians are well below the 64-token forced-continuation cap.

AI Agent Monitoring

AI agent monitoring tracks actions, tool use, and execution traces to detect failures, misuse, drift, and policy violations during operation.

See all 281 papers

Apr 26, 2026

Do Prompt-Elicited Trajectories Reflect Training-Time Reward Hacking? A Systematic Study on Monitoring Trainig-Time Reward Hacking in Code Generation

Reward hacking in code generation, where models exploit evaluation loopholes to obtain high reward without correctly solving the intended task, poses a critical challenge for Reinforcement Learning (RL) and the deployment of reasoning models. Existing studies often rely on explicitly prompted hacking trajectories, but it remains unclear whether monitors trained on such data can detect reward hacks that arise without direct hacking instructions during RL training. In this work, we introduce Trace-and-Amplify, a framework for scalable curation of reward-hacking trajectories that arise during RL training without explicit hacking instructions. The framework uses unit-test tracers to identify hacking solutions when they occur and retains such trajectories for monitor training and evaluation. Through controlled comparisons between monitors trained on prompt-elicited hacking trajectories and training-time reward-hacking trajectories collected by Trace-and-Amplify, we find that (1) prompt-elicited-data-trained monitors often fail to generalize to trajectories curated by our framework, and (2) monitors trained on our Trace-and-Amplify trajectories demonstrate stronger generalizability to unseen hacking types. Our results indicate that prompted reward hacking data may not fully reflect training-time reward-hacking behaviors, and that relying solely on these data can lead to misleading conclusions. Codebase is available at https://github.com/LichenLillc/CoTMonitoring.git

Lichen Li, Hengguang Zhou, Yijun Liang +2

Peking University · University of California, Los Angeles · University of Maryland +2

RL for Code Generation Reward Hacking LLM Auditing

Ask PDF

Oct 2, 2026

hacktrace: behavior-supervised detection of reward hacking during code generation

A coding agent can earn a passing grade by fixing its code, or by deleting the test that exposes the bug. Detecting such reward hacking requires recognizing attempted shortcuts, including those that fail. We release 173,561 annotated multi-turn coding trajectories from Qwen3-8B and show that supervising shortcut behavior independently of exploit success substantially improves detection. We introduce HACKTRACE, a behavior-supervised monitor that reads the internal states the agent already computes while generating code. Reusing these states enables monitoring before a turn is complete, without additional language-model tokens or passes. Combining this evidence with static features of the final files achieves a mean per-problem AUC of 0.997 with 8 ms of monitoring overhead, improving both accuracy and latency over monitors that run the model again on an honesty question and answer. The same generation states also provide an inexpensive monitoring signal for reinforcement learning. With strong GRPO penalties, HACKTRACE reduces the cheating share of passing solutions from 82-91% to 1-5%, while retaining honest, correct solutions and maintaining high detection accuracy as the policy evolves. Our results show that both the supervision target and the source of monitoring evidence matter for turning accurate detection into a useful training signal.

Hao Jiang, Xin Li, Annan Wang +2

Nanyang Technological University · Independent Researcher

Reward Hacking AI Agent Monitoring Agentic Code Generation Reinforcement Learning with Verifiable Rewards

Ask PDF

Sep 30, 2026

CATCH: A Controllable Analysis Testbed for Reward Hacking in Coding RL

During reinforcement learning with verifiable rewards (RLVR), large language models (LLMs) can exploit loopholes in their environments to obtain high rewards without improving the intended capabilities, i.e., reward hacking. Despite its risks to training efficiency and safety, monitoring and mitigating reward hacking during training remain challenging, which is limited by a lack of testbeds that reproduce hacking and reliably identify it. We introduce CATCH, a controllable testbed for studying reward hacking in coding RL. CATCH deliberately exposes environmental loopholes and provides execution-based gold labels by comparing success under a vulnerable evaluator with task correctness under an independent audit. It also can control the model's initial hacking tendency through supervised fine-tuning data mixtures and the difficulty of earning rewards through reward designing, enabling systematic comparisons of hacking dynamics and interventions. Experiments show that CATCH can produce diverse RL training trajectories with clear reward hacking, and analyses demonstrate that both initial models and reward difficulties shape the emergence of reward hacking. We further evaluate the effectiveness of different reward hacking detection and mitigation methods. A key finding is that a chain-of-thought monitor initially suppresses hacking, but this protection erodes as the policy model learn to mislead the monitor with code comments. This highlights the need to evaluate hacking mitigations throughout training with CATCH. The source code and resources are publicly released at https://github.com/THUAIS-Lab/CATCH.

Shouli Wang, Yanfeng Jia, Zhihao Ou +6

Tsinghua University · Beihang University · Southern University of Science and Technology +2

RL Benchmarks RL for Code Generation Reward Hacking Reinforcement Learning with Verifiable Rewards Reward Design for RL

Ask PDF

Abstract

AI Agent Monitoring

Explore similar work

Do Prompt-Elicited Trajectories Reflect Training-Time Reward Hacking? A Systematic Study on Monitoring Trainig-Time Reward Hacking in Code Generation

hacktrace: behavior-supervised detection of reward hacking during code generation

CATCH: A Controllable Analysis Testbed for Reward Hacking in Coding RL