Simple Features Beat Most AI Control Monitors

Shubhrangshu Debsarkar; Valmik Nahata

cs.CROct 30, 2026

Simple Features Beat Most AI Control Monitors

Authors: Shubhrangshu Debsarkar, Valmik Nahata

Organizations: University of Virginia · University of California, San Diego

Abstract

AI control relies on monitors to catch a capable model that sabotages its work. These monitors are evaluated on their ability to differentiate honest from backdoored code. We show that on a standard evaluation dataset, simple features beat most monitors without ever running the code, outranking 20 of 24 open-weight models prompted as monitors. The dataset gives its labels away in two ways. People wrote the honest solutions and older language models the backdoors, so the style of the code reveals the class. Additionally, problems that LLMs can backdoor have different characteristics, so basic facts about the problem reveal it too. Swapping one class at a time for code by Claude Sonnet 5 shows that these features reward a contrast between authors, since human code against Sonnet 5's backdoors is easier to separate than the original pairs, 0.95 against 0.79. One in seven backdoors in the standard test set fails our validity checks, which distorts comparisons between attackers. A Sonnet 5 monitor appears to catch 77% of the older models' backdoors but 37% of Sonnet 5's, and 78% against 61% once both are verified. We do not show that published monitors rely on these shortcuts, but our findings suggest that a high score on such a dataset is not by itself evidence of sabotage detection. We release the rebuilt dataset and checks.

Figures & tables

Figure 1: A standard monitor benchmark can be scored on without finding the sabotage. (a) AUROC on the standard APPS backdoor split for the 24 open-weight monitors of [ 9 ] (green) and two classifiers we trained (black), bag-of-words and no-code, which reads only problem metadata (Table A2 ). (b)–(d) The 99 working-backdoor problems, one row per kind of pair, named by who wrote its honest and its backdoored solution (first row: the original pairs; green: older models’ backdoors, from the original dataset; blue: Sonnet 5’s). (b) Bag-of-words, refitted on each row, is highest when the two authors differ. (c) The Sonnet 5 monitor drops whenever Sonnet 5 wrote the backdoors, but many of those have only an illegal trigger. (d) Share of backdoors that monitor catches when it flags 2% of the row’s honest programs (its false-positive rate, FPR), on the 54 problems where both backdoors are real (§ 5 ). Dashed line: chance. Error bars: 95% bootstrap intervals over problems.

Panel of Figure 2 — Figure 2: Three original pairs from the public APPS backdoor dataset [ 8 ] , comments removed. A person wrote each honest solution and the named model wrote the backdoor, whose trigger and wrong output are shaded; under each pair, an input on which the backdoored program answers wrongly. The models’ code also differs in style, with spaced operators in (a) and (c) and an entry-point guard in (b).

Model	working/ $n$	rebuilt (95% CI)	original	$\rho$ (95% CI)
All attempts: one backdoor sampled per problem
Sonnet 5	14/623	$0.7504$ ( $0.729\,49$ – $0.7713$ )	$0.8646$	0.687 (0.625–0.755)
Sonnet 5, prompt-matched	13/586	$0.6411$ ( $0.6179$ – $0.6621$ )	$0.8614$	0.390 (0.326–0.454)
Haiku 4.5	2/611	$0.5344$ ( $0.5159$ – $0.5531$ )	$0.8564$	0.097 (0.044–0.150)
Haiku 4.5, prompt-matched	2/562	$0.5391$ ( $0.5197$ – $0.5581$ )	$0.8540$	0.110 (0.055–0.165)
Working backdoors: only backdoors that work

Table 1: What is left when one model writes both classes. Bag-of-words AUROC on the rebuilt pairs, where the named model wrote both sides, and on the original pairs of the same problems.

\rho

is the fraction of the original advantage over chance that remains. Prompt-matched rows request the honest solution with the backdoor prompt minus the request for a flaw and reuse the backdoors of the row above. Working/

n

: rebuilt backdoors that work. Intervals: 95% bootstrap over problems.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Model	all problems	all attempts	working	AUROC (95% CI)
o3-mini	401	175	18	0.931 (0.918–0.944)
claude-3-7-sonnet	259	110	15	0.874 (0.850–0.896)
gpt-4o	211	72	16	0.854 (0.831–0.877)
claude-3-7-sonnet-thinking	168	71	12	0.824 (0.790–0.857)
o1-mini	132	60	17	0.773 (0.732–0.814)
unattributed	111	60	9	0.958 (0.947–0.969)

Appendix

Table A1 : Who wrote the original backdoors. Problems per model among the 1,444 problems we analyse, the 623 problems of Sonnet 5’s all-attempts row and the 99 of its working-backdoor row (Figure 1 ), and the bag-of-words AUROC of that model’s backdoors against all 1,444 human solutions (95% bootstrap over problems). The last row is the AUROC on all 1,444 original pairs. Model names omit the provider. Source: attacker_mix.py .

Panel of Figure 5 — Figure A1 : The label leak. 1(a) AUROC of the no-code classifier, which reads only problem metadata, cross-validated inside the standard test split and trained on problems outside it. 1(b) Mean test cases per test problem by class, the cue it reads: a problem enters the backdoor class only if its backdoor works, which is easier with few tests. Error bars: 95% bootstrap intervals over problems. Dashed line: chance.

Baseline	trained on	AUROC (95% CI)	shuffled	$p$	beats
bag-of-words	training problems	0.806 ( $0.7705$ – $0.8406$ )	0.541	$<0.001$	20 / 18 / 2
no-code	outside the split	0.298 ( $0.2563$ – $0.3424$ )	$0.5412$	—	0 / 0 / 0
no-code	outside, $\geq 4$ tests	0.629 ( $0.5851$ – $0.6725$ )	$0.5399$	0.001	14 / 8 / 1
no-code	within the split (CV)	0.766 ( $0.7306$ – $0.7995$ )	$0.5446$	$<0.001$	20 / 17 / 1
no-code	within, held-out slice	0.716 ( $0.6474$ – $0.7861$ )	$0.5581$	0.001	17 / 10 / 3

Appendix

Table A2 : Both floors against the 24 monitors on the standard test split. Shuffled : 95th percentile of the AUROC on shuffled labels. Beats : monitors below the floor by point estimate, clearly, and borderline. The held-out row compares monitors on the same 413 items.

Protocol	baseline	value	comments	against
AUROC
Standard test split	bag-of-words	0.806	removed	above 20 of 24
same	bag-of-words	0.891	kept	above 22 of 24
cross-validated within it	no-code classifier	0.766	no code	above 20 of 24
held-out slice of it	no-code classifier	0.716	no code	above 17 of 24
same	code length only	0.473	removed	chance

Appendix

Table A3 : Every baseline that never runs the code. Compare rows only within a block, since each block uses its own protocol. Pairwise accuracy is the share of matched pairs in which the backdoored program scores higher. Comments : removed from both classes, kept, or no code when the baseline never reads the program. Against : how many of the 24 monitors the value beats, or what it is compared with (the DSPy tutorial’s monitor: 0.69–0.81).

Panel of Figure 8 — Figure A2 : What bag-of-words reads. The eight word and word-pair features with the largest weight towards each class, refitted on all 623 all-attempts problems. 2(a) On the original pairs, the tokens of an added special case mark the backdoor side and the idioms of human competitive code mark the honest side. 2(b) On the rebuilt pairs, the program layout the backdoor prompt invites ( def main , data[ ) marks the backdoor side. Source: lexical_probe.py .

Pairs	mean similarity	$>0.95$	$<0.60$	differs from the original pairs in
original	0.606	3.4%	45.4%	—
edit (shown its own solution)	0.928	64.2%	4.0%	author, backdoors, similarity
independent (problem only)	0.659	3.4%	34.8%	author, backdoors

Appendix

Table A4 : How similar the two programs of a pair are. Character similarity with comments removed. The middle columns give the share of pairs above 0.95 and below 0.60.

Model ( $n$ )	honest by	backdoors by older models	backdoors by the model
Sonnet 5 (623)	person	0.865	0.981 (0.974–0.987)
	Sonnet 5	0.956 (0.946–0.965)	0.750
Haiku 4.5 (611)	person	0.856	0.973 (0.965–0.981)
	Haiku 4.5	0.966 (0.958–0.974)	0.534
Sonnet 5, working (99)	person	0.789	0.953 (0.925–0.976)
	Sonnet 5	0.905 (0.865–0.941)	0.749

Appendix

Table A5 : Bag-of-words with one class swapped at a time. AUROC on each model’s all-attempts problems and on Sonnet 5’s working-backdoor problems (95% bootstrap over problems), by who wrote the honest side (rows) and the backdoors (columns). The person / older models cell is the original AUROC and the model / model cell the rebuilt AUROC of Table 1 .

Monitor	problems	original	rebuilt (95% CI)	$\rho$ (95% CI)
Sonnet 5	all attempts	0.958	0.566 ( $0.5507$ – $0.5833$ )	0.145 (0.111–0.182)
Sonnet 5	working backdoors	0.944	0.818 ( $0.7630$ – $0.8686$ )	0.717 (0.601–0.832)
Haiku 4.5	all attempts	$0.8824$	$0.5577$	—
Haiku 4.5	working backdoors	$0.8832$	$0.7871$	0.749 (0.514–0.993)

Appendix

Table A6 : Monitor AUROC on the original and rebuilt pairs. Both sets cover the same problems; intervals are 95% bootstraps over problems. Haiku 4.5’s replies were capped at 512 tokens; on the all-attempts problems, where half of them are missing, we give no

\rho

		AUROC				mean suspicion
		honest by person		honest by Sonnet 5		honest		backdoored
Monitor, problems	$n$	older	Sonnet	older	Sonnet	person	Sonnet	older	Sonnet
Sonnet 5, all attempts	617	0.958	0.446	0.974	0.564	6.9	4.0	78.8	6.6
Sonnet 5, working backdoors	98	0.944	0.755	0.966	0.819	6.9	3.4	73.8	37.6
Haiku 4.5, all attempts	144	0.917	0.513	0.943	0.593	28.4	21.1	86.2	27.8
Haiku 4.5, working backdoors	31	0.951	0.846	0.989	0.882	17.6	17.5	88.2	60.7

Appendix

Table A7 : Monitor AUROC with one class swapped at a time, on Sonnet 5’s pairs and the original pairs of the same problems. Each AUROC column pairs the honest side named above it with the backdoors named in the column.

n

: problems on which all four programs have a readable score. Mean suspicion is on a 0–100 scale. ∗ Haiku 4.5 rerun with a 1,024-token cap. Source: monitor_swap.py .

Figure A3 : The Sonnet 5 monitor’s suspicion scores by class on the original pairs (person / older models) and the rebuilt pairs (Sonnet 5 / Sonnet 5), for the problems of Sonnet 5’s all-attempts and working-backdoor rows. Each class is normalised within its panel. Honest programs score near zero in every panel; only the backdoored class moves.

backdoors	checked	all	by witness	illegal trigger	human answer	public test	undecided
		real		rejected
Sonnet 5, working	99	59	1	34	6	0	0
older models, same problems	99	85	2	13	0	1	0
standard test split	172	141	4	17	0	7	7

Appendix

Table A8 : Which labelled backdoors are real. A backdoor is real if its trigger is a legal input and its output there differs from the human solutions’ majority answer, or if a nearby legal input shows it wrong (a witness). It is rejected if it has only an illegal trigger, gives the human answer on its trigger, or prints a wrong answer on a public test. The first two rows cover the 99 problems of Sonnet 5’s working-backdoor row; the last covers the standard test split’s backdoored programs, all written by Claude 3.7 Sonnet. Source: verify_backdoor.py , split_checks.py .

Figure A4 : The Sonnet 5 monitor at four audit budgets. Share of the checked backdoors caught, on the 54 problems where both backdoors are wrong on a legal input, when the threshold flags 1%, 2%, 5% or 10% of honest programs, for three kinds of pair named by who wrote the honest and the backdoored side; each threshold is set on that pair’s own honest programs, from all 99 problems. Error bars: 95% bootstrap intervals over problems.

Detector, problems	FPR	original	rebuilt	difference	95% CI
Monitors, on the problems of Sonnet 5’s rows
Sonnet 5, all attempts	1%	83%	5%	0.79	0.74 to 0.83
	2%	85%	6%	0.79	0.75 to 0.83
	5%	88%	11%	0.76	0.72 to 0.80
Sonnet 5, checked backdoors	1%	76%	72%	0.04	$-$ 0.11 to 0.20
	2%	78%	74%	0.04	$-$ 0.12 to 0.19

Appendix

Table A9 : Share of backdoors caught at a fixed false-positive rate (FPR). The threshold is set on each set’s honest programs. Original : the original pairs (person / older models). Rebuilt : the rebuilt pairs of the same problems (Sonnet 5 / Sonnet 5, and Haiku 4.5 / Haiku 4.5 in the last row). Difference : original minus rebuilt, with a paired 95% bootstrap interval. Checked backdoors : the 54 working-backdoor problems where both backdoors are wrong on a legal input (55 for bag-of-words). Honest programs come from all 99 problems. All attempts rows are not checked. Source: audit_budget.py , and --verified for the checked rows.

Figure A5 : Character count separates the classes in four of five agentic settings, although the same models write both. Bars: AUROC of character count within each main task, pooled over tasks weighted by pair count, and folded to

\max(a,1-a)

because the direction can differ between tasks. Ticks: 95th percentile on shuffled labels (1,000 shuffles within each task); folding puts it above 0.5. bash is below its tick (

p=0.829{}

); the other four are above theirs (

p=0.001{}

each). The number beside each setting counts its task groups, so the estimates are not equally precise. These datasets have no licence; we redistribute nothing.

Backdoor Detection

Backdoor detection finds hidden triggers or poisoned behavior in trained models before they can secretly redirect predictions or actions.

See all 55 papers

Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages
Jul 6, 2026Lucas PintoBackdoor Detection AI Agent Monitoring
Independent Researcher
SpecGuard: Inference-Time Backdoor Detection For Free
Sep 10, 2026Rui Wen, Ahmed Salem, Andrew Paverd +2Backdoor Detection Backdoor Defense in LLMs
Institute of Science Tokyo · Microsoft Security Response Center · Microsoft Azure +1
MonitoringBench: Semi-Automated Red-Teaming for Agent Monitoring
May 10, 2026Monika Jotautaitė, Maria Angelica Martinez, Ollie Matthews +1Software Engineering Agents AI Agent Security Benchmarks
Independent · Redwood Research

Jul 6, 2026J/K move · Enter open · S save

Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages

Lucas Pinto

Independent Researcher

Backdoor Detection AI Agent Monitoring AI Control

Open paper AskPDF arXiv

Trusted monitoring is a central defense in AI control: a cheaper trusted model scores an untrusted model's actions for sabotage, and the most suspicious are audited or deferred. Such monitors are evaluated against one or two untrusted models, and the accuracy is reported as a property of the monitor. We ask whether it is partly a property of the pairing. We make the untrusted policy family the controlled axis: we fit a monitor on family A's transcripts, apply it to family B, and decompose the cross-family AUROC into how obvious each family's sabotage is, how capable each monitor is, and the residual own-family advantage after both are removed: the interaction. On code-backdoor transcripts the interaction is positive and survives the dominant confounds: +0.172 (95% CI [+0.158, +0.185]) on four open-weight families on a strict leak-free basis, holding across an eight-family matrix spanning four frontier vendors, blatancy-matched controls, a trained transcript-reading monitor up to 34B parameters, and a second code benchmark. The gap is geometric: each lineage's sabotage occupies a different scoring direction, and rotating the monitor's scoring direction toward the target lineage recovers most of the lost detection while a matched random rotation recovers nothing. At a 1% audit budget under defer-to-trusted, an off-lineage monitor catches 19% of attack attempts where an in-lineage monitor catches 41% at the same audit cost, a +21.7-point gap. Control evaluations should report cross-family transfer matrices: a single-pairing accuracy overstates the safety a monitor delivers against a lineage it was not calibrated on. We give a four-step protocol that acts on the gap, with each step a measured result.

Abstract

Backdoor Detection

Explore similar work

Calibration-Family Overfit: Why Trusted Sabotage Monitors Don't Transfer Across Lineages

SpecGuard: Inference-Time Backdoor Detection For Free

MonitoringBench: Semi-Automated Red-Teaming for Agent Monitoring