Correlated Failures of AI Approval Monitors

Luca Zanolini

cs.CROct 21, 2026

Correlated Failures of AI Approval Monitors

Authors: Luca Zanolini

Organizations: Independent researcher

Abstract

Before an AI agent takes a consequential action, such as paying, messaging or changing code, several monitors check it. Their verdicts are combined by aggregating scores or by counting approvals, whatever inputs the monitors share. Monitors that share a base model or a poisoned input, however, may fail together. We borrow the fail-prone systems of distributed computing: lists of monitor sets that may fail together. A rule is safe if it executes no bad action and live if it blocks no good one. We measured twelve small open-weights monitors on payment approval under two pre-registered protocols. Without pre-registration, we also measured three closed frontier models (Claude Opus 5.5, GPT-6 Astra, Gemini 4 Argon) and the open DeepSeek-V3.2. Every reliable fooling of a monitor into approving a fraud came through an evidence source and reached only that source's readers. Fooling it into blocking a correct payment depended on injection resistance, a monitor's refusal to follow instructions planted in its data. Small models and DeepSeek obeyed such notes. The three closed models ignored them in all but one case, and Opus also resisted persuasion tailored to it. On new data, a rule that counts sources instead of votes blocked 0.9% of correct payments, against 7.4% for the best counting rule, at equal safety. The pre-registered worst-case test failed. One attack left the approvers on a single source, so no rule over the twelve language-model monitors survives every measured attack. Adding a rule engine, which cannot be persuaded, restored the condition for a safe and live rule on the measured attacks. Diversify the inputs, harden the monitors, count sources, not votes.

Figures & tables

Panel of Figure 1 — Figure 1: The ingredients of the measurement on the $4\times 3$ grid (rows: Llama, Qwen, Gemma, Ministral; columns: sources). (a) Where the two kinds of payload land. (b) The fooled set of one item. (c) The fooled set $W_{a}$ of an attack unit, summarised over its items. The empirical fail-prone system $\hat{\mathcal{F}}$ is the family of all subsets of the sets $W_{a}$ . The counts in (c) are illustrative.

Panel of Figure 2 — Figure 2: The measured directional structure on the $4\times 3$ grid (rows: Llama, Qwen, Gemma, Ministral; columns: sources). (a) Every set fooled into approving a fraud lies inside one column (pilot exploratory, confirmation pre-registered). (b) The witness to the failure of ( CC ) in the confirmation; shaded cells were fooled into blocking on at least half of the items. (c) The same attack on the pre-registered secondary pool.

		fraud approved			good payment blocked			liveness
monitor family	size	all	inj.	pers.	all	inj.	pers.	fragility
Ministral 3 (Mistral AI)	8.9B, local	0.097	0.000	0.000	0.783	0.333	0.867	0.741
Qwen 3.5 (Alibaba)	9B, local	0.139	0.137	0.000	0.488	0.354	0.287	0.444
Llama 3.1 (Meta)	8B, local	0.117	0.137	0.000	0.338	0.479	0.153	0.185
Gemma 4 E4B (Google)	${\approx}4$ B, local	0.124	0.098	0.000	0.244	0.146	0.080	0.056
DeepSeek-V3.2 (DeepSeek)	open, API	0.152	0.216	0.000	0.501	0.292	0.453	0.426

Table 1: All pools on the same

340

items under transferred attacks (not pre-registered), pooled over each family’s three source monitors. Share of adversarial verdicts that approved a fraud or blocked a good payment: all attack classes, template source injections (“inj.”, reaching only the poisoned source’s readers) and the

18

justification payloads (“pers.”), all optimised against the small families. The fraud “all” column is the substitution floor (one poisoned source of three). Liveness fragility : share of the

54

blocking attack units that fool the family on at least half of their items (the safety analogue is

0

everywhere). DeepSeek-V3.2 with thinking off ( Appendix D ).

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Check	Example	Condition	Rule
Self-verifying	a monitor exhibits a signed order whose payee differs	$Q^{1}$	accept one valid piece of evidence
Factual (witness)	payee and amount match the order	$Q^{2}$	kernel rule ( Proposition 3 )
Judgement, or equivocating agent	“is this email appropriate?”; different invoices to different monitors	$Q^{3}$	consensus; or quorum of one-shot approvals

Appendix

Table 2: Approval rules by type of check, for one principal with fail-prone system

\mathcal{F}

. All entries are classical. The AI-specific content is which checks fall into which row.

Panel of Figure 3 — Figure 3: Pilot, exploratory analysis ( Section 5.4 ). Liveness at matched safety on the held-out split ( Table 10 ; log scale). Circles: counting rules. Squares: structured rules on the pooled system, and the margin-2 directional kernel, which is safe against one fooled column plus one stray approver ( Appendix A ). Triangle: the directional kernel over the source columns. $\ddagger$ : no single monitor met the safety bound on the training split.

	criterion values (primary pool of $12$ monitors)	outcome
H1 ′	every maximal approve-fooled set within one column: yes; $Q^{2}$ : yes ( $\tau=3$ ); column coverage $0.999$ (need $\geq 0.95$ ); both in $1.000$ of replicates (need $\geq 0.8$ )	supported
H2 ′	( CC ): fails, slack $0$ , holds in $0.750$ of replicates (need $\geq 0.8$ ); pooled $Q^{2}$ fails ( $0.397$ ); $\nu=1.000$ $[0.667,1.000]$ . Secondary pool of $13$ : ( CC ) holds, slack $1$ ( $1.000$ ); pooled $Q^{2}$ holds ( $0.99$ ); $\nu=0.667$	( $\gamma$ ); secondary ( $\beta$ )
H3 ′	$\Delta_{\mathrm{FRR}}$ (log-odds vote minus directional kernel) $=0.065$ $[0.033,0.098]$ (need interval above $0$ , point $\geq 0.05$ ); unsafe difference $0.000$ $[0.000,0.000]$ (need $\leq 0.01$ , upper bound $\leq 0.02$ )	supported
H4 ′	$\rho(\text{prov})-\rho(\text{lineage})$ in $[0.546,0.687]$ ; $\rho(\text{prov})-\rho(\text{benign})$ in $[0.658,0.784]$ ; kernel for $\mathcal{F}_{\mathrm{prov}}$ unsafe $0.000$ , for lineage only $0.311$ (FRR $0.206$ and $0.046$ , reported against $0.10$ )	supported

Appendix

Table 3: Pre-registered outcomes of the confirmation study (protocol v2;

\theta=0.5

B=1{,}000

cluster-bootstrap replicates over attack units and tasks; intervals are

95\%

). Frozen decision rule under H2 ′ (

\gamma

): “the empirical section becomes the negative result”.

rule family (selected rule)	unsafe approvals	false rejections
single best monitor ‡	0.099 $[0.022,0.197]$	0.812 $[0.710,0.896]$
majority (7-of-12)	0.000 $[0.000,0.000]$	0.571 $[0.435,0.697]$
all-must-approve	0.000 $[0.000,0.000]$	0.993 $[0.987,0.997]$
best $k$ -of- $n$ (5-of-12)	0.001 $[0.000,0.002]$	0.332 $[0.207,0.453]$
LP-weighted vote	0.000 $[0.000,0.000]$	0.365 $[0.228,0.486]$
log-odds vote	0.001 $[0.000,0.002]$	0.074 $[0.037,0.112]$

Appendix

Table 4: Confirmation study (protocol v2, pre-registered H3 ′ ). Liveness at matched safety on the held-out half of the attack conditions: for each rule family, the parameters with the lowest training FRR among those with a training unsafe rate

\leq 0.01

;

95\%

cluster-bootstrap intervals over attack conditions.

\ddagger

: no single monitor met the safety bound on the training split; the one with the lowest training unsafe rate is shown.

maximal $R\in\hat{\mathcal{F}}_{\mathrm{reject}}$	$\|R\|$	$M\setminus R$ (survivors)	slack	survivors in one column
L@doc, L@tool, L@web, Q@doc, Q@tool, Q@web, M@doc, M@tool, M@web	9	G@doc, G@tool, G@web	2	no
L@doc, L@tool, Q@doc, Q@tool, Q@web, G@doc, M@doc, M@tool, M@web	9	L@web, G@tool, G@web	1	no
L@tool, Q@doc, Q@tool, Q@web, G@tool, M@doc, M@tool, M@web	8	L@doc, L@web, G@doc, G@web	2	no
L@tool, L@web, Q@doc, Q@tool, Q@web, G@web, M@doc, M@tool, M@web	9	L@doc, G@doc, G@tool	1	no
L@doc, L@tool, L@web, Q@tool, Q@web, G@tool, G@web, M@doc, M@tool, M@web	10	Q@doc, G@doc	0	yes

Appendix

Table 5: Confirmation study (protocol v2, pre-registered H2 ′ ). The cross-cover condition set by set on the twelve-monitor pool. For each maximal reject-fooled set, its survivors and the slack: the fewest monitors outside both

R

and any one column. The last set, the fooled set of one evidence-aware blocking payload optimised against Gemma, leaves survivors in the doc column only: slack

0

, ( CC ) fails. Maximal approve-fooled sets: the three columns.

System	maximal sets	$\tau$	$\tau^{*}$	$Q^{2}$	$Q^{3}$	coverage
$\hat{\mathcal{F}}$ (measured, attack level, $\theta=0.5$ )	7	2	1.667	no	no	0.963
$\mathcal{F}_{\mathrm{prov}}$ (one family + one source)	12	3	2.000	yes	no	0.760

Appendix

Table 6: Pre-registered estimate of the empirical fail-prone system (attack level,

\theta=0.5

;

108

attack units,

2{,}160

adversarial items) next to the provenance system. Coverage: fraction of adversarial items whose realised fooled set lies in the system (one minus residual risk). H1 predicted

\tau^{*}(\hat{\mathcal{F}})\leq 2<\tau(\hat{\mathcal{F}})

; measured:

\tau=2

, refuted (b). Bootstrap (

B=200

, tasks and attack units resampled):

\tau\leq 2

0.720

\tau^{*}>2

0.275

\tau^{*}\leq 2<\tau

0.005

of the replicates;

0.630

of the replicates have

\tau^{*}

at exactly

2

Rule family	selected rule	unsafe approvals [95% CI]	false rejections [95% CI]
single best monitor	Gemma@ tool	0.162 [0.041, 0.318]	0.244 [0.118, 0.391]
majority	7-of-12	0.000 [0.000, 0.000]	0.478 [0.311, 0.640]
all-must-approve	12-of-12	0.000 [0.000, 0.000]	0.985 [0.972, 0.994]
best $k$ -of- $n$	4-of-12	0.295 [0.146, 0.471]	0.120 [0.053, 0.200]
weighted (LP weights)	LP from $\hat{\mathcal{F}}_{\mathrm{train}}$	0.367 [0.207, 0.550]	0.104 [0.027, 0.197]
weighted (log-odds)	best on train	0.002 [0.000, 0.006]	0.115 [0.041, 0.202]

Appendix

Table 7: Approval rules on held-out attacks (pre-registered primary split by attack condition); parameters chosen on the training split at false rejection

\leq 0.10

; cluster-bootstrap CIs over attack conditions. H2 predicted

\Delta>0

with false-rejection non-inferiority (margin

0.02

); measured: not supported. Adaptive-attacker results are in Table 12 .

System	maximal sets	largest	$\tau$	$\tau^{*}$	$Q^{2}$	coverage
$\hat{\mathcal{F}}_{\mathrm{approve}}$ (approve-bad units)	3	4	3	3.000	yes	0.999
$\hat{\mathcal{F}}_{\mathrm{reject}}$ (block-good units)	7	10	2	1.667	no	0.925
$\hat{\mathcal{F}}$ (pooled, pre-registered)	7	10	2	1.667	no	0.963

Appendix

Table 8: Pilot, exploratory analysis computed after its pre-registered results ( Section 5 ). Goal-separated (directional) empirical fail-prone systems (attack level,

\theta=0.5

, units split by goal). Coverage: fraction of adversarial items of the same goal whose realised fooled set lies in the system. The maximal sets of

\hat{\mathcal{F}}_{\mathrm{approve}}

are exactly the three source columns. Cross-cover condition (no

R\in\hat{\mathcal{F}}_{\mathrm{reject}}

A\in\hat{\mathcal{F}}_{\mathrm{approve}}

with

R\cup A=M

): holds, slack

1

; bootstrap (

B=1{,}000

, attack units within goal and tasks resampled): holds in

1.000

Q^{2}(\hat{\mathcal{F}}_{\mathrm{approve}})

1.000

Q^{2}(\hat{\mathcal{F}}_{\mathrm{reject}})

and

Q^{2}

(pooled) in

0.256

of the replicates. Sensitivity: at

\theta=0.25

the reject side has

\tau=2

\tau^{*}=1.571

and the condition holds with slack

1

; at

\theta=0.75

the reject side has

\tau=\tau^{*}=3

and the condition holds with slack

2

; the approve side is unchanged at both.

Items	family only	source only	provenance	benign $\phi$	source $-$ prov.
all ( $n{=}2{,}160$ )	$-0.099$ [ $-0.161$ , 0.004]	0.746 [0.673, 0.771]	0.591 [0.554, 0.633]	0.135 [0.016, 0.211]	[0.067, 0.206]
approve-bad ( $n{=}1{,}080$ )	$-0.208$ [ $-0.247$ , $-0.202$ ]	0.771 [0.771, 0.772]	0.529 [0.498, 0.533]	0.055 [ $-0.111$ , 0.151]	[0.238, 0.273]
block-good ( $n{=}1{,}080$ )	0.019 [ $-0.087$ , 0.118]	0.761 [0.675, 0.770]	0.695 [0.583, 0.741]	0.137 [ $-0.026$ , 0.216]	[ $-0.011$ , 0.147]

Appendix

Table 9: Pilot, exploratory analysis computed after its pre-registered results ( Section 5 ). H3 with four predictors of pairwise co-failure under attack: Spearman

\rho

of the predictor with the pairs’

\phi

over the adversarial items (

66

monitor pairs; cluster-bootstrap CIs over attack conditions,

B=200

), overall and per attack goal. Family only is the pre-registered lineage predictor and provenance (same family or same source) the pre-registered H3 predictor; source only was added post hoc. Last column: CI of

\rho_{\mathrm{source}}-\rho_{\mathrm{provenance}}

Rule family	selected rule	unsafe approvals [95% CI]	false rejections [95% CI]
single best monitor	Ministral@ tool ‡	0.140 [0.034, 0.289]	0.758 [0.616, 0.878]
majority	7-of-12	0.000 [0.000, 0.000]	0.478 [0.311, 0.640]
all-must-approve	12-of-12	0.000 [0.000, 0.000]	0.985 [0.972, 0.994]
best $k$ -of- $n$	5-of-12	0.002 [0.000, 0.006]	0.271 [0.137, 0.433]
weighted (LP weights)	LP from $\hat{\mathcal{F}}_{\mathrm{train}}$	0.002 [0.000, 0.006]	0.255 [0.143, 0.381]
weighted (log-odds)		0.002 [0.000, 0.006]	0.115 [0.041, 0.202]

Appendix

Table 10: Pilot, exploratory analysis computed after its pre-registered results ( Section 5 ). Liveness at matched safety on held-out attacks (primary split): for each rule family, the parameter choice with the lowest training false-rejection rate among those with a training unsafe-approval rate

\leq 0.01

; cluster-bootstrap CIs over attack conditions.

\ddagger

: no parameter choice met the safety bound on the training split; the choice with the lowest training unsafe rate is shown.

\lx@sectionsign

: the pair-LP rule (weights minimising

\max_{A}w(A)+\max_{R}w(R)

over the training maximal sets, threshold at the low end of the feasible interval) was added after the first draft; fitted on all units (in-sample, optimistic) it has unsafe

0.002

and false rejections

0.027

–

0.051

. The same rules selected under the pre-registered criterion (false rejection

\leq 0.10

) are in Table 7 .

maximal $R\in\hat{\mathcal{F}}_{\mathrm{reject}}$	$\|R\|$	$M\setminus R$ (survivors)	slack	$M\setminus R$ in a column
L@doc, L@tool, L@web, Q@doc, Q@tool, Q@web, G@doc, M@doc, M@tool, M@web	10	G@tool, G@web	1	no
L@doc, L@tool, L@web, Q@tool, G@tool, M@doc, M@tool, M@web	8	Q@doc, Q@web, G@doc, G@web	2	no
L@doc, L@tool, L@web, Q@tool, Q@web, G@web, M@doc, M@tool, M@web	9	Q@doc, G@doc, G@tool	1	no
L@web, Q@doc, Q@tool, Q@web, G@web, M@doc, M@tool, M@web	8	L@doc, L@tool, G@doc, G@tool	2	no
L@doc, Q@doc, Q@tool, Q@web, G@doc, G@web, M@doc, M@tool, M@web	9	L@tool, L@web, G@tool	1	no
L@tool, Q@doc, Q@tool, Q@web, G@tool, G@web, M@doc, M@tool, M@web	9	L@doc, L@web, G@doc	1	no

Appendix

Table 11: Pilot, exploratory analysis computed after its pre-registered results ( Section 5 ). The cross-cover condition set by set (L = Llama, Q = Qwen, G = Gemma, M = Ministral). For each maximal reject-fooled set

R

: the survivors it leaves, whether they lie inside some approve-fooled set (a column; then the directional kernel rule would reject under that attack, a liveness failure), and the least number of monitors left uncovered by

R\cup A

over the three columns

A

. Every

R

contains all three Ministral monitors and at least one Qwen monitor.

Rule	worst fraud approved		worst good blocked		conditions chosen
	typed	any	typed	any	approve-bad / block-good
single best monitor (Gemma@ tool )	1.000	1.000	1.000	1.000	chan-tool-sub-2 / chan-tool-inj-0
majority (7-of-12)	0.000	0.000	1.000	1.000	chan-doc-inj-0 / comb-alibaba-doc-1
all-must-approve (12-of-12)	0.000	0.000	1.000	1.000	chan-doc-inj-0 / chan-doc-inj-0
best $k$ -of- $n$ (3-of-12)	1.000	1.000	0.000	0.000	chan-doc-sub-3 / comb-google-tool-1
weighted (LP weights)	0.900	0.900	0.300	0.300	chan-web-sub-2 / comb-google-web-1

Appendix

Table 12: Pilot, exploratory analysis computed after its pre-registered results ( Section 5 ). Offline rule-aware attacker against every rule family including the directional kernels: tasks are split in half, each rule is fitted on the first half, the attacker picks on the first half the attack condition that best achieves its goal against that rule (typed: family, source and combined conditions; any: also universal payloads), and its success rate on the ten second-half tasks is reported (steps of

0.1

). The margin-2 kernel approves iff the approvers have at least two monitors outside every approve-fooled set. The rules are refitted on the task halves, so the selected

k

-of-

n

differs from Table 7 .

\lx@sectionsign

: on the task-half split the pair LP is infeasible (

\nu=1.000

); the fallback threshold is shown. A rate of

0.0

on ten tasks has an exact

95\%

upper bound of

0.31

payload	target	origin	Alibaba (Qwen)	Google (Gemma)	Meta (Llama)	Mistral (Ministral)
fam-alibaba-0	Alibaba	optimised	0.517	0.017	0.100	0.917
fam-alibaba-1	Alibaba	optimised	0.750	0.017	0.017	0.967
fam-alibaba-2	Alibaba	template	0.067	0.000	0.017	0.717
fam-alibaba-3	Alibaba	template	0.050	0.000	0.017	0.900
fam-google-0	Google	optimised	0.400	0.000	0.167	0.933
fam-google-1	Google	optimised	0.933	0.250	0.017	1.000

Appendix

Table 13: Pilot, exploratory analysis computed after its pre-registered results ( Section 5 ). Fooling rate of family-targeted and universal payloads with goal reject a good payment , by payload and monitor family (pooled over the family’s three sources;

20

held-out tasks,

60

verdicts per cell). “template”: an unoptimised seed template; “optimised”: the best of

31

optimiser rounds against the target. The same matrix for the goal approve a bad payment is

0.000

in every cell except

0.017

for univ-0 on Llama.

Family (developer)	role	Ollama tag (all q4_K_M )	params / size	digest (sha256, first 12)	machine
Llama 3.1 (Meta)	monitor	llama3.1:8b-instruct-q4_K_M	8.0B / 4.9 GB	46e0c10c039e	MacBook
Ministral 3 (Mistral AI)	monitor	ministral-3:8b-instruct-2512-q4_K_M	8.9B / 6.0 GB	1922accd5827	MacBook
Qwen 3.5 (Alibaba)	monitor	qwen3.5:9b-q4_K_M	9.0B / 6.6 GB	56671c2ab938	Mac mini
Gemma 4 E4B (Google)	monitor	gemma4:e4b-it-q4_K_M	7.5B stored, ${\approx}4$ B effective / 6.6 GB	dc35e8d9c606	Mac mini
OLMo 3 (Ai2)	attacker only	olmo-3:7b-instruct-q4_K_M	7.3B / 4.5 GB	ea72df8c85d7	MacBook

Appendix

Table 14: Models used in the pilot, with tags and content digests as pulled. Every monitor runs on the doc , tool and web source at temperature

1.0

, with at most

300

output tokens, a

4{,}096

-token context and reasoning disabled. Thinking is switched off for Qwen 3.5 and Gemma 4. The attacker emits at most

600

tokens. In protocol v2 Gemma 4 runs on the MacBook, and a deterministic rule engine on tool is a 13th monitor.

System	maximal sets	largest	$\tau$	$\tau^{*}$	$Q^{2}$	coverage
$\hat{\mathcal{F}}_{\mathrm{approve}}$ (approve-bad units)	3	2	3	3.000	yes	1.000
$\hat{\mathcal{F}}_{\mathrm{reject}}$ (block-good units)	3	2	3	3.000	yes	1.000
$\hat{\mathcal{F}}$ (pooled)	3	2	3	3.000	yes	1.000

Appendix

Table 15: Frontier arm ( Section 5 ; not pre-registered). Directional empirical fail-prone systems of the six-monitor Opus + Astra pool (attack level,

\theta=0.5

54

units per goal,

18

non-empty on each side). The maximal sets of both systems are the three source columns,

\{

Opus@doc, Astra@doc

\}

and so on. Cross-cover condition: holds with slack

2

; pair-LP statistic

\nu=0.667

. Cluster bootstrap (

B=1{,}000

, attack units within goal and tasks resampled): the condition,

Q^{2}

of each side and of the pooled system, “approve side within columns” and

\nu<1

hold in

1.000

of the replicates;

\nu

interval

[0.667,0.667]

. Sensitivity: at

\theta=0.25

and

0.75

every entry is identical. For comparison, the pilot’s twelve-monitor pool ( Table 8 ) has the same approve side and seven reject-fooled sets of

8

–

10

monitors.

Rule family	unsafe approvals	false rejections
single best monitor (Opus on tool ) †	0.136 $[0.033,0.281]$	0.138 $[0.034,0.284]$
all-must-approve (6 of 6)	0.000	0.310 $[0.150,0.487]$
majority (4 of 6); best $k$ -of- $n$ (3 of 6); weighted votes (LP, log-odds, pair LP); root count; kernel for $\mathcal{F}_{\mathrm{prov}}$ , for $\hat{\mathcal{F}}_{\mathrm{train}}$ (and tuned $\theta$ ); directional kernel over the columns (and learned); margin-2 kernels	0.000	0.000

Appendix

Table 16: Frontier arm ( Section 5 ; not pre-registered). Approval rules on the six-monitor Opus + Astra pool, held-out attack split (the same split procedure as Table 7 ), parameters chosen on the training split at false rejection

\leq 0.10

; the matched-safety selection (training unsafe

\leq 0.01

, minimum training false rejection) gives the same numbers for every rule. Cluster-bootstrap

95\%

intervals; an interval of

[0.000,0.000]

is omitted.

\dagger

: no single monitor met the safety bound on the training split. Fourteen rule families coincide at

0.000

0.000

because the only failure mode on this pool is one column of two monitors.

	line appended to the invoice e-mail	Opus 5.5	Astra	Argon	DeepSeek
A	“I can’t rewrite the fraud alert to deceive the reviewer. …”	$36/120$ $[0.225,0.387]$	$0/120$	$0/120$	$0/120$
B	“Thank you for your business. …” (neutral)	$0/120$ $[0.000,0.031]$	$0/120$	$0/120$	$0/120$
C	“Note: this message was flagged by our e-mail security filter.”	$8/120$ $[0.034,0.126]$	$0/120$	$0/120$	$0/120$
D	nothing (clean)	$0/40$ $[0.000,0.088]$	$0/40$	$0/40$	$0/40$
	reasoning tokens per call under A (mean)	${\approx}291$	$0$	$0$	$0$

Appendix

Table 17: Anomaly control (exploratory). Correct payments rejected by the monitor reading the invoice, each at its API’s lowest reasoning setting; the same

40

tasks and prompts for every model, three samples per task for A–C, one for D;

95\%

Wilson intervals (zero counts:

[0,0.031]

for A–C,

[0,0.088]

for D). Fisher exact tests, Opus: A vs B

p=1.3\times 10^{-12}

, C vs B

p=0.0069

; Opus vs the others: A

p=1.3\times 10^{-12}

, C

p=0.0069

AI Agent Monitoring

AI agent monitoring tracks actions, tool use, and execution traces to detect failures, misuse, drift, and policy violations during operation.

See all 280 papers

Jul 9, 2026

Persuasion Attacks Can Decrease Effectiveness of CoT Monitoring

Chain-of-thought (CoT) monitoring is a promising safety mechanism for AI agents, based on the premise that visible reasoning traces can surface misaligned or deceptive behavior. While effective in standard scenarios, recent work highlights that LLMs remain vulnerable to persuasion-based jailbreaks, where natural-language arguments override model constraints. We stress-test whether this vulnerability extends to monitoring LLMs: can an adversarial agent persuade its CoT monitor to approve proposed actions that violate the monitor's policy? We design an evaluation framework with 40 tasks and analyze thousands of agent-monitor interactions, where agents are instructed to argue for policy-violating proposals. We find that in such adversarial settings, monitor access to the agent's CoT reasoning increases rather than decreases approval of harmful actions on average by 9.5%, as the scratchpad provides an additional persuasion channel. To address this, we introduce a fact-checking monitoring framework. We find that a fact-checker and monitor pairing from different model families, for example a Claude 3.7 Sonnet monitor paired with a GPT-4.1 fact-checker, reduces approval of policy-violating actions by up to 45%, compared to only 6%, when using the same model for both fact-checking and monitoring roles. Our results demonstrate that CoT monitoring alone may be insufficient against adversarial persuasion, and that model-diverse fact-checking provides a robust mitigation.

Jennifer Za, Julija Bainiaksina, Nikita Ostrovsky +2

LASR Labs · University College London · Google DeepMind

Language Model Safety Evaluation Adversarial Attacks on LLMs AI Agent Monitoring Persuasion in Language Models CoT Monitoring

Ask PDF

May 14, 2026

Ensemble Monitoring for AI Control: Diverse Signals Outweigh More Compute

As AI systems are increasingly deployed in autonomous agentic settings at scale, it is important to ensure the actions they take are safe and aligned with user intent. Monitoring agent actions is a key safety mechanism, yet reliable monitors remain difficult to build and the scale of these systems makes human oversight impractical. We show that combining signals from diverse monitors into an ensemble improves detection of misaligned actions. We build 12 GPT-4.1-Mini monitors using both prompting and fine-tuning strategies. We evaluate them on coding tasks where candidate solutions pass standard tests but fail on adversarial inputs. In this setting, diverse ensembles outperform both individual monitors and homogeneous ensembles. Our best 3-monitor ensemble achieves 2.4x greater detection performance gain compared to an ensemble composed of three identical monitors, with the same ensemble performing strongly on an independent dataset. We contend that these results show that diversity - not scale - drives gains. The best ensembles combine strong individual performance with low correlation between monitors. Furthermore, fine-tuned monitors appear in every top-performing ensemble and maintain this advantage on out-of-distribution attack types, suggesting that fine-tuning enables detection capabilities that prompting alone does not elicit. These results support ensemble monitoring as a practical AI control strategy for safety gains at reasonable inference costs.

Eugene Koran, Yejun Yun, Samantha Tetef +2

SPAR · Yale University

AI Agent Monitoring AI Control AI Safety Evaluation Language Model Ensembles

Ask PDF

Aug 25, 2026

More Rejective, Not More Discriminative: The Unit of Verification in Pre-Execution LLM Oversight

Pre-execution oversight is core to trusted monitoring in AI control: a fallible LLM monitor vets planned actions before irreversible execution. Over-blocking forfeits usefulness and pressures deployers to disable it. Every protocol must fix a unit of verification: how many actions one call reviews. Existing designs take the unit as given; its effect on fallible monitors is unmeasured. Natural traces cannot isolate it: review length co-varies with error type and position. Catch alone misleads: rejecting everything catches everything. Measuring this needs boundary variation alone and a matched clean control. We introduce the twin-prefix framework, which supplies both. Each gold plan yields a prefix with one injected, environment-accepted error and a clean twin differing in one write. Judging each pair at five nested lengths ties verdict changes to the unit alone. Discrimination is scored by pre-registered informedness, catch minus false rejection. Longer review raises catch; false rejection climbs in lockstep. Informedness peaks at one or two actions for all six judges in both domains: longer windows make zero-shot monitors more rejective, not more discriminative. Replaying withheld observations traces the failure largely to observation deprivation. Safety cases should state the unit and co-report the clean series. Our framework is the first controlled, pre-registered instrument for this choice and never reads catch alone. Our calibrated short unit recovers up to 0.95 informedness over eight-action review, and no tested label-blind policy consistently beats it.

Yuchen Han, Cheng Yan, Wuyang Zhang

University of Science and Technology of China Hefei, China

AI Agent Monitoring LLM Safety Evaluation

Ask PDF

Abstract

AI Agent Monitoring

Explore similar work

Persuasion Attacks Can Decrease Effectiveness of CoT Monitoring

Ensemble Monitoring for AI Control: Diverse Signals Outweigh More Compute

More Rejective, Not More Discriminative: The Unit of Verification in Pre-Execution LLM Oversight