More Rejective, Not More Discriminative: The Unit of Verification in Pre-Execution LLM Oversight

Yuchen Han; Cheng Yan; Wuyang Zhang

cs.AIAug 25, 2026

More Rejective, Not More Discriminative: The Unit of Verification in Pre-Execution LLM Oversight

Authors: Yuchen Han, Cheng Yan, Wuyang Zhang

Organizations: University of Science and Technology of China Hefei, China

Abstract

Pre-execution oversight is core to trusted monitoring in AI control: a fallible LLM monitor vets planned actions before irreversible execution. Over-blocking forfeits usefulness and pressures deployers to disable it. Every protocol must fix a unit of verification: how many actions one call reviews. Existing designs take the unit as given; its effect on fallible monitors is unmeasured. Natural traces cannot isolate it: review length co-varies with error type and position. Catch alone misleads: rejecting everything catches everything. Measuring this needs boundary variation alone and a matched clean control. We introduce the twin-prefix framework, which supplies both. Each gold plan yields a prefix with one injected, environment-accepted error and a clean twin differing in one write. Judging each pair at five nested lengths ties verdict changes to the unit alone. Discrimination is scored by pre-registered informedness, catch minus false rejection. Longer review raises catch; false rejection climbs in lockstep. Informedness peaks at one or two actions for all six judges in both domains: longer windows make zero-shot monitors more rejective, not more discriminative. Replaying withheld observations traces the failure largely to observation deprivation. Safety cases should state the unit and co-report the clean series. Our framework is the first controlled, pre-registered instrument for this choice and never reads catch alone. Our calibrated short unit recovers up to 0.95 informedness over eight-action review, and no tested label-blind policy consistently beats it.

Figures & tables

Figure 1: The twin-prefix framework (schematic). Left: construction, the gold plans of both domains, the bad/clean twins split at one anchor write, and the verify-before-execute loop. Center: one worked retail pair’s two verifier inputs, each scored by its own stateless call, so catch arrives with its paired false rejection,

J=\mathrm{catch}-\mathrm{FR}

. Right: the anchor slot closes all five nested windows; raising

L

widens the window and withholds in-window observations; what it buys is rejection, not discrimination.

Retail
Unit policy	Open-weight verifiers									Closed-weight judges
	Llama-3.3-70B			Qwen2.5-14B			Qwen2.5-72B			DeepSeek-V4-Pro			Claude Opus 4.8			GPT-5.6 Sol
	$L^{*}{=}2\,\|\,1$			$L^{*}{=}2\,\|\,1$			$L^{*}{=}1\,\|\,1$			$L^{*}{=}1\,\|\,1$			$L^{*}{=}1\,\|\,1$			$L^{*}{=}1\,\|\,1$
	catch $\uparrow$	$\mathrm{FR}$ $\downarrow$	$J$ $\uparrow$	catch $\uparrow$	$\mathrm{FR}$ $\downarrow$	$J$ $\uparrow$	catch $\uparrow$	$\mathrm{FR}$ $\downarrow$	$J$ $\uparrow$	catch $\uparrow$	$\mathrm{FR}$ $\downarrow$	$J$ $\uparrow$	catch $\uparrow$	$\mathrm{FR}$ $\downarrow$	$J$ $\uparrow$	catch $\uparrow$	$\mathrm{FR}$ $\downarrow$	$J$ $\uparrow$
$L{=}1$	.765	.419	.346	.342	.065	.277	.795	.129	.666	.948	.167	.781	.985	.032	.953	.955	.097	.858
$L{=}2$	.795	.194	.602	.712	.290	.422	.920	.419	.501	.974	.300	.674	1.000	.194	.806	1.000	.290	.710

Table 1: Bold marks each judge’s best

J

within that domain; shading covers that judge’s whole catch /

\mathrm{FR}

J

triple at its

L^{*}

. Headers:

L^{*}

, the max-

J

unit, Retail

|

AppWorld. Across both domains, every judge reaches its peak at either

L{=}1

L{=}2

. Open-weight columns are the frozen decision leg; closed-weight columns replicate under the score-mode prompt and per-provider transports (Appendix I), sharing shape, not serving envelope. Policy rows reuse the cell’s five-length verdict matrix: union, majority of five, LOTO per-tool router, label-aware any-/all-

L

bound. Retail

n{=}200

injected /

31

gold-clean (

n_{\mathrm{eff}}

21 to 31), AppWorld

190/200

(133 to 172); 14B policy rows

174/29

(Appendix C); rates and

J

round independently.

Figure 2: Left: the primary retail cell in (false rejection, catch) space (arrows: rising

L

): beyond the

L{=}2

peak (star) it slides along the chance diagonal toward reject-everything. Center/right:

J(L)

per cell, 95% item-bootstrap bands, stars at peaks.

Figure 3: Threshold-artifact test: ranking AUC at

L{=}2

(filled) vs

L{=}8

(open); blue (thick) = the paired

\Delta{=}\mathrm{AUC}(2){-}\mathrm{AUC}(8)

’s 95% CI excludes 0, gray (thin) = it does not. Retail 14B served Int4 as frozen (all lengths, exact

\Delta

[CI], bf16: Appendix I).

Figure 4: Observation deprivation, by intervention: gold-clean false rejection with in-window observations withheld (the frozen baseline) vs. provided (gold replay); 95% Wilson intervals,

n_{\mathrm{clean}}{=}31/200

; arms coincide at

L{=}1

by construction.

Figure 5: No label-blind policy closes the oracle gap: any-reject pool (cross), majority pool (triangle), LOTO per-tool router (circle), calibrated fixed

L^{*}

(star), label-aware oracle bound (diamond, 95% CI); shaded band = headroom above the fixed unit; only the AppWorld/14B router beats it, nothing enters the band (Appendix J).

claim (main text)	evidence	status
Catch rises with the reviewed length on every verifier (Sec. 4.2)	Table 1 top; Appendices F , G	registered
Gold-clean false rejection rises in lockstep, the mandatory co-report (Sec. 4.2)	Table 1 top; Table A19 ; Fig. A8	registered
$J$ peaks at $L\in\{1,2\}$ in all six verifier–domain cells (Sec. 4.2)	Fig. 2; Table A9 (peak probabilities)	registered
The short–long contrast $J(2){-}J(8)$ excludes 0 in every cell, both resamplings (Sec. 4.2)	Table A9 ; Appendix AF (power, size)	registered
The peak is interior ( $L{=}2$ ) for retail 70B and 14B (Sec. 4.2)	Table A10 rows 2–3; Appendix R	registered contrast, derived argmax
Ranking itself decays in retail, and recalibration does not rescue the long window (Sec. 4.3)	Fig. 3; Figure A5 ; Tables A11 , A12 ; Appendix T	amended

Table A1: Claim–evidence–status crosswalk. Un-prefixed Table, Figure, and Section numbers refer to the main paper; “registered” abbreviates pre-registered primary and “amended” abbreviates amendment-registered, both defined in Table A2 .

status	results
pre-registered primary (frozen before any measurement)	the two registered series $\mathrm{catch}(L)$ , $\mathrm{FR}(L)$ and derived $J(L)$ in all six verifier–domain cells under the direction-agnostic outcome map, with the mandatory clean co-report (Table 1, Fig. 2, Sec. 4.2; Appendices F , G ); one registered short–long contrast per cell, item-level CI primary, task-cluster CI co-reported (Sec. 4.2)
amendment-registered	score-mode pass with per- $L$ recalibration (Sec. 4.3) and each closed-weight judge leg (Fig. 3; Appendix I ); position sweep (Sec. 4.3) and its worst-case readings (Appendix U ); per-step control (Appendix C ); observation-provision intervention and its predicted/inert substitute arms (Sec. 4.3; Appendices M , V ); inert-padding controls, dose target re-registered after the frozen usage logs falsified the original (Appendix N ); draft-clean re-judge and the $K{=}8$ harvest mining rule (Sec. 4.4; Appendix P ); three-way defer probe (Appendix O ); trained monitor, short-only regime and all-length ceiling role (Sec. 4.4; Appendix S ); compositional corpus, census, and cross-verifier re-judgments (Appendix X ); exemplar-protocol decomposition (Appendix Y ); placeholder rendering (Appendix M ); groundability stratification and per-anchor jackknife (Appendices M , R ); reasoning-dose contrasts (Appendix W ); held-out recipe validation (Sec. 3.4; Appendix T ); episode, partial-commit, and cascade projections (Appendices Z – AB ); label-validity panel, four registered questions and registered dispute set (Appendix Q ); deterministic natural-error harvest (Appendix L )
exploratory / derived (secondary analyses; nominal, uncorrected CIs)	policy-table derivations from the registered-series judgments (oracle bound, router, pooling; Table 1; Appendix J ); the defer probe’s selective- $J$ difference CI (Appendix O ); sampled natural harvest, a disclosed design deviation (Appendix L ); sampled-draft extension and interior-position release rates (Appendices P , AA ); prediction fidelity by window depth (Appendix V ); per-mode argmax side-notes (Appendix W.2 ). Non-exhaustive: each such reading is flagged where stated

Table A2: Confirmatory status of each result; un-prefixed Table, Figure, and Section numbers refer to the main paper. “Amendment-registered” means a dated, append-only amendment to the frozen pre-registration fixed the protocol and reading branches before the corresponding judgments or joins ran; from the compositional cross-verifier re-judgment onward, each amendment was also committed to version control before the first judgment of the leg it registers.

Appendix figures & tables43 assets

Supplementary material from the paper’s appendix.

Appendix

symbol	meaning (defining point)
$L$	reviewed-prefix length: the unit of verification (Sec. 1)
$L^{*}$	fixed unit a deployed policy chooses (Secs. 1, 3.4)
$\mathrm{catch}(L)$	P(bad prefix rejected): safety proxy (Sec. 3.1)
$\mathrm{FR}(L)$	P(matched clean prefix rejected) (Sec. 3.1)
$J(L)$	informedness $\mathrm{catch}-\mathrm{FR}$ (Sec. 3.1)
$J^{*}(L)$	per- $L$ recalibrated $J$ , score mode (Sec. 3.3)

Appendix

Table A3: Symbols used in the main paper.

stratum	definition	example	$n$
S-A	cross-customer scope: the write targets an entity owned by a different customer	cancel another customer’s order	40
S-B	wrong-argument variant: same tool and target as a gold write, $\geq$ 1 consequential argument replaced	exchange the requested item to an unrequested variant	100
S-C	unrequested extra write on the task’s own customer, no gold counterpart	modify an address the goal never mentions	60

Appendix

Table A4: Injected-error severity strata (retail;

n

of 200 bad items; registered mixture 20/50/30% with a 50/50 near/far argument-distance split, both achieved exactly).

	$L{=}1$	$L{=}2$	$L{=}3$	$L{=}5$	$L{=}8$
70B holistic	0.419	0.194	0.548	0.806	0.935
70B per-step	0.161	0.355	0.548	0.935	0.935
14B holistic	0.069	0.276	0.586	0.966	1.000
14B per-step	0.323	0.419	0.645	0.903	1.000

Appendix

Table A5: Clean false-reject vs

L

under the holistic (headline) and per-step + any-reject protocols; retail. Clean denominators:

n{=}31

, except the 14B holistic row’s complete-matrix

n{=}29

(denominator-policy note in Appendix C ’s text).

verifier	$L$	interior positions	end	$\bar{c}(L)$	$\Delta(L)$ [95% CI]
70B	8	0.883 (p4), 0.950 (p6)	0.970	0.934	0.036 [0.001, 0.076]
72B	8	0.700 (p4), 1.000 (p6)	1.000	0.900	0.100 [0.061, 0.139]
14B	8	0.237 (p4), 0.661 (p6)	1.000	0.633	0.367 [0.311, 0.418]
70B	5	0.867 (p2), 0.917 (p3)	0.915	0.899	0.016 [ $-$ 0.030, 0.063]
72B	5	0.500 (p2), 0.707 (p3)	0.960	0.722	0.238 [0.178, 0.300]
14B	5	0.298 (p2), 0.368 (p3)	0.971	0.546	0.425 [0.363, 0.484]

Appendix

Table A6: Catch by error position within the review window.

\bar{c}(L)

= position-marginal catch;

\Delta(L)

= window-end catch minus

\bar{c}(L)

(95% item-bootstrap CI). Position feasibility is bounded by gold-plan length; position 2 at

L{=}8

admits no anchors.

Figure A1: Catch by error position, as a heatmap (blank cells: structurally infeasible positions). The window interior is a blind spot that darkens as the verifier weakens; AppWorld’s deep anchors admit position 1, where the 14B’s catch at $L{=}8$ is exactly 0.

	ARG		EVID		OBS
verifier (domain)	short	long	short	long	short	long
70B (retail)	0.74	0.41	0.05	0.35	0.21	0.24
14B (retail)	0.27	0.08	0.09	0.12	0.55	0.73
72B (retail)	0.65	0.29	0.06	0.17	0.18	0.49
70B (AppWorld)	0.74	0.56	0.12	0.16	0.11	0.27
14B (AppWorld)	0.28	0.03	0.52	0.33	0.16	0.62

Appendix

Table A7: Dominant clean-rejection reasons, short vs long windows. ARG = hallucinated argument/precondition mismatch; EVID = insufficient supporting evidence; OBS = demands an observation that is not available.

Figure A2: Clean-prefix rejection reasons, short (

L\in\{1,2\}

) vs long (

L\in\{5,8\}

) windows, per cell (100% stacked;

n

above each bar). In every cell the long-window mass shifts from concrete (often hallucinated) argument mismatches toward unverifiable-evidence refusals.

	Llama-3.3-70B			Qwen2.5-14B			Qwen2.5-72B
$L$	catch	FR	$J$	catch	FR	$J$	catch	FR	$J$
1	0.821	0.320	0.501	0.705	0.340	0.365	0.905	0.435	0.470
2	0.916	0.580	0.336	0.816	0.455	0.361	1.000	0.875	0.125
3	0.932	0.610	0.322	0.884	0.715	0.169	1.000	0.940	0.060
5	0.900	0.595	0.305	0.995	0.920	0.075	1.000	0.930	0.070
8	1.000	0.815	0.185	1.000	0.950	0.050	1.000	0.950	0.050

Appendix

Table A8: AppWorld: the two registered series and derived informedness per verifier. The shape matches retail: catch rises (or saturates) with

L

while clean false-reject climbs, with a milder long-window decline for the 70B (

J(8){=}0.185

vs retail’s collapse to

{\sim}0

cell	$\arg\max_{L}J$	$J$ at peak	$P(\arg\max)$	$J(2){-}J(8)$	item 95% CI	cluster 95% CI
retail / Llama-3.3-70B	2	0.602	0.984	0.567	[0.398, 0.726]	[0.452, 0.680]
retail / Qwen2.5-14B	2	0.422	0.916	0.422	[0.243, 0.587]	[0.302, 0.536]
retail / Qwen2.5-72B	1	0.666	0.980	0.501	[0.322, 0.677]	[0.307, 0.668]
AppWorld / Llama-3.3-70B	1	0.501	0.9996	0.151	[0.068, 0.233]	[0.070, 0.234]
AppWorld / Qwen2.5-14B	1	0.365	0.540	0.311	[0.221, 0.397]	[0.215, 0.408]
AppWorld / Qwen2.5-72B	1	0.470	1.000	0.075	[0.025, 0.125]	[0.024, 0.130]

Appendix

Table A9: Registered short-vs-long contrasts and peak probabilities, all six open-weight cells. AppWorld 14B’s peak probability is split across

L{=}1

(0.540) and

L{=}2

(0.460):

J(1){=}0.365

J(2){=}0.361

, an effective tie inside the short regime.

Figure A3: Operating-point trajectories for all six open-weight cells (stars mark the

J

peak). Every cell walks toward the reject-everything corner as

L

grows; only the retail 70B/14B cells first move toward the ideal corner (

L{=}1{\to}2

), the two interior-peak cells.

statistic	value
retail 70B peak $J(2)$ (Sec. 4.2)	$0.602$ ; cluster $[0.472,0.727]$ , item $[0.450,0.743]$
retail 70B $J(2){-}J(1)$ (Sec. 4.2)	${+}0.256$ ; cluster $[0.085,0.436]$ , item $[0.030,0.467]$
retail 14B $J(2){-}J(1)$ (Sec. 4.2)	${+}0.144$ ; cluster $[0.009,0.295]$ , item $[{-}0.048,0.324]$
registered catch-rise McNemar, 70B retail (Sec. 4.2)	$b{=}3$ , $c{=}44$ : items caught only at $L{=}1$ vs. $L{=}8$
paraphrase short $-$ long contrast, 70B (Sec. 4.4)	$0.487$ $[0.311,0.656]$
retail $J^{}(2){\to}J^{}(8)$ , 70B / 72B / 14B † (Sec. 4.3)	$0.596\to 0.205$ / $0.503\to 0.030$ / $0.391\to 0.040$

Appendix

Table A10: Interval registry: main-text statistics stated in compressed form in the prose, in full here (95% CIs; main-text site in parentheses). † in-sample

J^{*}

: a maximum of

\mathrm{TPR}-\mathrm{FPR}

over thresholds drawn from the same score arrays it is evaluated on, hence a bound on what per-

L

recalibration could achieve rather than an estimate of it; the held-out rows below refit on training clusters and read on held-out ones.

judge	AUC(2)	AUC(8)	$\Delta$ [95% CI]	$P$	AUC(2)	AUC(8)	$\Delta$ [95% CI]	$P$
	retail				AppWorld
Llama-3.3-70B	0.81	0.61	$+0.204$ [0.071, 0.326]	.998	0.69	0.70	$-0.008$ [ $-$ 0.070, 0.053]	.38
Qwen2.5-72B	0.79	0.52	$+0.266$ [0.148, 0.378]	1.000	0.58	0.55	$+0.030$ [ $-$ 0.029, 0.087]	.83
Qwen2.5-14B	0.70	0.52	$+0.184$ [0.046, 0.320]	.995	0.65	0.58	$+0.073$ [0.010, 0.136]	.987
bf16 re-serve	0.71	0.46	$+0.251$ [0.121, 0.375]	.9998	–	–	–	–
DeepSeek-V4-Pro	0.89	0.53	$+0.362$ [0.233, 0.479]	1.000	0.69	0.64	$+0.050$ [ $-$ 0.006, 0.106]	.958

Appendix

Table A11: Threshold-artifact test, exact values (the main text shows this contrast as Figure 3; all lengths: Figure A5 ): the registered contrast, ranking AUC at

L{=}2

L{=}8

, paired

\Delta

(item-bootstrap,

B{=}5000

;

P

: fraction

\Delta>0

). Retail 14B served Int4 as frozen; indented row: its retail-only bf16 re-serve quantization control (the AppWorld leg is bf16 already).

domain	judge	$L{=}1$	2	3	5	8
retail	DeepSeek-V4-Pro	0.915	0.894	0.708	0.598	0.532
retail	Claude Opus 4.8	0.9995	0.990	0.964	0.839	0.786
retail	GPT-5.6 Sol	0.990	0.972	0.880	0.729	0.662
AppWorld	DeepSeek-V4-Pro	0.752	0.694	0.684	0.641	0.644
AppWorld	Claude Opus 4.8	0.909	0.865	0.864	0.858	0.867
AppWorld	GPT-5.6 Sol	0.904	0.781	0.763	0.747	0.746

Appendix

Table A12: Closed-weight judges: full verbalized-score ranking AUC

(L)

. GPT-5.6 Sol’s AppWorld series loses most rank between

L{=}1

and

L{=}2

and is flat thereafter; the registered contrast is

L{=}2

L{=}8

(Table A11 ).

Figure A4: Ranking AUC at every frozen length, every judge and both domains (open-weight solid, closed-weight dotted; retail 14B is the frozen Int4 serving). In retail every judge falls, including the highest-starting ones (decline magnitude is not monotone in starting AUC); in AppWorld only the weak 14B

2{\to}8

contrast excludes 0 (Table A11 ), the other five do not. Per-point intervals omitted for legibility.

Figure A5: Ranking AUC per judge at every frozen length, both domains, values in-cell; retail 14B is the frozen Int4 serving. The color scale floors at 0.45 so chance (0.5) sits at the dark end, marked on the bar. The $L{\in}\{2,8\}$ columns are the registered contrast of Table A11 (shown in the main text as Figure 3); the same series appear as curves in Figure A4 .

cell	$L^{*}$	$J(L^{*})$	$J_{\mathrm{or}}$	headroom	router $J$
retail 70B	2	0.602	0.903	0.302	0.607
retail 14B	2	0.431	0.966	0.535	0.431
retail 72B	1	0.666	0.871	0.205	0.619
AppWorld 70B	1	0.501	0.730	0.229	0.484
AppWorld 14B	1	0.365	0.720	0.355	0.471
AppWorld 72B	1	0.470	0.615	0.145	0.462

Appendix

Table A13: Oracle bound vs fixed

L

vs deploy-observable per-tool routing (LOTO cross-validated), per cell.

J_{\mathrm{or}}

= oracle informedness; headroom

=J_{\mathrm{or}}-J(L^{*})

cell	$J(L^{*})$	$J_{\mathrm{maj}}$	$\Delta J_{\mathrm{maj}}$ [95% CI]	$\Delta J_{\mathrm{router}}$ [95% CI]
retail 70B	0.602	0.272	$-0.329$ [ $-0.506$ , $-0.153$ ]	$+0.005$ [ $+0.000$ , $+0.015$ ]
retail 14B	0.431	0.190	$-0.241$ [ $-0.431$ , $-0.058$ ]	$0.000$ [ $-0.103$ , $+0.103$ ]
retail 72B	0.666	0.379	$-0.287$ [ $-0.465$ , $-0.108$ ]	$-0.047$ [ $-0.169$ , $+0.053$ ]
AppWorld 70B	0.501	0.383	$-0.118$ [ $-0.196$ , $-0.042$ ]	$-0.017$ [ $-0.073$ , $+0.044$ ]
AppWorld 14B	0.365	0.216	$-0.149$ [ $-0.241$ , $-0.053$ ]	$+0.105$ [ $+0.024$ , $+0.188$ ]
AppWorld 72B	0.470	0.070	$-0.400$ [ $-0.481$ , $-0.315$ ]	$-0.008$ [ $-0.050$ , $+0.033$ ]

Appendix

Table A14: Label-blind verdict-pooling and routing vs fixed

L^{*}

, per cell. Majority = reject iff

\geq

3 of the 5 lengths reject.

\Delta J

CIs are paired item-bootstrap (

B{=}5000

) with the cell’s full-data

L^{*}

held fixed; for the router, the LOTO per-item

L

-assignments are also held fixed under resampling (conditional-on-fitted-policy CI).

cell	$w^{*}$	max long-window edge over always-reject
retail 70B	0.282	$+0.022$
retail 14B	0.406	$0.000$
retail 72B	0.186	$+0.028$
AppWorld 70B	0.381	$+0.080$
AppWorld 14B	0.385	$+0.025$
AppWorld 72B	0.000	$0.000$

Appendix

Table A15: Asymmetric-weight sensitivity per cell.

w^{*}

= smallest false-rejection weight above which the optimum stays at

L\in\{1,2\}

for all larger

w

; last column = the largest margin by which any long-window optimum beats free always-reject anywhere below

w^{*}

(0 = never beats it).

	baseline			obs-provided			clean McNemar
$L$	catch	FR	$J$	catch	FR	$J$	$b/c$ ( $p$ )
retail ( $n_{\text{bad}}{=}200$ , $n_{\text{clean}}{=}31$ )
1	0.765	0.419	0.346	0.765	0.419	0.346	$0/0$
2	0.795	0.194	0.601	0.760	0.161	0.599	$3/2$ (1.0)
3	0.895	0.548	0.347	0.800	0.258	0.542	$10/1$ (.012)
5	0.915	0.806	0.109	0.740	0.290	0.450	$16/0$ ( $3.1{\times}10^{-5}$ )

Appendix

Table A16: Observation-provision intervention, per length: baseline vs. observation-provided (obs) legs on the byte-identical frozen corpora, primary verifier. McNemar cells are (base-only rejects / obs-only rejects) among paired items;

L{=}1

is the byte-identity control (verdicts agree on all

231

retail and

390

AppWorld items). Baseline

J

’s last digit can differ from the main paper’s Table 1 by source-file rounding precision.

leg	$L$	FR ${}_{\text{froz}}$	FR ${}_{\text{pad}}$	$\Delta$ FR $[$ 95% CI $]$	$b/c$ ( $p$ )	catch ${}_{\text{froz}}$	catch ${}_{\text{pad}}$
T-a	1	0.419	0.226	$-0.194$ $[-0.314,-0.065]$	$6/0$ (.031)	0.765	0.730
T-a	2	0.194	0.258	$+0.065$ $[0.000,0.172]$	$0/2$ (.50)	0.795	0.760
T-b	8	0.935	0.968	$+0.032$ $[0.000,0.107]$	$0/1$ (1.0)	0.970	1.000

Appendix

Table A17: Inert-padding controls, item-paired against the frozen retail 70B rows (

n_{\text{clean}}{=}31

n_{\text{bad}}{=}200

per length).

\Delta

FR is the paired clean-side change (task-cluster bootstrap,

B{=}5000

); McNemar cells are frozen-only / pad-only clean rejects.

retail ( $n_{\text{bad}}{=}200$ , $n_{\text{clean}}{=}31$ )
$L$	defer ${}_{\text{clean}}$	defer ${}_{\text{bad}}$	cov ${}_{\text{clean}}$	sel. FR	sel. catch	sel. $J$
1	0.000	0.000	1.000	0.161	0.740	0.579
2	0.065	0.195	0.935	0.103	0.727	0.623
3	0.323	0.330	0.677	0.190	0.799	0.608
5	0.452	0.415	0.548	0.471	0.778	0.307
8	0.613	0.380	0.387	0.750	0.927	0.177

Appendix

Table A18: Three-way probe, per length: defer rates by side, decided coverage on clean windows, and selective (decided-subset) metrics. Within-leg readouts only: the prompt-vocabulary change makes this leg not byte-comparable to the frozen instrument.

retail
	Llama-3.3-70B		Qwen2.5-14B		Qwen2.5-72B
$L$	gold [95% Wilson]	draft	gold [95% Wilson]	draft	gold [95% Wilson]	draft
1	$0.419$ [0.26, 0.59]	$0.092$	$0.065$ [0.02, 0.21]	$0.007$	$0.129$ [0.05, 0.29]	$0.007$
2	$0.194$ [0.09, 0.36]	$0.215$	$0.290$ [0.16, 0.47]	$0.017$	$0.419$ [0.26, 0.59]	$0.168$
3	$0.548$ [0.38, 0.71]	$0.287$	$0.600$ [0.42, 0.75]	$0.055$	$0.581$ [0.41, 0.74]	$0.217$
5	$0.806$ [0.64, 0.91]	$0.282$	$0.967$ [0.83, 0.99]	—	$0.903$ [0.75, 0.97]	—

Appendix

Table A19: Clean false rejection per cell and length: registered gold-clean rate with its 95% Wilson interval (the upper bound), beside the draft-clean rate on harvested non-divergent executor drafts (the lower bound; “—” where no draft population exists). Retail denominators are

n{=}29

–

31

per length, AppWorld

n{=}200

; retail draft

n{=}414/298/272

L{=}1/2/3

and

163/119

L{=}5/8

(deterministic harvest, primary verifier only). Generated by p20_z2_fr_intervals.py .

training support	$L$	catch	clean FR	$J$ [95% CI]	AUC	draft FR
short-only $\{1,2\}$	1	.975	.355	.620 [.441,.783]	.971	.007
	2	.985	.484	.501 [.324,.646]	.912	.009
	3	.965	.645	.320 [.162,.465]	.865	.072
	5	.995	.871	.124 [-.010,.308]	.766	.122
	8	.975	.935	.040 [.000,.102]	.672	.186
all-length ceiling	1	.980	.290	.690 [.501,.846]	.973	.014

Appendix

Table A20: Three-seed ensemble, task-disjoint OOF results for the same-backbone monitor. Main clean FR uses matched gold twins; draft FR uses the held-out Qwen2.5-14B-executor clean panel. Short-only is the length-generalization experiment. All-length is a seen-length ceiling because labeled

L{=}8

twins enter training.

support	seed	catch	clean FR	$J$	AUC
short	20260720	.975/.980/.980/1.000/.985	.290/.452/.677/.871/.968	.685/.528/.303/.129/.017	.951/.889/.856/.752/.661
	20260721	.975/.975/.965/.985/.960	.452/.484/.677/.871/.871	.523/.491/.288/.114/.089	.951/.913/.858/.761/.658
	20260722	.975/.980/.975/.990/.975	.290/.419/.645/.839/.903	.685/.561/.330/.151/.072	.965/.916/.867/.767/.689
all	20260720	.945/.960/.955/.935/.910	.355/.323/.323/.355/.290	.590/.637/.632/.580/.620	.953/.911/.920/.878/.888
	20260721	.980/.985/.960/.935/.870	.258/.323/.387/.355/.290	.722/.662/.573/.580/.580	.980/.941/.918/.882/.882
	20260722	.970/.985/.970/.955/.940	.290/.323/.419/.387/.387	.680/.662/.551/.568/.553	.961/.920/.929/.894/.921

Appendix

Table A21: All registered seed curves. Each cell is the five-entry vector for

L=1/2/3/5/8

; seed-level metrics use that seed’s OOF predictions before ensembling.

Figure A6: The trained curves at a glance:

J(L)

for the zero-shot 14B reference (no interval in the source), the short-only monitor (trained on

L\in\{1,2\}

twins), and the all-length monitor (three-seed ensembles, task-disjoint folds; bands: 95% bootstrap CIs; values: Tables A20 , A21 ). Short-only training does not transfer to long windows; all-length supervision repairs the collapse within the seen length support.

cell	protocol	regret	$\Pr(\hat{L}^{*}{\leq}2)$	contrast [CI]	$J(\hat{L}^{*}){-}J(2)$
retail 70B	LOCO	0.063	1.000	0.498 [0.328, 0.669]	$+0.000$
retail 14B	LOCO	0.054	1.000	0.414 [0.278, 0.545]	$+0.000$
retail 72B	LOCO	0.082	1.000	0.602 [0.413, 0.782]	$+0.175$
AppWorld 70B	split	0.001	1.000	0.197 [0.121, 0.271]	$+0.163$
AppWorld 14B	split	0.031	1.000	0.301 [0.208, 0.402]	$-0.012$
AppWorld 72B	split	0.000	1.000	0.399 [0.313, 0.477]	$+0.345$

Appendix

Table A22: Held-out validation of the recipe. Regret

=\max_{L}J_{\mathrm{test}}(L)-J_{\mathrm{test}}(\hat{L}^{*})

(an upper estimate; protocol named per row). Contrast

=\max_{L\leq 3}J-\max_{L\geq 5}J

on held-out clusters, task-cluster 95% CI.

cell	$J_{\mathrm{w}}(1)$	$J_{\mathrm{w}}(8)$	LB(1)	LB(8)	$\Delta_{1-8}$ [CI]	$\bar{J}(8)$
retail 70B	0.346	$-0.052$	$+0.109$	$-0.233$	0.398 [0.200, 0.719]	$-0.001$
retail 14B	0.259	$-0.763$	$+0.043$	$-0.868$	1.021 [0.820, 1.195]	$-0.367$
retail 72B	0.666	$-0.300$	$+0.445$	$-0.454$	0.966 [0.746, 1.170]	$-0.100$
AppWorld 70B	0.501	0.018	$+0.373$	$-0.172$	0.483 [0.354, 0.640]	$+0.096$
AppWorld 14B	0.365	$-0.950$	$+0.229$	$-0.973$	1.315 [1.224, 1.408]	$-0.422$
AppWorld 72B	0.470	$-0.533$	$+0.351$	$-0.694$	1.004 [0.826, 1.212]	$-0.189$

Appendix

Table A23: Worst-case and randomized-offset readings.

J_{\mathrm{worst}}(L)=\min_{p}\mathrm{catch}(L,p)-\mathrm{FR}(L)

(plug-in); LB

=J^{\mathrm{LB}}_{\mathrm{worst}}

, the simultaneous bound;

\bar{J}(8)

= randomized-offset informedness.

L{=}1

is exhaustive by construction; the

L{=}8

minimum ranges over a 3-point grid (an upper bound on the true minimum).

$L$	baseline	inert	predicted	oracle
1	.765/.419/.346	.765/.419/.346	.765/.419/.346	.765/.419/.346
2	.795/.194/.602	.780/.194/.587	.805/.226/.579	.760/.161/.599
3	.895/.548/.347	.860/.484/.376	.910/.452/.458	.800/.258/.542
5	.915/.806/.108	.870/.774/.096	.880/.581/.299	.740/.290/.450
8	.970/.935/.035	.985/.903/.082	.985/.903/.082	.780/.290/.490

Appendix

Table A24: Observation substitutes, retail 70B: catch / FR /

J

per arm. All four arms render identically at

L{=}1

	$L{=}1$	2	3	5	8
AUC, highest	0.990	0.972	0.880	0.729	0.662
AUC, lowest	0.987	0.977	0.871	0.683	0.652
$J^{*}$ , highest	0.923	0.796	0.596	0.434	0.324
$J^{*}$ , lowest	0.945	0.888	0.607	0.253	0.238

Appendix

Table A25: Closed-judge effort dose, retail: full curves per arm (highest vs lowest admissible reasoning effort).

domain	mode	$L{=}1$	2	3	5	8
retail	think ON	0.725	0.721	0.496	0.315	0.029
retail	think OFF	0.516	0.289	0.374	0.091	0.035
AppWorld	think ON	0.493	0.305	0.304	0.271	0.231
AppWorld	think OFF	0.342	0.249	0.221	0.189	0.186

Appendix

Table A26: Open-weight reasoning verifier (Qwen3-32B),

J(L)

per mode and domain; complete-length item sets per leg.

	70B Llama-3.3	14B Qwen2.5 (Int4)	72B Qwen2.5
ALL	$+0.450$ [0.298, 0.605]	$+0.876$ [0.763, 0.957]	$+0.500$ [0.333, 0.629]
$s{=}1$	$+0.500$ [0.240, 0.781]	$+0.938$ [0.769, 1.000]	$+0.471$ [0.250, 0.633]
$s{=}2$	$+0.500$ [0.250, 0.765]	$+0.867$ [0.692, 1.000]	$+0.448$ [0.217, 0.667]
$s{=}4$	$+0.417$ [0.250, 0.647]	$+0.829$ [0.595, 1.000]	$+0.571$ [0.314, 0.829]

Appendix

Table A27: Compositional corpus,

\max_{L\leq 3}J-\max_{L\geq 5}J

with task-cluster 95% CIs;

s

= distance from the first error component to the judged anchor.

$L$	4-exemplar	control
1	.623/.000/.623	.755/.423/.332
2	.723/.192/.531	.774/.192/.581
3	.799/.462/.337	.899/.500/.399
5	.792/.462/.331	.931/.808/.123
8	.786/.385/.402	.969/.923/.045

Appendix

Table A28: One frozen 4-exemplar protocol vs the same-config zero-shot control (catch / FR /

J

; 15 clusters, 159 bad / 26 clean complete-length items).

	retail 70B			AppWorld 70B			draft 70B
$L$	$S_{\mathrm{bb}}$	$\mathbb{E}[c]$	cost	$S_{\mathrm{bb}}$	$\mathbb{E}[c]$	cost	$S_{\mathrm{bb}}$	cost
1	0.133	3.06	18.2	0.221	3.80	13.6	0.677	7.6
2	0.558	3.11	5.2	0.113	1.82	14.9	0.521	5.4
3	0.185	1.72	9.9	0.137	1.60	12.4	0.470	5.1
5	0.079	1.19	20.0	0.226	1.41	8.3	0.570	4.0
8	0.065	1.00	26.9	0.185	1.00	9.4	0.723	2.4

Appendix

Table A29: Episode projections at

N{=}8

under halt-on-reject accounting (gold-clean;

B{=}5000

task-cluster bootstrap).

S_{\mathrm{bb}}

= clean episode survival;

\mathbb{E}[c]

= expected calls issued before halting or completing; cost =

10^{3}

tokens per completed episode. Bold: the cheapest unit per column. Draft scenario: draft-clean FR spliced from two harvest populations (

L{\leq}3

L{\in}\{5,8\}

\hat{\rho}

proxied from gold-clean; point estimates only.

Figure A7: Verifier tokens per completed

N{=}8

episode under halt-on-reject accounting (log scale; stars: per-scenario minimum; whiskers: 95% cluster-bootstrap intervals where Table A29 ’s source reports them; the draft-clean scenario carries none). The retail

L{=}8

upper endpoint is unbounded: resampled FR reaches 1 in 11.7% of draws and the episode never completes.

$L$	$\ell_{\mathrm{PC}}$ [95% CI]	thr PC	thr B	cost PC	cost B	release
1	0.84 [0.70,0.94]	0.839	0.839	942	942	0.185
2	1.39 [1.13,1.63]	0.694	0.645	667	717	0.170
3	1.58 [1.13,2.07]	0.527	0.452	671	782	0.130
5	2.16 [1.56,2.89]	0.432	0.065	615	4,123	0.015
8	4.19 [3.44,4.92]	0.524	0.065	414	3,362	0.015

Appendix

Table A30: Partial commit vs whole-window binary on the frozen retail 70B per-step verdicts (

n_{\mathrm{clean}}{=}31

n_{\mathrm{bad}}{=}200

\ell

= committed clean steps per call; thr = committed fraction of clean steps; cost = tokens per committed clean action; release = terminal bad-write release (identical for PC and B).

cell	best fixed $J^{*}$	cascade $J^{*}$	$\Delta$ [95% CI]	cost
retail 70B	0.596 ( $L{=}2$ )	0.636	$+0.040$ [0.007,0.071]	$1.12\times$
retail 14B	0.423 ( $L{=}3$ )	0.406	$+0.017$ [0.000,0.061]	$1.24\times$
retail 72B	0.683 ( $L{=}1$ )	0.510	$+0.007$ [ $-$ 0.023,0.054]	$1.61\times$
AppWorld 70B	0.526 ( $L{=}1$ )	0.432	$+0.048$ [0.012,0.089]	$1.23\times$
AppWorld 14B	0.375 ( $L{=}1$ )	0.325	$+0.015$ [ $-$ 0.006,0.037]	$1.28\times$
AppWorld 72B	0.493 ( $L{=}1$ )	0.186	$+0.040$ [ $-$ 0.060,0.140]	$2.04\times$

Appendix

Table A31: Test-tuned score cascades vs test-tuned fixed-

L

(

B{=}5000

paired bootstrap at the tuned operating points).

\Delta

= best

L_{s}{=}2

cascade minus fixed(

2

); cost ratio = clean-side expected tokens vs fixed(

2

Figure A8: The two registered series, one panel per retail verifier:

\mathrm{catch}(L)

on injected-bad prefixes (

n{=}200

) and gold-clean false-reject(

L

) (

n{=}31

); error bars are 95% Wilson intervals per point. Catch rises monotonically on all three verifiers, but clean false-reject rises with it: at

L{=}8

the 70B rejects 94% of clean gold prefixes and the 14B/72B reach catch

=

false-reject

=1.000

exactly: blanket rejection, not better discrimination.

Figure A9: The lockstep rise and the deployment-facing clean series (primary verifier, retail; 95% Wilson intervals; main paper Section 4.4). Catch (

n{=}200

) and gold-clean false-reject (

n{=}31

) rise together; false-reject on correct executor drafts (

n{=}414/298/272

; cap

L{=}3

) is far lower but rises too.

Figure A10: Bootstrap distribution of the informedness argmax (retail): the fraction of

B{=}5000

paired resamples in which

\arg\max_{L}J(L)=L

. The peak location is decisive per verifier:

L{=}2

in 98.4% (70B) and 91.6% (14B) of resamples,

L{=}1

in 98.0% (72B).

Figure A11: The decision rule instantiated on the measured curves: amortized per-action supervision cost

s(L;N)=(F+iL)\,\lceil N/L\rceil/N

for an episode of

N

supervised actions (

F{\approx}655

i{\approx}135

tokens), plotted at

N{=}8

with whiskers spanning

N\in\{4,16\}

(the

\lceil N/L\rceil

term is the endpoint effect that makes

L{=}3,5

N

-sensitive), against

J(L)

. Extending the unit past the starred

L^{*}

pays only if informedness is priced below the chord’s own slope, and each printed

\bar{\lambda}_{N}

is that slope at the plotted

N{=}8

255/264/433

tokens per action per unit

J

for the 70B’s

L^{*}\!\to\!3/5/8

chords,

{\approx}1{,}980

for the 72B’s

1\to 2

extension. Main Eq. 7 gives the

N\to\infty

limit

F(1/L-1/L^{\prime})/\Delta J

, which the 70B reads as

428/399/433

; because every

L^{*}

here divides

N

while

L^{\prime}\in\{3,5\}

does not, a finite episode charges a partial final call at the longer unit and the chord can only fall, never rise (the two agree exactly wherever both lengths divide

N

, as at

L^{*}{\to}8

and for the 72B). Either reading, a safety-co-primary deployment never accepts an extension past the peak.

Figure A12: Retail 70B catch(

L

) by injected-error severity (Table A4 strata). The most severe stratum (S-A) is caught at 1.00 at every

L

(the 72B replicates this); the length signal lives entirely in the subtle strata, where the catch climb is bought with the clean false-reject rise.

Figure A13: The short-window advantage is not a severity-mixture artifact:

\Delta_{p}(w)=\max_{L\leq 3}J_{w}-\max_{L\geq 5}J_{w}

over the full mixture grid of subtle-stratum weight

w

(bands: pointwise 95% bootstrap CIs; markers: each verifier’s worst-case mixture; dashed line: the observed corpus mixture; strata: Table A4 ). A post-hoc, severity-only reading; S-A is excluded (zero observed in real E3).

Figure A14: Position sub-design at

L{=}8

(retail): catch with the injected error at step 4 vs step 8. On a long window the weaker verifier essentially inspects only the end of the prefix (Table A6 gives the full grid).

AI Agent Monitoring

AI agent monitoring tracks actions, tool use, and execution traces to detect failures, misuse, drift, and policy violations during operation.

See all 280 papers

Sep 16, 2026

Beyond Accuracy: How Procedural Traces Shift the Decision Criterion of LLM Overseers

Organizations increasingly use oversight loops where one large language model (LLM) audits another's outputs alongside procedural traces of claimed steps. A common concern about such LLM-as-a-judge pipelines is that detailed traces make overseers gullible. Using signal detection theory, we audit five LLM overseers on 19 compliance tasks (4,551 analyzed judgments), varying only trace detail and evidence labeling. With disconfirming evidence always visible, error detection remains near ceiling. Instead, elaborate traces shift the decision criterion toward rejection, increasing false alarms in susceptible overseers. Without option labels, human-validated reason coding shows about 60% of false alarms cite an inability to tie evidence to its option. Labels eliminate this stated reason, yet residual rejection of correct work persists in those overseers and rises with trace detail. Procedural traces thus act as governance artifacts that shape oversight decisions. AI auditors should be evaluated by their decision criterion and false-alarm behavior, alongside accuracy.

Zihan Chen, Di Zhu, Lei Zheng +1

Stevens Institute of Technology Hoboken, NJ, USA · University of Massachusetts Boston Boston, MA, USA · Stony Brook University Stony Brook, NY, USA

LLM-as-a-Judge LLM Auditing

Ask PDF

Sep 7, 2026

The Oversight Gap: What LLM Safety Monitors Miss, and Why It Is Not Capability

Several properties safety monitors are asked to certify, among them cross-tenant noninterference, sandbagging and evaluation awareness, are 2-safety hyperproperties, witnessed only by two executions. The standard consequence is a binary impossibility: one trace cannot decide them. We replace the binary with a measurement. A tight bound puts the balanced accuracy of any single-trace monitor at

\tfrac12+\tfrac12\,TV(P_0,P_1)

, turning undecidability into a graded detectability frontier and defining an oversight gap: a monitor's shortfall below it. On a leak family with closed-form

TV

, nine LLM monitors are optimal at

TV=0

but capture little signal as

TV

grows; at

TV=1

, where a 20-line membership check scores

100\%

, they average

60.9\%

. That shortfall is mostly not capability: naming what to check closes

61\%

of it while leaving the

TV=0

control at chance. The same split runs through a

2{\times}2

factorial: an imagined second run leaves monitors at chance (

50.4\%

) while the same rule on an executed second run reaches

90.0\%

, and a stored oracle without a comparison procedure yields only

68.2\%

. Information and procedure are each necessary and neither is capability. Under nondeterminism, replay tracks a closed-form

k

-replay curve only under the right projection, and a projection frontier shows the resulting dilemma is forced: narrow misses

98.6\%

of off-channel leaks, broad flags

75.7\%

of clean traffic, and attainable accuracy decays like

1/(qm)

in the benign-variation rate and the channel count. Finally, two frontier LLM judges certified an earlier version of our own benchmark as sound while a sign test found a directional bias (

p=2.7\times10^{-5}

) that invalidated three of our findings. Construction validity for hyperproperty benchmarks should be proved mechanically, not audited by models.

Xin Xu

Carnegie Mellon University

LLM Safety Benchmarks LLM Safety Evaluation Language Model Auditing

Ask PDF

May 7, 2026

PrefixGuard: Online Failure Warning and Trace-Grounded Diagnosis for LLM Agents

Large language model (LLM) agents execute multi-step tasks, but terminal verdicts arrive too late for intervention. Agent traces mix messages, tool calls, and feedback, making it difficult to learn failure signals from terminal outcomes. Even if a model learns to predict failure from execution prefixes, its predictions alone cannot show which actions, feedback, or unmet conditions need inspection. Existing work studies online failure prediction and diagnosis from completed trajectories, yet few methods connect timely warning to a compact symbolic model of execution behavior. We introduce PrefixGuard, a neuro-symbolic method that uses learned events to connect online failure warning with finite-state execution diagnosis. A gated recurrent unit (GRU) predicts horizon-specific failure risk from these events. A deterministic finite automaton (DFA) organizes their histories into compact risk-labeled paths for replay and rule-monitor composition. Across all 24 benchmark--horizon settings, PrefixGuard exceeds the strongest evaluated baseline in mean test area under the precision--recall curve (AUPRC). Average gains range from 12.0 to 21.9 percentage points. At a nominal 10% FAR budget, averaged test recall spans 56.1%--93.0% across benchmarks. On one frozen

τ^2

-Bench DFA, model checking five specifications identifies violations in 21 of 265 benchmark-successful runs.

Xinmiao Huang, Jinwei Hu, Qisong He +4

University of Liverpool · Université Grenoble Alpes

Fault Detection AI Agent Monitoring Finite-State Machine Neuro-Symbolic AI LLM Agent Reliability Fault Diagnosis

Ask PDF

Abstract

AI Agent Monitoring

Explore similar work

Beyond Accuracy: How Procedural Traces Shift the Decision Criterion of LLM Overseers

The Oversight Gap: What LLM Safety Monitors Miss, and Why It Is Not Capability

PrefixGuard: Online Failure Warning and Trace-Grounded Diagnosis for LLM Agents