When Better Policies Leave Harder Failures: Coverage Shift in VLM Success Monitoring

Isabel Guan; Dingyuan Zhang; Shipeng Lyu; I-Ming Chen

cs.ROOct 17, 2026

When Better Policies Leave Harder Failures: Coverage Shift in VLM Success Monitoring

Authors: Isabel Guan, Dingyuan Zhang, Shipeng Lyu, I-Ming Chen

Organizations: The Hong Kong University of Science and Technology · Nanyang Technological University · The Hong Kong Polytechnic University

Abstract

Can a better robot policy become harder to monitor? We study a fixed vision-language model (VLM) that judges success while a policy improves. In a controlled 20-seed tray-placement experiment, training selects only simulator-verified successes. Success rises from 66.8% to 97.0%, and accepted hover failures fall from 5.0% to 0.9% of executions. Yet the judge accepts 50.0% of remaining hovers, versus 31.1% under untrained sampling. These residual failures more often lie below the tray rim, which can conceal the gap beneath a grasped cube. A separate boundary probe accepts 148/160 below-rim hovers and 0/160 above-rim hovers. We call this change in conditional monitoring sensitivity coverage shift: learning changes which failures reach a fixed judge. A broader diagnosis covers nine judges and 1,392 stimuli. Judge-filtered experiments across eight settings further show heterogeneous effects of state-dependent acceptance. The results distinguish task improvement from failure detectability and motivate reporting absolute misses, conditional misses, and audit coverage together.

Figures & tables

Figure 2: How successful learning can leave less detectable failures. (a) C1 fits a controller-parameter distribution using simulator-confirmed successes. VLM verdicts are recorded only for monitoring. (b) The author’s schematic shows a grasped cube above and below the tray rim; both remain off the floor and fail the task. F1 counts come from 40 paired scenes. The bars instead use the revised 20-seed audit and report equal-seed means over rounds 1–5. (c) In that audit, accepted hovers decrease per execution but increase as a fraction of hovers. The judge, prompt, camera, and threshold stay fixed. F1 and the revised audit are separate evidence, and the drawing illustrates geometry rather than identifying the judge’s internal computation.

Figure 3: Errors depend on failure type and view. Main-split acceptance percentages for six selected judges, reproduced from the archived diagnostic figure. Stack is above tray. Held_on is excluded; the missing stack-edge type is hatched and uses the pooled task failure rate in prediction. Claude Opus 5.5 has two views only. Shared physical states across views make camera-dependent ambiguity visible. The full nine-judge protocol and coverage are in the supplement.

Judge	Stack grasp weight	Tray grasp weight
Gemini 3.6-flash	$.10\ [.02,.20]$	$-.29\ [-.47,-.17]$
GPT-6 Astra	$-.02\ [-.22,.17]$	$-.05\ [-.24,.12]$
Claude Fable 5.1	$.09\ [.02,.15]$	$.02\ [-.11,.32]$
Claude Opus 5.5	$-.32\ [-.49,-.21]$	$-1.47\ [-2.33,-1.06]$
Gemini 3.8-flash	$-.41\ [-.56,-.32]$	$-1.41\ [-2.20,-1.26]$

Table 1: Descriptive logistic cue weights, with 95% scene-bootstrap intervals (1,000 resamples). Ideal-observer weights are

-.86/-.57

for stack/tray; for Opus’s two-view subset they are

-.78/-.53

. The registered weak-use criterion is an upper bound above

-.3

with a negative ideal weight, not a test of perceptual or causal equivalence.

Figure 4: A coverage gap near the success boundary. Left: historical T1 hover composition at V1; map values precede smoothing (the all-zero M1/tray/V1 prediction is uniform after smoothing for KL). Right: the preregistered F1 diagnostic probe accepts 0/160 above-rim hovers (D) and 148/160 below-rim hovers (D+), across 40 valid scenes. This is a state-coverage comparison, not evidence that the judge changed over time.

Metric (%)	N0	R0	C1
True success	66.81	66.69	96.99
Hover incidence	15.92	17.64	1.75
Accepted given hover	31.09	41.39	49.98
Below rim given hover	31.26	45.72	52.65
Hover and accepted / execution	4.97	7.21	0.86
Hover count	1,592	1,764	175

Table 2: Common-reset mechanism run, M1/tray/V1. Percentages are equally weighted seed-rate means (20 seeds, 500 observations per seed and arm, rounds 1–5). Counts pool seeds. In particular,

86/175=49.14\%

differs from the seed mean

49.98\%

Figure 5: The shift appears across seeds. Each point is one seed’s ratio pooled over rounds 1–5; horizontal lines are equal-seed means. Truth filtering (C1) leaves 4–15 hovers per seed. The untrained (N0), refit-only (R0), and truth-filtered arms use the revised common-reset protocol.

Cell	Frozen failure set	C3 failure	C2* failure	C3 $-$ C2* [95% interval]	C3 success	C2* success
M1 / stack / V1	behind	0.4	0.2	$+0.2\ [+0.0,+0.6]$	95.2	95.2
M1 / stack / V2	hover	5.6	2.0	$+3.6\ [+1.6,+5.6]$	87.0	93.6
M1 / tray / V1	hover	3.2	3.0	$+0.2\ [-3.2,+3.2]$	94.4	95.0
M1 / tray / V2	edge, hover	16.0	10.0	$+6.0\ [+4.6,+7.8]$	83.4	88.2
J1 / stack / V1	behind, hover	4.2	2.6	$+1.6\ [-1.4,+5.4]$	92.4	94.4
J1 / stack / V2	hover	5.6	2.4	$+3.2\ [+0.8,+5.8]$	85.6	93.8

Table 3: Revised C3/C2* experiment: independent final evaluation, five paired seeds per cell, 100 rollouts per arm and seed. All values are percentages except differences in pp. Intervals are descriptive paired-bootstrap intervals, not multiplicity-adjusted. M1/tray/V1’s hover endpoint is a control type; no blind spot was detected there in the original main-set map.

Figure 6: Two controls for error structure, with different protocols. (a) Historical final target failure differences: C3 minus C1 (circles and seed-bootstrap intervals) and C2 minus C1 (squares, means only); the no-blind-spot cell has no T2 test. (b) Later C3 minus C2* differences. Each interval uses five paired seeds and is unadjusted. Panels must not be added or interpreted as a single decomposition.

Cell	C3	C4	C6	C3t	CV
J1 / stack / V2	85.6	96.8	97.0	97.0	97.0
M1 / stack / V2	87.0	97.8	97.6	97.6	97.4
M1 / tray / V2	84.6	91.6	95.4	95.4	95.6
J1 / tray / V1	75.6	84.4	89.2	81.2	94.0
J1 / tray / V2	96.4	96.4	97.8	96.6	97.4
M1 / stack / V1	95.2	97.2	97.8	95.6	94.8

Table 4: Historical final true success (%, five-seed means). C1 is 97.0% throughout; C2 ranges from 94.6–96.4%. C3t equals C6 by construction except in the three inseparable cells. All filters use the same training budget; CV has extra visual information and query cost.

Figure 7: Multi-view diagnosis and the complete training controls. Left: one executed stacking failure across four views with archived M1 scores. Right: the filtering loop and controls used across the eight settings. This broader overview complements the C1 mechanism schematic in Fig. 2 . Six training rounds precede independent final evaluation in the historical and permutation loops; the common-reset mechanism audit instead pools rounds 1–5.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Class	Indices	$S$	Release offset (cm)	$h$ (cm)	Release	Retreat (cm)	Support
stack	SUCC	c00–c05	2, 2, 1	$r=0,\ 0.7,\ 1.4$ along $\phi$	0.2	yes	2 or 10	goal
	F-hover	c06–c09	2, 1	$r=0,\ 1.6$ along $\phi$	1.4 or 2.4	no	–	goal
	F-fell	c10–c11	1	$r=2.4,\ 3.0$ along $\phi$	0.4	yes	8	goal
	F-behind	c12–c13	2, 1	$(\Delta x,\Delta y)=(-4.7,0),\ (-6.0,0)$	0.2	yes	8	table
tray	SUCC	c00–c05	2, 2, 1	$r=0,\ 1.0,\ 2.0$ along $\phi$	0.4	yes	2 or 10	goal
	F-held	c06–c09	2, 1	$r=0,\ 1.5$ along $\phi$	3.0 or 5.0	no	–	goal

Appendix

Table 5: Design conditions per scene: 14 in stack, 15 in tray.

r

is the release offset from the goal centre along the scene’s direction

\phi

;

h

is the nominal height of the cube bottom above the support (in tray the rim is 2.0 cm above the floor, so F-held holds the cube 1 or 3 cm above it); a retreat of 5 cm or more parks the arm.

S

(success-like appearance: 2 high, 1 mid, 0 low) is given per offset or distance, in the order listed. Indices run over the offsets and, within each, the near and far retreat (SUCC), and over the heights and, within each, the offsets (F-hover, F-held).

Cue	Question	Correct answer is Yes when
stack
contact	Is the red cube touching the green cube?	$m<0.04$ and $\|p_{z}-g_{z}-0.04\|<0.003$ ; or on_table holds and $m-0.04<0.003$
grasped	Is the robot gripper currently holding the red cube?	the grasp flag is set
on_top	Is the red cube resting directly on top of the green cube?	$m<0.04$ , $\|p_{z}-g_{z}-0.04\|<0.003$ , and not grasped
on_table	Is the red cube resting on the table surface?	$\|p_{z}-0.02\|<0.004$ and tilt below $10^{\circ}$
tray

Appendix

Table 6: Cue probes and their simulator truth. Each question is sent followed by “Answer with Yes or No.”

m=\max(|l_{x}|,|l_{y}|)

, where

(l_{x},l_{y})

is the red cube’s horizontal offset from the goal in the goal’s frame, rotated by the goal yaw.

View	Eye offset	Eye position	Image up
V1 front-low	$(0.42,0,0.10)$	$(0.47,0,0.13)$	$+z$
V2 oblique	$(0.30,0.30,0.30)$	$(0.35,0.30,0.33)$	$+z$
V3 top-down	$(0,0,0.50)$	$(0.05,0,0.53)$	$-x$
V4 level side	$(0,0.42,0.035)$	$(0.05,0.42,0.065)$	$+z$
Lighting	Ambient	Key light: direction, intensity	Fill
preset 0	0.30	$(1,1,-1)$ , 1.0	1.0

Appendix

Table 7: Cameras and lighting presets. Every camera looks at

(0.05,0,0.03)

, 3 cm above the nominal goal centre; the eye offset is relative to that point, the eye position in world coordinates (m). All cameras render

512\times 512

pixels with a field of view of 0.45 rad and clipping planes at 0.01 and 100 m. Lights are directional and grey; the key light casts shadows (shadow map

2048\times 2048

), and the fill light points straight down,

(0,0,-1)

Figure 8: Stimulus catalogue: one main-set scene per task (stack seed 1, tray seed 6) and, in it, one end state per design class, each rendered from the four views (centre crops); labels give the simulator’s outcome type. Numbers: M1’s success probability $p$ , in bold where M1 accepts ( $p\geq 0.5$ ). These examples illustrate view-dependent evidence, rather than population error rates.

Set	Scene seeds	Use	Task	End states	success	hover	held_on	behind	beside	edge
v1	0–11	diagnosis	stack	168	73	48	0	32	15	0
			tray	180	76	48	0	28	16	12
v2	20–31	replication	stack	168	73	48	0	24	23	0
			tray	180	80	48	0	24	19	9
N1	100–199	calibration	stack	100	57	10	11	9	13	0
			tray	100	64	20	1	5	6	4

Appendix

Table 8: Stimulus sets and simulator outcomes per task. In v1, seeds 0–7 form the main set (112 stack and 120 tray end states) and seeds 8–11 are held out. Stimuli per set: 1,392 in v1 and in v2, 800 in N1, 2,400 in N2.

Cell	Class	AUROC	$\tau_{c}$	$\mathcal{B}_{c}$	$v^{\prime}_{c}$
J1/stack/V2	separable	.988	.998	hover	V4
M1/stack/V2	separable	.970	.984	hover	V4
M1/tray/V2	separable	.980	.915	hover, edge	V1
J1/tray/V1	inseparable	.485	.993	behind	V4
J1/tray/V2	inseparable	.820	.993	edge, hover	V1
M1/stack/V1	inseparable	.652	.967	behind	V2

Appendix

Table 9: Frozen cell parameters, computed from the v1 main set before any loop data. AUROC separates true successes from the dominant type

f^{\star}_{c}

(listed first in

\mathcal{B}_{c}

) at the deployment view;

\tau_{c}

is the C6 threshold and

v^{\prime}_{c}

the CV check view. No failure type passes in M1/tray/V1; its

\tau_{c}

and

v^{\prime}_{c}

are computed for hover.

Claim or result	Evidence	Prespecification	Replication unit
Type/view dependence; cue use	Truth-fixed diagnosis; H3	Dated/amended protocols; later H3 intervals	Scenes; 12/task in diagnosis, 300/task in N2
Missing below-rim support	Historical F1 D/D+	Hash-locked before F1 data	40 paired valid scenes
Fewer but harder residual hovers	Revised N0/R0/C1	Revised after historical observations; reuses F1 seed IDs	20 paired seeds; pooled rounds
State-specific filtering effects	Revised C3/C2*	Later protocol, frozen before its run	5 paired seeds per cell
Retention and safeguard outcomes	Historical T2–T4	Frozen before main loop; deviations retained	5 seeds per cell
Static-ratio prediction fails	Later ratio/trajectory audit	Exploratory, post hoc	8 related cells

Appendix

Table 10: Evidence scope at a glance. Prespecification refers to local dated or hash-locked documents. Revised and historical protocols are analyzed separately; arm-round records are not independent replicates.

Document	Frozen	SHA-256 (prefix) / content
Diagnosis v1	10-02	hypotheses H1–H3, stimulus design
v1-1 to v1-5	10-02	visibility table, pools N1/N2, cue analysis, J2 threshold
v1-6	10-03	0a41f9a5 mid-size judge M1
Pilot v2-0	10-03	cfa29ad7 closed-loop pilot
v1-7	10-03	283f23cd first closed judges
v1-8	10-05	531595d9 Claude batch route

Appendix

Table 11: Preregistration documents (dates in 2026).

Judge	$\rho_{S}$	Map KL	B1 KL	B2 KL	B1 $-$ map [95% CI]	B2 $-$ map [95% CI]
J1 Qwen3-VL-8B	.707	.560	1.124	2.133	$.563\ [-.043,.908]$	$1.573\ [.347,1.934]$
M1 Qwen3.8-27B	.582	.702	.919	1.245	$.217\ [-.094,.514]$	$.544\ [-.366,1.543]$
RoboReward-8B	.684	1.061	1.116	2.031	$.055\ [-.970,.788]$	$.970\ [-1.262,1.990]$
Claude Opus 5.5	.583	.387	1.099	.477	$.712\ [.230,1.033]$	$.090\ [-.526,.469]$
Gemini 3.6-flash	.451	1.616	1.152	.708	$-.464\ [-1.409,.159]$	$-.908\ [-1.879,-.272]$
Gemini 3.8-flash	.446	1.255	1.832	1.190	$.577\ [-.530,.935]$	$-.066\ [-1.196,.215]$

Appendix

Table 12: Complete H3 point estimates and later fixed-forecast scene-bootstrap intervals. Advantage is baseline KL minus map KL; positive favors the map. Rounding can make differences of displayed KLs differ from displayed advantages. J4 is excluded from the eight-way pass count:

\rho=.894

, map/B2 KL

.637

, B1 KL

1.204

, at V2 only. Intervals are unadjusted.

Figure 9: Follow-up F1 (M1/tray/V1; rounds 1–5 of 20 seeds). (a) Cumulative distribution of the measured heights of hovers per arm; dots: share below the tray rim (2 cm); triangles: measured heights of the diagnostic hovers, two design heights each for D+ and D. (b) Share of hovers accepted by M1 at V1 per height bin (bins as in Tab. 13 ; Wilson 95% intervals), and of the diagnostic stimuli of D and D+ (diamonds).

				accepted / hovers, by measured height
Set	hovers	below rim	accepted	$<1.5$ cm	1.5–2.0 cm	2.0–2.5 cm	$\geq 2.5$ cm
Round 0 (shared)	322	96	96	57/62	25/34	14/68	0/158
C3, rounds 1–5	469	246	225	169/182	39/64	17/67	0/156
C1, rounds 1–5	176	92	87	61/64	22/28	4/24	0/60
N0, rounds 1–5	1592	499	490	271/295	156/204	54/276	9/817
D	160	0	0				0/160

Appendix

Table 13: Follow-up F1 (M1/tray/V1; 20 seeds, 2,000 rollouts per arm and round; diagnostic sets at V1): hovers, hovers below the rim (measured height below 2 cm) and hovers accepted by M1. The true success rate in round 5 was 0.963 under C3, 0.975 under C1 and 0.682 under N0, against 0.663 in round 0.

	N0				R0				C1
Seed	$H$	$A$	$L$	$S$ (%)	$H$	$A$	$L$	$S$ (%)	$H$	$A$	$L$	$S$ (%)
5	80	22	24	69.6	62	24	22	68.6	9	4	4	96.2
6	70	26	25	67.8	83	47	42	66.4	10	6	7	96.4
7	71	18	19	69.0	114	60	59	63.6	8	4	4	96.2
8	90	31	32	64.8	104	42	50	65.8	6	4	5	97.6
9	70	16	18	67.6	70	30	36	72.0	9	4	4	98.2

Appendix

Table 14: Revised mechanism run, all 20 seeds.

H

: hover count;

A

: accepted hovers;

L

: hovers below the rim;

S

: true success percentage. Every arm has 500 observations per seed. These counts reproduce both seed means and pooled ratios.

Cell	Success accepted	Target accepted	Ratio
M1 / stack / V1	42/49	11/18	.713
M1 / stack / V2	49/49	17/32	.531
M1 / tray / V1	51/51	0/32	.000
M1 / tray / V2	51/51	18/40	.450
J1 / stack / V1	49/49	24/50	.480
J1 / stack / V2	49/49	19/32	.594

Appendix

Table 15: Static ratio inputs. The outcome is the continuous C3–C2* final target-failure difference in Tab. 3 , not a significance-based effective/ineffective label.

C3 trajectory	Static type	Round-0 type	Unchanged
Historical main	4.99	3.83	7.83
Revised protocol	5.08	3.87	7.77

Appendix

Table 16: Mean absolute target-failure trajectory error (pp). The historical and revised experiments are related and are not two independent validations.

Cell	Seed 0	1	2	3	4
M1 / stack / V1	1/0	0/0	0/0	1/1	0/0
M1 / stack / V2	6/0	7/1	10/6	3/2	2/1
M1 / tray / V1	7/2	2/1	2/8	4/3	1/1
M1 / tray / V2	16/7	18/13	10/6	13/6	23/18
J1 / stack / V1	11/2	1/1	4/7	4/2	1/1
J1 / stack / V2	8/1	7/1	8/6	2/3	3/1

Appendix

Table 17: Final target failure counts for C3/C2* per seed; each denominator is 100. Full success and other-failure seed rows accompany the manuscript as CSV. This table exposes small counts hidden by cell means.

Visual Evidence Sensitivity in VLMs

See all 28 papers

Early Warning Signals for OpenVLA Failure under Visual Distribution Shift
Jun 29, 2026Dipesh Tharu Mahato, Rachel RenRobot Failure Detection Distribution Shift
New York University
Robot Critics that Sweat the Small Stuff
Jun 19, 2026Sruthi Sudhakar, Junbang Liang, Sreehari Rammohan +3Robot Failure Detection VLM Adaptation
Columbia University · Toyota Research Institute
FailBench: How Reliable are VLMs at Judging Robot Task Success?
Sep 3, 2026Zaruhi Navasardyan, Tatul Danielyan, Hrant DavtyanVLM Evaluation Robot Failure Detection
Metric AI Lab

Jun 29, 2026J/K move · Enter open · S save

Early Warning Signals for OpenVLA Failure under Visual Distribution Shift

Dipesh Tharu Mahato, Rachel Ren

New York University

Robot Failure Detection Distribution Shift Fault Detection VLM Interpretability Robustness of VLA Models

Open paper AskPDF arXiv

Vision Language Action models combine perception, language grounding, and control in a single policy, but their failures are hard to diagnose once visual conditions shift. We test whether OpenVLA feedforward activations contain linearly decodable information about near term task failure in LIBERO manipulation rollouts. The policy is fixed throughout. We log internal activations during execution and fit lightweight monitors after the rollouts are collected. Occlusion is the main controlled stress test. It reduces OpenVLA success from

57\%

17\%

over

100

episodes per condition. Under this shift, a logistic probe at layer 16 reaches AUROC

0.972

and AUPRC

0.352

for predicting failure within a

15

step horizon. It outperforms both a mean difference direction and an action disagreement baseline. A sparse layer sweep finds uneven decodability across depth: layer 16 is strongest among the tested layers, layer 8 remains informative, and layer 10 is weaker. To check whether the monitor is just an occlusion detector, we also evaluate color shift and camera jitter without refitting. Color shift produces no failures in this setting, so it is a benign control rather than a failure benchmark. Camera jitter does induce failures, and the occlusion trained monitor remains above random. The result is deliberately limited: OpenVLA internal states contain failure relevant structure under controlled perceptual shift, but these experiments do not establish a causal mechanism, task held out generalization, or a deployable recovery system.

Abstract

Visual Evidence Sensitivity in VLMs

Explore similar work

Early Warning Signals for OpenVLA Failure under Visual Distribution Shift

Robot Critics that Sweat the Small Stuff

FailBench: How Reliable are VLMs at Judging Robot Task Success?