OGAM: Connecting Systematic Testing to Runtime Assurance through Object-Grounded Attention Monitoring for VLA Policies

Haki Darwish; Xiangyu Yin; Changwen Li; Rongjie Yan; Francisco Gomes de Oliveira Neto; Chih-Hong Cheng

cs.LGOct 5, 2026

OGAM: Connecting Systematic Testing to Runtime Assurance through Object-Grounded Attention Monitoring for VLA Policies

Authors: Haki Darwish, Xiangyu Yin, Changwen Li, Rongjie Yan, Francisco Gomes de Oliveira Neto, Chih-Hong Cheng

Organizations: Carl von Ossietzky University of Oldenburg, Germany · Huadian Electric Power Research Institute, China · Institute of Software, Chinese Academy of Science, China · Chalmers University of Technology, Sweden

Abstract

Benchmarks expose vision-language-action (VLA) policies to few canonical instructions, while exhaustive deployment testing is impossible. We introduce Object-Grounded Attention Monitoring (OGAM), connecting systematic testing to runtime assurance: testing reveals attention divergence between successful and failed executions, and OGAM uses this signal to stop failures beyond the finite suite. We generate scene-grounded instructions through pairwise combinations of action templates and objects, and separately test meaning-preserving paraphrases. All 87 out-of-benchmark cases reveal problematic behavior across OpenVLA, OpenVLA-OFT, UniVLA, and $π_{0.5}$ : none completes any of the 24 feasible instructions, while infeasible or hazardous requests also trigger behavior substitution. At each action query, we project gradient-weighted visual attention through object masks and group it by instruction role for comparison across tasks and policies. Dynamic time warping aligns this course with a successful reference despite speed differences; conformal calibration on successful episodes sets the early-stopping threshold for sustained deviations, with a nominal false-stop target of $α=0.05$ . Across four policies, OGAM stops 87-100% of failed episodes at median times of 5-12s within a 20s budget, with observed false-stop rates of 3-5%, without failure-labeled training. Finite testing thus identifies attention patterns that support online intervention before failure fully unfolds.

Figures & tables

Fig. 1: Combinatorial instruction testing for VLA policies

Fig. 2: The monitor. Offline, once per policy: (1) successful episodes $S$ supply one reference per task; (2) leave-one-out scoring produces one sustained-deviation score $r_{j}$ per success. Sorting these scores sets the shared height $q$ using the nominal false-stop target $\alpha=0.05$ . Online: (3) object-grounded attention forms the running trace; (4) open-end DTW supplies on-track and completion scores from normalized row and column minima; (5) either readout triggers a stop after remaining above its success-relative band for the full sustain window after the initial settling period. The $\alpha$ annotation denotes a nominal target (Sec. III-D ).

LLM label	$n$	followed	trained task	unnamed object	hazard	froze or idle
feasible (expect success)	24	0	17	8	4	5
infeasible (expect safe failure)	62	0	26	33	9	6
hazardous (expect refusal)	1	0	0	1	0	1

TABLE I: Expected versus observed behavior of OpenVLA on the 87 generated instructions, by LLM label. Behavior tags from the review notes are not exclusive.

Policy	Detected	False stops	Med. stop	Saved
OpenVLA	58/67 (87%)	3/63 (4.8%)	10.3 s	39%
OpenVLA-OFT	13/15 (87%)	5/115 (4.3%)	9.6 s	41%
UniVLA	24/25 (96%)	3/105 (2.9%)	11.5 s	41%
$\pi_{0.5}$	11/11 (100%)	5/119 (4.2%)	5.0 s	72%

TABLE II: Early stopping with nominal

\alpha=0.05

and 3 s sustain. Detected/false stops: fractions of failures/successes stopped. Median stop time: detected failures only; saved: failing-episode rollout time avoided.

Fig. 3: Monitored OpenVLA episodes. Top row: the reference at the same moments and at its end. Middle row: the paraphrased instruction-failing episode before the stop, at the stop (red frame), and, dashed, continued without the monitor. Bottom: the monitored signals and their divergence.

Score	OpenVLA	OFT	UniVLA	$\pi_{0.5}$	worst
Ablations of the monitor
Full monitor (on-track $\cup$ completion)	87	87	96	100	87
on-track readout only	63	67	96	100	63
completion readout only	72	67	72	100	67
target channel only, no role channels	66	60	88	100	60
no time warping (fixed time index)	69	73	60	100	60

TABLE III: Ablations and other monitors: detection in % per policy and worst case. Same rule, nominal

\alpha=0.05

, burn-in, and sustain; observed false stops: 3–5%. The split bound is marginal under exchangeability.

\dagger

: failure-labeled training; –: signal unavailable.

Explore similar work

FATE-VLA:Failue-aware test generation for vision-language-action models
Jun 1, 2026Arusa Kanwal, Pablo Valle, Shaukat Ali +1VLM Evaluation Vision-Language-Action Models
Mondragon University Mondragon, Spain · Simula Research Laboratory Oslo, Norway
Early Warning Signals for OpenVLA Failure under Visual Distribution Shift
Jun 29, 2026Dipesh Tharu Mahato, Rachel RenRobot Failure Detection Distribution Shift
New York University
BOKBO (Best of K Bad Options): Calibrated Abstention for VLA Policies
May 28, 2026Anya Singh, Cabrel Happi, Jai Relan +2Selective Prediction Conformal Prediction
Relling Systems

Jun 1, 2026, cs.ROJ/K move · Enter open · S save

FATE-VLA:Failue-aware test generation for vision-language-action models

Arusa Kanwal, Pablo Valle, Shaukat Ali, Aitor Arrieta

Mondragon University Mondragon, Spain · Simula Research Laboratory Oslo, Norway

VLM Evaluation Vision-Language-Action Models Robustness of VLA Models

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models are increasingly used as generalist robot policies, yet their evaluation still relies largely on static benchmarks that randomly sample task scenes. In high-dimensional embodied spaces, failures are sparse and clustered, so static benchmarking can underestimate robustness risks. We reframe VLA evaluation as an active failure-discovery problem and propose a failure-aware test-generation approach that combines diversity-driven exploration with surrogate models learned from observed executions. The method steers testing toward high-risk yet diverse scene regions. Across four state-of-the-art VLA models, it uncovers substantially more failures (up to +29.7 % over selected baselines) while revealing more diverse failure modes. This mean that, for instance, in the case of GR00T-N1.6, success rate dropped from 64.4% to 34.7%. More broadly, our findings call for a shift in VLA evaluation: from passive measurement on fixed task suites to adaptive, failure-seeking test generation that exposes the structure of model weaknesses before deployment.

Abstract

Explore similar work

FATE-VLA:Failue-aware test generation for vision-language-action models

Early Warning Signals for OpenVLA Failure under Visual Distribution Shift

BOKBO (Best of K Bad Options): Calibrated Abstention for VLA Policies