Vision-Language-Action (VLA) models integrate visual perception, language understanding, and action generation to control robots. Testing these systems produces trajectories and task outcomes, but existing approaches provide limited guidance on using these recorded trajectories to assess subsequent executions or inform policy repair. We conjecture that, for a given policy and task, traversing certain workspace regions is associated with a higher likelihood of failure. To investigate this conjecture, we propose VoxelSBFL, a black-box adaptation of Spectrum-Based Fault Localization (SBFL) in which workspace voxels replace program elements as units of analysis. Recorded tool-center-point (TCP) trajectories provide binary voxel coverage, which is combined with execution outcomes to construct suspiciousness maps using established SBFL formulas. These maps identify failure-associated regions and score unseen trajectories without requiring internal model access. We evaluate 34 spectrum-based techniques across four VLA models using 96,000 Pick-Up trajectories in simulation and compare the leading techniques against eight action- and trajectory-based metrics. VoxelSBFL obtains an overall macro ROC-AUC of 0.704, significantly outperforming state-of-the art baselines. Sensitivity analyses show that performance varies with spatial granularity and training-set size. These findings support spatial spectra for VLA execution assessment and motivate future research on runtime monitoring, self-healing and targeted policy improvement.
Figures & tables
Figure 1. Spatial Distribution of Successful and Failed Robot Trajectories
Figure 2. Overview of the Architecture of a VLA-Enabled Robot
ID
Program block
t1
t2
t3
t4
t5
NCF
NCS
NF
NS
Suspiciousness
Rank
b1
fee = 0 if priority:
∙
∙
∙
∙
∙
2
3
2
3
0.500
3
b2
fee += 5
∙
–
∙
∙
–
1
2
2
3
0.429
6
b3
if amount_based:
∙
∙
∙
∙
∙
2
3
2
3
0.500
3
b4
if amount > 200:
∙
∙
∙
–
–
2
1
2
3
0.750
2
b5
fee += 0.20 * amount
∙
∙
–
–
–
2
0
2
3
1.000
1
b6
else: fee += 0.10 * amount
–
–
∙
–
–
0
1
2
3
0.000
7
Table 1. SBFL example using Tarantula. A bullet ( ∙ ) denotes execution and a dash (–) denotes non-execution; F and P denote failing and passing tests, respectively. The faulty block and its score are highlighted in bold.
Figure 3. Overview of VoxelSBFL. Successful and failed tool-center-point trajectories are mapped onto a voxel grid. SBFL ranks visited voxels by their association with failed executions. The resulting suspiciousness map is used to score unseen trajectories and predict whether they are going to fail.
Figure 4. Example visual observations from SimplerEnv: (a) a can on a tabletop for a “Pick-up” task; and (b) a sponge, a can, and a bottle on a tabletop for a “Move Near” task.
VLA
RQ1 pass (%)
Nmmin
Selected pass
Selected fail
Selected Nm
π0
33.592
298
100
198
298
GR00T
40.654
246
100
146
246
EO1
44.875
223
100
123
223
SpatialVLA
29.246
342
100
242
342
Table 2. Pooled RQ1 success proportions and selected target-task evaluation samples. Nmmin denotes the reference budget required for at least 100 expected observations of the minority class under the pooled RQ1 proportions.
Technique
EO1
GR00T
Pi0
SpatialVLA
Overall
Harmonic Mean
0.665 (1)
0.673 (1)
0.746 (1)
0.732 (1)
0.704 (1)
Rogot2
0.665 (1)
0.673 (1)
0.746 (1)
0.732 (1)
0.704 (1)
Dennis
0.646 (5)
0.660 (3)
0.738 (3)
0.708 (3)
0.688 (3)
Phi (Geometric Mean)
0.638 (6)
0.653 (4)
0.737 (4)
0.699 (4)
0.682 (4)
Tarantula
0.659 (3)
0.645 (6)
0.671 (9)
0.644 (8)
0.655 (5)
Tarwid
0.659 (4)
0.645 (5)
0.671 (10)
0.644 (7)
0.655 (6)
Table 3. RQ1 – Macro ROC-AUC of the 34 SBFL techniques for each VLA model. Parentheses report tie-aware ranks. Bold denotes rank 1 and underlining denotes the remaining top-five ranks. Harmonic Mean and Rogot2 are tied because they produce identical results in every reported column.
Technique
ROC-AUC [95% CI]
Gap [95% CI]
Holm p
rrb
Equivalent
Harmonic Mean
0.704 [0.680, 0.728]
Ref.
–
–
Ref.
Rogot2
0.704 [0.680, 0.728]
0.000 [0.000, 0.000]
1.000
0.000
Yes
Dennis
0.688 [0.667, 0.710]
0.016 [0.007, 0.025]
0.279
0.838
No
Phi (Geometric Mean)
0.682 [0.659, 0.705]
0.022 [0.012, 0.032]
0.134
0.882
No
Tarantula
0.655 [0.636, 0.675]
0.049 [0.031, 0.069]
0.018
0.985
No
Tarwid
0.655 [0.636, 0.675]
0.049 [0.031, 0.069]
0.018
0.985
No
Table 4. RQ1 – Statistical comparison of the techniques reaching a top-five rank for at least one VLA. The AUC gap is the leader’s ROC-AUC minus the candidate’s ROC-AUC. Confidence intervals are pointwise 95%; p -values are adjusted using Holm correction over all 561 pairwise comparisons among the 34 SBFL techniques, and rrb is the paired rank-biserial effect size.
Method
EO1
GR00T
Pi0
SpatialVLA
Overall
Optimal Trajectory †
0.6666
0.7131
0.8029
0.8408
0.7559
Harmonic Mean ⋆
0.6652
0.6730
0.7461
0.7315
0.7039
Rogot2 ⋆
0.6652
0.6730
0.7461
0.7315
0.7039
Dennis ⋆
0.6462
0.6602
0.7383
0.7075
0.6880
Phi (Geometric Mean) ⋆
0.6375
0.6533
0.7370
0.6994
0.6818
TCP Acceleration Instability
0.6995
0.5598
0.6671
0.6934
0.6550
Table 5. RQ2 – Trajectory-level failure-discrimination ROC-AUC. Harmonic Mean is the VoxelSBFL reference technique selected in RQ1. Bold denotes the best result and underlining denotes the second-best result in each column. † denotes a task-aware metric using object and goal information and ⋆ indicates an SBFL technique .
VLA
0.01
0.02
0.05
Descriptive best size
EO1
0.6711
0.6652
0.6253
0.01
GR00T
0.6574
0.6730
0.6815
0.05
Pi0
0.7101
0.7461
0.7416
0.02
SpatialVLA
0.7398
0.7315
0.7235
0.01
Macro
0.6946
0.7039
0.6930
0.02
Table 6. RQ3 – Voxel-size sensitivity of Harmonic Mean, the frozen RQ1 reference technique. Values are macro ROC-AUC over the four trajectory-generation conditions for each VLA. Bold denotes the descriptive best size within a row; inferential tests are reported separately in Table 9 .
Technique
0.01 ROC-AUC (rank)
0.02 ROC-AUC (rank)
0.05 ROC-AUC (rank)
(a) EO1
Harmonic Mean
0.6711 (1)
0.6652 (1)
0.6253 (5)
Rogot2
0.6711 (1)
0.6652 (1)
0.6253 (4)
Tarantula
0.5460 (16)
0.6590 (3)
0.7089 (1)
Tarwid
0.5460 (15)
0.6590 (4)
0.7089 (2)
Dennis
0.6555 (4)
0.6462 (5)
0.6680 (3)
Table 7. RQ3 – Voxel-size sensitivity of techniques reaching a top-five rank for at least one size within each VLA. Each cell reports macro ROC-AUC and the rank in parentheses. Ranks use unrounded values. Bold denotes rank 1 and underlining denotes the remaining top-five ranks.
VLA
Size
Known
Ties
Train voxels
Artifact (MB)
Fit (s)
Score (s)
EO1
0.01
92.4%
90.1%
30,791
2.12
0.146
0.244
0.02
98.8%
67.2%
5,651
0.93
0.061
0.057
0.05
99.8%
84.6%
572
0.15
0.028
0.014
GR00T
0.01
92.8%
90.7%
37,545
2.86
0.180
0.296
0.02
98.8%
59.1%
7,231
1.18
0.080
0.071
0.05
99.8%
75.0%
736
0.18
0.034
0.016
Table 8. RQ3 – Coverage and computational diagnostics by VLA and voxel size for Harmonic Mean. Values are averaged over the 80 held-out folds in each VLA-size cell. Artifact sizes use decimal megabytes.
(a) Voxel-size effect for Harmonic Mean
Size
ROC-AUC [95% CI]
Δ vs. 0.02 [95% CI]
Equivalent
0.01
0.6946 [0.6727, 0.7178]
-0.0093 [-0.0219, +0.0030]
Yes
0.02
0.7039 [0.6796, 0.7280]
+0.0000 [+0.0000, +0.0000]
Yes
0.05
0.6930 [0.6652, 0.7188]
-0.0110 [-0.0271, +0.0033]
No
Table 9. RQ3 – Global statistical analysis over the 16 matched VLA–trajectory-condition blocks. Panel (a) reports the frozen Harmonic Mean size comparison (Friedman χF2(2)=1.625 , p=0.444 ). Panel (b) contains interactions satisfying Holm-corrected p<0.05 and ∣ΔΔAUC∣≥0.01 . Panel (c) reports tie-aware Spearman rank correlations across all 34 techniques.
VLA
n
Calibration cov. (%)
Held-out cov. (%)
Harmonic Mean AUC (rank)
Leading technique
Leading AUC
EO1
10
8.2
8.2
0.5608 (10)
Ochiai2
0.5684
20
15.3
15.3
0.5688 (16)
Rogot1 / Scott
0.5824
50
31.7
31.7
0.5938 (9)
Dennis
0.6029
100
49.3
49.3
0.6174 (3)
Dennis
0.6199
200
67.6
67.6
0.6377 (2.5)
Zoltar
0.6381
GR00T
10
9.1
9.1
0.5213 (7)
Dennis
0.5226
Table 10. RQ4 learning-curve coverage and held-out performance by VLA. Coverage values are percentages. The Harmonic Mean cell reports ROC-AUC and its within-VLA, within-size rank among the 34 techniques in parentheses; leaders were determined from unrounded ROC-AUC.
Technique
EO1
GR00T
Pi0
SpatialVLA
Macro
AUC@200
Δ AUC 10→200
Dennis
0.6000 (1)
0.5646 (1)
0.6302 (3)
0.6026 (5)
0.5994 (1)
0.6501
+0.0958
Rogot2
0.5941 (2)
0.5644 (3)
0.6232 (15)
0.6072 (4)
0.5972 (2)
0.6534
+0.1024
Harmonic Mean
0.5938 (3)
0.5645 (2)
0.6227 (16)
0.6072 (3)
0.5971 (3)
0.6534
+0.1036
Phi (Geometric Mean)
0.5929 (4)
0.5632 (5)
0.6310 (2)
0.5956 (6)
0.5957 (4)
0.6466
+0.0969
Mountford
0.5910 (9)
0.5638 (4)
0.6130 (17)
0.6092 (2)
0.5942 (5)
0.6501
+0.1032
Arithmetic Mean
0.5909 (10)
0.5585 (7)
0.6317 (1)
0.5795 (7)
0.5902 (6)
0.6294
+0.0796
Table 11. RQ4 sample efficiency for techniques reaching a top-five NAULC rank for at least one VLA. VLA and macro cells report NAULC with rank among the 34 techniques in parentheses. Bold denotes rank 1 and underlining the remaining top-five ranks. Endpoint and gain are macro ROC-AUC.
Transition
Mean Δ AUC
Median Δ AUC
Positive (%)
Holm p
rrb
10–20
+0.0187
+0.0170
100.0
0.00012
+1.000
20–50
+0.0296
+0.0288
100.0
0.00012
+1.000
50–100
+0.0268
+0.0245
100.0
0.00012
+1.000
100–200
+0.0285
+0.0299
100.0
0.00012
+1.000
Table 12. RQ4 matched adjacent-size effects for Harmonic Mean over the 16 VLA–trajectory-generation blocks. Tests used paired Wilcoxon signed-rank tests with one Holm correction.
Technique
Mean effect
Median effect
Positive blocks (%)
Holm p
rrb
Zoltar
+0.0047
+0.0063
50.0
1.0000
+0.206
Gower
+0.0017
+0.0034
56.2
1.0000
+0.118
Mountford
+0.0009
+0.0046
56.2
1.0000
-0.015
Braun-Banquet
+0.0006
+0.0035
62.5
1.0000
+0.118
Dennis
+0.0004
-0.0014
43.8
1.0000
+0.029
Cohen
-0.0012
-0.0053
37.5
1.0000
-0.074
Table 13. Complete conditional coverage analysis for RQ4. Effects are ROC-AUC changes per ten percentage points of calibration coverage after centering within configuration, outer fold, and suite size. Holm correction covered all 34 techniques.
Technique
EO1
GR00T
Pi0
SpatialVLA
Overall
Zoltar
0.604 (15)
0.571 (4)
0.677 (1)
0.504 (3)
0.589 (1)
Mountford
0.593 (20)
0.568 (5)
0.632 (2)
0.494 (7)
0.572 (2)
Harmonic Mean
0.618 (6)
0.574 (2)
0.585 (4)
0.511 (1)
0.572 (3)
Rogot2
0.618 (6)
0.574 (2)
0.585 (4)
0.511 (1)
0.572 (3)
Gower
0.535 (33)
0.605 (1)
0.606 (3)
0.497 (6)
0.561 (5)
Dennis
0.650 (2)
0.537 (9)
0.521 (9)
0.502 (4)
0.552 (6)
Table 14. RQ5 – Macro ROC-AUC of the 34 SBFL techniques for each VLA model on the Move-Near task. Parentheses report tie-aware ranks. Bold denotes rank 1 and underlining denotes the remaining top-five ranks. Harmonic Mean and Rogot2 are tied because they produce identical results in every reported column.
Method
EO1
GR00T
Pi0
SpatialVLA
Overall
Rank
Optimal Trajectory †
0.8980
0.8742
0.9473
0.7644
0.8710
1
TCP Acceleration Instability
0.7881
0.5416
0.6708
0.4682
0.6172
2
Trajectory Instability
0.7858
0.5529
0.6615
0.4645
0.6162
3
TCP Velocity Instability
0.7752
0.5618
0.6285
0.4286
0.5985
4
Zoltar ⋆
0.6041
0.5714
0.6771
0.5044
0.5892
5
Action Position Instability
0.7724
0.6568
0.5065
0.4125
0.5871
6
Table 15. RQ5 baseline trajectory-level failure-discrimination ROC-AUC on the selected RQ5 evaluation samples. Overall is the equal-weight mean across the four models. Bold denotes the best value and underlining the second-best value per column. † indicates a task-aware metric using object/goal state and ⋆ indicates an SBFL technique.
Figure 5. Three-dimensional distribution of failure-associated voxel suspiciousness for the four VLA models. Colors represent the median within-map percentile of the Harmonic Mean suspiciousness score at voxel size 0.02, computed across the 80 RQ1 maps available for each VLA. Warmer colors indicate higher suspiciousness, while marker size and opacity increase with episode support.
Vision-Language-Action (VLA) models are increasingly used as generalist robot policies, yet their evaluation still relies largely on static benchmarks that randomly sample task scenes. In high-dimensional embodied spaces, failures are sparse and clustered, so static benchmarking can underestimate robustness risks. We reframe VLA evaluation as an active failure-discovery problem and propose a failure-aware test-generation approach that combines diversity-driven exploration with surrogate models learned from observed executions. The method steers testing toward high-risk yet diverse scene regions. Across four state-of-the-art VLA models, it uncovers substantially more failures (up to +29.7 % over selected baselines) while revealing more diverse failure modes. This mean that, for instance, in the case of GR00T-N1.6, success rate dropped from 64.4% to 34.7%. More broadly, our findings call for a shift in VLA evaluation: from passive measurement on fixed task suites to adaptive, failure-seeking test generation that exposes the structure of model weaknesses before deployment.
Arusa Kanwal, Pablo Valle, Shaukat Ali +1
Mondragon University Mondragon, Spain · Simula Research Laboratory Oslo, Norway
Vision-Language-Action (VLA) models enable robots to follow natural language instructions and generalize across diverse tasks, but they remain vulnerable to execution failures that compromise reliability in real-world deployment. Detecting such failures during execution is therefore critical for the robust deployment of embodied systems. Existing failure detection methods either rely on expensive action resampling or external models, while alternatives propagate trajectory-level labels uniformly across every timestep, obscuring localized failure signals. In this paper, we propose Hide-and-Seek, a framework that formulates VLA failure detection as a coarsely supervised learning problem. By combining inter-trajectory and intra-trajectory contrastive objectives, Hide-and-Seek localizes failure-indicative actions and induces temporally structured failure signals from trajectory-level supervision alone, without any step-level annotation. We evaluate Hide-and-Seek on LIBERO, VLABench, and a real-world robotic platform across three representative VLA policies: OpenVLA, π0, and π0.5.Our method achieves state-of-the-art multi-task failure detection performance with a practical accuracy--timeliness trade-off under conformal prediction, and generalizes well to both seen and unseen tasks.
Seongheon Park, Wendi Li, Changdae Oh +4
University of Wisconsin–Madison · Georgia Institute of Technology
Vision-language-action (VLA) models map visual observations and natural-language instructions to robotic actions, but distribution shifts can compromise their reliability. Because these models may still succeed under out-of-distribution (OOD) conditions, detecting OOD inputs alone is insufficient to predict execution failure. In this paper, we introduce VLA-Scope, a two-stage framework that combines input-shift characterization with execution history to predict failure during OOD rollouts. The first stage uses pooled image and language representations to detect OOD inputs and classify their shift categories. For inputs flagged as OOD, the second stage combines the predicted category, action-prefix features, and execution progress features. A logistic regression model shared across shift categories updates failure risk as execution proceeds. We evaluate the framework with OpenVLA on ten LIBERO-Spatial tasks using leave-one-group-out cross-validation. OOD detection achieves a ROC-AUC of 0.9454, and shift classification achieves 91% accuracy. Evaluated independently of the OOD gate on all 1,400 OOD rollouts, the failure predictor achieves a ROC-AUC of 0.8497 after 60 executed actions, compared with 0.7906 without execution progress features. It also achieves a higher ROC-AUC than the evaluated ActProbe and SAFE-MLP baselines. These results suggest that combining action features with temporally aggregated execution step representations improves failure prediction under input shifts.
Kaiwen Zhu, Dongfang Liu, Liangkai Liu
Department of Computer Science, Texas Tech University, Lubbock, TX, USA · Purdue University, West Lafayette, IN, USA