Vision-Language-Action (VLA) models integrate visual perception, language understanding, and action generation to control robots. Testing these systems produces trajectories and task outcomes, but existing approaches provide limited guidance on using these recorded trajectories to assess subsequent executions or inform policy repair. We conjecture that, for a given policy and task, traversing certain workspace regions is associated with a higher likelihood of failure. To investigate this conjecture, we propose VoxelSBFL, a black-box adaptation of Spectrum-Based Fault Localization (SBFL) in which workspace voxels replace program elements as units of analysis. Recorded tool-center-point (TCP) trajectories provide binary voxel coverage, which is combined with execution outcomes to construct suspiciousness maps using established SBFL formulas. These maps identify failure-associated regions and score unseen trajectories without requiring internal model access. We evaluate 34 spectrum-based techniques across four VLA models using 96,000 Pick-Up trajectories in simulation and compare the leading techniques against eight action- and trajectory-based metrics. VoxelSBFL obtains an overall macro ROC-AUC of 0.704, significantly outperforming state-of-the art baselines. Sensitivity analyses show that performance varies with spatial granularity and training-set size. These findings support spatial spectra for VLA execution assessment and motivate future research on runtime monitoring, self-healing and targeted policy improvement.
Figures & tables
Figure 1. Spatial Distribution of Successful and Failed Robot Trajectories
Figure 2. Overview of the Architecture of a VLA-Enabled Robot
ID
Program block
t1
t2
t3
t4
t5
NCF
NCS
NF
NS
Suspiciousness
Rank
b1
fee = 0 if priority:
∙
∙
∙
∙
∙
2
3
2
3
0.500
3
b2
fee += 5
∙
–
∙
∙
–
1
2
2
3
0.429
6
b3
if amount_based:
∙
∙
∙
∙
∙
2
3
2
3
0.500
3
b4
if amount > 200:
∙
∙
∙
–
–
2
1
2
3
0.750
2
b5
fee += 0.20 * amount
∙
∙
–
–
–
2
0
2
3
1.000
1
b6
else: fee += 0.10 * amount
–
–
∙
–
–
0
1
2
3
0.000
7
Table 1. SBFL example using Tarantula. A bullet ( ∙ ) denotes execution and a dash (–) denotes non-execution; F and P denote failing and passing tests, respectively. The faulty block and its score are highlighted in bold.
Figure 3. Overview of VoxelSBFL. Successful and failed tool-center-point trajectories are mapped onto a voxel grid. SBFL ranks visited voxels by their association with failed executions. The resulting suspiciousness map is used to score unseen trajectories and predict whether they are going to fail.
Figure 4. Example visual observations from SimplerEnv: (a) a can on a tabletop for a “Pick-up” task; and (b) a sponge, a can, and a bottle on a tabletop for a “Move Near” task.
VLA
RQ1 pass (%)
Nmmin
Selected pass
Selected fail
Selected Nm
π0
33.592
298
100
198
298
GR00T
40.654
246
100
146
246
EO1
44.875
223
100
123
223
SpatialVLA
29.246
342
100
242
342
Table 2. Pooled RQ1 success proportions and selected target-task evaluation samples. Nmmin denotes the reference budget required for at least 100 expected observations of the minority class under the pooled RQ1 proportions.
Technique
EO1
GR00T
Pi0
SpatialVLA
Overall
Harmonic Mean
0.665 (1)
0.673 (1)
0.746 (1)
0.732 (1)
0.704 (1)
Rogot2
0.665 (1)
0.673 (1)
0.746 (1)
0.732 (1)
0.704 (1)
Dennis
0.646 (5)
0.660 (3)
0.738 (3)
0.708 (3)
0.688 (3)
Phi (Geometric Mean)
0.638 (6)
0.653 (4)
0.737 (4)
0.699 (4)
0.682 (4)
Tarantula
0.659 (3)
0.645 (6)
0.671 (9)
0.644 (8)
0.655 (5)
Tarwid
0.659 (4)
0.645 (5)
0.671 (10)
0.644 (7)
0.655 (6)
Table 3. RQ1 – Macro ROC-AUC of the 34 SBFL techniques for each VLA model. Parentheses report tie-aware ranks. Bold denotes rank 1 and underlining denotes the remaining top-five ranks. Harmonic Mean and Rogot2 are tied because they produce identical results in every reported column.
Technique
ROC-AUC [95% CI]
Gap [95% CI]
Holm p
rrb
Equivalent
Harmonic Mean
0.704 [0.680, 0.728]
Ref.
–
–
Ref.
Rogot2
0.704 [0.680, 0.728]
0.000 [0.000, 0.000]
1.000
0.000
Yes
Dennis
0.688 [0.667, 0.710]
0.016 [0.007, 0.025]
0.279
0.838
No
Phi (Geometric Mean)
0.682 [0.659, 0.705]
0.022 [0.012, 0.032]
0.134
0.882
No
Tarantula
0.655 [0.636, 0.675]
0.049 [0.031, 0.069]
0.018
0.985
No
Tarwid
0.655 [0.636, 0.675]
0.049 [0.031, 0.069]
0.018
0.985
No
Table 4. RQ1 – Statistical comparison of the techniques reaching a top-five rank for at least one VLA. The AUC gap is the leader’s ROC-AUC minus the candidate’s ROC-AUC. Confidence intervals are pointwise 95%; p -values are adjusted using Holm correction over all 561 pairwise comparisons among the 34 SBFL techniques, and rrb is the paired rank-biserial effect size.
Method
EO1
GR00T
Pi0
SpatialVLA
Overall
Optimal Trajectory †
0.6666
0.7131
0.8029
0.8408
0.7559
Harmonic Mean ⋆
0.6652
0.6730
0.7461
0.7315
0.7039
Rogot2 ⋆
0.6652
0.6730
0.7461
0.7315
0.7039
Dennis ⋆
0.6462
0.6602
0.7383
0.7075
0.6880
Phi (Geometric Mean) ⋆
0.6375
0.6533
0.7370
0.6994
0.6818
TCP Acceleration Instability
0.6995
0.5598
0.6671
0.6934
0.6550
Table 5. RQ2 – Trajectory-level failure-discrimination ROC-AUC. Harmonic Mean is the VoxelSBFL reference technique selected in RQ1. Bold denotes the best result and underlining denotes the second-best result in each column. † denotes a task-aware metric using object and goal information and ⋆ indicates an SBFL technique .
VLA
0.01
0.02
0.05
Descriptive best size
EO1
0.6711
0.6652
0.6253
0.01
GR00T
0.6574
0.6730
0.6815
0.05
Pi0
0.7101
0.7461
0.7416
0.02
SpatialVLA
0.7398
0.7315
0.7235
0.01
Macro
0.6946
0.7039
0.6930
0.02
Table 6. RQ3 – Voxel-size sensitivity of Harmonic Mean, the frozen RQ1 reference technique. Values are macro ROC-AUC over the four trajectory-generation conditions for each VLA. Bold denotes the descriptive best size within a row; inferential tests are reported separately in Table 9 .
Technique
0.01 ROC-AUC (rank)
0.02 ROC-AUC (rank)
0.05 ROC-AUC (rank)
(a) EO1
Harmonic Mean
0.6711 (1)
0.6652 (1)
0.6253 (5)
Rogot2
0.6711 (1)
0.6652 (1)
0.6253 (4)
Tarantula
0.5460 (16)
0.6590 (3)
0.7089 (1)
Tarwid
0.5460 (15)
0.6590 (4)
0.7089 (2)
Dennis
0.6555 (4)
0.6462 (5)
0.6680 (3)
Table 7. RQ3 – Voxel-size sensitivity of techniques reaching a top-five rank for at least one size within each VLA. Each cell reports macro ROC-AUC and the rank in parentheses. Ranks use unrounded values. Bold denotes rank 1 and underlining denotes the remaining top-five ranks.
VLA
Size
Known
Ties
Train voxels
Artifact (MB)
Fit (s)
Score (s)
EO1
0.01
92.4%
90.1%
30,791
2.12
0.146
0.244
0.02
98.8%
67.2%
5,651
0.93
0.061
0.057
0.05
99.8%
84.6%
572
0.15
0.028
0.014
GR00T
0.01
92.8%
90.7%
37,545
2.86
0.180
0.296
0.02
98.8%
59.1%
7,231
1.18
0.080
0.071
0.05
99.8%
75.0%
736
0.18
0.034
0.016
Table 8. RQ3 – Coverage and computational diagnostics by VLA and voxel size for Harmonic Mean. Values are averaged over the 80 held-out folds in each VLA-size cell. Artifact sizes use decimal megabytes.
(a) Voxel-size effect for Harmonic Mean
Size
ROC-AUC [95% CI]
Δ vs. 0.02 [95% CI]
Equivalent
0.01
0.6946 [0.6727, 0.7178]
-0.0093 [-0.0219, +0.0030]
Yes
0.02
0.7039 [0.6796, 0.7280]
+0.0000 [+0.0000, +0.0000]
Yes
0.05
0.6930 [0.6652, 0.7188]
-0.0110 [-0.0271, +0.0033]
No
Table 9. RQ3 – Global statistical analysis over the 16 matched VLA–trajectory-condition blocks. Panel (a) reports the frozen Harmonic Mean size comparison (Friedman χF2(2)=1.625 , p=0.444 ). Panel (b) contains interactions satisfying Holm-corrected p<0.05 and ∣ΔΔAUC∣≥0.01 . Panel (c) reports tie-aware Spearman rank correlations across all 34 techniques.
VLA
n
Calibration cov. (%)
Held-out cov. (%)
Harmonic Mean AUC (rank)
Leading technique
Leading AUC
EO1
10
8.2
8.2
0.5608 (10)
Ochiai2
0.5684
20
15.3
15.3
0.5688 (16)
Rogot1 / Scott
0.5824
50
31.7
31.7
0.5938 (9)
Dennis
0.6029
100
49.3
49.3
0.6174 (3)
Dennis
0.6199
200
67.6
67.6
0.6377 (2.5)
Zoltar
0.6381
GR00T
10
9.1
9.1
0.5213 (7)
Dennis
0.5226
Table 10. RQ4 learning-curve coverage and held-out performance by VLA. Coverage values are percentages. The Harmonic Mean cell reports ROC-AUC and its within-VLA, within-size rank among the 34 techniques in parentheses; leaders were determined from unrounded ROC-AUC.
Technique
EO1
GR00T
Pi0
SpatialVLA
Macro
AUC@200
Δ AUC 10→200
Dennis
0.6000 (1)
0.5646 (1)
0.6302 (3)
0.6026 (5)
0.5994 (1)
0.6501
+0.0958
Rogot2
0.5941 (2)
0.5644 (3)
0.6232 (15)
0.6072 (4)
0.5972 (2)
0.6534
+0.1024
Harmonic Mean
0.5938 (3)
0.5645 (2)
0.6227 (16)
0.6072 (3)
0.5971 (3)
0.6534
+0.1036
Phi (Geometric Mean)
0.5929 (4)
0.5632 (5)
0.6310 (2)
0.5956 (6)
0.5957 (4)
0.6466
+0.0969
Mountford
0.5910 (9)
0.5638 (4)
0.6130 (17)
0.6092 (2)
0.5942 (5)
0.6501
+0.1032
Arithmetic Mean
0.5909 (10)
0.5585 (7)
0.6317 (1)
0.5795 (7)
0.5902 (6)
0.6294
+0.0796
Table 11. RQ4 sample efficiency for techniques reaching a top-five NAULC rank for at least one VLA. VLA and macro cells report NAULC with rank among the 34 techniques in parentheses. Bold denotes rank 1 and underlining the remaining top-five ranks. Endpoint and gain are macro ROC-AUC.
Transition
Mean Δ AUC
Median Δ AUC
Positive (%)
Holm p
rrb
10–20
+0.0187
+0.0170
100.0
0.00012
+1.000
20–50
+0.0296
+0.0288
100.0
0.00012
+1.000
50–100
+0.0268
+0.0245
100.0
0.00012
+1.000
100–200
+0.0285
+0.0299
100.0
0.00012
+1.000
Table 12. RQ4 matched adjacent-size effects for Harmonic Mean over the 16 VLA–trajectory-generation blocks. Tests used paired Wilcoxon signed-rank tests with one Holm correction.
Technique
Mean effect
Median effect
Positive blocks (%)
Holm p
rrb
Zoltar
+0.0047
+0.0063
50.0
1.0000
+0.206
Gower
+0.0017
+0.0034
56.2
1.0000
+0.118
Mountford
+0.0009
+0.0046
56.2
1.0000
-0.015
Braun-Banquet
+0.0006
+0.0035
62.5
1.0000
+0.118
Dennis
+0.0004
-0.0014
43.8
1.0000
+0.029
Cohen
-0.0012
-0.0053
37.5
1.0000
-0.074
Table 13. Complete conditional coverage analysis for RQ4. Effects are ROC-AUC changes per ten percentage points of calibration coverage after centering within configuration, outer fold, and suite size. Holm correction covered all 34 techniques.
Technique
EO1
GR00T
Pi0
SpatialVLA
Overall
Zoltar
0.604 (15)
0.571 (4)
0.677 (1)
0.504 (3)
0.589 (1)
Mountford
0.593 (20)
0.568 (5)
0.632 (2)
0.494 (7)
0.572 (2)
Harmonic Mean
0.618 (6)
0.574 (2)
0.585 (4)
0.511 (1)
0.572 (3)
Rogot2
0.618 (6)
0.574 (2)
0.585 (4)
0.511 (1)
0.572 (3)
Gower
0.535 (33)
0.605 (1)
0.606 (3)
0.497 (6)
0.561 (5)
Dennis
0.650 (2)
0.537 (9)
0.521 (9)
0.502 (4)
0.552 (6)
Table 14. RQ5 – Macro ROC-AUC of the 34 SBFL techniques for each VLA model on the Move-Near task. Parentheses report tie-aware ranks. Bold denotes rank 1 and underlining denotes the remaining top-five ranks. Harmonic Mean and Rogot2 are tied because they produce identical results in every reported column.
Method
EO1
GR00T
Pi0
SpatialVLA
Overall
Rank
Optimal Trajectory †
0.8980
0.8742
0.9473
0.7644
0.8710
1
TCP Acceleration Instability
0.7881
0.5416
0.6708
0.4682
0.6172
2
Trajectory Instability
0.7858
0.5529
0.6615
0.4645
0.6162
3
TCP Velocity Instability
0.7752
0.5618
0.6285
0.4286
0.5985
4
Zoltar ⋆
0.6041
0.5714
0.6771
0.5044
0.5892
5
Action Position Instability
0.7724
0.6568
0.5065
0.4125
0.5871
6
Table 15. RQ5 baseline trajectory-level failure-discrimination ROC-AUC on the selected RQ5 evaluation samples. Overall is the equal-weight mean across the four models. Bold denotes the best value and underlining the second-best value per column. † indicates a task-aware metric using object/goal state and ⋆ indicates an SBFL technique.
Figure 5. Three-dimensional distribution of failure-associated voxel suspiciousness for the four VLA models. Colors represent the median within-map percentile of the Harmonic Mean suspiciousness score at voxel size 0.02, computed across the 80 RQ1 maps available for each VLA. Warmer colors indicate higher suspiciousness, while marker size and opacity increase with episode support.