Vision-Language-Action (VLA) models enable general-purpose robotic policies by mapping visual observations and language instructions to low-level actions, but they often lack reliable introspection. A common practice is to compute a token-level uncertainty signal and take its mean over a rollout. However, mean aggregation can dilute short-lived but safety-critical uncertainty spikes in continuous control. In particular, successful rollouts may contain localized high-entropy segments due to benign noise or non-critical micro-adjustments, while failure rollouts can appear low-entropy for most timesteps and only exhibit brief spikes near the onset of failure. We propose a unified uncertainty quantification approach for predicting rollout success versus failure that (1) uses max-based sliding window pooling to preserve transient risk signals, (2) applies motion-aware stability weighting to emphasize high-frequency action oscillations associated with unstable behaviors, and (3) performs DoF-adaptive calibration via Bayesian Optimization to prioritize kinematically critical axes. Experiments on the LIBERO benchmark show that our method substantially improves failure prediction accuracy and yields more reliable signals for failure detection, which can support downstream human-in-the-loop interventions.
Figures & tables
Figure 1: Overview of the proposed uncertainty quantification framework. Standard global averaging often masks failure signals. We propose: (A) Sliding Window Pooling (SW) to capture transient uncertainty spikes; (B) Action Transfer Reweighting (ATR) to prioritize uncertainty during oscillatory actions; and (C) Bayesian Optimization (BO) to learn adaptive weights for kinematically critical DoFs.
Figure 2: Empirical evidence of the Averaging Trap on LIBERO-10. The plots show the probability density of the global mean entropy SAvg(τ) for success (blue) and failure (red) rollouts. The significant overlap between the two distributions, coupled with near-random AUROC scores (0.51 on train and 0.47 on test), demonstrates that global averaging masks critical failure signals and fails to distinguish between successful and failed executions.
LIBERO-SPATIAL
LIBERO-OBJECT
LIBERO-GOAL
LIBERO-10
Method (best params)
AUROC ↑
Acc ↑
AUROC ↑
Acc ↑
AUROC ↑
Acc ↑
AUROC ↑
Acc ↑
Baseline ( meant,d )
0.845
0.892
0.542
0.744
0.641
0.755
0.468
0.622
SW ( w=70 )
0.908
0.912
0.765
0.744
0.837
0.846
0.699
0.691
ATR ( α=0.9 )
0.847
0.868
0.567
0.768
0.670
0.819
0.508
0.606
SW+ATR ( w=60,α=0.9 )
0.919
0.948
0.772
0.768
0.841
0.843
0.738
0.711
SW+ATR+BO ( w⋆,α⋆,β⋆ )
0.936
0.939
0.793
0.692
0.811
0.807
0.838
0.743
Table 1: Rollout-level failure prediction on LIBERO suites with OpenVLA. We report test AUROC and test Accuracy (threshold chosen by Youden’s index on the train split). Bold denotes the best result and underline denotes the second-best result for each metric within a suite.
Figure 3: Ablation studies on LIBERO-10. (a) SW window size ablation on LIBERO-10. (b) ATR stability contrast α ablation on LIBERO-10. (c) Joint SW+ATR ablation on LIBERO-10; the red star indicates the best-performing (w,α) pair. (d) Effect of DoF-adaptive calibration, comparing SW+ATR with and without Bayesian Optimization across LIBERO suites.
Target Suite
Global Mean
Zero-Shot Transfer
Oracle BO
LIBERO-OBJECT
0.542
0.780
0.793
LIBERO-GOAL
0.641
0.819
0.811
LIBERO-10
0.468
0.715
0.838
Table 2: Cross-suite zero-shot transfer test AUROC. Parameters calibrated solely on the LIBERO-SPATIAL train split transfer robustly to unseen suites.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Suite
w⋆
α⋆
β⋆ (7-DoF)
LIBERO-SPATIAL
93
0.95
[1.0, 2.6, 10.0, 1.0, 1.0, 1.0, 10.0]
LIBERO-OBJECT
62
0.05
[2.6, 1.0, 1.2, 1.0, 10.0, 7.5, 10.0]
LIBERO-GOAL
72
0.05
[1.0, 1.0, 4.6, 1.0, 10.0, 1.0, 10.0]
LIBERO-10
100
0.95
[1.6, 6.9, 10.0, 1.0, 1.0, 1.0, 10.0]
Appendix
Table A1 : Best SW+ATR+BO hyperparameters found on the train split of each suite. All β⋆ values are rounded to one decimal place for readability.
Figure A1 : DoF importance learned by Bayesian Optimization. We visualize the optimized DoF weights β⋆ across LIBERO suites. Gripper and Δz consistently receive high weights, while Δpitch is emphasized in LIBERO-OBJECT and LIBERO-GOAL, supporting DoF-adaptive calibration.
Figure A2 : Per-task success rates (%) of OpenVLA baseline across LIBERO suites. Darker cells indicate higher success. Task difficulty varies significantly both across and within suites, motivating the need for reliable failure prediction via uncertainty quantification.
Figure A3 : Ablation Study on LIBERO-SPATIAL. (a) Impact of window size w on AUROC. (b) Impact of stability contrast α on AUROC. (c) Joint heatmap of AUROC across w and α . (d-e) Detailed metrics (Accuracy, Precision, Recall, F1) for SW and ATR components respectively.
Figure A4 : Ablation Study on LIBERO-OBJECT. (a) Impact of window size w on AUROC. (b) Impact of stability contrast α on AUROC. (c) Joint heatmap of AUROC across w and α . (d-e) Detailed metrics (Accuracy, Precision, Recall, F1) for SW and ATR components respectively.
Figure A5 : Ablation Study on LIBERO-GOAL. (a) Impact of window size w on AUROC. (b) Impact of stability contrast α on AUROC. (c) Joint heatmap of AUROC across w and α . (d-e) Detailed metrics (Accuracy, Precision, Recall, F1) for SW and ATR components respectively.
Figure A6 : Ablation Study on LIBERO-10. (a) Impact of window size w on AUROC. (b) Impact of stability contrast α on AUROC. (c) Joint heatmap of AUROC across w and α . (d-e) Detailed metrics (Accuracy, Precision, Recall, F1) for SW and ATR components respectively.
Vision-Language-Action (VLA) models have shown strong performance in robotic manipulation, but reliable uncertainty quantification remains challenging, particularly under distribution shift. Unlike autoregressive policies, many modern VLA models generate continuous actions through regression or flow-based generation, where explicit predictive probabilities are unavailable. Moreover, stochastic action sampling primarily captures action-generation variability under a fixed model, while failure detection under distribution shift can benefit from capturing uncertainty in the model itself. Motivated by Bayesian perspectives on local model variations, we propose perturbation-based failure detection (PFD), a training-free framework for estimating epistemic uncertainty in VLA models through low-rank weight perturbations. Specifically, we inject random low-rank perturbations into selected transformer weight matrices and estimate epistemic uncertainty from disagreement across perturbed action predictions. Experiments on LIBERO-PRO show that PFD achieves the highest average AUROC and balanced accuracy among the evaluated methods while consistently outperforming stochastic action sampling across distribution shifts. Real-world robot experiments further demonstrate that PFD provides a competitive failure-detection signal under an unseen object shift.
Yousung Lee, Dongsoo Har
Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea.
Vision-language-action (VLA) models have emerged as generalist robotic controllers that map visual observations and language instructions to continuous actions. However, pretrained VLAs typically provide limited information about the reliability of their generated actions, making deployment challenging when execution failures must be anticipated. We address this problem by introducing SentryVLA, a lightweight runtime wrapper for frozen VLA policies. At the action level, SentryVLA uses conformalized quantile regression to estimate a calibrated upper bound on the deviation between sampled VLA actions and expert-like behavior under the calibration distribution. The predicted deviation is then used to rank multiple stochastic candidates for action selection. At the execution-monitoring level, SentryVLA applies a Mahalanobis-distance-based anomaly score to the contextualized latent execution state extracted before final action prediction, providing a complementary signal for detecting abnormal execution states during closed-loop control. The framework does not require retraining or architectural modification of the underlying VLA policy. Experiments in simulation and on a physical robot show that SentryVLA improves action selection and provides useful runtime warning signals for abnormal execution states. These findings demonstrate the practical value of post-hoc reliability scoring for improving the runtime behavior of pretrained VLA policies. Project webpage: https://robotic-vision-lab.github.io/sentryvla/.
Zongyao Lyu, Lingling Chen, William J. Beksi
Department of Computer Science and Engineering, The University of Texas at Arlington, Arlington, TX, USA.
Vision-language-action models (VLAs) combine vision-language backbones with expressive generative action heads trained via flow matching on large-scale robotic datasets. Despite their strong empirical performance in robotic manipulation, VLAs lack mechanisms to quantify confidence in their predictions and to detect when their actions may be unreliable. This presents a critical limitation for real-world deployment in non-stationary environments, where models inevitably encounter scenarios outside their pretraining distribution and may fail without warning. To address this, we derive an efficient method for quantifying epistemic uncertainty in flow-matching models by leveraging velocity-field disagreement (VFD) across a small ensemble. We successfully use this uncertainty estimate for failure detection during deployment and active fine-tuning of flow-based VLAs. To this end, we propose SAVE, a framework for uncertainty-guided active multitask fine-tuning that reduces the number of costly expert demonstrations required to adapt VLAs to new tasks. Through extensive experiments on the LIBERO benchmark, we demonstrate that VFD yields better-calibrated uncertainty estimates predictive of downstream performance, that VFD achieves strong performance in detecting failures, and that uncertainty-guided data acquisition with SAVE requires at least 22% fewer samples than baselines. In summary, our work shows that quantifying epistemic uncertainty in flow-based VLAs improves both failure awareness and adaptation. Project website: tum-lsy.github.io/uq_vla/.