Confidence Calibration in Vision-Language-Action Models

Thomas P Zollo; Richard Zemel

cs.ROJul 23, 2025

Confidence Calibration in Vision-Language-Action Models

Authors: Thomas P Zollo, Richard Zemel

Organizations: Columbia University

Abstract

Trustworthy robot behavior requires not only high levels of task success but also that the robot can reliably quantify how likely it is to succeed. To this end, we present a first-of-its-kind study of confidence calibration in vision-language-action (VLA) foundation models, which map visual observations and natural language instructions to low-level robot motor commands. We establish a confidence estimation baseline for VLAs, examine how task success relates to calibration error and how calibration evolves over time, and introduce two lightweight techniques to remedy the miscalibration we observe: prompt ensembles and action-wise Platt scaling. Our aim in this study is to begin to develop the tools and conceptual understanding necessary to render VLAs trustworthy via reliable uncertainty quantification.

Figures & tables

Figure 1: To be trustworthy, a robotic system must be able to reliably express its confidence in its ability to successfully complete a task trial, especially in high-stakes and open-world domains. A well-calibrated robot policy produces confidence estimates that align with its probability of trial success, ideally updating its estimate as the task progresses. For example, the robot should succeed on 95% of instances for which it expresses 95% confidence.

Figure 2: Overview of confidence estimation and calibration for token-based VLAs. Given an input image and text instruction, a token-based VLA generates distributions over discrete action representations, which may correspond to, e.g., action dimensions, latent action codes, or action chunks depending on the model architecture. These distributions can be converted into scalar confidence estimates by aggregating the probability assigned to the selected tokens. Given an uncalibrated confidence estimate, recalibration methods such as Platt scaling use a small calibration dataset to learn a map from uncalibrated confidence estimates to calibrated ones. Calibration error can then be measured by comparing confidence estimates to actual task success rates.

Figure 3: Visualization of task error rates compared against 4 different calibration error measurements for 4 VLA variants (OpenVLA, MolmoAct, UniVLA, and NORA) and 4 LIBERO task suites (Spatial, Object, Goal, 10), as well as OpenVLA 8- and 4-bit versions on Spatial, Object, and Goal. Vertical error bars report 95% percentile-bootstrap intervals for the calibration metrics obtained from 2,000 episode-level resamples with replacement within each model–task-suite setting. All models exhibit a roughly monotonic relationship between task error and the discriminative measures (Brier score and NLL). ECE shows differences between model families, potentially due to architecture and objective differences.

Panel of Figure 4 — Figure 4: Empirical study of calibration error across the task time. In the top row (a), the left two plots quantify how calibration evolves with task progress, while the right two plots show the average confidence by task time, grouped by successful and failed trials. Shaded regions indicate $\pm 1$ standard error: ECE uncertainty is estimated from 500 episode-level bootstrap resamples shared across task-completion points, while Brier and confidence-curve errors are computed analytically across episodes. The bottom rows provide qualitative reliability diagrams for two representative aggregation rules: current-step confidence (b) and averaging all observed confidences so far (c); we omit the sliding-window variant for visual clarity because its behavior is intermediate between these two. Overall, these results illustrate that calibration can improve as the task progresses and more information is gathered, suggesting opportunities for context-aware uncertainty interventions.

Panel of Figure 5 — Figure 5: Qualitative examples of a context-aware confidence monitoring strategy applied to a task from the Goal suite. Here, the task is to “put the wine bottle on the rack”. The red dashed line represents the 10% quantile of the confidence estimates output by the model across the task time horizon, offering a potential threshold below which the robot may abort execution.

Figure 6: Each bar shows the percent change in a calibration metric after applying Reprompt, relative to the baseline confidence estimate from the original instruction; negative values indicate improved calibration. Results are grouped by OpenVLA task suite and by VLA model on the Spatial suite. Error bars denote $\pm 1$ prompt-bootstrap standard error, obtained by resampling the 20 instruction rephrasings. $\text{ECE}_{2}$ and NLL always decrease, while $\text{ECE}_{1}$ and Brier score decrease in 5 of 6 cases.

Panel of Figure 7 — Figure 7: For each model/task setting, the left plot shows ECE for each action dimension using the baseline per-dimension confidence, while the right plot compares recalibration methods (temperature scaling is excluded from MolmoAct results, as the variable-length action encodings preclude a straightforward adaptation). Lower values indicate better calibration in both plots. Error bars in the right plots report $\pm 1$ standard error across the 1,000 random calibration–test splits. Calibration varies substantially across action dimensions, and action-wise Platt scaling consistently improves over global recalibration baselines.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Example Image	Example Instruction
Spatial		Pick up the black bowl between the plate and the ramekin and place it on the plate.
Object		Pick up the alphabet soup and place it in the basket.
Goal		Open the middle drawer of the cabinet.
10		Put both the alphabet soup and the tomato sauce in the basket.

Appendix

Table 1: Task examples from the LIBERO task suites: Spatial, Object, Goal, and 10.

Instruction	Pick up the black bowl between the plate and the ramekin and place it on the plate
Rephrasings	(1) Lift the black bowl located between the plate and the ramekin and set it on the plate.
	(2) Grasp the black bowl found between the plate and the ramekin and move it to the plate.
	(3) Take the black bowl positioned between the plate and the ramekin and position it on the plate.

Appendix

Table 2: Example of multiple paraphrases of a single VLA instruction (from our experiments with LIBERO Spatial). The rephrasings carry the same meaning as the original instruction, enabling a reprompting approach in which we ensemble over predictions conditioned on lexically different yet semantically equivalent instructions.

	Prompt
Prompt 1	You are generating alternative phrasings of a robotic task instruction while preserving its exact meaning. ### Task Instruction: ’[TASK DESCRIPTION]’ ### Instructions: - Generate 20 alternative ways to phrase the task instruction. - Keep each instruction concise and unambiguous. - Ensure the instructions remain suitable for a robot, not a human. - Only make semantically meaningless changes (e.g., word order, synonyms, slight rewording). - Double-check that the new instructions mean the same exact thing for the robot; do not just substitute synonyms without considering context. - Do not introduce additional steps, remove essential details, or alter the action. ### Output Format: Each rephrased instruction should be wrapped in ‘[instruction]‘ and ‘[/instruction]‘ tags, like this: [instruction] Rephrased instruction 1 [/instruction] [instruction] Rephrased instruction 2 [/instruction]
Prompt 2	You are generating alternative phrasings of a robotic task instruction while preserving its exact meaning. ### Task Instruction: ’[TASK DESCRIPTION]’ ### Instructions: - Generate 20 alternative ways to phrase the task instruction. - Keep each instruction concise and unambiguous. - Ensure the instructions remain suitable for a robot, not a human. - Only make semantically meaningless changes (e.g., word order, synonyms, slight rewording). - Double-check that the new instructions mean the same exact thing for the robot; do not just substitute synonyms without considering context. - Do not introduce additional steps, remove essential details, or alter the action. - The first word of the instruction should be ‘PICK’, and then it should also include the word ‘PLACE’. ### Output Format: Each rephrased instruction should be wrapped in ‘[instruction]‘ and ‘[/instruction]‘ tags, like this: [instruction] Rephrased instruction 1 [/instruction] [instruction] Rephrased instruction 2 [/instruction]
Prompt 3	You are generating alternative phrasings of a robotic task instruction while preserving its exact meaning. ### Task Instruction: ’[TASK DESCRIPTION]’ ### Instructions: - Generate 20 alternative ways to phrase the task instruction. - Keep each instruction concise and unambiguous. - Ensure the instructions remain suitable for a robot, not a human. - Only make semantically meaningless changes (e.g., word order, synonyms, slight rewording). - Double-check that the new instructions mean the same exact thing for the robot; do not just substitute synonyms without considering context. - Do not introduce additional steps, remove essential details, or alter the action. - Make the changes as minor as possible, as the robot’s language system is not very robust to rephrasing. - The first word of the instruction should be ‘PICK’, and then it should also include the word ‘PLACE’. ### Output Format: Each rephrased instruction should be wrapped in ‘[instruction]‘ and ‘[/instruction]‘ tags, like this: [instruction] Rephrased instruction 1 [/instruction] [instruction] Rephrased instruction 2 [/instruction]

Appendix

Table 3: Prompts for GPT-4o-mini to rephrase LIBERO task instructions. These prompts are used for Spatial and Object. There is a small modification for Goal.

Model	Method	$ECE_{1}$	$ECE_{2}$	Brier score	NLL
NORA	Arithmetic Mean	0.140	0.158	0.142	0.483
	Geometric Mean	0.162	0.185	0.150	0.501
	Max	0.148	0.152	0.148	2.006
	Min	0.442	0.468	0.354	0.936
MolmoAct	Arithmetic Mean	0.116	0.127	0.137	0.573
	Geometric Mean	0.113	0.124	0.136	0.568

Appendix

Table 4: Comparison of different strategies for aggregating confidence across the set of tokens produced by the VLA models. Results are presented (1) per model, averaged across task suites and (2) averaged across all models and task suites. Using the arithmetic mean performs best on average for

ECE_{1}

and the proper scoring rules, but the geometric mean offers comparable performance.

Dataset	Model	Quant	$\text{ECE}_{1}$	$\text{ECE}_{2}$	Brier	NLL	Succ. Rate
Spatial	NORA	-	0.099	0.115	0.105	0.386	0.894
Object	NORA	-	0.210	0.229	0.090	0.343	0.960
Goal	NORA	-	0.065	0.077	0.111	0.391	0.880
10	NORA	-	0.184	0.212	0.260	0.811	0.674
Spatial	MolmoAct	-	0.113	0.123	0.134	0.625	0.860
Object	MolmoAct	-	0.046	0.059	0.062	0.296	0.936

Appendix

Table 5: All task success and calibration results for 22 VLA/task suite combinations.

Figure 8: Calibration-in-the-large across model–task suite pairs. Left: mean first-step confidence versus empirical success rate, with the dashed line denoting perfect marginal calibration. Right: marginal calibration error $|\bar{c}-\bar{y}|$ versus task error, with model-family trend lines.

Figure 9: Calibration metrics relative to the constant mean-confidence control. We report $\Delta M=M(\mathbf{c},\mathbf{y})-M(\bar{c},\mathbf{y})$ ; positive values indicate higher error for the original trial-level confidence estimates.

Figure 10: Calibration metrics for the original trial-level confidence estimates and the corresponding constant mean-confidence controls across model families and task suites.

Model	Dataset	Method	$\text{ECE}_{1}$	$\text{ECE}_{2}$	Brier score	NLL
OpenVLA	Spatial	Baseline	0.0879	0.1062	0.1498	0.5330
		Reprompt	0.0525	0.0685	0.1449	0.4766
	Object	Baseline	0.0602	0.0727	0.1076	0.4010
		Reprompt	0.0363	0.0527	0.1046	0.3610
	Goal	Baseline	0.1513	0.1701	0.2065	0.7072
		Reprompt	0.1155	0.1322	0.1974	0.6195

Appendix

Table 6: Calibration error measurements using all 4 models (and the OpenVLA 8-bit and 4-bit fine-tuned model versions) and 3 different LIBERO task suites, for 2 different methods of confidence estimation: (1) baseline (2) ensembling over semantically equivalent prompts (“Reprompt”). The prompt ensemble method improves most measures.

Figure 11: Ablation results for the prompt ensemble method, where different prompts are given to GPT-4o-mini for producing the rephrasings of the original instruction. Prompt 1 is used in the main experiments, while Prompts 2 and 3 are variants. Results show that improvements in calibration error from Reprompt are robust to different rephrasing prompts.

Panel of Figure 12 — Figure 12: Ablating the number of prompts in the prompt ensemble for the Spatial, Goal, and Object task suites. Increasing the number of prompts generally improves calibration error.

Task	Dimension agreement	Exact 6-D agreement	Within one bin	Gripper agreement	MAE
Spatial	49.56%	13.40%	16.30%	99.85%	0.0453
Object	65.20%	42.70%	45.66%	99.18%	0.0216
Goal	51.76%	10.38%	13.32%	99.98%	0.0384

Appendix

Table 7: Agreement between first actions produced by OpenVLA under the original instruction and 20 semantically equivalent rephrasings. Dimension agreement averages exact agreement across the six non-gripper motion dimensions; exact 6-D agreement requires all six motion coordinates to match. “Within one bin” allows each motion coordinate to differ by at most one OpenVLA token bin. MAE is measured for the 6 motion dimensions on the normalized

[-1,1]

action scale.

Panel of Figure 13 — Figure 13: Empirical study of calibration error across task time horizon for the Spatial task suite and the Quant-8 OpenVLA model.

Panel of Figure 14 — Figure 14: Empirical study of calibration error across task time horizon for the Object task suite and the full precision OpenVLA model.

Panel of Figure 15 — Figure 15: Empirical study of calibration error across task time horizon for the Object task suite and the Quant-8 OpenVLA model.

Panel of Figure 16 — Figure 16: Empirical study of calibration error across task time horizon for the Goal task suite and the full precision OpenVLA model.

Panel of Figure 17 — Figure 17: Empirical study of calibration error across task time horizon for the Goal task suite and the Quant-8 OpenVLA model.

Panel of Figure 18 — Figure 18: Empirical study of calibration error across task time horizon for the Spatial task suite and the UniVLA model.

Panel of Figure 19 — Figure 19: Empirical study of calibration error across task time horizon for the Object task suite and the UniVLA model.

Panel of Figure 20 — Figure 20: Empirical study of calibration error across task time horizon for the Goal task suite and the UniVLA model.

Panel of Figure 21 — Figure 21: Qualitative examples of a context-aware confidence calibration strategy for the task “put the wine bottle on the rack”. The red dashed line represents the 10% quantile of the confidence estimates output by the model across the task horizon, representing a potential threshold below which the robot may abstain from performing the task.

Panel of Figure 22 — Figure 22: On the left of each pair of plots, we compare miscalibration across action dimensions. On the right, we compare the performance of typical Platt scaling, temperature scaling, histogram binning, and Platt binning to action-wise Platt scaling.

Uncertainty Estimation for VLA Models

Uncertainty estimation for Vision-Language-Action (VLA) models predicts when robotic actions or task outcomes may be unreliable.

See all 32 papers

VLAConf: Calibrated Task-Success Confidence for Vision-Language-Action Models
May 28, 2026Dehao Huang, Aoxiang Gu, Chengjie Zhang +5Confidence Estimation for VLMs Uncertainty Estimation for VLA Models
Department of Electronic and Electrical Engineering, Southern University of Science and Technology, Shenzhen, China. · Zhongguancun Academy, Beijing, China · National Cybersecurity Academy, Wuhan University, Wuhan, China.
SentryVLA: Uncertainty-Guided Action Selection and State Monitoring for Vision-Language-Action Robotic Control
Apr 17, 2026Zongyao Lyu, Lingling Chen, William J. BeksiUncertainty Estimation for VLA Models Robot Failure Detection
Department of Computer Science and Engineering, The University of Texas at Arlington, Arlington, TX, USA.
Uncertainty Quantification for Flow-Based Vision-Language-Action Models
Jun 16, 2026Ralf Römer, Maximilian Seeliger, Saida Liu +5Flow Matching Uncertainty Estimation for VLA Models
TU Munich · ETH Zurich · TU of Munich +1

May 28, 2026J/K move · Enter open · S save

VLAConf: Calibrated Task-Success Confidence for Vision-Language-Action Models

Dehao Huang, Aoxiang Gu, Chengjie Zhang, Bolin Zou+4

Department of Electronic and Electrical Engineering, Southern University of Science and Technology, Shenzhen, China. · Zhongguancun Academy, Beijing, China · National Cybersecurity Academy, Wuhan University, Wuhan, China.

Confidence Estimation for VLMs Uncertainty Estimation for VLA Models Probability Calibration Robotic Manipulation Vision-Language-Action Models One-Class Classification

Open paper AskPDF arXiv

Confidence estimation for Vision-Language-Action (VLA) models is essential for robots to perform manipulation tasks in the open world, providing crucial signals for risk-sensitive decision-making and failure anticipation. Existing confidence estimation methods typically rely on ensemble-based paradigms or action-token probabilities to predict the likelihood of task success. However, they still encounter challenges in computational efficiency and cross-architecture generalizability. These methods usually require repeated sampling, leading to inference inefficiency, and are restricted to VLA models with discrete action outputs, making them difficult to apply to continuous action spaces. To address this issue, we propose VLAConf, a one-class discriminative confidence framework. By leveraging frozen pretrained VLA internal representations, VLAConf directly estimates step-wise anomaly scores in a single forward pass using a lightweight confidence head, thereby eliminating the overhead of exhaustive resampling. We additionally use step-conditioned modeling to encode rollout-phase information along the manipulation trajectory. Experiments on the LIBERO benchmark demonstrate that VLAConf significantly improves the quality of the confidence signal constructed for post-hoc calibration, outperforming existing baselines by a large margin in inference efficiency. The effectiveness of VLAConf is further validated in real-robot experiments. To access the source code and supplementary videos, visit https://sites.google.com/view/vlaconf.

Abstract

Uncertainty Estimation for VLA Models

Explore similar work

VLAConf: Calibrated Task-Success Confidence for Vision-Language-Action Models

SentryVLA: Uncertainty-Guided Action Selection and State Monitoring for Vision-Language-Action Robotic Control

Uncertainty Quantification for Flow-Based Vision-Language-Action Models