Decision-Aware Uncertainty Evaluation of Vision-Language Model-Based Early Action Anticipation for Human-Robot Interaction

Zhaoda Du; Michael Bowman; Qiaojie Zheng; Xiaoli Zhang

cs.ROMar 9, 2026

Decision-Aware Uncertainty Evaluation of Vision-Language Model-Based Early Action Anticipation for Human-Robot Interaction

Authors: Zhaoda Du, Michael Bowman, Qiaojie Zheng, Xiaoli Zhang

Organizations: Colorado School of Mines, Golden, CO 80401, USA · Postdoctoral Fellow in the Cancer Biology Department at the University of Pennsylvania, Philadelphia, PA 19104, USA

Abstract

Robots in shared workspaces must interpret human actions from partial, ambiguous observations, where overconfident early predictions can lead to unsafe or disruptive interaction. This challenge is amplified in egocentric views, where viewpoint changes and occlusions increase perceptual noise and ambiguity. As a result, downstream human-robot interaction modules require not only an action hypothesis but also a trustworthy estimate of confidence under partial observation. Recent vision-language model-based approaches have been proposed for short-term action recognition due to their open-vocabulary and context-aware reasoning, but their uncertainty reliability in the temporal-prefix regime is largely uncharacterized. We present the first systematic evaluation of uncertainty in vision-language model-based short-term action recognition for human-robot interaction. We introduce a temporal-prefix evaluation protocol and metrics for calibration and selective prediction. We also characterize miscalibration patterns and failure modes under partial observations. Our study provides the missing reliability evidence needed to use vision-language model predictions in confidence-gated human-robot interaction modules.

Figures & tables

Fig. 1: Comparison between traditional accuracy-based Top-K selection and our confidence-aware selection framework for short-term action anticipation. Given observed short context, a VLM produces Top-K action hypotheses. Traditional methods select the highest-scoring action and commit early to a single hypothesis. In contrast, our approach models confidence and preserves multiple plausible intents, enabling delayed commitment, active clarification, and uncertainty-calibrated execution.

Fig. 2: From stochastic variability to structured uncertainty for HRI decision-making. Multiple stochastic decoding runs produce unstable Top-K action hypotheses under partial observation. Aggregation operators transform these discrete prediction sets into a structured confidence distribution over candidate actions. Different aggregation strategies reshape the geometry of confidence in distinct ways, influencing calibration behavior and confidence-based decision gating in downstream human–robot interaction systems.

Method	Rank-wise	Verbalized Conf.	Global
Consistency	✓	$\times$	$\times$
Confidence-weighted	✓	✓	$\times$
Pairwise Ranking	$\times$	$\times$	✓

TABLE I: Comparison of aggregation strategies.

Attribute	Role in HRI	Verified Property	Metrics
Correctness	Candidate validity	Ranking quality of predicted Top-K actions	Top-1 Recall@K
Uncertainty Reliability	Probabilistic fidelity	Alignment between confidence and empirical correctness	Top-1 ECE Set-ECE vs $K$
Selective Decision Utility	Decision gating	Effectiveness of confidence-based abstention	Coverage-Threshold Accuracy-Threshold
Confidence Geometry	Ambiguity modeling	Structural distribution of confidence mass	Rank-wise Conf Distribution. Normalized Entropy

TABLE II: Decision-Aware Evaluation Framework for Top-K Uncertainty in HRI

Dataset	Method	Top-1 $\uparrow$	Recall@10 $\uparrow$	Top-1 ECE $\downarrow$	Set-ECE@10 $\downarrow$	Entropy
EGTEA Gaze+	Pairwise Ranking	0.151	0.487	0.6977	0.387	0.184
	Consistency	0.162	0.401	0.6009	0.027	0.958
	Confidence-weighted	0.162	0.403	0.6175	0.101	0.964
	Single run	0.160	0.425	0.2404	0.322	0.749
EPIC-KITCHENS-100	Pairwise Ranking	0.026	0.124	0.6377	0.027	0.419
	Consistency	0.030	0.114	0.5984	0.232	0.958

TABLE III: Main results on Dataset A and Dataset B (

K=10

, bins=10).

Fig. 3: Comparison of aggregation strategies on two egocentric action benchmarks (top: EGTEA Gaze+; bottom: EPIC-KITCHENS-100). From left to right: Recall@K, selective accuracy and coverage under confidence thresholding , set-level calibration, and normalized Top-10 entropy. Ranking performance differences are modest. Calibration behavior and confidence geometry exhibit clearer structural differences across aggregation strategies, impacting selective decision characteristics.

Fig. 4: Rank-wise confidence distributions across aggregation strategies on EGTEA Gaze+ (top) and EPIC-KITCHENS-100 (bottom). Boxplots show the distribution of confidence assigned to each rank position within the Top-K set. PairRank sharply concentrates confidence on top ranks, whereas consistency and confidence-weighted methods yield smoother, higher-entropy distributions.

Fig. 5: Illustrative example of Top- $K$ ( $K=5$ ) prediction with confidence thresholding ( $\tau=0.39$ ) for a temporal prefix, used to examine interaction responses in HRI. When only Top- $K$ rankings are considered, the aggregation strategies appear behaviorally similar. However, incorporating confidence via threshold-based selection leads to markedly different clarification behaviors. Sharp distributions (e.g., PairRank) may produce overconfident misprediction, whereas smoother aggregations(e.g., Consistency) expand the clarification space and increase interaction burden. Low-confidence single-run outputs may suppress interaction under thresholding. This example highlights the importance of uncertainty-aware evaluation for interaction-level decision-making.

Uncertainty Estimation for VLA Models

Uncertainty estimation for Vision-Language-Action (VLA) models predicts when robotic actions or task outcomes may be unreliable.

See all 32 papers

Confidence Calibration in Vision-Language-Action Models
Jul 23, 2025Thomas P Zollo, Richard ZemelUncertainty Estimation for VLA Models Vision-Language-Action Models
Columbia University
SentryVLA: Uncertainty-Guided Action Selection and State Monitoring for Vision-Language-Action Robotic Control
Apr 17, 2026Zongyao Lyu, Lingling Chen, William J. BeksiUncertainty Estimation for VLA Models Robot Failure Detection
Department of Computer Science and Engineering, The University of Texas at Arlington, Arlington, TX, USA.
Shifting Uncertainty to Critical Moments: Towards Reliable Uncertainty Quantification for VLA Model
Mar 18, 2026Yanchuan Tang, Taowen Wang, Yuefei Chen +3Uncertainty Estimation for VLA Models Uncertainty Calibration
Rutgers University · Kent State University

Jul 23, 2025J/K move · Enter open · S save

Confidence Calibration in Vision-Language-Action Models

Thomas P Zollo, Richard Zemel

Columbia University

Uncertainty Estimation for VLA Models Vision-Language-Action Models

Open paper AskPDF arXiv

Trustworthy robot behavior requires not only high levels of task success but also that the robot can reliably quantify how likely it is to succeed. To this end, we present a first-of-its-kind study of confidence calibration in vision-language-action (VLA) foundation models, which map visual observations and natural language instructions to low-level robot motor commands. We establish a confidence estimation baseline for VLAs, examine how task success relates to calibration error and how calibration evolves over time, and introduce two lightweight techniques to remedy the miscalibration we observe: prompt ensembles and action-wise Platt scaling. Our aim in this study is to begin to develop the tools and conceptual understanding necessary to render VLAs trustworthy via reliable uncertainty quantification.

Abstract

Uncertainty Estimation for VLA Models

Explore similar work

Confidence Calibration in Vision-Language-Action Models

SentryVLA: Uncertainty-Guided Action Selection and State Monitoring for Vision-Language-Action Robotic Control

Shifting Uncertainty to Critical Moments: Towards Reliable Uncertainty Quantification for VLA Model