Act, Think or Abstain: Complexity-Aware Adaptive Inference for Vision-Language-Action Models

Riccardo Andrea Izzo; Gianluca Bardaro; Matteo Matteucci

cs.CVMar 5, 2026

Act, Think or Abstain: Complexity-Aware Adaptive Inference for Vision-Language-Action Models

Authors: Riccardo Andrea Izzo, Gianluca Bardaro, Matteo Matteucci

Organizations: All authors are with the Department of Electronics, Information, and Bioengineering, Politecnico di Milano, Milan, Italy.

Abstract

Current research on Vision-Language-Action (VLA) models predominantly focuses on enhancing generalization through reasoning techniques. While effective, these improvements increase computational complexity and inference latency. Furthermore, these mechanisms are typically applied indiscriminately, wasting resources on trivial tasks while failing to provide the uncertainty estimation necessary to prevent catastrophic failure on out-of-distribution scenarios. Inspired by human cognition, we propose an adaptive framework that dynamically routes VLA execution based on the complexity of the perceived state. Our approach transforms the VLA's vision-language backbone into an active detection tool by projecting latent embeddings into a set of parametric and non-parametric estimators. This allows the system to execute known tasks immediately (Act), reason about ambiguous scenarios (Think), and preemptively halt execution when encountering physical or semantic anomalies (Abstain). We find that a Gaussian Mixture Model fitted to fused vision-language embeddings provides the most reliable task-complexity signal, combining visual novelty with instruction context and cross-modal compatibility. Evaluated on the LIBERO and LIBERO-PRO benchmarks as well as on a real robot, our fused configuration achieves up to 87.5% F1-score across two VLA backbones (SmolVLA and $π_0$ ), retains 83% with as little as 5% of training data, and surpasses state-of-the-art failure detectors.

Figures & tables

Fig. 1: Overview of our framework. Given an image observation and a language instruction, the system extracts text and vision embeddings from the frozen VLA backbone (i.e., SmolVLA). These embeddings are then scored using a set of GMMs and kNN. The resulting scores are consolidated into a unified score vector and processed by a custom MLP to select the optimal execution strategy. High confidence triggers direct execution ( Act ), while ambiguous scenarios result in additional reasoning ( Think ). Tasks which are entirely out-of-distribution are preemptively halted ( Abstain ). The Think branch is invoked at most once per episode, after which the task resolves to either Act or Abstain .

Panel of Figure 2 — Fig. 2: Think prompt template. The VLM is queried with the current observation and the structured prompt (top). The output is appended to the task instruction (bottom) and re-scored by the fused GMM.

Fig. 3: Scores and learned routing decisions. GMM (fused) scores for held-out ID, partially OOD, and fully OOD samples from

\mathcal{D}

, shown as unit-area Kernel Density Estimates (KDEs). Dashed lines show MLP decision boundaries, while hollow and filled markers with arrows illustrate post- Think transitions to Act or Abstain .

Fig. 4: Data scaling. Macro F1 with increasing training data fractions.

	SmolVLA			$\pi_{0}$
Method	P	R	F1	P	R	F1
MLP (Raw features)	80.08	69.70	62.34	80.73	69.97	62.77
MLP (PCA-reduced features)	77.19	70.79	65.47	79.06	71.38	66.04
MLP + GMM (text + vision + fused)	72.57	70.67	67.76	74.04	71.45	68.42
MLP + GMM (all) + kNN	76.01	73.54	71.41	78.05	75.19	73.32
MLP + GMM (text)	49.90	64.99	54.76	50.15	65.42	54.98

TABLE I: Component analysis. Precision (P), Recall (R), and Macro F1 (F1) on the three-class Act/Think/Abstain decision.

Fig. 5: Rollout examples on simulation. Representative episodes from the LIBERO and LIBERO-PRO benchmarks: known tasks are executed with high confidence (left), ambiguous scenarios trigger additional reasoning leading to success (center), out-of-distribution tasks are preemptively halted (right).

	LIBERO				LIBERO-PRO
Method	Bal-Acc	TPR	TNR	ROC-AUC	Bal-Acc	TPR	TNR	ROC-AUC
logpZO [ 15 ]	65.58	55.41	75.75	77.67	75.42	75.46	75.37	80.91
SAFE-MLP [ 16 ]	76.33	70.27	82.40	82.23	89.50	90.01	88.98	89.45
Ours	83.66	79.73	87.59	92.23	93.58	95.36	91.80	94.12

TABLE II: Failure detection baselines. Binary execute/abstain decision with the

\pi_{0}

backbone. Abstain is the positive class; Think episodes count as their final resolved decision.

SmolVLA

\pi_{0}

Baseline

Ours

Baseline

Ours

Suite

Variant

SR (%)

\uparrow

T_{inf}

(s)

\downarrow

SR (%)

\uparrow

A (T) / Ab (T)

T_{inf}

(s)

\downarrow

SR (%)

\uparrow

T_{inf}

(s)

\downarrow

SR (%)

\uparrow

A (T) / Ab (T)

T_{inf}

(s)

\downarrow

Goal

base

83.33

\pm

15.28

49.71

86.67

\pm

15.28

29 (1) / 1 (0)

52.78

93.33

\pm

5.77

58.99

96.67

\pm

5.77

30 (1) / 0 (0)

61.02

object

60.00

\pm

43.59

97.73

60.00

\pm

43.59

19 (2) / 11 (3)

61.42

93.33

\pm

5.77

105.41

93.33

\pm

5.77

29 (1) / 1 (0)

104.23

position

0.00

\pm

0.00

163.46

0.00

\pm

0.00

0 (0) / 30 (1)

4.33

3.33

\pm

5.77

188.63

10.00

\pm

10.00

3 (0) / 27 (3)

7.30

semantic

90.00

\pm

10.00

52.89

93.33

\pm

11.55

28 (2) / 2 (1)

53.80

86.67

\pm

5.77

55.11

90.00

\pm

10.00

28 (1) / 2 (1)

46.97

TABLE III: Simulation evaluation. Performance on LIBERO and LIBERO-PRO with the SmolVLA and

\pi_{0}

backbones.

Fig. 6: Real-robot rollout examples. SO-ARM 101 with SmolVLA performing tabletop manipulation tasks.

Regime	SR (%) $\uparrow$	$T_{inf}$ (s) $\downarrow$	SR (%) $\uparrow$	PF	A (T) / Ab (T)	$T_{inf}$ (s) $\downarrow$
	Baseline		Ours
ID	90.0 $\pm$ 10.0	114.8	96.7 $\pm$ 5.8	0	30 (4) / 0 (0)	108.6
Partially OOD	36.7 $\pm$ 5.8	190.3	50.0 $\pm$ 0.0	11	15 (11) / 15 (8)	78.2
Fully OOD	6.7 $\pm$ 5.8	207.3	13.3 $\pm$ 5.8	23	5 (5) / 25 (9)	25.1

TABLE IV: Real-robot evaluation. SmolVLA on the SO-ARM 101.

Explore similar work

ATA: Bridging Implicit Reasoning with Attention-Guided and Action-Guided Inference for Vision-Language Action Models
Mar 2, 2026Cheng Yang, Jianhao Jiao, Lingyi Huang +8Vision-Language-Action Models Reasoning for VLA Models
Rutgers University · University College London · Rice University +1
Think Less, Act Early: Reinforced Latent Reasoning with Early Exit in Vision-Language-Action Models
Jun 13, 2026Dianqiao Lei, Lianlei ShanEfficient Inference Efficient VLA Model Inference
1Tsinghua University, Beijing, China.
VLA-FAIL: Efficient Task Failure Detection for Finetuned Vision-Language-Action Models
Jun 19, 2026Florian Seligmann, Emiliyan Gospodinov, Enes Ulas Dincer +1Mahalanobis Distance Vision-Language-Action Models
Autonomous Learning Robots Lab, Karlsruhe Institute of Technology · FZI Forschungszentrum Informatik, Karlsruhe

Mar 2, 2026J/K move · Enter open · S save

ATA: Bridging Implicit Reasoning with Attention-Guided and Action-Guided Inference for Vision-Language Action Models

Cheng Yang, Jianhao Jiao, Lingyi Huang, Jinqi Xiao+7

Rutgers University · University College London · Rice University+1

Vision-Language-Action Models Reasoning for VLA Models Efficient VLA Model Inference Robustness of VLA Models

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models rely on current observations, including images, language instructions, and robot states, to predict actions and complete tasks. While accurate visual perception is crucial for precise action prediction and execution, recent work has attempted to further improve performance by introducing explicit reasoning during inference. However, such approaches face significant limitations. They often depend on data-intensive resources such as Chain-of-Thought (CoT) style annotations to decompose tasks into step-by-step reasoning, and in many cases require additional visual grounding annotations (e.g., bounding boxes or masks) to highlight relevant image regions. Moreover, they involve time-consuming dataset construction, labeling, and retraining, which ultimately results in longer inference sequences and reduced efficiency. To address these challenges, we propose ATA, a novel training-free framework that introduces implicit reasoning into VLA inference through complementary attention-guided and action-guided strategies. Unlike CoT or explicit visual-grounding methods, ATA formulates reasoning implicitly by integrating attention maps with an action-based region of interest (RoI), thereby adaptively refining visual inputs without requiring extra training or annotations. ATA is a plug-and-play implicit reasoning approach for VLA models, lightweight yet effective. Extensive experiments show that it consistently improves task success and robustness while preserving, and even enhancing, inference efficiency.

Abstract

Explore similar work

ATA: Bridging Implicit Reasoning with Attention-Guided and Action-Guided Inference for Vision-Language Action Models

Think Less, Act Early: Reinforced Latent Reasoning with Early Exit in Vision-Language-Action Models

VLA-FAIL: Efficient Task Failure Detection for Finetuned Vision-Language-Action Models