Self-Refining Vision Language Model for Robotic Failure Detection and Reasoning

Carl Qi; Xiaojie Wang; Silong Yong; Stephen Sheng; Huitan Mao; Sriram Srinivasan; Manikantan Nambi; Amy Zhang

cs.ROFeb 12, 2026

Self-Refining Vision Language Model for Robotic Failure Detection and Reasoning

Authors: Carl Qi, Xiaojie Wang, Silong Yong, Stephen Sheng, Huitan Mao, Sriram Srinivasan, Manikantan Nambi, Amy Zhang,

Organizations: UT Austin · Amazon Robotics · Carnegie Mellon University

Abstract

Reasoning about failures is crucial for building reliable and trustworthy robotic systems. Prior approaches either treat failure reasoning as a closed-set classification problem or assume access to ample human annotations. Failures in the real world are typically subtle, combinatorial, and difficult to enumerate, whereas rich reasoning labels are expensive to acquire. We address this problem by introducing ARMOR: Adaptive Round-based Multi-task mOdel for Robotic failure detection and reasoning. We formulate detection and reasoning as a multi-task self-refinement process, where the model iteratively predicts detection outcomes and natural language reasoning conditioned on past outputs. During training, ARMOR learns from heterogeneous supervision - large-scale sparse binary labels and small-scale rich reasoning annotations - optimized via a combination of offline and online imitation learning. At inference time, ARMOR generates multiple refinement trajectories and selects the most confident prediction via a self-certainty metric. Experiments across diverse environments show that ARMOR achieves state-of-the-art performance by improving over the previous approaches by up to 30% on failure detection rate and up to 100% in reasoning measured through LLM fuzzy match score, demonstrating robustness to heterogeneous supervision and open-ended reasoning beyond predefined failure modes. We provide additional visualizations on our website: https://sites.google.com/utexas.edu/armor

Figures & tables

Figure 1: We present ARMOR: adaptive round-based multi-task model for robotic failure detection and reasoning. Prior approaches reduce failure reasoning to closed-set classification of pre-defined modes (e.g., “spillage” or “unstable grasp”). In contrast, ARMOR performs open-ended, iterative refinement by jointly refining its detection and reasoning predictions. This enables accurate detection with nuanced and human-like reasoning that capture real-world failures beyond fixed categories.

Figure 2: Overview of ARMOR. (a) Our failure data consist of heterogeneous supervision, with large-scale binary detection labels and scarce free-form reasoning labels. (b) A vision-language model (VLM) with multi-task heads jointly predicts detection $l$ via a classification head and reasoning $e$ via a language decoder, trained with binary cross-entropy (BCE) and next-token prediction (NTP) losses. (c) We fine-tune the VLM with both offline imitation and online refinement: the model conditions on dataset labels ( $l,e$ ) or its prior predictions ( $\hat{l},\hat{e}$ ) as well as auxiliary prompts $p$ to generate a new round of outputs. In online refinement, this process is repeated $T$ times. The model’s predictions (denoted by $\hat{l}$ and $\hat{e}$ in purple rectangles) are supervised with available labels. (d) At inference, ARMOR performs iterative self-refinement, rolling out multiple stochastic trajectories. We select best prediction $m^{\star}$ from the final predictions with the lowest entropy score.

Models	Evaluation Datasets	Detect Acc. $\uparrow$	LLM Fuzzy $\uparrow$	ROUGE L $\uparrow$
Qwen2.5-VL	RLBench	0.376	0.255	0.353
	Sparrow	0.453	0.240	0.137
	Maniskill	0.548	0.268	0.112
	ARMBench	0.500	0.349	0.208
Cosmos-Reasoning	RLBench	0.317	0.220	0.140
	Sparrow	0.510	0.346	0.311

Table 1: Quantitative Results on Failure Detection and Reasoning. Metrics include detection accuracy and reasoning quality (LLM Fuzzy and ROUGE L ). Our method achieves higher performance across different domains, accurately detecting failures and producing high quality reasoning.

Ablation	Offline	Offline Expert	Online	Refinement	Detection / Reasoning
Ablation	Warmup	Condition	Imitation	(Inference)	Detection / Reasoning
Multitask Prediction	✓	✗	✗	✗	0.897 / 0.460
Refinement Only	✓	✗	✗	✓	0.803 / 0.488
Offline Imitation Only	✓	✓	✗	✓	0.853 / 0.658
Online Imitation Only	✗	✗	✓	✓	0.850 / 0.683
ARMOR (ours)	✓	✓	✓	✓	0.917 / 0.718

Table 2: Ablation of ARMOR’s components. Each row shows the impact of different stages on detection accuracy and reasoning quality. ARMOR combines all components to achieve the best overall performance.

Panel of Figure 3 — Figure 3: Refinement analysis in ARMOR. (a) Effect of refinement rounds on entropy and performance: detection and reasoning entropy decrease steadily across rounds, while combined plots show that refinement improves both detection accuracy and reasoning quality (values computed over 300 test datapoints). (b) Example of ARMOR’s multi-round refinement process, where predictions are iteratively updated to improve the quality and consistency between detection and reasoning outputs.

Refinement Rounds	Reasoning (LLM Fuzzy)
Round 0	0.475 $\pm$ 0.016
Round 1	0.676 $\pm$ 0.025 (+42.4%)
Round 2	0.703 $\pm$ 0.013 (+4.0%)
Round 3	0.717 $\pm$ 0.002 (+2.0%)

Table 3: ARMOR’s reasoning performance across refinement rounds. Mean and std are reported across 4 seeds. Each round shows improvements with decreasing variance, demonstrating consistent gains.

Refinement Rounds	Avg. Memory Usage per GPU (GB)	Wall Clock Time (s)
Round 0	3.48	7.95
Round 1	5.84	9.30
Round 2	6.12	10.50
Round 3	6.31	10.95

Table 4: Memory usage and wall clock time across refinement rounds. ARMOR adds minimal computational overhead, with approximately 1s increase per round and modest memory growth.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Value
Global batch size (videos)	16
Global batch size (collaged images)	64
Offline epochs	3
Online epochs	10
Horizon $T$	3
Frames per second (FPS)	5.0

Appendix

Table 5: Hyperparameters for training and inference.

Figure 4: ARMOR model architecture. We select the intermediate layer representation from the LM decoder of Qwen2.5-VL ( Bai et al., 2025 ) and attach a classification head for detection, while using the original LM decoder for reasoning. Conditioning prompts describe the previous outputs for each task, enabling iterative multi-task refinement.

Dataset	Domain	Sparse	Dense	Test	Unseen task	Unseen workspace	Unseen failure
RLBench	Simulation	8000	1000	300	✓	✗	✗
Maniskill (R $\rightarrow$ M)	Simulation	0	600	100	✓	✓	✗
Sparrow	Real world	3000	600	300	✗	✗	✓
ARMBench (S $\rightarrow$ A)	Real world	0	600	200	✗	✓	✓

Appendix

Table 6: Details of datasets. Dataset domains, number of sparse and dense samples used for training, and types of distribution shifts between train and test data.

Sparse / Dense Ratio	2	(5)	10	30
Detection / Reasoning	0.813 / 0.831	(0.725 / 0.698)	0.640 / 0.609	0.620 / 0.427

Appendix

Table 7: Performance across varying sparse-dense data ratios. The column in parentheses represents the setting used in the main paper.

Model	Detection / Reasoning
Qwen2.5-VL 7B	0.453 / 0.240
Qwen2.5-VL 32B	0.563 / 0.268
ARMOR 7B (ours)	0.733 / 0.503
ARMOR 32B (ours)	0.765 / 0.562

Appendix

Table 8: Performance comparison across model sizes on Sparrow dataset.

Figure 5: Hockey-stick task from RLBench Multi-view frames (front, wrist, overhead) across timesteps showing the robot attempting to pick up a hockey stick.

Figure 6: ARMBench transport task. The ground truth failure is item damage caused by detached book cover.

Robot Failure Detection

Robot failure detection identifies when autonomous systems are failing, predicts emerging faults, and supports diagnosis or recovery.

See all 47 papers

Guardian: Detecting Robotic Planning and Execution Errors with Vision-Language Models
Dec 1, 2025Paul Pacaud, Ricardo Garcia, Shizhe Chen +1Failure Detection for VLA Models Robot Manipulation Datasets
Inria, École normale supérieure, CNRS, PSL Research University
RAFAIL: Relationship-Aware Failure Detection for Robotic Manipulation
Sep 16, 2026Loris Schneider, Edgar Welte, Rania RayyesRobot Failure Detection Fault Detection
Institute for Material Handling and Logistics (IFL), Karlsruhe Institute of Technology, 76131, Germany.
FailBench: How Reliable are VLMs at Judging Robot Task Success?
Sep 3, 2026Zaruhi Navasardyan, Tatul Danielyan, Hrant DavtyanVLM Evaluation Robot Failure Detection
Metric AI Lab

Dec 1, 2025J/K move · Enter open · S save

Guardian: Detecting Robotic Planning and Execution Errors with Vision-Language Models

Paul Pacaud, Ricardo Garcia, Shizhe Chen, Cordelia Schmid

Inria, École normale supérieure, CNRS, PSL Research University

Failure Detection for VLA Models Robot Manipulation Datasets Synthetic Data Generation for Robotics Generalization in Robotic Manipulation Reasoning for VLA Models

Open paper AskPDF arXiv

Robust robotic manipulation requires reliable failure detection and recovery. Although recent Vision-Language Models (VLMs) show promise in robot failure detection, their generalization is severely limited by the scarcity and narrow coverage of failure data. To address this bottleneck, we propose an automatic framework for generating diverse robotic planning and execution failures across both simulated and real-world environments. Our approach perturbs successful manipulation trajectories to synthesize failures that reflect realistic failure distributions, and leverages VLMs to produce structured step-by-step reasoning traces. This yields GuardianFail-36k, a large-scale failure reasoning dataset built upon the RLBench simulator and the BridgeDataV2 real-robot dataset. Using GuardianFail-36k, we train Guardian, a multi-view reasoning VLM for unified planning and execution verification. Guardian achieves state-of-the-art performance on three unseen real-world benchmarks: RoboFail, RoboVQA, and our newly introduced UR5-Fail. When integrated with a state-of-the-art LLM-based manipulation policy, it consistently boosts task success rates in both simulation and real-world deployment. These results demonstrate that scaling high-quality failure reasoning data is critical for improving generalization in robotic failure detection. Code, Data, and Models are available at https://www.di.ens.fr/willow/research/guardian/.

Abstract

Robot Failure Detection

Explore similar work

Guardian: Detecting Robotic Planning and Execution Errors with Vision-Language Models

RAFAIL: Relationship-Aware Failure Detection for Robotic Manipulation

FailBench: How Reliable are VLMs at Judging Robot Task Success?