Reasoning about failures is crucial for building reliable and trustworthy robotic systems. Prior approaches either treat failure reasoning as a closed-set classification problem or assume access to ample human annotations. Failures in the real world are typically subtle, combinatorial, and difficult to enumerate, whereas rich reasoning labels are expensive to acquire. We address this problem by introducing ARMOR: Adaptive Round-based Multi-task mOdel for Robotic failure detection and reasoning. We formulate detection and reasoning as a multi-task self-refinement process, where the model iteratively predicts detection outcomes and natural language reasoning conditioned on past outputs. During training, ARMOR learns from heterogeneous supervision - large-scale sparse binary labels and small-scale rich reasoning annotations - optimized via a combination of offline and online imitation learning. At inference time, ARMOR generates multiple refinement trajectories and selects the most confident prediction via a self-certainty metric. Experiments across diverse environments show that ARMOR achieves state-of-the-art performance by improving over the previous approaches by up to 30% on failure detection rate and up to 100% in reasoning measured through LLM fuzzy match score, demonstrating robustness to heterogeneous supervision and open-ended reasoning beyond predefined failure modes. We provide additional visualizations on our website: https://sites.google.com/utexas.edu/armor
Figures & tables
Figure 1: We present ARMOR: adaptive round-based multi-task model for robotic failure detection and reasoning. Prior approaches reduce failure reasoning to closed-set classification of pre-defined modes (e.g., “spillage” or “unstable grasp”). In contrast, ARMOR performs open-ended, iterative refinement by jointly refining its detection and reasoning predictions. This enables accurate detection with nuanced and human-like reasoning that capture real-world failures beyond fixed categories.
Figure 2: Overview of ARMOR. (a) Our failure data consist of heterogeneous supervision, with large-scale binary detection labels and scarce free-form reasoning labels. (b) A vision-language model (VLM) with multi-task heads jointly predicts detection l via a classification head and reasoning e via a language decoder, trained with binary cross-entropy (BCE) and next-token prediction (NTP) losses. (c) We fine-tune the VLM with both offline imitation and online refinement: the model conditions on dataset labels ( l,e ) or its prior predictions ( l^,e^ ) as well as auxiliary prompts p to generate a new round of outputs. In online refinement, this process is repeated T times. The model’s predictions (denoted by l^ and e^ in purple rectangles) are supervised with available labels. (d) At inference, ARMOR performs iterative self-refinement, rolling out multiple stochastic trajectories. We select best prediction m⋆ from the final predictions with the lowest entropy score.
Models
Evaluation Datasets
Detect Acc. ↑
LLM Fuzzy ↑
ROUGE L ↑
Qwen2.5-VL
RLBench
0.376
0.255
0.353
Sparrow
0.453
0.240
0.137
Maniskill
0.548
0.268
0.112
ARMBench
0.500
0.349
0.208
Cosmos-Reasoning
RLBench
0.317
0.220
0.140
Sparrow
0.510
0.346
0.311
Table 1: Quantitative Results on Failure Detection and Reasoning. Metrics include detection accuracy and reasoning quality (LLM Fuzzy and ROUGE L ). Our method achieves higher performance across different domains, accurately detecting failures and producing high quality reasoning.
Ablation
Offline
Offline Expert
Online
Refinement
Detection / Reasoning
Warmup
Condition
Imitation
(Inference)
Multitask Prediction
✓
✗
✗
✗
0.897 / 0.460
Refinement Only
✓
✗
✗
✓
0.803 / 0.488
Offline Imitation Only
✓
✓
✗
✓
0.853 / 0.658
Online Imitation Only
✗
✗
✓
✓
0.850 / 0.683
ARMOR (ours)
✓
✓
✓
✓
0.917 / 0.718
Table 2: Ablation of ARMOR’s components. Each row shows the impact of different stages on detection accuracy and reasoning quality. ARMOR combines all components to achieve the best overall performance.
Figure 3: Refinement analysis in ARMOR. (a) Effect of refinement rounds on entropy and performance: detection and reasoning entropy decrease steadily across rounds, while combined plots show that refinement improves both detection accuracy and reasoning quality (values computed over 300 test datapoints). (b) Example of ARMOR’s multi-round refinement process, where predictions are iteratively updated to improve the quality and consistency between detection and reasoning outputs.
Refinement Rounds
Reasoning (LLM Fuzzy)
Round 0
0.475 ± 0.016
Round 1
0.676 ± 0.025 (+42.4%)
Round 2
0.703 ± 0.013 (+4.0%)
Round 3
0.717 ± 0.002 (+2.0%)
Table 3: ARMOR’s reasoning performance across refinement rounds. Mean and std are reported across 4 seeds. Each round shows improvements with decreasing variance, demonstrating consistent gains.
Refinement Rounds
Avg. Memory Usage per GPU (GB)
Wall Clock Time (s)
Round 0
3.48
7.95
Round 1
5.84
9.30
Round 2
6.12
10.50
Round 3
6.31
10.95
Table 4: Memory usage and wall clock time across refinement rounds. ARMOR adds minimal computational overhead, with approximately 1s increase per round and modest memory growth.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Global batch size (videos)
16
Global batch size (collaged images)
64
Offline epochs
3
Online epochs
10
Horizon T
3
Frames per second (FPS)
5.0
Appendix
Table 5: Hyperparameters for training and inference.
Figure 4: ARMOR model architecture. We select the intermediate layer representation from the LM decoder of Qwen2.5-VL ( Bai et al., 2025 ) and attach a classification head for detection, while using the original LM decoder for reasoning. Conditioning prompts describe the previous outputs for each task, enabling iterative multi-task refinement.
Dataset
Domain
Sparse
Dense
Test
Unseen task
Unseen workspace
Unseen failure
RLBench
Simulation
8000
1000
300
✓
✗
✗
Maniskill (R → M)
Simulation
0
600
100
✓
✓
✗
Sparrow
Real world
3000
600
300
✗
✗
✓
ARMBench (S → A)
Real world
0
600
200
✗
✓
✓
Appendix
Table 6: Details of datasets. Dataset domains, number of sparse and dense samples used for training, and types of distribution shifts between train and test data.
Sparse / Dense Ratio
2
(5)
10
30
Detection / Reasoning
0.813 / 0.831
(0.725 / 0.698)
0.640 / 0.609
0.620 / 0.427
Appendix
Table 7: Performance across varying sparse-dense data ratios. The column in parentheses represents the setting used in the main paper.
Model
Detection / Reasoning
Qwen2.5-VL 7B
0.453 / 0.240
Qwen2.5-VL 32B
0.563 / 0.268
ARMOR 7B (ours)
0.733 / 0.503
ARMOR 32B (ours)
0.765 / 0.562
Appendix
Table 8: Performance comparison across model sizes on Sparrow dataset.
Figure 5: Hockey-stick task from RLBench Multi-view frames (front, wrist, overhead) across timesteps showing the robot attempting to pick up a hockey stick.
Figure 6: ARMBench transport task. The ground truth failure is item damage caused by detached book cover.
Robust robotic manipulation requires reliable failure detection and recovery. Although recent Vision-Language Models (VLMs) show promise in robot failure detection, their generalization is severely limited by the scarcity and narrow coverage of failure data. To address this bottleneck, we propose an automatic framework for generating diverse robotic planning and execution failures across both simulated and real-world environments. Our approach perturbs successful manipulation trajectories to synthesize failures that reflect realistic failure distributions, and leverages VLMs to produce structured step-by-step reasoning traces. This yields GuardianFail-36k, a large-scale failure reasoning dataset built upon the RLBench simulator and the BridgeDataV2 real-robot dataset. Using GuardianFail-36k, we train Guardian, a multi-view reasoning VLM for unified planning and execution verification. Guardian achieves state-of-the-art performance on three unseen real-world benchmarks: RoboFail, RoboVQA, and our newly introduced UR5-Fail. When integrated with a state-of-the-art LLM-based manipulation policy, it consistently boosts task success rates in both simulation and real-world deployment. These results demonstrate that scaling high-quality failure reasoning data is critical for improving generalization in robotic failure detection. Code, Data, and Models are available at https://www.di.ens.fr/willow/research/guardian/.
Paul Pacaud, Ricardo Garcia, Shizhe Chen +1
Inria, École normale supérieure, CNRS, PSL Research University
Detecting failures during execution is essential for reliable robotic manipulation. Vision-language models (VLMs) can assess task outcomes semantically but add runtime computation, whereas out-of-distribution (OOD) detectors may respond to harmless scene variations rather than failure-relevant deviations. We introduce RAFAIL, a framework for detecting execution failures during robotic manipulation. RAFAIL identifies failures by detecting anomalies in task-relevant relationships between entities, such as a gripper and an object or an object and its target. By focusing OOD detection on relevant parts of the observation, RAFAIL reduces sensitivity to task-irrelevant scene variation. Offline, a VLM annotates successful demonstrations with task progress and relationship importance, which are used to learn point-cloud-based relationship representations without relying on policy-internal features. At runtime, relationship-specific OOD detectors evaluate these representations while relationship importance and task progress are predicted without VLM inference. RAFAIL requires no failure data and achieves 73.4% balanced accuracy across three real-world robotic manipulation tasks, outperforming the strongest evaluated OOD- and uncertainty-based baselines.
Loris Schneider, Edgar Welte, Rania Rayyes
Institute for Material Handling and Logistics (IFL), Karlsruhe Institute of Technology, 76131, Germany.
Vision-Language Models (VLMs) are increasingly used to evaluate robot manipulation outcomes, but existing benchmarks offer limited evidence of cross-domain generalization. We introduce FailBench, a benchmark for robot failure detection comprising 2,197 manipulation attempts across 14 public sources (12 real-world, 2 simulated). In FailBench, 75% of failures occur naturally, and six real-world sources come from non-failure-detection datasets. Evaluating 13 VLM-based detectors, we find the best model achieves only 0.77 mean balanced accuracy. Notably, models fine-tuned for failure detection consistently underperform general-purpose VLMs and their own pretrained baselines. Performance depends heavily on required visual evidence: models approach saturation when outcomes depend on observable object motion, but degrade to near-chance (<0.60 balanced accuracy) on contact-intensive assembly tasks. Error analysis reveals a systematic bias toward predicting success under ambiguous evidence, which persists even with increased reasoning effort. Finally, we show that input-level intervention--spatially localizing and cropping outcome-relevant regions--improves the top detector by 2.4 percentage points without extra training.