Guardian: Detecting Robotic Planning and Execution Errors with Vision-Language Models

Paul Pacaud; Ricardo Garcia; Shizhe Chen; Cordelia Schmid

cs.RODec 1, 2025

Guardian: Detecting Robotic Planning and Execution Errors with Vision-Language Models

Authors: Paul Pacaud, Ricardo Garcia, Shizhe Chen, Cordelia Schmid

Organizations: Inria, École normale supérieure, CNRS, PSL Research University

Abstract

Robust robotic manipulation requires reliable failure detection and recovery. Although recent Vision-Language Models (VLMs) show promise in robot failure detection, their generalization is severely limited by the scarcity and narrow coverage of failure data. To address this bottleneck, we propose an automatic framework for generating diverse robotic planning and execution failures across both simulated and real-world environments. Our approach perturbs successful manipulation trajectories to synthesize failures that reflect realistic failure distributions, and leverages VLMs to produce structured step-by-step reasoning traces. This yields GuardianFail-36k, a large-scale failure reasoning dataset built upon the RLBench simulator and the BridgeDataV2 real-robot dataset. Using GuardianFail-36k, we train Guardian, a multi-view reasoning VLM for unified planning and execution verification. Guardian achieves state-of-the-art performance on three unseen real-world benchmarks: RoboFail, RoboVQA, and our newly introduced UR5-Fail. When integrated with a state-of-the-art LLM-based manipulation policy, it consistently boosts task success rates in both simulation and real-world deployment. These results demonstrate that scaling high-quality failure reasoning data is critical for improving generalization in robotic failure detection. Code, Data, and Models are available at https://www.di.ens.fr/willow/research/guardian/.

Figures & tables

Figure 1: Illustration of our Guardian model - a VLM fine-tuned on our constructed failure datasets. It detects planning failures (top) and execution failures (bottom) in robotic manipulation.

Figure 2: Failure Data Generation Framework. We construct failure cases both online in simulation (RLBench, left), and offline on the real-world dataset (BridgeDataV2, right). For each positive example, given its correct plan and successful trajectory, we generate a corresponding incorrect plan and unsuccessful trajectory.

Figure 3: Planning and execution examples with CoT.

		Training		Validation		Test
Dataset	Env.	Exec	Plan	Exec	Plan	Exec	Plan
GuardianFail (Ours)
RLBench-Fail	Sim	12358	5808	1000	500	1000	500
BridgeDataV2-Fail	Real	7830	4880	1000	500	1000	500
Real-World Robot Failure Detection Benchmarks
UR5-Fail (Ours)	Real	-	-	-	-	140	140

Table 1: Statistics of the GuardianFail-36k dataset and three real-world robot failure detection benchmarks. Our constructed datasets (GuardianFail and UR5-Fail) contain balanced success and failure cases covering both execution and planning errors.

Figure 4: Failure mode distributions in real executions and our constructed data.

Figure 5: Left: Overview of the Guardian model architecture. Right: Integration of Guardian model into a robot manipulation pipeline for planning and execution verification. The failure reasoning trace is reinjected to help with replanning.

Model	Trained on	RoboFail [ 33 ]		UR5-Fail		RVQA [ 42 ]
	GuardianFail	Exec	Plan	Exec	Plan	Exec
Closed-Source VLM
GPT-4o	✗	0.80	0.67	0.77	0.85	0.79
GPT-4o +Sentinel-Video-QA [ 1 ]	✗	0.80	0.63	0.76	0.62	0.66
Robotic Failure Detection VLMs
RoboFAC-7B [ 34 ]	✗	0.25	0.05	0.54	0.02	0.52

Table 2: Comparison of failure detection models on unseen real-world benchmarks. Execution and planning accuracies are reported. ∗ denotes numbers from the original paper.

Figure 6: GuardianFail training data size impact on the binary accuracy averaged over planning and execution.

	Output		Avg. Accuracy		Inf.
Model	Train	Test	Exec	Plan	Time
Vanilla	A	A	0.80	0.78	0.68s
Dropout	CoT , A $\vee$ A	A	0.81	0.83	0.68s
Thinking	CoT , A	CoT , A	0.83	0.84	4.3s

Table 4: Comparison of training and test-time strategies with and without CoT ( Sec. 4.3 ). “A” denotes the final answer. We report binary accuracy and inference time (seconds/sample) on one H100, averaged across all test sets.

Views	Image Format	Exec	Plan
Single	concat	0.69	0.82
Single	separated	0.72	0.82
Multi	concat	0.74	0.82
Multi	separated	0.83	0.84

Table 5: Impact of image representation. We report accuracy averaged across all datasets, for both Execution and Planning. AHA [ 14 ] and I-Fail-Sense [ 20 ] use multi-view concatenation.

Figure 7: Qualitative examples of Guardian on UR5-Fail. The left and middle examples show successful plan and execution verification, while the right example shows a failure case.

Verifier	Trained on GuardianFail	CoT	Put food		Arrange fruits		Stack cups
Verifier	Trained on GuardianFail	CoT	Norm	Pert	Norm	Pert	Norm	Pert
✗	✗	✗	15/20	4/20	10/20	3/20	9/20	2/20
✓	✗	✗	12/20	10/20	9/20	9/20	7/20	6/20
✓	✓	✗	17/20	13/20	14/20	12/20	10/20	8/20
✓	✓	✓	18/20	15/20	14/20	12/20	12/20	11/20

Table 7: Success rate of 3D-LOTUS++ on real robot experiments without and with different VLM-based failure detectors. The last row denotes our full Guardian model.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Execution Verification. Qualitative examples of Guardian on RoboFail [ 33 ] (left), RoboVQA [ 42 ] (middle), and BridgeDataV2-Fail (right). The left and middle are correct detection examples, the right is a failure case of the model.

Figure 10: Planning Verification. Qualitative examples of Guardian on RoboFail [ 33 ] (left and middle), BridgeDataV2-Fail (right). The left is a correct detection example, the middle and right are failure cases of the model.

Figure 11: Confusion matrices for Guardian’s failure detection on execution (top) and planning (bottom). For each scenario, the left matrix reports binary success/failure prediction accuracy, and the right matrix breaks this accuracy down per ground-truth failure mode. Values are percentages; rows denote the ground-truth success or failure mode, and columns the binary class predicted.

	Faithfulness	Error Attribution
Benchmark	( $\%$ )	Perception ( $\%$ )	Reasoning ( $\%$ )
UR5-Fail	100	75	25
RoboFail [ 33 ]	100	68	32
RoboVQA [ 42 ]	100	93	7

Appendix

Table 8: Study of Guardian’s reasoning traces. Faithfulness is the rate at which the final answer matches the conclusion of the thinking step. Perception and Reasoning report the share of misclassifications whose first failing stage is an inaccurate image description or an incorrect analysis of an otherwise-correct description.

Figure 12: Our real-robot setup includes a UR5 robotic arm equipped with a RG6 gripper, with three RealSense D435 cameras.

Training
Base model	InternVL3-8B (InternViT-300M-448px-V2.5 + Qwen2.5-7B)
Loss	Next-token prediction
Optimizer	AdamW
Learning rate	$4\times 10^{-5}$
LR schedule	Cosine, warmup ratio 0.03
Weight decay	0.05

Appendix

Table 9: Training and inference hyperparameters.

Failure Detection for VLA Models

See all 16 papers

FailSafe: Reasoning and Recovery from Failures in Vision-Language-Action Models
Oct 2, 2025Zijun Lin, Jiafei Duan, Haoquan Fang +4Robot Failure Recovery Synthetic Data Generation for Robotics
1Nanyang Technological University · Centre for Frontier AI Research, A*STAR · 3Allen Institute for AI +1
FailBench: How Reliable are VLMs at Judging Robot Task Success?
Sep 3, 2026Zaruhi Navasardyan, Tatul Danielyan, Hrant DavtyanVLM Evaluation Robot Failure Detection
Metric AI Lab
RoboFAC: A Comprehensive Framework for Robotic Failure Analysis and Correction
May 18, 2025Zewei Ye, Weifeng Lu, Minghao Ye +4Robot Failure Recovery Robustness of VLA Models
School of AI, Shanghai Jiao Tong University · Harbin Institute of Technology, Shenzhen

Oct 2, 2025J/K move · Enter open · S save

FailSafe: Reasoning and Recovery from Failures in Vision-Language-Action Models

Zijun Lin, Jiafei Duan, Haoquan Fang, Dieter Fox+3

1Nanyang Technological University · Centre for Frontier AI Research, A*STAR · 3Allen Institute for AI+1

Robot Failure Recovery Synthetic Data Generation for Robotics Robotics Vision-Language-Action Models Robustness of VLA Models

Open paper AskPDF arXiv

Recent advances in robotic manipulation have integrated low-level robotic control into Vision-Language Models (VLMs), extending them into Vision-Language-Action (VLA) models. Although state-of-the-art VLAs achieve strong performance in downstream robotic applications, supported by large-scale crowd-sourced robot training data, they still inevitably encounter failures during execution. Enabling robots to reason and recover from unpredictable and abrupt failures remains a critical challenge. Existing robotic manipulation datasets, collected in either simulation or the real world, primarily provide only ground-truth trajectories, leaving robots unable to recover once failures occur. Moreover, the few datasets that address failure detection typically offer only textual explanations, which are difficult to utilize directly in VLA models. To address this gap, we introduce FailSafe, a novel failure generation and recovery system that automatically produces diverse failure cases paired with executable recovery actions. FailSafe can be easily adapted to a wide range of manipulation tasks in simulators with motion planning support, enabling scalable creation of failure-action data. To demonstrate its effectiveness, we fine-tune LLaVA-OneVision-7B (LLaVA-OV-7B) to build FailSafe-VLM. Experimental results show that FailSafe-VLM successfully helps robotic arms detect and recover from potential failures, improving the performance of three state-of-the-art VLA models (Pi-0-FAST, OpenVLA, OpenVLA-OFT) by up to 22.6% on average across several tasks in ManiSkill. Furthermore, FailSafe-VLM could generalize across different spatial configurations, camera viewpoints, object and robotic embodiments.

Abstract

Failure Detection for VLA Models

Explore similar work

FailSafe: Reasoning and Recovery from Failures in Vision-Language-Action Models

FailBench: How Reliable are VLMs at Judging Robot Task Success?

RoboFAC: A Comprehensive Framework for Robotic Failure Analysis and Correction