Robot failures are difficult to collect at scale, limiting the range of failure scenarios represented in robot learning datasets. We present FailureGen, a method for generating synthetic videos with temporal event annotations by reusing action trajectories from recorded failures. The method comprises four stages: (1) fine-tuning an action-conditioned Cosmos world model on successful and failed real executions; (2) annotating source recordings to identify failure-associated donor segments and compatible successful hosts; (3) transferring donor motion into host contexts, with optional spatial offsets, to generate continuous video rollouts; and (4) annotating and filtering the generated samples according to observed events and visual quality. We study two implementations with different tasks and robot configurations, with visual failure detection serving as one downstream application. In a preliminary comparison, a detector trained on real and generated data achieves an average precision of 0.51, compared with 0.41 using real data alone, and detects 33 of 36 failure intervals, compared with 28 of 36. By reusing observed failures in additional contexts, FailureGen expands the data available for studying robot failures and developing downstream learning methods.
Figures & tables
Figure 1: FailureGen architecture. Successful and failed source executions support world-model adaptation through forward- and inverse-dynamics supervision. At generation time, a successful host supplies visual context and nominal motion, while a failure donor supplies motion for a selected intervention. Motion transfer and optional spatial offsets construct the conditioning sequence for the adapted world model. Generated videos are labeled according to their observed events, with a cleanup filter for unreliable examples. Each retained example includes the video, its associated motion sequence, and temporal event labels.
Figure 2: Failure transfer in industrial manipulation. Top: a real donor execution in which a reel falls outside the bin. Middle: the real host recording, which supplies the initial scene and object appearance. Bottom: the FailureGen continuation, which preserves the host context while reproducing the donor’s drop behavior with different timing and object motion. Columns show elapsed clip time; red outlines highlight the falling or dropped object.
Training condition
Events
Avg Precision
Clean alarms
No fine-tuning
25/36
0.0412
41/41
Real-only
28/36
0.4085
5/41
Real + FailureGen
33/36
0.5119
0/41
Table 1: Industrial failure recognition on the shared real evaluation subset. Events are detected failure intervals out of 36; clean alarms are clean episodes containing an alert out of 41. Real-only reports the stronger of the evaluated real-only configurations. The full configuration comparison is provided in the appendix.
Figure 3: Failure transfer for object placement. Top: the donor releases a can outside the box. Middle: the host carries a bottle toward the box. Bottom: the generated continuation preserves the host’s bottle and scene while depicting premature release onto the table. Times are relative to each clip; red outlines highlight the released object.
Qwen3-VL-2B
Qwen3-VL-8B
Task
Accuracy
Failure recall
Accuracy
Failure recall
Object into black box
94.1→100.0
93.3→100.0
98.5→100.0
98.3→100.0
Vial/test-tube rack
58.5→58.5
53.3→53.3
49.6→54.1
43.3→48.3
Drawer/can
84.4→97.0
82.5→96.7
94.8→97.0
94.2→96.7
Table 2: Failure detection on the three-task ViFailback subset. Values show real-only → FailureGen-augmented performance (%). Bold values indicate improvements.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Quantity
Real 1:1
Real 5:1
Aug.
Real positive targets
105
105
105
Synthetic positive targets
0
0
128
Real negative targets
105
525
233
Configured updates
135
400
295
Events detected
21/36
28/36
33/36
Average precision
0.3090
0.4085
0.5119
Appendix
Table 3: Industrial fine-tuning configurations. Update counts describe the configured five-epoch runs. Real 1:1 and Real 5:1 denote negative-to-positive sampling ratios; Aug. denotes real plus FailureGen data.
Figure 4: Failure transfer for drawer placement. Rows show the real donor, real host, and generated continuation. The generated can is released outside the drawer while the host scene is retained. Wrist-camera insets show the release obscured in the main view. Times are relative to each clip; red outlines highlight the released can.
Robust robotic manipulation requires reliable failure detection and recovery. Although recent Vision-Language Models (VLMs) show promise in robot failure detection, their generalization is severely limited by the scarcity and narrow coverage of failure data. To address this bottleneck, we propose an automatic framework for generating diverse robotic planning and execution failures across both simulated and real-world environments. Our approach perturbs successful manipulation trajectories to synthesize failures that reflect realistic failure distributions, and leverages VLMs to produce structured step-by-step reasoning traces. This yields GuardianFail-36k, a large-scale failure reasoning dataset built upon the RLBench simulator and the BridgeDataV2 real-robot dataset. Using GuardianFail-36k, we train Guardian, a multi-view reasoning VLM for unified planning and execution verification. Guardian achieves state-of-the-art performance on three unseen real-world benchmarks: RoboFail, RoboVQA, and our newly introduced UR5-Fail. When integrated with a state-of-the-art LLM-based manipulation policy, it consistently boosts task success rates in both simulation and real-world deployment. These results demonstrate that scaling high-quality failure reasoning data is critical for improving generalization in robotic failure detection. Code, Data, and Models are available at https://www.di.ens.fr/willow/research/guardian/.
Paul Pacaud, Ricardo Garcia, Shizhe Chen +1
Inria, École normale supérieure, CNRS, PSL Research University
Synthetic data generated by video generative models has shown promise for robot learning as a scalable pipeline, but it often suffers from noisy action labels due to imperfect video generation or inaccurate action prediction. Recently, vision-language models (VLMs) have been leveraged to validate video quality, but they cannot directly evaluate the generated actions themselves. To tackle this issue, we introduce RoboCurate, a novel framework for action-verified synthetic robot data that addresses inconsistencies between generated videos and annotated actions. Specifically, RoboCurate executes the actions predicted by an inverse dynamics model (IDM) in a simulator and verifies action consistency by measuring the motion agreement between the simulator rollout and the generated video. To better leverage such a framework, we further expand the scene diversity of synthetic data beyond the available dataset through off-the-shelf generative models with structured editing axes. We then use a VLM to pre-filter physically implausible generated videos before verifying their action labels. Across diverse manipulation scenarios, RoboCurate yields substantial relative improvements in success rates compared to using only real data, achieving +90.3% on GR-1 Tabletop (300 demos), +29.1% on DexMimicGen in the pre-training setup on simulation benchmarks, and +179.9% and +42.9% in the co-finetuning setup on challenging real-world ALLEX and OpenArm humanoid benchmarks, respectively.
Action-conditioned world models offer a promising foundation for generalist robotic intelligence by predicting the consequences of actions. However, existing robotic datasets provide limited coverage of the diverse interaction dynamics required for generalizable world modeling. Large-scale egocentric human videos offer complementary interaction experience, yet the embodiment gap between human hands and robotic manipulators hinders effective cross-embodiment learning. Meanwhile, the predominance of successful demonstrations leaves failure dynamics substantially underrepresented, limiting the ability of world models to anticipate unsuccessful executions. To address these limitations, we introduce HF-WM, an action-conditioned world model that combines Human interaction learning with Failure-aware post-training. To bridge the embodiment gap, we develop a dual-branch framework that couples video generation with auxiliary pointflow prediction, using stabilized pointflow as a unified motion representation across human and robot interactions while suppressing egocentric camera motion. To improve failure modeling, we further perform Group Relative Policy Optimization (GRPO) post-training on perturbed actions derived from successful demonstrations, guided by action alignment and physical plausibility rewards, enabling the model to learn plausible failure dynamics without paired ground-truth videos. Extensive experiments demonstrate that HF-WM achieves state-of-the-art performance in action-conditioned video generation and exhibits strong utility for policy evaluation in downstream robotic tasks.
Peiyan Tu, Cong Wang, Hanxin Zhu +2
Zhejiang University · Beijing Zhongguancun Academy · Institute of Automation, Chinese Academy of Sciences +2