FailureGen: Scalable Robot Failure Data Generation with World Models
Organizations: NVIDIA
Abstract
Robot failures are difficult to collect at scale, limiting the range of failure scenarios represented in robot learning datasets. We present FailureGen, a method for generating synthetic videos with temporal event annotations by reusing action trajectories from recorded failures. The method comprises four stages: (1) fine-tuning an action-conditioned Cosmos world model on successful and failed real executions; (2) annotating source recordings to identify failure-associated donor segments and compatible successful hosts; (3) transferring donor motion into host contexts, with optional spatial offsets, to generate continuous video rollouts; and (4) annotating and filtering the generated samples according to observed events and visual quality. We study two implementations with different tasks and robot configurations, with visual failure detection serving as one downstream application. In a preliminary comparison, a detector trained on real and generated data achieves an average precision of 0.51, compared with 0.41 using real data alone, and detects 33 of 36 failure intervals, compared with 28 of 36. By reusing observed failures in additional contexts, FailureGen expands the data available for studying robot failures and developing downstream learning methods.
Figures & tables
| Training condition | Events | Avg Precision | Clean alarms |
|---|---|---|---|
| No fine-tuning | 25/36 | 0.0412 | 41/41 |
| Real-only | 28/36 | 0.4085 | 5/41 |
| Real + FailureGen | 33/36 | 0.5119 | 0/41 |
| Qwen3-VL-2B | Qwen3-VL-8B | |||
|---|---|---|---|---|
| Task | Accuracy | Failure recall | Accuracy | Failure recall |
| Object into black box | ||||
| Vial/test-tube rack | ||||
| Drawer/can | ||||
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Quantity | Real 1:1 | Real 5:1 | Aug. |
| Real positive targets | 105 | 105 | 105 |
| Synthetic positive targets | 0 | 0 | 128 |
| Real negative targets | 105 | 525 | 233 |
| Configured updates | 135 | 400 | 295 |
| Events detected | 21/36 | 28/36 | 33/36 |
| Average precision | 0.3090 | 0.4085 | 0.5119 |
Synthetic Data Generation for Robotics
Synthetic data generation for robotics creates simulated scenes, sensor readings, and action trajectories to train and evaluate robots.
See all 327 papers