Learning Actionable Manipulation Recovery via Counterfactual Failure Synthesis

Dayou Li; Jiuzhou Lei; Hao Wang; Lulin Liu; Yunhao Yang; Zihan Wang; Bangya Liu; Minghui Zheng

cs.ROMar 13, 2026

Learning Actionable Manipulation Recovery via Counterfactual Failure Synthesis

Authors: Dayou Li, Jiuzhou Lei, Hao Wang, Lulin Liu, Yunhao Yang, Zihan Wang, Bangya Liu, Minghui Zheng,

Organizations: Texas A&M University · University of Minnesota · University of Texas at Austin · Abaka AI ·

Abstract

While recent foundation models have significantly advanced robotic manipulation, these systems still struggle to autonomously recover from execution errors. Current failure-learning paradigms rely on either costly and unsafe real-world data collection or simulator-based perturbations, which introduce a severe sim-to-real gap. Furthermore, existing visual analyzers predominantly output coarse, binary diagnoses rather than the executable, trajectory-level corrections required for actual recovery. To bridge the gap between failure diagnosis and actionable recovery, we introduce Dream2Fix, a framework that synthesizes photorealistic, counterfactual failure rollouts directly from successful real-world demonstrations. By perturbing actions within a generative world model, Dream2Fix creates paired failure-correction data without relying on simulators. To ensure the generated data is physically viable for robot learning, we implement a structured verification mechanism that strictly filters rollouts for task validity, visual coherence, and kinematic safety. This engine produces a high-fidelity dataset of over 120k paired samples. Using this dataset, we fine-tune a vision-language model to jointly predict failure types and precise recovery trajectories, mapping visual anomalies directly to corrective actions. Extensive real-world robotic experiments show our approach achieves state-of-the-art correction accuracy, improving from 19.7% to 81.3% over prior baselines, and successfully enables zero-shot closed-loop failure recovery in physical deployments.

Figures & tables

Fig. 1: Dream2Fix is a data generation pipeline that synthesizes large-scale, photorealistic failure rollouts with paired corrections from successful demonstrations and curates them with physical and visual verification.

Fig. 2: Overview of Dream2Fix pipeline . Dream2Fix generates diverse failure cases from successful demonstrations via keyframe-level action perturbations, then validates and curates them with physical and visual verifiers. The verified rollouts are auto-labeled into a structured schema to instruction-tune a VLM that predicts actionable corrections for real-world recovery.

Source	Dream2Fix	AHA [ 9 ]	REFLECT [ 12 ]

Quantity	120K	49K	130
Input	Instruction: Put the lid on the pot. Provide RESULT and (if needed) a fix suggestion.	For the given sub-tasks, first determine if it has succeeded by choosing from [“yes”, “no”] and then explain the reason why the current sub-tasks have failed.	RGB-D video frames, audio, robot proprioception
Label	RESULT=FAIL; TYPE=translation; STAGE=pre_grasp; FIX_DIR_X=-x; FIX_N_X=2; FIX_DIR_Y=+y; FIX_N_Y=3; The execution failed due to a translation misalignment before grasping. To fix it, nudge the end-effector in -x for 2 steps and in +y for 3 steps in the keyframe.	No, the robot gripper rotated with an incorrect roll angle.	At 03:00, the robot failed to properly place the pot filled with water on the stove burner. The failure was caused by the robot dropping something on the ground at 02:43, which might have been the pot or its contents. Failure timestep: 02:43

TABLE I: Comparison to existing datasets. We compare Dream2Fix with AHA [ 9 ] and REFLECT [ 12 ] in terms of dataset scale, input modalities, and label style. Dream2Fix pairs synthesized failure clips with structured labels that specify failure type and stage, together with executable corrections, enabling vision-language models to produce grounded failure diagnoses and executable corrections.

Fig. 3: Real-world experimental workspace. A Franka Research 3 arm with a Franka Hand gripper, and a fixed-view Intel RealSense D435i RGB-D camera are equipped for real-world evaluation.

Metric	GT	Generated
$S_{\text{smooth}}$	0.814	0.823
$S_{\text{vis}}$	0.999	0.998
$S_{\text{topo}}$	0.977	0.976
$S_{\text{global}}$	0.689	0.692
$\mathrm{MAE}_{xyz}(\widehat{\Delta s},\Delta s)$	0.008	0.015
$\mathrm{MAE}_{rpy}(\widehat{\Delta s},\Delta s)$	0.034	0.064

TABLE II: Generated data quality results. We compare verifier metrics for ground-truth demonstrations (GT) and generated failure cases, where closer values indicate more similar distributions and higher-quality generated data.

Fig. 4: Real world robot execution. For each task, we show the initial failed execution (left) and the corrected execution (right). Across diverse tasks, the correction adjusts the action to recover from the failure under the same real-robot setup.

Models	Dream2Fix Dataset (Test Set)					Real-World Benchmark (Zero-Shot)
Models	ROUGE ${}_{L}\uparrow$	Cos. Sim. $\uparrow$	BinSucc(%) $\uparrow$	Fuzzy Match $\uparrow$	Acc. $\uparrow$	ROUGE ${}_{L}\uparrow$	Cos. Sim. $\uparrow$	BinSucc(%) $\uparrow$	Fuzzy Match $\uparrow$	Acc. $\uparrow$
LLaVA-NeXT-7B [ 27 ]	0.080	0.099	0.111	0.322	0.022	0.000	0.000	0.000	0.354	0.000
LLaVA-NeXT-34B [ 27 ]	0.130	0.127	0.153	0.453	0.044	0.090	0.090	0.300	0.128	0.022
Qwen2.5-VL-7B [ 28 ]	0.170	0.391	0.551	0.241	0.113	0.052	0.176	0.250	0.105	0.008
Qwen3-VL-8B [ 29 ]	0.248	0.386	0.567	0.248	0.143	0.061	0.478	0.930	0.167	0.183
Gemini-3-Flash [ 30 ]	0.406	0.518	0.679	0.496	0.179	0.467	0.589	0.980	0.250	0.374

TABLE III: Results on the Dream2Fix test set and real-world benchmarks. We compare baselines with Dream2Fix-VLM (ours) under the same evaluation protocol on two settings: (i) Dream2Fix (Test Set) and (ii) real-world benchmark (Zero-Shot). Metrics include text similarity (ROUGE-L and cosine similarity), match-based success measures (BinSucc and fuzzy match), and correction accuracy (Acc.).

\uparrow

indicates higher value is better.

Failure Source	Recovery Rate (%) $\uparrow$
Failure Source	Dream2Fix-VLM	Gemini3-Flash
Real-World Benchmark	46%	36%
OpenVLA Deployment	40%	30%

TABLE IV: Real-world results. We report recovery rate (

\uparrow

is better) for two methods, Dream2Fix-VLM and Gemini3-Flash, evaluated on failures from our Real-World Benchmark and OpenVLA deployment.

Robot Failure Recovery

Robot failure recovery studies how robots detect, diagnose, and respond to errors, damage, and unexpected physical interactions.

See all 72 papers

RoboFAC: A Comprehensive Framework for Robotic Failure Analysis and Correction
May 18, 2025Zewei Ye, Weifeng Lu, Minghao Ye +4Robot Failure Recovery Robustness of VLA Models
School of AI, Shanghai Jiao Tong University · Harbin Institute of Technology, Shenzhen
F4R: Failure-Driven Recognition, Reconstruction, Refinement, and Redeployment for Continual Robot Self-Improvement
Sep 28, 2026Zhuoyuan Yu, Jiacheng Wang, Tianle Liu +13Continual Robot Learning Sim-to-Real Transfer
Nanyang Technological University · Xi’an Jiaotong University · Dexmal
FailSafe: Reasoning and Recovery from Failures in Vision-Language-Action Models
Oct 2, 2025Zijun Lin, Jiafei Duan, Haoquan Fang +4Robot Failure Recovery Synthetic Data Generation for Robotics
1Nanyang Technological University · Centre for Frontier AI Research, A*STAR · 3Allen Institute for AI +1

May 18, 2025J/K move · Enter open · S save

RoboFAC: A Comprehensive Framework for Robotic Failure Analysis and Correction

Zewei Ye, Weifeng Lu, Minghao Ye, Tao Lin+3

School of AI, Shanghai Jiao Tong University · Harbin Institute of Technology, Shenzhen

Robot Failure Recovery Robustness of VLA Models Vision-Language-Action Models Fault Diagnosis

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models have recently advanced robotic manipulation by translating natural-language instructions and visual observations into control actions. However, existing VLAs are primarily trained on successful expert demonstrations and lack structured supervision for failure diagnosis and recovery, limiting robustness in open-world scenarios. To address this limitation, we propose the Robotic Failure Analysis and Correction (RoboFAC) framework. We construct a large-scale failure-centric dataset comprising 9,440 erroneous manipulation trajectories and 78,623 QA pairs across 53 scenes in both simulation and real-world environments, with systematically categorized failure types. Leveraging this dataset, we develop a lightweight multimodal model specialized for task understanding, failure analysis, and failure correction, enabling efficient local deployment while remaining competitive with large proprietary models. Experimental results demonstrate that RoboFAC achieves a 34.1% higher failure analysis accuracy compared to GPT-4o. Furthermore, we integrated RoboFAC as an external supervisor in a real-world VLA control pipeline, yielding a 29.1% relative improvement across four tasks while significantly reducing latency relative to GPT-4o. These results demonstrate that RoboFAC enables systematic failure diagnosis and recovery, significantly enhancing VLA recovery capabilities. Our model and dataset are publicly available at https://github.com/MINT-SJTU/RoboFAC.

Abstract

Robot Failure Recovery

Explore similar work

RoboFAC: A Comprehensive Framework for Robotic Failure Analysis and Correction

F4R: Failure-Driven Recognition, Reconstruction, Refinement, and Redeployment for Continual Robot Self-Improvement

FailSafe: Reasoning and Recovery from Failures in Vision-Language-Action Models