cs.ROOct 1, 2026

Recova: Agent-Guided Failure Recovery for Autonomous Robotic Manipulation

Authors: Isabella Liu, An-Chieh Cheng, Johan Bjorck, Zhiding Yu, Hongxu Yin, Jan Kautz, Linxi Fan, Yuke Zhu, +1 more

Organizations: University of California, San Diego · NVIDIA · University of Texas at Austin

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Manipulation failures can leave scenes in states from which a task policy cannot recover. Learning corrective behaviors requires scalable failure exploration and physical grounding. We present Recova, an agent-guided framework that jointly develops task execution and recovery in a reconstructed digital twin, then verifies and refines both through real-world experience. In the twin, the agent diagnoses failures, tests corrective programs, and collects successful task and recovery rollouts for separate policies. During deployment, it monitors progress, invokes a learned or programmatic recovery, verifies scene restoration, and resumes execution. When no suitable recovery is available, a human demonstration resolves the failure and enters the learning loop, allowing the system to expand its recovery capabilities. Physical rollouts and human demonstrations are routed to the corresponding policy for DAgger training. Across six LIBERO-Pro settings and four MolmoSpaces categories, Recova achieves 78.8% and 64.9% mean success, compared with 71.7% and 38.0% for the strongest baselines. With parallel collection across four real-robot workstations, DAgger fine-tuning raises mean success from 23.8% to 77.5%, and recovery skills further raise it to 87.5%. Over four collection rounds on one task, observed human intervention falls from 87.5% to 0%. Together, these results show how agent-guided recovery turns failures into reusable capabilities, improving robustness while progressively reducing human intervention. Project page: https://www.liuisabella.com/Recova

Figures & tables

Figure 2: (A) Simulation: Recova builds a digital twin from camera observations, calibration, and robot trajectories. Successful task and recovery rollouts train separate policies, while recovery programs populate a code-as-policy skill library. (B) Real-robot rollout: Recova monitors execution, invokes recovery policies or programs, verifies restoration, and requests human demonstrations when needed. Task successes, verified recoveries, and human demonstrations update the corresponding policies, while new failures guide skill-library expansion. Paired frames show rollout endpoints; the inset tracks human intervention across DAgger rounds.
Figure 3: Recovery skills in simulation. The failure (left) and recovery (right) moments of one LIBERO-Pro (top) or MolmoSpaces (bottom) episode: the main camera view, with the wrist camera inset at the top right, under the recovery instruction.
 MolmoSpaces ↑\uparrow
MethodsPickP&POpenCloseAvg.
Agentic code-as-policy methods
CaP-Agent0 ( 16 )23.011.014.036.021.0
RATs ( 67 )37.022.020.073.038.0
Vision-language-action policies
π0\pi_{0} ( 4 )14.07.010.047.019.5
Table 1: MolmoSpaces task success (%). P&P denotes pick-and-place. Avg. is the unweighted mean across the four simulation task categories. Bold marks the best reported value in each column. Baseline rows reproduce published results.
 Object ↑\uparrowGoal ↑\uparrowSpatial ↑\uparrow 
MethodsPos.TaskPos.TaskPos.TaskAvg. ↑\uparrow
Agentic code-as-policy methods
CaP-Agent0 ( 16 )27.031.029.016.013.023.023.2
RATs ( 67 )61.063.043.036.029.031.043.8
ASPIRE ( 41 )98.095.081.045.051.060.071.7
Vision-language-action policies
Table 2: LIBERO-Pro task success (%). Object, Goal, and Spatial each include initial-position swaps (Pos.) and task perturbations (Task). Avg. is the unweighted mean of all six columns. Bold marks the best reported value in each column. Baseline rows reproduce published results.
Figure 4: Parallel automated DAgger data collection. Left: Four robot stations operate in parallel with varied scene configurations. Middle: Station activity during one collection session, distinguishing task-policy execution, recovery-policy execution, human control, and waiting or reset periods. Right: Observed task success and human intervention rates across four collection rounds. As policies are fine-tuned on accumulated data, task success increases while human intervention decreases.
PolicyPencil boxStack ringsDraw tileDiscard tileAvg. ↑\uparrow
Base task policy20.035.025.015.023.8
+ DAgger70.075.085.080.077.5
+ Recovery skills85.090.090.085.087.5
Table 3: Real-robot task success (%) over 20 trials per task and configuration. Rows add DAgger fine-tuning of the task policy, then recovery skills at deployment. Bold marks the best result for each task.
Figure 5: Real-robot tasks and their recovery skills. For each task, the gray panel shows the start and result of a task-policy rollout. Each recovery skill appears under its name as two frames of one rollout, the action and then the result, developed in the digital twin (blue) or executed on the robot by a learned recovery policy (green). The list on the right names further recovery skills.
Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Monitor prompts , abridged: each query’s attached text and images, prompt, and required JSON reply. Symbols follow Alg. 1 .
Figure 7: Recorded monitor verdicts while drawing a mahjong tile. Each image is the four-camera composite sent to the monitor. (a) The monitor reuses a known recovery skill verbatim. (b) A stalled task in an intact scene leads to a task demonstration. (c) The recovery policy executes the proposed instruction, and the restoration query confirms the scene. (d) The completion query accepts a finished task.
TaskSuccess criterion
Pencil boxThe pencil lies in its tray, and the tray is slid back into the box sleeve.
Stack ringsThe four rings sit on the peg in size order.
Draw tileExactly one more tile stands upright at the end of the hand row, facing the front camera; no tile has fallen, and the tile wall is otherwise unchanged.
Discard tileThe specified hand tile lies face-up in the central discard area, the other hand tiles are undisturbed, and both grippers are empty.
Appendix
Table 4: Real-robot tasks and success criteria.
SettingValue
Initializationπ0.5\pi_{0.5} base checkpoint
Actions14-D joint positions (joint offsets, absolute grippers)
Action chunk50 steps (1.67 s at 30 Hz)
Image inputstop, left wrist, right wrist; 224×224224\times 224
OptimizerAdamW, gradient clipping 1.0
Learning ratecosine decay from 2.5×10−52.5\times 10^{-5} to 2.5×10−62.5\times 10^{-6}
Appendix
Table 5: Policy fine-tuning hyperparameters , shared by all task and recovery policies.
Figure 8: Simulated recovery skills for pencil-box packing and ring stacking. Each blue panel shows one recovery skill developed in the task’s digital twin, as four frames of one rollout, from the failure to the result. Frames come from the twin’s inspection camera, except Move the idle arm clear (top camera).
Figure 9: Simulated recovery skills for drawing and discarding a mahjong tile. Panels follow the conventions of Fig. 8 .
Figure 10: Recovery examples on LIBERO-Pro (top) and MolmoSpaces (bottom). Each panel shows one episode’s failure and recovery moments under its recovery instruction and task, as in Fig. 3 . Red rings in some LIBERO-Pro frames are markers from the source recordings.

Explore similar work

CardsList

  1. LIBERO-RECOVER: Beyond Task Success Towards Failure Recovery in Robotic Manipulation Models

    Sep 4, 2026Lin Liu, Zhicheng Bao, Lu Zhang +7Libero Manipulation BenchmarkRobotic Manipulation

    Beta Infinity · Beijing Jiaotong University · School of Information and Communication Engineering Dalian University of Technology +1

  2. EgoRecovery: Acquiring Failure Recovery Ability Through Human Recovery Demonstration

    Jul 22, 2026Zuhao Ge, Yuchen Zhou, Weitao Zhou +8Robot-Free DemonstrationsEgocentric Dataset

    Institute of Trustworthy Embodied AI (TEAI), Fudan University · 2Shanghai Key Laboratory of Multimodal Embodied AI · 3Simple AI

  3. MAGMA-GEN: Validated Recovery Supervision from Ambiguous Failures via Counterfactual Re-Execution

    Sep 17, 2026Loan Bernat, Matthieu Grard, Ariane Herbulot +1Long-Horizon Manipulation

    LAAS-GEPETTO · Sil´eane, LAAS-CNRS · LAAS-RAP +2