Disturbances can invalidate the conditions required by a robot's next action even when much of its task program remains useful. A recovery system must therefore choose a response suited to the current state. We present DiMuRe (diagnosis-guided multilevel recovery), a framework for robot-use agents that uses phase-specific evidence and method preconditions to select program reuse or predefined recovery skills, including bounded vision-language-action (VLA) grasping. After the original program exits, a multimodal large language model (MLLM) can generate corrective code from images and execution feedback. Local verification checks the outcome of each recovery attempt. We also introduce LIBERO-Disturb, a benchmark for robustness and recovery under mid-execution disturbances, supporting program-based agents and VLA policies. On 801 scheduled disturbed Encore development cases, Local DiMuRe (program reuse and recovery skills) achieves 77.65% complete-task success, compared with 58.68% for the original program under the same physical-step allowance. Full DiMuRe achieves 81.90% with terminal replanning, an additional step allowance, and early handoff. Local DiMuRe also uses fewer physical steps per scheduled episode than the original program.
Figures & tables
Fig. 2: DiMuRe system architecture and recovery flow. Phase-specific diagnosis uses RGB-D observations, robot state, and program context, and requests new observations when evidence is insufficient. The dashed box shows local grasp recovery. The system considers template replay (L1), geometry-based regrasp (L2), and bounded VLA grasping (L2) in that order, checking admission ( am ), executing the method ( πm ), and verifying the grasp ( vm ). It skips ineligible candidates and tries the next candidate if an attempt does not produce a verified grasp. A verified grasp ends the local search. MLLM-assisted replanning (L3) starts separately after the original program exits and its entry checks pass. The planner revises corrective code using new observations and execution feedback.
Fig. 3: Nine mid-execution disturbances in LIBERO-Disturb, shown in recorded Encore episodes: (a)–(d) object location and motion, (e)–(f) object pose, (g)–(h) grasp and robot state, and (i) receiving-region location. The three images in each example use the same camera and crop. Blue dashed wireframes mark the approximate pre-intervention position. Orange annotations indicate position ( Δp ), velocity ( Δv ), orientation ( ΔR ), or joint-position ( Δq ) changes and their visible effects; grey chevrons indicate time order. Values in parentheses are relative native control-step indices Δk . Zero marks the first recorded post-injection frame, captured after one native control step following the first intervention. The overlays are qualitative visual guides.
Configuration
Clean SR (%)
Disturbed SR (%)
Encore
Original
100.00
58.68
Local DiMuRe (L1+L2)
100.00
77.65
Full DiMuRe
100.00
81.90
TABLE I: Native task SR (%) on 90 clean and 801 scheduled disturbed Encore cases per configuration.
(a) Task outcomes and execution costs
Configuration
SR (%)
Δ SR (pp)
Mean task time (s)
Mean physical steps
All cases
Successful cases
All cases
Successful cases
Encore: shared native allowance
Original program
58.68
0.00
4.65
2.93
328.90
213.46
L1-only
73.66
+14.98
5.24
3.68
317.32
229.20
L2-only
65.54
+6.87
5.91
3.95
349.72
229.49
TABLE II: Recovery-level ablation, execution costs, and process analysis. Panel (a) uses 801 scheduled disturbed cases; Δ SR is relative to Original. All-case means include failures; successful-case means use each configuration’s successful subset. Full has an additional allowance and distinct handoff.
Appendix figures & tables14 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol
Meaning
Value
Unit
τmove
Minimum tool displacement for evaluating attachment
0.025
m
τfollow
Maximum accepted object/tool displacement residual
0.035
m
gmin
Lower admissible gripper opening
0.003
m
gmax
Upper admissible gripper opening
0.070
m
Appendix
TABLE S1: Thresholds for grasp verification during original-program execution.
Symbol
Applicable procedure or invocation
Value
Unit
Amax
Shared candidate-attempt limit across the task
3
attempts
bmin
Grasp-recovery entry
250
steps
bmin
Placement-retry entry
230
steps
r
Grasp recovery and placement retry
150
steps
Cmax
Grasp-recovery procedure
450
steps
Cmax
Placement-retry procedure
500
steps
Appendix
TABLE S2: Candidate-attempt limits and physical-step budgets.
Symbol
Meaning
Value
Unit
bentryplan
Minimum remaining budget for planner entry
80
steps
broundplan
Minimum remaining budget for beginning a round
40
steps
Kplan
Maximum planning rounds within the invocation
32
rounds
Kreview
Shared cap on generic reviews of unverified stop proposals
2
requests
Kcmd
Maximum counted physical-command calls per fragment
12
commands
Texec
Wall-clock execution limit per generated fragment
45
s
Appendix
TABLE S3: Terminal MLLM planner entry thresholds and execution limits.
Original/Local/Full: 90 clean and 891 disturbed each; L1-only/L2-only: 801 disturbed each
Controlled components
Fixed task programs, perception models, and disturbance settings
Disturbance protocol
Trigger rules, perturbation parameters, and validity checks in Appendix C
Recovery configurations
Original: no added recovery; L1-only: program reuse; L2-only: geometry/VLA skills; Local: L1+L2; Full: Local plus terminal replanning
Appendix
TABLE S4: Run settings and recovery configurations. Original-program checks and retries are retained throughout.
Metric
Definition and denominator
SR and Δ SR
Native task successes divided by all scheduled episodes, including unreached or ineffective disturbances. Δ SR is the difference from Original in percentage points, computed from unrounded rates.
Recovery and retention
Successful outcomes among Original failures and among Original successes, respectively, paired by scheduled case.
Mean task time and steps
Arithmetic means over all scheduled episodes, including failures, or over each configuration’s successful subset as labeled. Successful subsets can differ between configurations.
Recorded recovery time
Sum of logged stage intervals within the task interval, averaged over episodes entering that stage before the endpoint, including failed episodes. Local and terminal stages are measured separately.
Anomaly-response coverage
Auditable first-fault windows containing a qualifying alarm or target adaptation, divided by all auditable windows; repeated responses count once.
Task SR after response
Native task successes strictly after the first qualifying response, divided by responding windows.
Appendix
TABLE S5: Metric definitions and statistical populations. All task-success measures use native complete-task success; local grasp verification is separate.
Evaluation set
N
Original
Local
Full
Main-text subset (without 6 cm)
801
470/801
622/801
656/801
Full disturbed set
891
549/891
689/891
725/891
Receiving-region shift, 6 cm
90
79/90
67/90
69/90
Receiving-region shift, 12 cm
90
49/90
77/90
77/90
Appendix
TABLE S6: Native task success on the full disturbed development set, the main-text subset, and each receiving-region shift amplitude. Entries are successful episodes / scheduled episodes. Cases remain in the denominator whether or not the intervention has a verified effect.
Configuration
Applied
Effect verified
Effect check missing
Final check passed
Original
631
105
696
470
L1-only
632
105
696
590
L2-only
632
105
696
523
Local DiMuRe
632
105
696
620
Full DiMuRe
632
105
696
654
Appendix
TABLE S7: Disturbance application, effect verification, and final-time checks on 801 scheduled cases per configuration. Application records are complete. Effect checks are either positive or missing; missing checks remain unknown.
Configuration
Original failures recovered (%)
Original successes retained (%)
Full DiMuRe
194/331 (58.61)
462/470 (98.30)
Local DiMuRe (no L3)
159/331 (48.04)
463/470 (98.51)
L1-only
125/331 (37.76)
465/470 (98.94)
L2-only
69/331 (20.85)
456/470 (97.02)
Appendix
TABLE S8: Paired native outcomes on the 801-case disturbed subset. Counts and percentages use Original’s failures (left) or successes (right) as the denominator.
Disturbance
N
Original
L1-only
L2-only
Local
Full
Arm retraction
90
83
83
84
84
84
Grasp slip
90
38
72
49
70
76
Object bump
90
90
90
90
90
90
Object pose change
81
26
26
47
47
47
Receiving-region shift (12 cm)
90
49
77
49
77
77
Object retrieval
90
24
58
29
58
69
Appendix
TABLE S9: Native task successes by scheduled disturbance type on the 801-case development subset. Each entry is a success count with the row denominator N . Full has a larger allowance and uses early handoff. Scheduled cases remain in the denominator even when no disturbance is applied.
Fault family
Response (both)
Correct diagnosis (both)
Task success after response
Task success after correct diagnosis (both)
Local
Full
Grasp not established
31/34
15/34
23/31
24/31
15/15
Holding lost
30/34
12/34
15/30
21/30
11/12
Receiving region moved
31/36
30/36
28/31
28/31
28/30
Target state changed
34/57
0/57
28/34
28/34
N/A
Total
126/161
57/161
94/126
101/126
54/57
Appendix
TABLE S10: Anomaly responses, formal diagnoses, and subsequent native task success on 161 auditable windows per configuration. Entries show numerator / denominator. The shared columns have identical observed counts for Local and Full; success after response and after correct diagnosis use different populations.
Configuration
L3 requests
Input tokens
Output tokens
Cached input tokens
Full DiMuRe
1.93
204,421
570
30,825
Appendix
TABLE S11: Full DiMuRe terminal-model use, averaged over all 801 scheduled episodes, including those without planner entry. Cached input is included in input tokens. Values are rounded from recorded request usage.
Disturbance
Trigger or target selection
Nominal intervention
Object XY displacement
Step 30; declared task target
Random horizontal shift of 6–10 cm; bounded by workspace, separation, and penetration checks
Object bump
Step 40; random free object excluding the nearest to the tool
Horizontal velocity increment of 0.3–0.6 m/s in a random direction
Slow object drift
Step 20; object nearest to the tool
Horizontal translation of 1 mm per step for 60 steps in a random direction
Object retrieval
Poll every 5 steps from step 80; held object displaced more than 8 cm in XY
Reset the object’s position to its initial position and zero its velocity
Object pose change
Poll every 10 steps from step 20 to 520; eligible resting target
Apply the predefined orientation assigned to the case, subject to clearance and settling checks
Object tipover
Step 40; object nearest to the tool
Rotate by 90∘ about a random horizontal axis and raise by 3 cm
Appendix
TABLE S12: Nine disturbance types in the evaluated Encore protocol. Steps are native physical control steps. The settings describe intended interventions, not guaranteed physical effects or task failures.
Setting
Value or rule
Initial state
Official state index 0 for each task
Random seeds
Environment 7; policy 7; disturbance 51
Native physical-step limits
Spatial: 220; Object: 280; Goal: 300; Long: 520
Initialization and action queue
10 native wait steps; 5-action chunks
Trigger polling
Begin after 20 policy steps; retry after 10 steps if rejected
TABLE S13: Fixed settings of the native π0.5 evaluation.
Condition
Runs
Successes
SR (%)
Applied
Safety stops
Clean
40
39
97.5
0
0
Empty hook
40
39
97.5
0
0
Receptacle relocation
40
36
90.0
22
1
Appendix
TABLE S14: Complete native-task outcomes in the reported π0.5 subset. SR is the complete-task success percentage. Applied counts indicate that an intervention began, without implying that its intended effect was verified. Safety-stopped runs count as unsuccessful.
Manipulation failures can leave scenes in states from which a task policy cannot recover. Learning corrective behaviors requires scalable failure exploration and physical grounding. We present Recova, an agent-guided framework that jointly develops task execution and recovery in a reconstructed digital twin, then verifies and refines both through real-world experience. In the twin, the agent diagnoses failures, tests corrective programs, and collects successful task and recovery rollouts for separate policies. During deployment, it monitors progress, invokes a learned or programmatic recovery, verifies scene restoration, and resumes execution. When no suitable recovery is available, a human demonstration resolves the failure and enters the learning loop, allowing the system to expand its recovery capabilities. Physical rollouts and human demonstrations are routed to the corresponding policy for DAgger training. Across six LIBERO-Pro settings and four MolmoSpaces categories, Recova achieves 78.8% and 64.9% mean success, compared with 71.7% and 38.0% for the strongest baselines. With parallel collection across four real-robot workstations, DAgger fine-tuning raises mean success from 23.8% to 77.5%, and recovery skills further raise it to 87.5%. Over four collection rounds on one task, observed human intervention falls from 87.5% to 0%. Together, these results show how agent-guided recovery turns failures into reusable capabilities, improving robustness while progressively reducing human intervention. Project page: https://www.liuisabella.com/Recova
Isabella Liu, An-Chieh Cheng, Johan Bjorck +6
University of California, San Diego · NVIDIA · University of Texas at Austin
Robot-policy benchmarks increasingly cover diverse tasks and preset out-of-distribution conditions, but typically evaluate complete trajectories from predefined initial states. These evaluations often focus on the initialized scene and the final outcome, while paying less attention to the dynamic interaction process. During closed-loop execution, actions and contacts can alter object relations and task progress, producing off-nominal intermediate states that need recovery. Recovery requires a policy to infer how task progress has changed, correct the relevant relations, and continue the original goal. We introduce RoboRecover, a benchmark for robot policy recovery under execution deviations. RoboRecover selects deviation states from trajectories, reconstructs them by replaying action prefixes, and evaluates policies on the original task. RoboRecover contains 2,000 scenarios across RoboTwin and LIBERO, with 1,000 scenarios and a fixed 800/200 train/test split on each platform. Results show that initial-state performance does not determine recovery performance and policies exhibit different recovery strengths across scenarios. Using its training split, RoboRecover further supports study on recovery interventions. RoboRecover establishes recovery from execution-induced intermediate states as a distinct dimension of robot policy evaluation.
Yang Li, Chen Zhao, Zhuoran Wang +5
School of Information, Renmin University of China, Beijing, China · Key Laboratory of Data Engineering and Knowledge Engineering, Beijing, China · University of Science and Technology of China +1
Robotic manipulation poses fundamental challenges due to uncertainty, long-horizon execution, and compounding errors, which can easily destabilize execution and lead to task failure. Although recent vision-language-action (VLA) models exhibit strong generalization, they typically lack explicit mechanisms to assess execution stability and to recover when execution deviates from its nominal behavior. In this paper, we propose: (1) two complementary metrics to assess execution quality at runtime, and (2) an agentic reinforcement learning framework that learns to restore effective execution through high-level decision-making rather than directly learning low-level actions. In this framework, an agentic policy reasons over recent execution history and selects among a small set of execution modes to regulate the execution process. Under execution degradation, it triggers appropriate recovery mechanisms to restore the robot to previously visited nominal states, enabling the task to continue. We evaluate the proposed method on the LIBERO benchmark, achieving up to a 13.7% improvement in success rate under standard settings and up to a 39.2% improvement under disturbance settings, demonstrating substantially enhanced execution robustness.
Xiaopeng Zhang, Yueyang Weng, Qi Liu +2
School of Inteligence Science and Engineering, the Harbin Institute of Technology Shenzhen, 518055, China. · Faculty of Robot Science and Engineering, Northeastern University, Shenyang, 110819, China.