Simulation enables scalable training of Vision-Language-Action policies by using privileged experts to generate visual demonstrations without requiring every trajectory to be collected through manual teleoperation. However, such pipelines typically retain successful demonstrations while failed rollouts are discarded, even though they expose precisely the off-nominal states from which recovery must be learned. We introduce Kintsugi-VLA, a framework for converting failed rollouts into targeted synthetic recovery data by exploiting exact state restoration and branching in simulation. For a fixed privileged expert, we define interventional recoverability as the probability of completing the original task after the simulator is restored to a given state, estimate it using adaptive Monte Carlo continuations with pointwise Wilson intervals, and characterize its non-monotonic evolution along failed trajectories. These estimates identify an observed terminal low-recoverability frontier-the point after which measured recoverability remains below a threshold-which is then used to select informative recovery starting states. In a simulated Franka manipulation task, targeted recovery data yield aggregate SmolVLA recovery success of 34.6% and 38.4% under difficulty- and frame-budget matching, respectively, 5.8 and 6.7 percentage points above uniform sampling within the same recovery window. The same ordering is observed under disturbed end-to-end execution and shifted clutter and physics conditions, while clean-task success decreases from 76.8% to 74.7%. Kintsugi-VLA demonstrates how failed simulator rollouts can be transformed from discarded experience into structured recovery-training data through direct interventional measurement.
Figures & tables
Fig. 1: Failed rollouts as recovery-data sources. Kintsugi-VLA restores saved simulator states and repeatedly tests expert continuations to identify states suitable for synthetic recovery demonstrations.
Fig. 2: Kintsugi-VLA method overview. Saved simulator states are restored and branched under a privileged expert. Recoverability measurements guide recovery-data collection, followed by difficulty- and frame-budget-matched VLA comparisons.
Fig. 3: Schematic expert-relative recoverability curve. A recovery island follows an earlier threshold crossing, while the frontier precedes the terminal suffix whose upper interval endpoints remain below τ . Shading is illustrative, not a simultaneous confidence band.
Profile
Static friction
Dynamic friction
Mass multiplier
High friction
1.25–1.35
1.05–1.15
1.00–1.20
Heavy object
0.70–0.90
0.55–0.75
1.55–1.70
TABLE I: Protocol G physics profiles. Friction coefficients and mass multipliers are dimensionless; target scale is 1.23.
Fig. 4: Protocol G calibration pool. Bin assignment uses 15 trials per state; the frozen evaluation set contains 30 states from each hard, medium, and easy bin.
Fig. 5: Historical trajectory structure across 127 failed episodes at τ=0.2 . Categories overlap; non-monotonicity uses the separated-rise criterion in Sec. III .
Training seed
Nominal success
Probe recovery
42
200/200
630/1280 (49.2%)
1234
201/201
701/1280 (54.8%)
2026
201/201
689/1280 (53.8%)
TABLE II: Historical teacher acceptance and recovery probe. Each probe uses five attempts from 256 shared states.
Fig. 6: Teacher dependence on the same 256-state probe. Bars show state-wise Spearman correlation and binary agreement at τ=0.2 .
Clutter seed
Profile
States
Ordinary
Recovery
20261706
HF
12
4/4
46/60
20261706
HO
12
4/4
40/60
20261707
HF
12
4/4
50/60
20261707
HO
12
3/4
43/60
20261708
HF
12
4/4
38/60
20261708
HO
12
4/4
40/60
TABLE III: Protocol G teacher outcomes. HF: high friction; HO: heavy object. Recovery uses five attempts per frozen state.
Fig. 7: Protocol G teacher recovery on the frozen balanced set. Each difficulty bin contains 30 states with five evaluation attempts per state; bin assignment uses separate calibration trials.
Protocol A
Protocol B
Protocol G
Variant
Difficulty
Frames
Disturbed
Clean
Recovery
V0
12.5
12.5
55.7
76.8
9.6
V1
25.9
28.8
58.6
75.8
22.1
V2
28.8
31.7
59.5
75.8
25.0
V3
34.6
38.4
63.4
74.7
28.8
TABLE IV: Closed-loop VLA success (%). Bold values indicate the highest aggregate rate in each column.
Vision-language-action (VLA) policies provide strong priors for language-conditioned manipulation, but remain brittle in off-nominal states requiring targeted recovery. We propose ReCoVLA -- a failure-conditioned residual recovery framework that keeps a pretrained VLA policy frozen, uses an external vision-language model (VLM) to infer the failure mode and recovery stage, and compiles a structured reward from task-relevant components. Rather than using the VLM to generate actions or rewards directly, ReCoVLA uses it as a semantic reward selector: it predicts a recovery descriptor and reward mask for in-simulation residual-policy training, followed by zero-shot sim-to-real deployment of the trained recovery policies. This decouples high-level failure understanding from low-level corrective control to support different VLAs. Experiments across short-horizon, long-horizon, and contact-rich manipulation tasks show that ReCoVLA outperforms the tested baselines on average. In simulation, our reward compiler improves average success from 36.7% for the fine-tuned π0.5 baseline to 66.7%. In physical zero-shot sim-to-real experiments, ReCoVLA achieves the best average performance, with 61.7% success.
Haodi Hu, Chung-Ta Huang, Jing Liu +4
University of Southern California, USA · Mitsubishi Electric Research Laboratories (MERL), USA · Harvard University, USA
Vision-Language-Action (VLA) models remain brittle in long-horizon, contact-rich manipulation because success-only imitation provides little supervision for execution drift, while failed rollouts are often discarded. We introduce RePO-VLA, a recovery-driven policy optimization framework that assigns distinct roles to success, recovery, and failure trajectories. RePO-VLA first applies Recovery-Aware Initialization (RAI), slicing recovery segments and resetting history so corrective actions depend on the current adverse state rather than the preceding failure. It then learns a Progress-Aware Semantic Value Function (PAS-VF), aligning spatiotemporal trajectory features with instructions and successful references. The resulting labels salvage useful failure prefixes via reliability decay, while low-value labels mark drift and terminal breakdowns, teaching differences among nominal, failed, and corrective actions. The data engine turns adverse states into planner-generated or human-collected corrective rollouts, teaching recovery to the success manifold. Value-Conditioned Refinement (VCR) trains the policy to prefer high-progress actions. At deployment, a fixed high value (v=1.0) biases actions toward the learned success manifold without online failure detectors or heuristic retries. We introduce FRBench, with standardized error injection and recovery-focused evaluation. Across simulated and real-world bimanual tasks, RePO-VLA improves robustness, raising adversarial success from 20% to 75% on average and up to 80% in scaled real-world trials.
Weijia Liufu, Xiaoyu Guo, Ruiyi Chen +16
1Sun Yat-sen University · 2South China University of Technology · 3Peng Cheng Laboratory +3
Vision-language-action (VLA) models have improved the flexibility and generality of robotic manipulation, yet they remain fragile to online disruptions, such as changes in task goal, scene configuration, or robot state. Existing recovery methods often require failure data, policy retraining, or external corrective agents, introducing additional data requirements and execution risks. We propose Counterfactual Realignment (CoRe), a training-free framework that recovers a frozen VLA at inference time without failure data. Upon detecting a deviation, CoRe imagines how the policy would continue toward the current goal from a recent viable state, using synthesized observations in place of physical execution, and then minimally realigns the robot and scene to rejoin this imagined continuation before returning control to the policy. Recovery is therefore planned without physical trial-and-error, preserves completed task progress, and handles both mid-episode instruction changes and physical perturbations in a unified manner. Extensive experiments across multiple simulators, VLA backbones, and real-world settings show that CoRe improves success rates by up to 85.0 percentage points to near-nominal levels while reducing physical restorations by 42.2%, without policy fine-tuning or failure-specific recovery training.
Yanyan Zhang, Disheng Liu, Kai Ye +6
Case Western Reserve University Cleveland, OH, USA