Imagining Recovery: Inference-Time Counterfactual Realignment for Vision-Language-Action Models

Yanyan Zhang; Disheng Liu; Kai Ye; Chaoda Song; Xinpeng Li; Mohsen Hariri; Vikash Singh; Yu Yin

cs.ROAug 14, 2026

Imagining Recovery: Inference-Time Counterfactual Realignment for Vision-Language-Action Models

Authors: Yanyan Zhang, Disheng Liu, Kai Ye, Chaoda Song, Xinpeng Li, Mohsen Hariri, Vikash Singh, Yu Yin, +1 more

Organizations: Case Western Reserve University Cleveland, OH, USA

Abstract

Vision-language-action (VLA) models have improved the flexibility and generality of robotic manipulation, yet they remain fragile to online disruptions, such as changes in task goal, scene configuration, or robot state. Existing recovery methods often require failure data, policy retraining, or external corrective agents, introducing additional data requirements and execution risks. We propose Counterfactual Realignment (CoRe), a training-free framework that recovers a frozen VLA at inference time without failure data. Upon detecting a deviation, CoRe imagines how the policy would continue toward the current goal from a recent viable state, using synthesized observations in place of physical execution, and then minimally realigns the robot and scene to rejoin this imagined continuation before returning control to the policy. Recovery is therefore planned without physical trial-and-error, preserves completed task progress, and handles both mid-episode instruction changes and physical perturbations in a unified manner. Extensive experiments across multiple simulators, VLA backbones, and real-world settings show that CoRe improves success rates by up to 85.0 percentage points to near-nominal levels while reducing physical restorations by 42.2%, without policy fine-tuning or failure-specific recovery training.

Figures & tables

Figure 1: OOD recovery paradigms for VLA models. (a) Post-training on failure data and (b) delegating recovery to an external module both hinge on costly retraining or a reliable auxiliary system. (c) CoRe instead plans recovery virtually and returns control to the same frozen policy.

Figure 2: Overview of CoRe. An online perturbation trips the suffix OOD detector and interrupts the frozen VLA (top). Recovery runs through the virtual planning stages (bottom), passing the recovery anchor, the rejoin state $c^{*}$ , and the restoration subset $S^{*}$ left to right. Only the final rearrangement and handoff produces physical motion before control returns to the frozen VLA.

Figure 3: Synthesized observations. Exocentric and egocentric views at imagined ticks, arm rendered by FK, residual holes inpainted.

Figure 4: Overview of the LangSwitch Tasks , spanning the Re-grounding , Re-skilling , and Conflict switch families defined in the Experimental Setup.

	Method	No-Switch	Re-grounding			Re-skilling			Conflict			Average
	Method	No-Switch	Pk $\to$ Pk	Ps $\to$ Ps	Pc $\to$ Pc	Pk $\to$ Ps	Pc $\to$ Pl	St $\to$ Pk	Ps $\to$ Pl	Pc $\to$ Ps	St $\to$ St	Switch	All
Base	$\pi_{0}$ ( Black et al. 2024 )	90.6	8.3 $\pm$ 13.6	0.7 $\pm$ 0.6	0.0 $\pm$ 0.0	7.7 $\pm$ 9.0	12.3 $\pm$ 7.5	27.0 $\pm$ 20.9	0.7 $\pm$ 1.2	91.0 $\pm$ 8.2	17.0 $\pm$ 12.1	18.3 $\pm$ 5.9	44.1
	$\pi_{0.5}$ ( Intelligence et al. 2025 )	97.8	29.0 $\pm$ 50.2	21.3 $\pm$ 37.0	0.0 $\pm$ 0.0	33.0 $\pm$ 39.8	56.3 $\pm$ 5.5	65.7 $\pm$ 56.9	7.0 $\pm$ 12.1	85.0 $\pm$ 11.5	22.3 $\pm$ 17.1	35.5 $\pm$ 21.3	57.8
	GR00T N1.7 ( NVIDIA 2026 )	88.4	5.0 $\pm$ 2.6	4.0 $\pm$ 5.3	0.0 $\pm$ 0.0	4.7 $\pm$ 5.7	27.7 $\pm$ 1.2	12.7 $\pm$ 11.0	7.7 $\pm$ 12.4	88.7 $\pm$ 4.0	17.0 $\pm$ 15.6	18.6 $\pm$ 4.6	43.5
Ours	$\pi_{0}$ + CoRe	90.0	98.7 $\pm$ 1.5	87.7 $\pm$ 1.5	73.0 $\pm$ 2.6	95.7 $\pm$ 0.6	88.7 $\pm$ 2.9	95.3 $\pm$ 4.7	83.3 $\pm$ 8.1	97.0 $\pm$ 2.0	68.0 $\pm$ 7.9	87.5 $\pm$ 0.6	88.4
	$\pi_{0.5}$ + CoRe	97.2	97.7 $\pm$ 0.6	98.3 $\pm$ 1.5	87.0 $\pm$ 1.0	99.7 $\pm$ 0.6	96.0 $\pm$ 1.0	92.7 $\pm$ 1.5	92.3 $\pm$ 7.4	98.7 $\pm$ 1.5	72.0 $\pm$ 11.8	92.7 $\pm$ 1.3	94.3

Table 1: Success rate (%) on the LangSwitch tasks with an xArm6 robot. Each switch pair is evaluated at 3 switch timings (shallow, mid, deep), 100 episodes each; per-pair cells and the Switch average report the mean

\pm

sample standard deviation across the 3 timings. No-Switch reports the mean over the 5 base skills (100 episodes each), and All is the macro average over all 14 task settings (5 No-Switch

+

9 switch pairs). Pk=Pick, Ps=Push, Pc=Place, Pl=Pull, St=Stack.

Figure 5: Overview of the LIBERO-Long with Perturbation Tasks , spanning the Arm Collision , Grasp Failure , Object Displace , and Object Tip perturbation families defined in the Experimental Setup.

	Method	Perturbation Type				Average
	Method	Arm Coll.	Grasp Fail.	Obj. Disp.	Obj. Tip	Average
Baselines	OpenVLA ( Kim et al. 2024 )	3.3 $\pm$ 2.5	41.3 $\pm$ 1.6	3.2 $\pm$ 1.0	3.3 $\pm$ 0.6	12.8 $\pm$ 0.3
	OpenVLA-OFT ( Kim et al. 2025 )	18.7 $\pm$ 4.3	65.0 $\pm$ 9.5	13.5 $\pm$ 6.4	10.3 $\pm$ 3.9	26.9 $\pm$ 1.8
	$\pi_{0.5}$ ( Intelligence et al. 2025 )	37.0 $\pm$ 8.7	65.0 $\pm$ 1.7	43.8 $\pm$ 7.0	28.0 $\pm$ 2.8	43.5 $\pm$ 2.3
	VLA-Adapter ( Wang et al. 2026 )	21.3 $\pm$ 5.8	65.0 $\pm$ 4.5	10.8 $\pm$ 1.9	10.5 $\pm$ 2.8	26.9 $\pm$ 2.4
	X-VLA ( Zheng et al. 2026 )	60.5 $\pm$ 5.2	51.2 $\pm$ 7.3	6.2 $\pm$ 0.6	9.7 $\pm$ 0.8	31.9 $\pm$ 2.8

Table 2: Post-perturbation success rate (%) on LIBERO-Long with a Franka Panda robot. Each of the 3 perturbation timings (shallow, mid, deep) is evaluated over 200 episodes (10 tasks

\times

20 episodes); cells report mean

\pm

sample standard deviation across the 3 timings. For CoRe, the Average additionally reports in parentheses the fully autonomous Arm-Only variant, which weights each restoration by the arm’s measured execution feasibility.

Condition	$\pi_{0.5}$	+CoRe	$\Delta$ [n (pp)]
$\bullet$ Language Switch
Re-grounding (pick $\to$ pick)	6/20	18/20	$+12$ ( $+60$ )
Re-skilling (pick $\to$ push)	1/20	18/20	$+17$ ( $+85$ )
Conflict (place $\to$ push)	1/20	18/20	$+17$ ( $+85$ )
$\bullet$ Physical Perturbation
Arm Collision (place)	9/20	17/20	$+8$ ( $+40$ )

Table 3: Real-world xArm6 results , evaluated using layout-paired trials with stratified perturbation timings.

Figure 6: Effect of injection timing. Success rate at each injection timing, bare versus with CoRe. Bare policies are timing-sensitive under language switches yet nearly timing-invariant under physical perturbations, and CoRe lifts and flattens all curves.

Figure 7: Overview of the Real-World Tasks. We validate the feasibility of CoRe beyond simulation on a physical xArm6 across multiple task families and objects.

Figure 8: Fidelity of synthesized rollouts. Endpoint TCP divergence between synthesized and GT rollouts from the same anchor (bars: median, whiskers: 90th percentile).

Figure 9: Mechanism ablation on LangSwitch. Top , overall comparison of the 4 recovery variants. #Restored is the number of objects physically moved per episode, and Post-hand Steps counts the control steps executed after handoff. Bottom , #Restored split by switch family.

Figure 10: Virtual probing versus physical execution. Probing a recovery hypothesis in imagination is 2-5 times cheaper than testing it with the arm.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Family	Pair	Target relation
Re-grounding	pick $\to$ pick	different color
Re-grounding	push $\to$ push	different color
Re-grounding	place $\to$ place	different color and bin
Re-skilling	pick $\to$ push	different color
Re-skilling	place $\to$ pull	different color
Re-skilling	stack $\to$ pick	leftover cube

Appendix

Table A1: The nine LangSwitch pairs. Colors are drawn per episode from eight colors.

Object	Upright	Tipped / Inverted
alphabet soup	100	100
tomato sauce	100	100
butter	100	100
chocolate pudding	100	100
cream cheese	98	100
black book	100	0

Appendix

Table A2: Oracle grasp success rates (%) per object and orientation, exactly as consumed by the Arm-Only weighting. Rates come from 50 motion-planned trials per tested object–orientation pair, 900 trials in total. Non-upright poses share one tested rate per object. The bowl entry lists its side and upside-down rates separately, and its side rate is carried over from the upright grasp rather than measured. The zero entries are orientations whose required grasp width exceeds the gripper aperture and are geometrically infeasible.

Backbone	pick	push	pull	place	stack
$\pi_{0}$	59	61	63	102	285
$\pi_{0.5}$	63	55	62	108	299
GR00T N1.7	60	42	61	89	292

Appendix

Table A3: Per-backbone minimum completion times

M_{A}

(control steps) used for switch timing.

Parameter	Value	Notes
Detection samples $K$	8	mean-pooled
Trigger $n_{\text{consec}}$	3	3-of-3 raw scores
Plateau backup	5 @ $0.9\tau_{k}$	sustained scores
Refractory period	16 queries	between triggers
$\tau_{k}$ quantile	0.95	held-out successes
Bank size (LS)	50 eps/task	successes only

Appendix

Table A4: CoRe hyperparameters. LS denotes LangSwitch and LB denotes LIBERO-Long.

Figure A1: Synthesized observations, as consumed by the frozen policy. Columns are the three domains, rows are the two viewpoints. LangSwitch shows a frame of an imagined continuation, in which the target is rendered in mid-air because the counterfactual has already grasped it. LIBERO shows the dual-view synthesis of a single imagined state. The real-robot column shows the repaint a subset probe is evaluated on, with the candidate objects removed and the vacated pixels filled from the background store.

Figure A2: Anchor-divergence threshold sweep on the validation pairs. Success peaks at

\delta=0.09

and collapses for later anchors.

Figure A3: Trajectory-equivalence threshold sweep on the validation pairs. The shaded band is the admissible plateau, and

\epsilon^{*}=0.050

minimizes restorations within it.

$\pi_{0}$
		Bare			+CoRe
Family	Pair	S	M	D	S	M	D
Re-grounding	pick $\to$ pick	24.0	1.0	0.0	99.0	97.0	100.0
Re-grounding	push $\to$ push	1.0	0.0	1.0	86.0	89.0	88.0
Re-grounding	place $\to$ place	0.0	0.0	0.0	74.0	75.0	70.0
Re-skilling	pick $\to$ push	18.0	2.0	3.0	96.0	96.0	95.0

Appendix

Table A5: LangSwitch per-cell success rates (%). Each cell is 100 episodes. S/M/D denote shallow/mid/deep switch timing.

	Arm collision			Grasp failure			Object displace			Object tip
Method	S	M	D	S	M	D	S	M	D	S	M	D	Avg
OpenVLA	1.0	3.0	6.0	42.5	42.0	39.5	4.0	3.5	2.0	3.0	4.0	3.0	12.8 $\pm$ 0.3
RobustVLA	10.0	10.5	19.0	34.0	50.5	57.0	3.0	1.5	3.5	2.5	3.5	4.0	16.6 $\pm$ 4.3
GR00T N1.7	8.5	18.0	20.5	40.5	49.5	48.0	19.5	17.0	12.5	13.5	14.5	18.5	23.4 $\pm$ 2.5
OpenVLA-OFT	15.5	17.0	23.5	54.5	67.5	73.0	20.5	12.0	8.0	13.5	6.0	11.5	26.9 $\pm$ 1.8
VLA-Adapter	15.0	22.5	26.5	60.5	65.0	69.5	13.0	10.0	9.5	11.0	7.5	13.0	26.9 $\pm$ 2.4

Appendix

Table A6: LIBERO-Long per-family per-timing success rates (%). Each entry aggregates 200 episodes. S/M/D denote shallow/mid/deep perturbation timing, and Avg reports mean and sample standard deviation across the three timings.

	$\pi_{0}$		$\pi_{0.5}$		GR00T N1.7
Skill	Bare	+CoRe	Bare	+CoRe	Bare	+CoRe
pick	100.0	100.0	98.0	100.0	99.0	100.0
push	94.0	97.0	100.0	100.0	91.0	92.0
place	90.0	91.0	97.0	93.0	81.0	89.0
pull	93.0	91.0	98.0	99.0	95.0	94.0
stack	76.0	71.0	96.0	94.0	76.0	76.0

Appendix

Table A7: No-Switch per-skill success rates (%) on LangSwitch. Each cell aggregates 100 episodes.

Figure A4: Real-robot success rate per condition, $20$ episodes each, for the bare policy and for the same frozen policy wrapped in CoRe.

Figure A5: Recovery composition under Scene Rearrangement: of the objects displaced above tolerance, the share the probe restores and the share it declines to touch.

	Bare	+CoRe
Condition	all	S	M	D
Re-grounding pick $\to$ pick	6/20	7/7	5/7	6/6
Re-skilling pick $\to$ push	1/20	6/7	6/7	6/6
Conflict place $\to$ push	1/20	6/7	7/7	5/6
Arm collision	9/20	6/7	5/7	6/6
Scene rearrangement	4/20	7/7	6/7	6/6

Appendix

Table A8: Real-robot success by injection timing. S/M/D denote shallow/mid/deep, at the same

0.30/0.55/0.75

fractions of the recalibrated real phase.

	Bare	+CoRe
Collisions or emergency stops	21/100	5/100

Appendix

Table A9: Physical safety over the

100

layout-paired episodes. CoRe removes three quarters of the unsafe terminations in absolute count.

Skill	pick	place	push
Nominal success	19/20	19/20	20/20

Appendix

Table A10: Unperturbed control condition, frozen checkpoint,

20

episodes per skill.

Explore similar work

RePO-VLA: Recovery-Driven Policy Optimization for Vision-Language-Action Models
May 10, 2026Weijia Liufu, Xiaoyu Guo, Ruiyi Chen +16VLM Robustness Robot Failure Recovery
1Sun Yat-sen University · 2South China University of Technology · 3Peng Cheng Laboratory +3
ReCoVLA: VLM-Guided Reward Compilation for Failure Recovery in Vision-Language-Action Policies
Jun 8, 2026Haodi Hu, Chung-Ta Huang, Jing Liu +4Robot Failure Recovery Vision-Language-Action Models
University of Southern California, USA · Mitsubishi Electric Research Laboratories (MERL), USA · Harvard University, USA
ForkCast: Predictive Recovery for Vision-Language-Action Policies via Counterfactual Futures
Oct 13, 2026Chenru Wen, Huajian Zeng, Haokai Ding +5Vision-Language-Action Models VLMs for Robotics
Mohamed bin Zayed University of Artificial Intelligence (MBZUAI), UAE. · Virginia Tech, USA.

May 10, 2026J/K move · Enter open · S save

RePO-VLA: Recovery-Driven Policy Optimization for Vision-Language-Action Models

Weijia Liufu, Xiaoyu Guo, Ruiyi Chen, Jingzhi Liu+15

1Sun Yat-sen University · 2South China University of Technology · 3Peng Cheng Laboratory+3

VLM Robustness Robot Failure Recovery Robotics Vision-Language-Action Models Robustness of VLA Models

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models remain brittle in long-horizon, contact-rich manipulation because success-only imitation provides little supervision for execution drift, while failed rollouts are often discarded. We introduce RePO-VLA, a recovery-driven policy optimization framework that assigns distinct roles to success, recovery, and failure trajectories. RePO-VLA first applies Recovery-Aware Initialization (RAI), slicing recovery segments and resetting history so corrective actions depend on the current adverse state rather than the preceding failure. It then learns a Progress-Aware Semantic Value Function (PAS-VF), aligning spatiotemporal trajectory features with instructions and successful references. The resulting labels salvage useful failure prefixes via reliability decay, while low-value labels mark drift and terminal breakdowns, teaching differences among nominal, failed, and corrective actions. The data engine turns adverse states into planner-generated or human-collected corrective rollouts, teaching recovery to the success manifold. Value-Conditioned Refinement (VCR) trains the policy to prefer high-progress actions. At deployment, a fixed high value (

v=1.0

) biases actions toward the learned success manifold without online failure detectors or heuristic retries. We introduce FRBench, with standardized error injection and recovery-focused evaluation. Across simulated and real-world bimanual tasks, RePO-VLA improves robustness, raising adversarial success from 20% to 75% on average and up to 80% in scaled real-world trials.

Abstract

Explore similar work

RePO-VLA: Recovery-Driven Policy Optimization for Vision-Language-Action Models

ReCoVLA: VLM-Guided Reward Compilation for Failure Recovery in Vision-Language-Action Policies

ForkCast: Predictive Recovery for Vision-Language-Action Policies via Counterfactual Futures