If at First You Don't Succeed, Backtrack: Lightweight Recovery for Frozen VLAs

Thomas Landeg; Thierry Tambe

cs.ROOct 23, 2026

If at First You Don't Succeed, Backtrack: Lightweight Recovery for Frozen VLAs

Authors: Thomas Landeg, Thierry Tambe

Organizations: Stanford University · ETH Zurich

Abstract

Vision-Language-Action (VLA) models perform well on robotic manipulation, yet remain brittle once their own actions carry them into states their training data barely covers. Recovering from such failures usually requires recovery demonstrations, policy retraining, or a separate model at inference time. We introduce Backtrack, a lightweight failure detection and recovery mechanism for frozen VLAs that needs none of these. A small failure prediction head, trained only on outcome-labeled rollouts of the frozen policy, reads the VLA's existing internal representations to estimate whether the current episode will fail. When the predicted failure probability exceeds a threshold, a controller pauses the policy, retraces the arm's recently achieved motion, and hands control back to the unchanged policy so it can retry from an earlier configuration. The head adds only 0.48-0.88% to the policy's parameter count. We evaluate Backtrack on LIBERO with three VLAs spanning two action-generation families: SmolVLA and $π$ 0.5, which use flow-matching action experts, and the autoregressive $π$ 0-FAST. On LIBERO-Long, the suite of long-horizon multi-step tasks and the one on which all three policies fail most often, Backtrack reduces each policy's failure rate by 19-48%, raising success by up to 7.4 percentage points; on the three shorter suites, where control success is already 85-99%, its effect is smaller. On MetaWorld, all twelve controller configurations tested with SmolVLA improve on the unassisted policy.

Figures & tables

Figure 1: Overview of Backtrack. A small failure prediction head reads features already computed by the frozen VLA and estimates the probability that the current episode will fail. When the predicted failure probability exceeds the threshold

t

, the Backtrack controller suppresses the policy’s proposed actions and instead retraces recent achieved arm motion stored in a buffer. Otherwise, the policy’s actions are executed normally. The resulting state is observed by the unchanged VLA and the control loop continues. Achieved motion is added to the buffer during normal policy execution; motion executed during backtracking is not. The diagram abstracts over differences in action generation between the evaluated VLAs.

POLICY	PARAMS	HEAD PARAMS	HEAD / POLICY
SmolVLA	450.0M	3.97M	0.88%
$\pi$ 0-FAST	3450.0M	17.35M	0.50%
$\pi$ 0.5	3616.8M	17.35M	0.48%

Table 1: Parameter count of the failure prediction head, relative to the frozen policy it attaches to. All three heads read the post feature tap.

POLICY	EPISODES	FAILED	SAMPLED FRAMES
SmolVLA	20,000	3,464 (17.3%)	100,000
$\pi$ 0-FAST	7,953	812 (10.2%)	100,000
$\pi$ 0.5	14,000	676 (4.8%)	100,000

Table 2: Rollout data behind each failure prediction head. The Episodes column gives the number of rollouts of the frozen policy pooled across all four LIBERO suites; the failure rate is the fraction of those episodes that ended in failure. Sampled frames is the number drawn from that pool to train and validate the head.

SUITE	POLICY	CONTROL	UPLIFT	95% CI	RECOVERY
LIBERO-Long	SmolVLA	61.00	$+7.40$	$[+4.52,+10.28]$	19.0%
	$\pi$ 0.5	94.72	$+1.60$	$[+0.06,+3.14]$	30.3%
	$\pi$ 0-FAST	85.32	$+7.08$	$[+5.91,+8.25]$	48.2%
all four suites	SmolVLA	83.26	$+3.40$	$[+2.04,+4.76]$	20.3%
	$\pi$ 0.5	96.98	$+0.79$	$[+0.05,+1.53]$	26.2%
	$\pi$ 0-FAST	91.31	$+1.70$	$[+0.97,+2.43]$	19.6%

Table 3: Recovery on the long-horizon LIBERO-Long suite, and aggregated over all four LIBERO suites. Five seeds (1000–5000), 50 episodes per task. Uplift is in percentage points and intervals are 95% confidence intervals over the five per-seed differences. Recovery rate is the uplift divided by the base failure rate, measuring the net relative reduction in failures.

POLICY	ctrl	upl	ctrl	upl	ctrl	upl	ctrl	upl
	SPATIAL		OBJECT		GOAL		LONG
SmolVLA	84.64	$+5.12$	94.72	$-0.20$	92.68	$+1.28$	61.00	$+7.40$
$\pi$ 0.5	98.24	$+0.16$	99.20	$+0.16$	95.76	$+1.24$	94.72	$+1.60$
$\pi$ 0-FAST	90.92	$-0.36$	96.80	$-0.20$	92.20	$+0.28$	85.32	$+7.08$

Table 4: Per-suite control success rate and uplift, five seeds, 50 episodes per task.

CONFIGURATION	SR	UPLIFT	95% CI	RECOVERY
Threshold, fixed retreat ( $s{=}50$ , $cd{=}50$ )
$t{=}0.90$	95.56	$+0.84$	$[+0.11,+1.57]$	15.9%
$t{=}0.95$	95.00	$+0.28$	$[-1.24,+1.80]$	5.3%
$t{=}0.97$	96.24	$+1.52$	$[+0.02,+3.02]$	28.8%
$t{=}0.99$ (headline)	96.32	$+1.60$	$[+0.06,+3.14]$	30.3%
$t{=}0.995$	95.56	$+0.84$	$[-0.36,+2.04]$	15.9%

Table 5: Threshold and retreat rule on

\pi

0.5, LIBERO-Long, post head, five seeds, 50 episodes per task, paired by seed. Control success is 94.72%, a failure rate of 5.28%. The first two blocks sweep the trigger threshold under each retreat rule; the third varies retreat depth at

t{=}0.99

. The adaptive rule retreats between 50 and 150 steps, stopping once the predicted failure probability falls below

0.2

CONFIGURATION	SR	UPLIFT	95% CI	RECOVERY	FIRE
Threshold, adaptive retreat (50–150 steps, $cd{=}50$ )
$t{=}0.90$	65.76	$+4.76$	$[+1.19,+8.33]$	12.2%	49.8%
$t{=}0.95$	66.44	$+5.44$	$[+2.78,+8.10]$	13.9%	47.5%
$t{=}0.97$ (headline)	68.40	$+7.40$	$[+4.52,+10.28]$	19.0%	42.6%
$t{=}0.99$	65.60	$+4.60$	$[+2.43,+6.77]$	11.8%	42.5%
$t{=}0.995$	65.00	$+4.00$	$[+0.69,+7.31]$	10.3%	39.2%

Table 6: Threshold and retreat rule on SmolVLA, LIBERO-Long, post head, five seeds, 50 episodes per task, paired by seed. Control success is 61.00%, a failure rate of 39.00%. The first block sweeps the trigger threshold under SmolVLA’s reported adaptive rule, which retreats between 50 and 150 steps, stopping once the predicted failure probability falls below

0.2

; the second holds the threshold at the reported

t{=}0.97

and substitutes a fixed retreat. Fire rate is the proportion of episodes in which the controller triggers at least once. Unlike for

\pi

0.5 (Table 5 ), we did not sweep the threshold under the fixed rule.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

CONFIGURATION	SR	UPLIFT	95% CI	RECOVERY	FIRE
Threshold, reported shape (adaptive, $a{=}0.3$ , 25–100 steps, $cd{=}50$ )
$t{=}0.90$ (headline)	92.40	$+7.08$	$[+5.91,+8.25]$	48.2%	33.7%
$t{=}0.95$	87.48	$+2.16$	$[+0.45,+3.87]$	14.7%	31.4%
$t{=}0.99$	90.84	$+5.52$	$[+3.42,+7.62]$	37.6%	20.8%
$t{=}0.995$	89.44	$+4.12$	$[+2.38,+5.86]$	28.1%	19.6%
Threshold, SmolVLA/ $\pi$ 0.5 shape (adaptive, $a{=}0.2$ , 50–150 steps, $cd{=}50$ )

Appendix

Table 7: Threshold and retreat shape on

\pi

0-FAST, LIBERO-Long, post head, five seeds, 50 episodes per task, paired by seed. Control success is 85.32%, a failure rate of 14.68%. The first block varies the threshold under the reported retreat shape; the second varies it under the retreat shape used for the SmolVLA and

\pi

0.5 threshold sweeps (Tables 5 and 6 ).

a

is the restart threshold. Fire rate is the proportion of episodes in which the controller triggers at least once.

FLOW-MATCHING STEPS	METAWORLD	LIBERO
1	$64.88\pm 1.40$	75.25
2 (ours)	$71.92\pm 1.29$	82.50
3	$72.04\pm 0.57$	85.00
5	$71.60\pm 0.93$	85.00
7	$70.48\pm 0.97$	85.75
10 (default)	$68.88\pm 0.73$	83.75

Appendix

Table 8: Success rate against the number of flow-matching steps, SmolVLA. MetaWorld uses seeds 1000–5000 with a 500-step episode budget and 500 episodes per seed, 10 on each of the 50 tasks, with per-seed standard deviations. LIBERO is a single seed, 100 episodes per suite over the four suites, and carries no error bars.

POLICY	520	624	728	832	936	1040	$\Delta$
SmolVLA	57.28	58.64	59.40	60.20	60.80	61.00	$+3.72$
$\pi$ 0-FAST	83.00	83.76	84.12	84.24	85.12	85.32	$+2.32$
$\pi$ 0.5	91.88	93.36	93.88	94.44	94.64	94.72	$+2.84$

Appendix

Table 9: Control success rate against step budget, LIBERO-Long, five seeds, 500 episodes per seed. 520 steps is the wrapper default and 1040 is the budget used throughout the paper. Computed offline from the per-episode step counts of the control arms, so no additional rollouts were run; the 1040 column reproduces the control success rates in Table 3 .

POLICY	SPATIAL	OBJECT	GOAL	LONG	TOTAL
SmolVLA	8,000	4,000	4,000	4,000	20,000
$\pi$ 0-FAST	2,000	2,000	2,000	1,953	7,953
$\pi$ 0.5	3,000	3,000	3,000	5,000	14,000

Appendix

Table 10: Rollout episodes per suite in each policy’s failure-head training pool. All counts are episodes of the frozen policy, pooled across seeds.

\pi

0-FAST’s LIBERO-Long count is 47 episodes short of a round 2,000 because one collection run ended early.

POLICY	TAP	EPS/TASK	CONTROL	UPLIFT	95% CI
SmolVLA	pre	10	62.20	$+3.00$	$[-3.33,+9.33]$
	post (3 seeds)	10	62.67	$+8.67$	$[-0.72,+18.06]$
	post_expert	10	60.40	$+8.20$	$[+0.28,+16.12]$
	post (headline)	50	61.00	$+7.40$	$[+4.52,+10.28]$
$\pi$ 0-FAST	pre	10	85.20	$+5.80$	$[+0.08,+11.52]$
	post	10	85.20	$+7.20$	$[+4.37,+10.03]$

Appendix

Table 11: Feature taps evaluated per policy, at each policy’s headline controller configuration on LIBERO-Long, five seeds unless noted. Rows at 10 episodes per task are screening runs, comparable only with the other 10-episode rows of the same policy; rows marked (headline) are the full-protocol arms reported in Table 3 . Uplift is in percentage points, paired by seed against each arm’s own control; intervals are 95% confidence intervals over the per-seed differences. Controls differ between runs, so compare uplifts rather than success rates across rows.

RECOVERY	SR	UPLIFT	95% CI
control	70.56
executable reversal	73.36	$+2.80$	$[+1.87,+3.73]$
instantaneous restoration	76.28	$+5.72$	$[+3.37,+8.07]$

Appendix

Table 12: Executable reversal against privileged instantaneous restoration, SmolVLA on MetaWorld, seeds 6000–10000, a 500-step episode budget, and 500 episodes per seed. Both recovery mechanisms use the same operating point (

t{=}0.8

, 100-step retreat, 100-step cooldown) and are paired by seed against a shared control. All three arms come from the same evaluation sweep; uplifts are relative to its control.

CAP	1000	2000	3000	MEAN	SD	UPLIFT	95% CI
	SUCCESS RATE BY SEED
control	93.20	95.00	94.00	94.07	0.90
1	96.80	95.60	95.40	95.93	0.76	$+1.87$	$[-1.99,+5.73]$
2	96.60	95.20	95.40	95.73	0.76	$+1.67$	$[-2.35,+5.68]$
3	96.20	97.00	95.60	96.27	0.70	$+2.20$	$[+0.41,+3.99]$
4	96.80	96.00	95.80	96.20	0.53	$+2.13$	$[-1.17,+5.44]$

Appendix

Table 13: Retreat cap on

\pi

0.5, LIBERO-Long, post head, fixed 50-step retreat, 50-step cooldown,

t=0.99

. Per-seed success rates at 50 episodes per task, and the seed-paired uplift against the shared control. Intervals are 95% confidence intervals over the three per-seed differences, using

t_{0.975,2}=4.303

; they are wide because

n=3

, and the comparison of interest is between caps rather than against zero. Control is the main evaluation’s control restricted to seeds 1000–3000, giving 94.07% success and a

+2.20

pp uplift for the headline configuration, compared with 94.72% and

+1.60

pp over all five seeds (Table 3 ). The independently evaluated cap-3 arm matches the headline arm’s per-seed success rates on these three seeds.

$t$	$a$	$s$	$cd$	SR	UPLIFT	95% CI
Control				72.32
Fixed retreat
0.8	–	100	50	74.76	$+2.44$	$[-0.65,+5.53]$
0.8	–	100	100	76.28	$+3.96$	$[+1.62,+6.30]$
0.9	–	100	50	75.44	$+3.12$	$[+1.14,+5.10]$
0.9	–	100	100	75.96	$+3.64$	$[+1.57,+5.71]$

Appendix

Table 14: Controller sweep on MetaWorld, SmolVLA, pre head. All arms use seeds 1000–5000, a 1000-step episode budget including retreat steps, and 500 episodes per seed (10 per task across all 50 tasks). Control success is 72.32%. SR is success rate (%); uplifts and intervals are in percentage points relative to this sweep’s control. Intervals are 95% confidence intervals over the five per-seed differences. Here

t

is the trigger threshold,

a

the adaptive restart threshold,

s

the fixed retreat length or adaptive minimum–maximum, and

cd

the cooldown; lengths and cooldowns are in steps.

Vision-Language-Action Models

Vision-language-action models turn visual observations and natural-language instructions into actions for robots and other embodied agents.

See all 1,186 papers

Imagining Recovery: Inference-Time Counterfactual Realignment for Vision-Language-Action Models
Aug 14, 2026Yanyan Zhang, Disheng Liu, Kai Ye +6Vision-Language-Action Models Robustness of VLA Models
Case Western Reserve University Cleveland, OH, USA
RePO-VLA: Recovery-Driven Policy Optimization for Vision-Language-Action Models
May 10, 2026Weijia Liufu, Xiaoyu Guo, Ruiyi Chen +16VLM Robustness Robot Failure Recovery
1Sun Yat-sen University · 2South China University of Technology · 3Peng Cheng Laboratory +3
What Frozen VLAs Already Know About Success: A Probing Study of Value-Like Structure in Foundation Robot Policies
May 27, 2026Jiachen Zhang, Junnan Nie, Junyi Lao +4Value Function Estimation Robotic Manipulation
Peking University · China Agricultural University · 1Peking University. +1

Aug 14, 2026J/K move · Enter open · S save

Imagining Recovery: Inference-Time Counterfactual Realignment for Vision-Language-Action Models

Yanyan Zhang, Disheng Liu, Kai Ye, Chaoda Song+5

Case Western Reserve University Cleveland, OH, USA

Vision-Language-Action Models Robustness of VLA Models

Open paper AskPDF arXiv

Vision-language-action (VLA) models have improved the flexibility and generality of robotic manipulation, yet they remain fragile to online disruptions, such as changes in task goal, scene configuration, or robot state. Existing recovery methods often require failure data, policy retraining, or external corrective agents, introducing additional data requirements and execution risks. We propose Counterfactual Realignment (CoRe), a training-free framework that recovers a frozen VLA at inference time without failure data. Upon detecting a deviation, CoRe imagines how the policy would continue toward the current goal from a recent viable state, using synthesized observations in place of physical execution, and then minimally realigns the robot and scene to rejoin this imagined continuation before returning control to the policy. Recovery is therefore planned without physical trial-and-error, preserves completed task progress, and handles both mid-episode instruction changes and physical perturbations in a unified manner. Extensive experiments across multiple simulators, VLA backbones, and real-world settings show that CoRe improves success rates by up to 85.0 percentage points to near-nominal levels while reducing physical restorations by 42.2%, without policy fine-tuning or failure-specific recovery training.

Abstract

Vision-Language-Action Models

Explore similar work

Imagining Recovery: Inference-Time Counterfactual Realignment for Vision-Language-Action Models

RePO-VLA: Recovery-Driven Policy Optimization for Vision-Language-Action Models

What Frozen VLAs Already Know About Success: A Probing Study of Value-Like Structure in Foundation Robot Policies