FailPatch: Failure Residual Patching for Vision-Language-Action Models

Peng Yu; Jiacheng Wang; Ziheng Zhang; Xuchong Zhang; Baoting Li; Zhuoyuan Yu; Yuxiang Chen; Tiancai Wang

cs.ROSep 28, 2026

FailPatch: Failure Residual Patching for Vision-Language-Action Models

Authors: Peng Yu, Jiacheng Wang, Ziheng Zhang, Xuchong Zhang, Baoting Li, Zhuoyuan Yu, Yuxiang Chen, Tiancai Wang, +1 more

Organizations: Xi’an Jiaotong University · Dexmal · Nanjing University

Abstract

Vision-Language-Action (VLA) policies are typically adapted using successful demonstrations, which provide direct action supervision but rarely cover failure-prone states. Deployment failures expose these states, yet lack the corrective actions needed for conventional supervised learning. We propose FailPatch, a failure-driven residual patching framework that decouples action supervision from execution-reliability supervision. Successful demonstrations ground how the policy should act, while deployment trajectories indicate when its behavior becomes unreliable. We further observe that action hidden representations exhibit clear linear separability between reliable and failure-associated states while directly conditioning action generation. Building on these insights, FailPatch introduces a Null-gated Residual Expert Bank into the action hidden space of a frozen VLA policy. A unified Preserve--Redirect--Trust objective retains the original policy in reliable states, selects residual experts in failure-associated states and redirects representations from failure regions toward success-associated regions under bounded intervention. With only 0.52% trainable parameters, FailPatch improves success rates by 11.0 percentage points on four long-horizon RoboTwin tasks under clean evaluation, 9.5 percentage points under clean-to-random generalization, and 16.7 percentage points over the baseline across three real-world tasks. Project and code: https://github.com/yupeng-2003/FailPatch.

Figures & tables

Figure 1 : Comparison of VLA policy-learning paradigms. Success-only SFT learns from demonstrations, rollout-based RL optimizes from rewarded interaction, and FailPatch jointly exploits successful demonstrations and deployment rollouts through lightweight residual patching.

Figure 2 : Overview of FailPatch. (A) A frozen VLA policy collects deployment rollouts, which an offline VLM converts into chunk-level reliability labels. (B) A Null-gated Residual Expert Bank is trained in the action hidden space using action supervision from successful demonstrations and Preserve–Redirect–Trust constraints from deployment experience. (C) At inference, the router either retains the base representation through the Null expert or applies a selected residual patch.

Method	Blocks Ranking	Stack Bowls	Put Object Cabinet	Place Bread Basket	Avg.
Without rollout trajectories
50-Task Base	38	68	40	52	49.50
LoRA SFT	44	68	36	54	50.50
Full SFT	44	70	42	55	52.75
Demo-Only REB	42	64	36	55	49.25
With rollout trajectories

Table 1: Success rates (%) under clean evaluation.

Method	Blocks Ranking	Stack Bowls	Object Cabinet	Bread Basket	Avg.
50-Task Base	12	39	25	36	28.00
LoRA SFT	13	33	20	33	24.75
Full SFT	23	45	29	34	32.75
Demo-Only REB	20	39	31	32	30.50
Outcome SFT (Ep.)	10	29	28	38	26.25
Outcome SFT (Ch.)	16	35	32	32	28.75

Table 2: Clean-to-random success rates (%).

Figure 3 : Adaptation performance and efficiency. Position denotes training time and average clean success rate, while bubble size represents the number of trainable parameters.

Figure 4 : Illustrations of the three real-world manipulation tasks. From top to bottom: Blocks Ranking, Stack Bowls, and Fold Dishcloth.

Method	Blocks	Stack	Fold	Avg.
Method	Ranking	Bowls	Dishcloth	Avg.
Three-Task Base	36.7	53.3	33.3	41.1
LoRA SFT	46.7	50.0	40.0	45.6
Demo-Only REB	43.3	50.0	36.7	43.3
FailPatch	56.7	66.7	50.0	57.8

Table 3 : Real-world success rates (%) over 30 trials per task.

Variant	Avg. Success (%)
Objective components
Demo-Only REB	49.25
+ Preserve	52.50
+ Redirect	56.50
+ Trust	50.25
+ Preserve + Redirect	59.00

Table 4: Ablation results averaged across simulation tasks.

Figure 5 : Linear separability of nominal and failure-associated chunks across representation spaces. Shuffled labels serve as a chance-level control.

Panel of Figure 6 — Figure 6 : (a) Soft routing probabilities along a representative episode. (b) Distance to the local success prototype before and after patching.

Panel of Figure 7 — Figure 7 : Residual-expert diversity at localized failure onsets.

Figure 8 : Representative Blocks Ranking rescue episode. Left: chunks 5–7 from the same initial configuration. Right: 2D TCP trajectories, where FailPatch makes a small local correction near the green block.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Algorithm 1: FailPatch offline preparation, joint training, and inference
Input: frozen policy $F_{\phi}$ and action projection $W_{v}$ ; demonstrations $\mathcal{D}_{\mathrm{demo}}$ ; deployment rollouts $\mathcal{B}$ ; $K$ experts $\{f_{k}\}_{k=1}^{K}$ ; router $g_{\theta}$ .
1	Given the terminal outcome $y_{i}$ , annotate each rollout $i\in\mathcal{B}$ to obtain failure onset $\widehat{t}_{i}^{\mathrm{fail}}$ for failed rollouts and success reliability $\gamma_{i}$ for successful rollouts.
2	Cache $h_{i,t}=F_{\phi}(o_{i,t},\ell_{i})$ for all deployment chunks. Form $\mathcal{H}^{+}$ from reliable-success chunks and failed-rollout pre-onset chunks; form $\mathcal{H}^{-}$ from onset/post-onset chunks.
3	For each task $\tau$ , construct $\mathcal{P}_{\tau}$ from normalized chunks of reliable successful rollouts only.
4	For $n=1,\ldots,30{,}000$ :
5	Sample task-balanced chunks for the demonstration, Preserve, and Redirect branches according to the implementation, without trajectory-duration reweighting.

Appendix

Table 13

Configuration	Value
Number of residual experts $K$	8
Nearest neighbors $K_{\mathrm{nn}}$	20
Demo routing weight $\lambda_{rd}$	1.0
Preserve weight $\lambda_{P}$	0.2
Redirect weight $\lambda_{R}$	0.2
Trust weight $\lambda_{T}$	0.05

Appendix

Table 5: Default FailPatch training configuration.

Setting	Task	Successful rollouts	$\gamma=1$	$\gamma=0$
Simulation	Blocks Ranking	50	44	6
Simulation	Stack Bowls	50	46	4
Simulation	Put Object Cabinet	50	43	7
Simulation	Place Bread Basket	50	45	5
Real world	Blocks Ranking	50	44	6
Real world	Stack Bowls	50	45	5

Appendix

Table 6 : Reliable-success filtering statistics.

Task	$N$	Exact	$\|\Delta\|\leq 1$	$\|\Delta\|\leq 2$	MAE	Bias
Blocks Ranking	50	28	40	47	0.72	$+0.14$
Stack Bowls	50	30	42	48	0.68	$+0.10$
Put Object Cabinet	50	26	39	48	0.80	$+0.20$
Place Bread Basket	50	28	39	47	0.76	$+0.12$
Overall	200	112	160	190	0.74	$+0.14$

Appendix

Table 7 : Agreement between GPT-5.5 and human failure-onset annotations. Errors are measured in action chunks.

Figure 10 : Representative execution sequences under clean and random evaluation for Blocks Ranking, Stack Bowls, Put Object Cabinet, and Place Bread Basket. Object pose and identity vary in both settings, while the random setting also changes background textures, clutter, table height, and lighting.

Task	Max steps	Max chunks	Success predicate
Blocks Ranking	1200	24	For the two adjacent block pairs (red, green) and (green, blue), $\|\Delta x\|<0.13$ m and $\|\Delta y\|<0.03$ m; the centers satisfy $x_{\mathrm{red}}<x_{\mathrm{green}}<x_{\mathrm{blue}}$ ; and both grippers are open.
Stack Bowls	1200	24	After sorting the bowls by height, every pair has planar center distance below 0.04 m; their heights are within 0.02 m of $[0.74,0.77,0.81]+\texttt{table\_z\_bias}$ ; and both grippers are open.
Put Object Cabinet	700	14	The planar distance from the object to the drawer functional point is below 0.05 m; its relative height satisfies $0.007<\Delta z<0.12$ m; and the executing gripper is open.
Place Bread Basket	700	14	Every bread object’s planar distance to the basket center is below 0.05 m, its height exceeds $0.73+\texttt{table\_z\_bias}$ , and both grippers are open.

Appendix

Table 8: Simulation horizons and exact success predicates.

Task	Randomized task instance
Blocks Ranking	Each block is initialized with $x\in[-0.28,0.28]$ m, $y\in[-0.08,0.05]$ m, $z=0.765$ m, and rotation about the vertical axis up to 0.75 rad. The block side length is sampled uniformly from $[0.03,0.05]$ m. Target $x$ ranges are $[-0.09,-0.08]$ , $[-0.01,0.01]$ , and $[0.08,0.09]$ m for red, green, and blue, respectively, with common target $y\in[-0.20,-0.10]$ m.
Stack Bowls	Bowl centers are initialized with $x\in[-0.30,0.30]$ m and $y\in[-0.15,0.15]$ m. Bowl identities are assigned according to their initial $y$ ordering, and the target stack center is $(0,-0.10)$ m.
Put Object Cabinet	The cabinet has $x\in[-0.05,0.05]$ m and $y=0.155$ m. The manipulated object and model instance are sampled from ten object categories. The object has $x\in[-0.25,0.25]$ m, $y\in[-0.20,-0.10]$ m, and yaw up to $\pi/3$ ; when the initially sampled $\|x\|<0.20$ m, the implementation widens the admissible $x$ range to $[-0.32,0.32]$ m.
Place Bread Basket	The basket center is fixed at $(0,-0.20)$ m, while its yaw is sampled from $[0,\pi]$ and its model ID from $\{0,1,2,3,4\}$ . Each instance contains one or two bread objects with $x\in[-0.27,0.27]$ m, $y\in[-0.20,0.05]$ m, yaw up to $\pi/4$ , and model ID sampled from $\{0,1,3,5,6\}$ .

Appendix

Table 9: Task-instance randomization applied independently for every seed in both clean and random evaluation.

Attribute	Clean	Random
Background	Default wall and table textures; clean-background probability 1.0.	Unseen evaluation texture library; clean-background probability 0.02.
Table clutter	Disabled.	Enabled, with up to approximately ten additional distractor objects.
Table height	No perturbation.	Downward height perturbation of up to 0.03 m.
Lighting	Fixed.	Randomized; extreme lighting is sampled with probability 0.02.
Head-camera distance	No distance perturbation.	No distance perturbation.
Camera observations	One D435 head view and two wrist views.	Identical camera configuration.

Appendix

Table 10: Difference between the clean and random simulation configurations. Object pose/type randomization is active in both settings.

Method	Trainable params	Fraction	Time	Offline rollouts/task	Fresh rollouts
LoRA SFT	50.0M	1.49%	8.3 h	0	0
Full SFT	3353.4M	100%	11.2 h	0	0
Demo-Only REB	17.3M	0.52%	3.1 h	0	0
Outcome-Prompt SFT (Episode)	17.3M	0.52%	5.5 h	100	0
Outcome-Prompt SFT (Chunk)	17.3M	0.52%	4.7 h	100	0
PPO	694M	20.7%	20.8 h	0	12,800

Appendix

Table 11: Trainable parameters, wall-clock adaptation time, and rollout usage. The PPO rollout count is the total over 100 updates.

Prototype variant	Blocks	Stack	Cabinet	Bread	Average
Default: $K_{\mathrm{nn}}=20$ , median	54	77	49	62	60.50
$K_{\mathrm{nn}}=1$	48	71	44	58	55.25
$K_{\mathrm{nn}}=10$	53	76	48	61	59.50
$K_{\mathrm{nn}}=50$	52	75	47	61	58.75
Mean aggregation	52	75	47	60	58.50
Unnormalized features	51	74	46	60	57.75

Appendix

Table 12: Prototype-construction ablations under clean simulation evaluation. The default uses a task-specific reliable-success pool, normalized features, Euclidean retrieval, and coordinate-wise median aggregation.

Hyperparameter	Evaluated values	Default	Average success (%)
Experts $K$	4, 8, 16	8	54.25, 60.50 , 58.00
Preserve weight $\lambda_{P}$	0.1, 0.2, 0.4	0.2	57.50, 60.50 , 58.25
Redirect weight $\lambda_{R}$	0.1, 0.2, 0.4	0.2	56.00, 60.50 , 59.25
Trust weight $\lambda_{T}$	0.02, 0.05, 0.10	0.05	58.75, 60.50 , 57.00
Demo temperature $T_{d}$	annealed, 0.2, 1.0	annealed	60.50 , 59.00, 55.50
Failure temperature $T$	0.1, 0.2, 0.5	0.2	59.25, 60.50 , 57.75

Appendix

Table 13: One-dimensional hyperparameter sensitivity under clean simulation evaluation.

Variant	Blocks	Stack	Cabinet	Bread	Average
Demo-Only REB	42	64	36	55	49.25
$+$ Preserve	50	63	38	59	52.50
$+$ Redirect	52	68	42	64	56.50
$+$ Trust	44	62	37	58	50.25
$+$ Preserve $+$ Redirect	54	74	48	60	59.00
Parameter-Matched Single Expert	50	71	44	58	55.75

Appendix

Table 14: Per-task objective, architecture, and annotation ablations under clean simulation evaluation.

Figure 11 : Effect of failure-data scale on the average success rate across the four simulation tasks. The number of failed deployment episodes is varied independently for each task, and the 50-episode result corresponds to the full FailPatch setting.

Task	Language instruction	Success criterion
Blocks Ranking	“Position red block, green block, and blue block in a left-to-right sequence, forming a row.”	The three blocks form one row in red–green–blue order from left to right.
Stack Bowls	“Stack the three bowls on the top of each other.”	The task ends successfully once all three bowls have been stacked; no additional fixed-duration stability test is imposed.
Fold Dishcloth	“Place the dishcloth in the center of the table, then fold it in half twice.”	The dishcloth is first placed at the table center and then successfully folded in half twice.

Appendix

Table 15: Language instructions and success criteria for the real-world tasks.

Feature	Balanced accuracy	ROC-AUC
Vision	0.73	0.79
Action hidden	0.74	0.83
Predicted action	0.62	0.69
Shuffled labels	0.51	0.52

Appendix

Table 16: Aggregate linear-probe results.

Figure 12 : Real-robot platform. The annotations identify the head-mounted D435i, dual arms, and two wrist cameras.

Explore similar work

RePO-VLA: Recovery-Driven Policy Optimization for Vision-Language-Action Models
May 10, 2026Weijia Liufu, Xiaoyu Guo, Ruiyi Chen +16Frictive Policy Optimization Imitation
1Sun Yat-sen University · 2South China University of Technology · 3Peng Cheng Laboratory +3
ReCoVLA: VLM-Guided Reward Compilation for Failure Recovery in Vision-Language-Action Policies
Jun 8, 2026Haodi Hu, Chung-Ta Huang, Jing Liu +4Residual Policy
University of Southern California, USA · Mitsubishi Electric Research Laboratories (MERL), USA · Harvard University, USA
Failing Forward: Adaptive Failure-Informed Learning for Vision-Language-Action Models
May 8, 2026Meng Zheng, Samhita Marri, Anwesa Choudhuri +6Action Generation Negative Results
United Imaging Intelligence, Boston, MA, USA · University of Illinois Urbana-Champaign, Urbana, IL, USA

May 10, 2026, cs.ROJ/K move · Enter open · S save

RePO-VLA: Recovery-Driven Policy Optimization for Vision-Language-Action Models

Weijia Liufu, Xiaoyu Guo, Ruiyi Chen, Jingzhi Liu+15

1Sun Yat-sen University · 2South China University of Technology · 3Peng Cheng Laboratory+3

Frictive Policy Optimization Imitation

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models remain brittle in long-horizon, contact-rich manipulation because success-only imitation provides little supervision for execution drift, while failed rollouts are often discarded. We introduce RePO-VLA, a recovery-driven policy optimization framework that assigns distinct roles to success, recovery, and failure trajectories. RePO-VLA first applies Recovery-Aware Initialization (RAI), slicing recovery segments and resetting history so corrective actions depend on the current adverse state rather than the preceding failure. It then learns a Progress-Aware Semantic Value Function (PAS-VF), aligning spatiotemporal trajectory features with instructions and successful references. The resulting labels salvage useful failure prefixes via reliability decay, while low-value labels mark drift and terminal breakdowns, teaching differences among nominal, failed, and corrective actions. The data engine turns adverse states into planner-generated or human-collected corrective rollouts, teaching recovery to the success manifold. Value-Conditioned Refinement (VCR) trains the policy to prefer high-progress actions. At deployment, a fixed high value (

v=1.0

) biases actions toward the learned success manifold without online failure detectors or heuristic retries. We introduce FRBench, with standardized error injection and recovery-focused evaluation. Across simulated and real-world bimanual tasks, RePO-VLA improves robustness, raising adversarial success from 20% to 75% on average and up to 80% in scaled real-world trials.

Abstract

Explore similar work

RePO-VLA: Recovery-Driven Policy Optimization for Vision-Language-Action Models

ReCoVLA: VLM-Guided Reward Compilation for Failure Recovery in Vision-Language-Action Policies

Failing Forward: Adaptive Failure-Informed Learning for Vision-Language-Action Models