Where Success Breaks: Failure-Boundary Learning for Robust Vision-Language-Action Models

Yanzhe Chen; Zhijun Cao; Mike Zheng Shou

cs.ROSep 5, 2026

Where Success Breaks: Failure-Boundary Learning for Robust Vision-Language-Action Models

Authors: Yanzhe Chen, Zhijun Cao, Mike Zheng Shou

Organizations: Showlab, National University of Singapore

Abstract

Vision-language-action (VLA) models adapted through supervised fine-tuning (SFT) inherit a structural asymmetry: expert demonstrations teach the policy where success behavior lies, but provide no signal about where it ceases to be reliable. We argue that robust VLA adaptation should therefore be viewed not as further demonstration fitting, but as Failure-Boundary Learning -- the problem of Discovering, Localizing, and Shaping the boundary between recoverable deviations and task failure. To instantiate this view, we propose DLS: built on a real-grounded behavioral prior from few real demonstrations and simulated co-training, DLS discovers failure boundaries at scale through on-policy digital twin rollouts. Rather than reducing each rollout to a binary label, semantic progress localization uses privileged simulator states to assign progress-aware signals that capture where the failure boundary is crossed, not merely whether. These signals drive directional boundary shaping in the flow dynamics -- reinforcing success-producing denoising directions and suppressing failure-producing ones, without action likelihoods or auxiliary critics. Across real-robot manipulation tasks, DLS improves robustness over SFT and online RL baselines, especially under randomized initial states and unseen visual conditions.

Figures & tables

Figure 1: Failure-Boundary Learning for Flow-based VLAs. Three bottlenecks cripple current post-SFT adaptation: asymmetric supervision , missing progress signal , and generation-aware mismatch (top). Our DLS pipeline resolves them in turn (bottom): scalable on-policy rollouts of a real-grounded prior, simulator-derived progress labels, and push–pull shaping in the velocity field.

Figure 2: Overview of DLS Pipeline. The rollout policy from real-grounded behavioral prior interacts with the digital twin; semantic progress localization assigns progress-aware labels to trajectories. Two mirrored branches are constructed around the stored rollout velocity, whose Mahalanobis errors drive directional boundary shaping , pulling the flow toward success-producing directions and pushing it away from failure-producing ones, anchored by real-data SFT regularization.

Figure 3: Visualization of Real-robot Tasks Execution.

Model	Training Setting	Pick & Place		Sweep		Plugin		Avg. ( $\uparrow$ )
Model	Training Setting	ID.	Unseen.	ID.	Unseen.	ID.	Unseen.	ID.	Unseen.
$\pi_{0}$	# Full SFT
	Real-only ( N = 50 )	20/30	18/30	16/30	14/30	22/30	13/30	64.4 [54.1, 73.6]	50.0 [39.9, 60.1]
	Real-only ( N = 100 )	25/30	19/30	20/30	15/30	24/30	15/30	76.7 [66.9, 84.2]	54.4 [44.2, 64.3]
	# Few-shot SFT + RL
	Sim-Real SFT ( N = 50 )	20/30	15/30	13/30	12/30	17/30	9/30	55.6 [45.3, 65.4]	40.0 [30.5, 50.3]

Table 1: Main real-robot results across three manipulation tasks under in-distribution ( ID. ) and unseen conditions ( Unseen. ). Task cells: successes out of 30 real-world rollouts; Avg.: success rate (%) over the 90 rollouts with its Wilson 95% interval (per-cell intervals in App. Tab. 6 );

N

= real demonstrations. Rewards: +GRPO and +PPO optimize the sparse binary task outcome, with group-normalized and critic-estimated advantages respectively; DLS uses SPL labels (Eq. ( 7 )). Within each backbone, bold : highest per column; underline : second best.

Panel of Figure 4 — Figure 4: SPL vs. Binary Reward: Training Curves on $\pi_{0.5}$ . Simulation success rate over training (128 rollouts per evaluation; 40 gradient updates per step). Shaded regions denote $\pm 1$ standard deviation. SPL achieves consistently faster convergence and higher final success across all tasks.

Method	Pick & Place		Sweep		Plugin
Method	Unseen SR. ( $\uparrow$ )	Hor. ( $\downarrow$ )	Unseen SR. ( $\uparrow$ )	Hor. ( $\downarrow$ )	Unseen SR. ( $\uparrow$ )	Hor. ( $\downarrow$ )
Step-count Penalty	$60.0\pm 3.3$	279.5	$53.3\pm 3.3$	296.1	$56.7\pm 3.3$	283.9
Distance-based Potential	$66.7\pm 10.0$	287.5	$50.0\pm 9.0$	301.4	$63.3\pm 9.0$	292.0
SPL (ours)	$\mathbf{80.0}\pm 3.3$	280.3	$\mathbf{66.7}\pm 3.3$	300.1	$\mathbf{70.0}\pm 3.3$	287.4

Table 2: SPL vs. alternative progress-aware designs on

\pi_{0.5}

. Success rate ( Unseen SR. , %, mean

\pm

std over 3 seeds) and average episode horizon ( Hor. ) across all tasks; bold : highest SR. per task.

Signal	Agree. (%) $\uparrow$	$\kappa\,\uparrow$	$\|\Delta i\|\,\downarrow$	Unseen SR $\uparrow$
Majority phase	52.1	0.02	41	—
Step-count	41.2	0.18	62	56.7
Distance potential	46.4	0.31	34	60.0
SPL (ours)	78.1	0.68	19	72.2

Table 3: Boundary localization probe (

\pi_{0.5}

, 192 failed rollouts): agreement with the true failure phase

z^{\star}_{\mathrm{rec}}

, Cohen’s

\kappa

, median step gap

|\Delta i|

to the crossing, and average Unseen SR (Tab. 2 ).

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Visualization of Task Execution in Digital Twin. We visualize the tasks in constructed digital twins, showing for each task a real data reference ( Real ), a representative simulated rollout ( Simulation ), and the corresponding task instruction prompt ( Task Prompt ).

Task	$z$	$w_{z}$	Phase	Gate Predicate ( $z\to z{+}1$ or Success)	$\phi_{z}(s)$
Pick & Place [2pt] $\mathrm{obj}(1)$ :Block $\mathrm{obj}(2)$ :Coaster	1	0.10	Reach	$\\|p_{\mathrm{tcp}}-p_{\mathrm{obj}(1)}\\|\leq 0.015$	$1-\\|p_{\mathrm{tcp}}-p_{\mathrm{obj}(1)}\\|\,/\,d_{1}$
	2	0.25	Grasp	$d_{\mathrm{grasp}}\leq 0.04\,\wedge\,F_{\mathrm{grasp}}>0$	$1-\|d_{\mathrm{grasp}}-0.04\|\,/\,0.04$
	3	0.20	Transit	G $\,\wedge\,\\|p_{\mathrm{tcp}}^{xy}-p_{\mathrm{obj}(2)}^{xy}\\|\leq 0.08\,\wedge\,r_{\mathrm{obj}(1)}^{\mathrm{roll/pitch}}\leq 45^{\circ}$	$1-\\|p_{\mathrm{tcp}}^{xy}-p_{\mathrm{obj}(2)}^{xy}\\|\,/\,d_{3}$
	4	0.45	Place	$\mathbb{I}_{\mathrm{goal}}:\\|p_{\mathrm{obj}(1)}^{xy}-p_{\mathrm{obj}(2)}^{xy}\\|\leq 0.025\,\wedge\,r_{\mathrm{obj}(1)}^{\mathrm{roll/pitch}}\leq 5^{\circ}$	$1-\\|p_{\mathrm{obj}(1)}^{z}-0.025\\|\,/\,d_{4}$
Sweep [2pt] $\mathrm{obj}(1)$ :Broom $\mathrm{obj}(2)$ :Cube $\mathrm{obj}(3)$ :Dustpan	1	0.08	Reach	$\\|p_{\mathrm{tcp}}^{yz}-p_{\mathrm{obj}(1)}^{yz}\\|\leq 0.01\,\wedge\,\\|p_{\mathrm{tcp}}^{x}-p_{\mathrm{obj}(1)}^{x}\\|\leq 0.02$	$1-\\|p_{\mathrm{tcp}}^{xyz}-p_{\mathrm{obj}(1)}^{xyz}\\|\,/\,d_{1}$
	2	0.20	Grasp	$d_{\mathrm{grasp}}\leq 0.03\,\wedge\,F_{\mathrm{grasp}}>0$	$1-\|d_{\mathrm{grasp}}-0.03\|\,/\,0.03$

Appendix

Table 4: SPL Phase Specifications Per Task . Gate predicate : condition at phase

z

that triggers transition to

z{+}1

(or task success). Each pose

p=(xyz,\,r)

decomposes into position

p^{xyz}_{\mathrm{obj}}\in\mathbb{R}^{3}

and Euler-angle orientation

r_{\mathrm{obj}}\in\mathbb{R}^{3}

; superscripts on

p

select position components (e.g.

p^{xy}

). Notation:

d_{z}

initial TCP–target distance at phase

z

entry;

F_{\mathrm{grasp}}

grasping contact force;

\mathcal{G}

the grasped mark;

\dot{p}_{\mathrm{obj(s)}}

object velocity; All distances in metres; velocities in m/s; angles in degrees.

Parameter	Pick & Place	Sweep	Plugin
Model Config
Denoising steps $T$	5	5	5
Action chunk / horizon	8 / 8	8 / 12	8 / 8
Control frequency	10 Hz (stride 2)	10 Hz (stride 2)	10 Hz (stride 2)
Sim-Real SFT
Base model	lerobot/pi05_base	lerobot/pi05_base	lerobot/pi05_base

Appendix

Table 5: Training Hyperparameters. Parameters are grouped into three blocks: shared model configuration ( Model Config ), stage 1 ( Sim-Real SFT ), and stage 2 ( DBS ).

Figure 9: Visualization of Real-robot Rollouts. Each row shows one task with three representative trials. The leftmost column of each trial depicts the initial object position; subsequent columns show the unfolding rollout, capturing key interaction stages through to task completion.

Figure 10: SFT Prior Failure Modes under Unseen Conditions. (a)–(b): Grasp -phase failure from lighting-induced approach offset. (c): pre-contact positioning failure before Contact phase. (d): terminal push direction misalignment at Push phase. (e)–(f): terminal near-miss at Insert phase.

Training Setting	Pick & Place		Sweep		Plugin		Avg.
Training Setting	ID.	Unseen.	ID.	Unseen.	ID.	Unseen.	ID.	Unseen.
$\pi_{0}$
Real-only ( N = 50 )	48.8–80.8	42.3–75.4	36.1–69.8	30.2–63.9	55.6–85.8	27.4–60.8	54.1–73.6	39.9–60.1
Real-only ( N = 100 )	66.4–92.7	45.5–78.1	48.8–80.8	33.2–66.8	62.7–90.5	33.2–66.8	66.9–84.2	44.2–64.3
Sim-Real SFT ( N = 50 )	48.8–80.8	33.2–66.8	27.4–60.8	24.6–57.7	39.2–72.6	16.7–47.9	45.3–65.4	30.5–50.3
+ GRPO [ 29 ]	55.6–85.8	52.1–83.3	42.3–75.4	27.4–60.8	62.7–90.5	42.3–75.4	61.0–79.5	47.5–67.5

Appendix

Table 6: Wilson 95% intervals (%) for Tab. 1 . Task cells:

n{=}30

; averages:

n{=}90

Variant	Agree. $\uparrow$	Sim. SR $\uparrow$
$\mathbf{w}$ : 12 Dirichlet draws	—	78.0 $\pm$ 5.4
$\mathbf{w}$ : $\pm 30\%$ / $\pm 50\%$	—	77.5 / 74.8
$\mathbf{w}$ : order reversed	—	63.3
Gate thresholds $\times[0.5,1.5]$	66.2	75.2
Merged phases ( $Z{=}2$ )	44.7	71.0
SPL (ours) / Terminal-only	78.1 / —	79.4 / 67.7

Appendix

Table 7: Design sensitivity of SPL (

\pi_{0.5}

): boundary-localization agreement (%, App. B.5 ) and simulated success rate (%).

Explore similar work

Failing Forward: Adaptive Failure-Informed Learning for Vision-Language-Action Models
May 8, 2026Meng Zheng, Samhita Marri, Anwesa Choudhuri +6Action Generation Negative Results
United Imaging Intelligence, Boston, MA, USA · University of Illinois Urbana-Champaign, Urbana, IL, USA
FailPatch: Failure Residual Patching for Vision-Language-Action Models
Sep 28, 2026Peng Yu, Jiacheng Wang, Ziheng Zhang +6Photometric Supervision Robot State
Xi’an Jiaotong University · Dexmal · Nanjing University
RePO-VLA: Recovery-Driven Policy Optimization for Vision-Language-Action Models
May 10, 2026Weijia Liufu, Xiaoyu Guo, Ruiyi Chen +16Frictive Policy Optimization Imitation
1Sun Yat-sen University · 2South China University of Technology · 3Peng Cheng Laboratory +3

May 8, 2026, cs.ROJ/K move · Enter open · S save

Failing Forward: Adaptive Failure-Informed Learning for Vision-Language-Action Models

Meng Zheng, Samhita Marri, Anwesa Choudhuri, Benjamin Planche+5

United Imaging Intelligence, Boston, MA, USA · University of Illinois Urbana-Champaign, Urbana, IL, USA

Action Generation Negative Results

Open paper AskPDF arXiv

Vision-language-action (VLA) models provide a promising paradigm for scalable robotic manipulation, yet their reliance on success-only behavioral cloning leaves them brittle; lacking corrective training signals, minor execution errors rapidly compound into unrecoverable, out-of-distribution failures. To address this limitation, we propose Adaptive Failure-Informed Learning (AFIL), an end-to-end framework that leverages failure trajectories as adaptive negative guidance for diffusion- and flow-based VLA policies. AFIL uses a pretrained VLA to generate failure rollouts online, avoiding the need for handcrafted failure-mode design or human-in-the-loop recovery. It then jointly trains Dual Action Generators (DAGs) for successful and failed behaviors while sharing a common vision-language backbone, enabling efficient failure-aware policy learning with limited parameter overhead. During sampling, the failure generator adaptively steers action generation away from failure-prone regions and toward more reliable success modes, with guidance strength determined by the per-diffusion-step distance between success and failure distributions. Experiments across in-domain and out-of-domain robotic manipulation tasks, covering both short- and long-horizon settings, show that AFIL consistently improves task success rates and robustness over existing VLA baselines, demonstrating its effectiveness, efficiency, and generality.

Abstract

Explore similar work

Failing Forward: Adaptive Failure-Informed Learning for Vision-Language-Action Models

FailPatch: Failure Residual Patching for Vision-Language-Action Models

RePO-VLA: Recovery-Driven Policy Optimization for Vision-Language-Action Models