Beyond Policy Patches: Distilling RL Specialists into Vision-Language-Action Policies

Zhennan Jiang; Tingying Xie; Shilong Ji; Shu'ang Yu; Tianxing Zhou; Yuhui Chen; Jiahuan Gao; Kaizhi Xie

cs.RONov 2, 2026

Beyond Policy Patches: Distilling RL Specialists into Vision-Language-Action Policies

Authors: Zhennan Jiang, Tingying Xie, Shilong Ji, Shu'ang Yu, Tianxing Zhou, Yuhui Chen, Jiahuan Gao, Kaizhi Xie,

Organizations: University of Chinese Academy of Sciences · Institute of Automation, Chinese Academy of Sciences · Zhongguancun Academy · Striding AI ·

Abstract

Vision-language-action (VLA) policies can handle noncritical stages of long-horizon manipulation yet fail during short precision-critical phases. Lightweight reinforcement learning (RL) specialists can address these failures without directly applying RL to the full VLA, but retain auxiliary controllers and switching mechanisms at deployment. We study how to transfer their capabilities into a single flow-matching VLA. Direct imitation of specialist rollouts can reproduce action jitter and leave the VLA unprepared for states encountered during its own execution. We introduce REPAIR, a value-guided distillation framework that combines offline initialization with on-policy refinement. At each critical-phase observation, REPAIR queries the corresponding specialist for multiple action candidates and weights them using its critic to construct a supervision distribution. A shared weighted flow-matching objective trains the VLA first on observations from specialist-assisted rollouts, then on observations collected through its own execution. Specialists provide supervision without taking control during on-policy collection and are removed at deployment. Across six real-robot tasks with 30 evaluation trials per task, REPAIR achieves 100% success on all six tasks, with lower mean end-effector jerk than specialist-assisted execution on every task.

Figures & tables

Figure 1: Overview of REPAIR. Local RL expertise provides value-guided supervision for offline initialization and on-policy repair. The VLA acts without specialist takeover during online distillation and is deployed as a single policy.

Figure 2: Specialist deployment across task horizons. (a) Peg-insertion settings. (b) Success rates for full and critical-phase takeover. (c) Position traces with local magnification.

Figure 3: State coverage after offline distillation. PCA of RL-token observations. Shading marks 95% KDE regions.

Figure 4: Training framework of REPAIR. (a) Human demonstrations initialize the base VLA $\pi_{0.5}$ . (b) With this VLA frozen, critical-phase RL trains multiple specialist actors and their critics. (c) Offline observations from $\mathcal{D}_{E}$ and on-policy observations from $\mathcal{D}_{\pi}$ feed a shared value-refined supervision mechanism. For each critical observation, the selected phase specialist generates the action candidates and its critic determines their relative weights. The VLA learns from these targets in critical phases and replayed VLA actions in noncritical phases. Only the VLA is deployed.

Panel of Figure 5 — Figure 5: The tasks in our experiments.

Method	SR $\uparrow$	Jerk $\downarrow$	SR $\uparrow$	Jerk $\downarrow$	SR $\uparrow$	Jerk $\downarrow$	SR $\uparrow$	Jerk $\downarrow$	SR $\uparrow$	Jerk $\downarrow$	SR $\uparrow$	Jerk $\downarrow$
	Stick P&P		Single peg		Dual peg		Bagging		Table clearing		Drawer
Base SFT ( $\pi_{0.5}$ )	30.0	$4.2\pm 1.2$	33.3	$3.3\pm 0.5$	0.0	–	13.3	$4.7\pm 0.9$	20.0	$4.7\pm 0.8$	0.0	–
RLT	100.0	$15.7\pm 2.0$	100.0	$4.7\pm 0.8$	100.0	$10.3\pm 0.9$	100.0	$7.2\pm 0.8$	100.0	$10.4\pm 2.5$	100.0	$16.7\pm 5.1$
EXPO-FT	80.0	$13.5\pm 0.8$	93.3	$4.2\pm 1.0$	33.3	$9.0\pm 1.3$	0.0	–	0.0	–	0.0	–
ConRFT	93.3	$12.8\pm 2.3$	100.0	$3.9\pm 0.6$	20.0	$7.1\pm 2.4$	0.0	–	0.0	–	0.0	–
HG-DAgger	100.0	$5.9\pm 1.2$	60.0	$3.1\pm 0.6$	63.3	$6.3\pm 1.1$	90.0	$4.4\pm 0.8$	63.3	$4.4\pm 0.6$	80.0	$10.1\pm 1.8$

Table 1: Main results on six real-robot tasks (30 trials per method per task). SR: success rate (%, higher is better). Jerk: mean

\pm

standard deviation over successful trials (

\mathrm{m\,s^{-3}}

, lower is better), with identical sampling and filtering across methods. Dashes denote no successful trials.

Figure 6: Dual-arm peg insertion episode lengths (30 trials per method; 100% success).

Table 8

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Value
Specialist candidates per critical observation	8
Q-weight temperature $\tau$	0.1
Reference dropout probability	0.5
Optimizer	AdamW
Learning rate / schedule	$5\times 10^{-6}$ / constant
AdamW $(\beta_{1},\beta_{2})$ / $\epsilon$	$(0.9,0.95)$ / $10^{-8}$

Appendix

Table 4: Distillation hyperparameters for REPAIR. Batch size counts observations before candidate expansion.

Setting	Value
RL-token	2048
proprioceptive dimensions	20
Action dimension	20
chunk horizon	10
Actor / critic hidden widths	$(512,512,512)$ / $(512,512,512)$
Reference dropout probability	0.5

Appendix

Table 5: RLT training hyperparameters.

Figure 7: Single-Arm Peg Insertion failure. Misalignment prevents the peg from entering the opening and completing insertion within the allotted time.

Panel of Figure 8 — Figure 8: Stick Pick-and-Place. Left: the red boundaries mark the object initialization region. Right: the gripper fails to establish a secure grasp. Dimensions and region annotations are retained from the supplied task illustration.

Figure 9: Dual-Arm Peg Insertion failure. Misalignment between the held components prevents the peg from entering the socket despite repeated attempts.

Panel of Figure 10 — Figure 10: Toy Stowing in a Drawer failures. Left: the toy is deposited on top of the drawer housing. Right: the drawer remains partially open after the closing attempt.

Panel of Figure 11 — Figure 11: Grocery Bagging setup and failures. Top left: the red region defines randomized object initialization. Top right: the right gripper fails to secure the chip container. Bottom left: inaccurate alignment prevents placing the container through the bag opening. Bottom right: a chip packet presses against the right rim of the bag and fails to enter. The bottom-right panel is an AI-enhanced illustration based on the supplied experimental frame.

Panel of Figure 12 — Figure 12: Bimanual Table Clearing setup and failures. Top: the red region defines bowl initialization. Bottom left: the right gripper misses the blue bowl’s rim. Bottom right: the left gripper fails to secure the bowl during handover, causing it to fall when released.

Human trajectories	Dual peg SR $\uparrow$	Table clearing SR $\uparrow$
50	0.0	–
100	0.0	20.0
150	–	26.7
200	0.0	–
300	0.0	30.0

Appendix

Table 6: Human-only SFT scaling on two long-horizon tasks (30 evaluation trials per setting). Trajectory counts refer to human teleoperation demonstrations. The main experiments use 100 demonstrations for each of these tasks; other buffer sizes are supplementary controls. SR is reported in %; dashes denote untested settings.

Robotic Manipulation

Robotic manipulation studies how robots perceive, plan, and control physical interactions to grasp, move, assemble, and use objects.

See all 1,736 papers

RePO-VLA: Recovery-Driven Policy Optimization for Vision-Language-Action Models
May 10, 2026Weijia Liufu, Xiaoyu Guo, Ruiyi Chen +16VLM Robustness Robot Failure Recovery
Sun Yat-sen University · South China University of Technology · Peng Cheng Laboratory +3
Generalists Act, Specialists Intervene: Modular Stage-Selective Reinforcement Learning for Vision-Language-Action Manipulation
Sep 22, 2026Chongyu Zhu, Hyegang Kim, Jaden Hinds +3RL for Robotics Robot Policy Learning
Department of Mechanical and Industrial Engineering, University of Toronto, Toronto, ON, Canada.
VLA-OPD: Bridging Offline SFT and Online RL for Vision-Language-Action Models via On-Policy Distillation
Mar 27, 2026Zhide Zhong, Haodong Yan, Junfeng Li +3Vision-Language-Action Models On-Policy Distillation
HKUST (GZ)

May 10, 2026J/K move · Enter open · S save

RePO-VLA: Recovery-Driven Policy Optimization for Vision-Language-Action Models

Weijia Liufu, Xiaoyu Guo, Ruiyi Chen, Jingzhi Liu+15

Sun Yat-sen University · South China University of Technology · Peng Cheng Laboratory+3

VLM Robustness Robot Failure Recovery Robotics Vision-Language-Action Models Robustness of VLA Models

Open paper AskPDF arXiv

Vision-Language-Action (VLA) models remain brittle in long-horizon, contact-rich manipulation because success-only imitation provides little supervision for execution drift, while failed rollouts are often discarded. We introduce RePO-VLA, a recovery-driven policy optimization framework that assigns distinct roles to success, recovery, and failure trajectories. RePO-VLA first applies Recovery-Aware Initialization (RAI), slicing each corrective suffix into a new episode so corrective actions depend on the current adverse state rather than the preceding failure. It then learns a Progress-Aware Semantic Value Function (PAS-VF), aligning spatiotemporal trajectory features with instructions and successful references. The resulting labels salvage useful failure prefixes via reliability decay, while low-value labels mark drift and terminal breakdowns, teaching differences among nominal, failed, and corrective actions. The data engine turns adverse states into planner-generated or human-collected corrective rollouts, teaching recovery to the success manifold. Value-Conditioned Refinement (VCR) trains the policy to prefer high-progress actions. At deployment, a fixed high-value token biases actions toward the learned success manifold without online failure detectors or heuristic retries. We introduce the Failure-Recovery Benchmark (FRBench), with standardized error injection and recovery-focused evaluation. Across simulated and real-world bimanual tasks, RePO-VLA improves robustness, raising adversarial success from 20% to 75% on average and up to 80% in scaled real-world trials. Code and video results are available at https://1018weijia.github.io/RePO-VLA-Web/.

Abstract

Robotic Manipulation

Explore similar work

RePO-VLA: Recovery-Driven Policy Optimization for Vision-Language-Action Models

Generalists Act, Specialists Intervene: Modular Stage-Selective Reinforcement Learning for Vision-Language-Action Manipulation

VLA-OPD: Bridging Offline SFT and Online RL for Vision-Language-Action Models via On-Policy Distillation