DriftOPD: Sequence-Level Reverse-KL Distillation for One-Step VLA Policies

Youngjun Jun; Kyumin Choi; Youngmin Kim; Seonghyun Jin; Sunwoo Park; Jangho Park; Jong Chul Ye

cs.ROSep 29, 2026

DriftOPD: Sequence-Level Reverse-KL Distillation for One-Step VLA Policies

Authors: Youngjun Jun, Kyumin Choi, Youngmin Kim, Seonghyun Jin, Sunwoo Park, Jangho Park, Jong Chul Ye

Organizations: KAIST, Republic of Korea · Sungkyunkwan University, Republic of Korea

Abstract

Vision-Language-Action (VLA) models increasingly rely on action experts that generate short action chunks under receding-horizon control. While chunk-level training is convenient across robot embodiments, it optimizes local action likelihood without explicitly accounting for long-horizon task success. Sequence-level reinforcement learning can address this limitation, but typically requires policy rollouts and closed-loop interaction, which are costly for real-robot manipulation. We introduce DriftOPD, a teacher-free, rollout-free framework for sequence-level on-policy distillation of continuous VLA action experts. We show that the sequence-level reverse Kullback-Leibler (KL) divergence decomposes into a chunk-level reverse-KL term and a future-potential term that captures the long-horizon effect of the current action. DriftOPD optimizes these two terms using a one-step drifting objective and a Q-function critic learned from offline demonstrations, respectively, enabling sequence-level optimization with only offline data and one-step action generation. Across multiple VLA architectures in simulation and real-world manipulation, DriftOPD generally outperforms existing one-step distillation baselines while achieving task success performance comparable to multi-step teacher policies. These results demonstrate that long-horizon behavior can be effectively distilled into one-step VLA action experts without online interaction or a separate teacher.

Figures & tables

Figure 1: Chunk-level VLA training versus our sequence-level formulation

Figure 2: Comparison of VLA training paradigms. (a) Standard VLA training uses demonstration-supervised forward KLD, (b) existing one-step distillation optimizes chunk-level reverse KLD, and (c) RL relies on closed-loop rollouts, whereas (d) DriftOPD approximates sequence-level reverse KLD using a drifting objective and critic guidance without online rollouts.

Model	Action prediction	Fine-tuning strategy	Teacher steps	Suite	Teacher	1-step
Model	Action prediction	Fine-tuning strategy	Teacher steps	Suite	Teacher	Teacher	sCD	MFD	Drift	DriftOPD
$\pi_{0.5}$	Flow matching	Full FT	10	atomic_seen (18)	39.6	31.30	28.33	34.44	32.22	38.33
				composite_seen (16)	7.1	0.83	1.04	2.50	5.00	5.42
				composite_unseen (16)	1.2	0.83	0.83	0.83	1.04	1.67
$\pi_{0.5}$	Flow matching	LoRA ( $r=16$ )	10	atomic_seen (18)	39.6	31.30	29.26	28.33	34.07	39.26
				composite_seen (16)	7.1	0.83	1.67	2.71	3.12	5.00

Table 1: Average task success rates on RoboCasa365, reported separately for each evaluation suite. Each task uses 30 episodes, totaling 540, 480, and 480 per suite.

Figure 3: Analysis of critic guidance for sequence-level VLA policy optimization on RoboCasa365 with GR00T N1.6. (a) Critic $Q$ over episode progress for demonstrations, successful rollouts, and failed student rollouts. (b) Task success rate versus $\lambda$ on the atomic_seen suite.

Model	Action objective	Fine-tuning strategy	Teacher steps	Suite	Teacher	1-step
Model	Action objective	Fine-tuning strategy	Teacher steps	Suite	Teacher	Teacher	sCD	MFD	Drift	DriftOPD
$\pi_{0.5}$ -LoRA	Flow matching	LoRA ( $r=16$ )	10	short (18)	60.74	61.30	64.81	54.44	66.30	67.78
				medium (21)	67.30	56.35	61.11	48.25	61.90	64.13
				long (11)	55.15	40.61	36.97	16.36	48.48	46.67
ABot-M0	Direct clean action	Joint FT (VLM + action expert)	10	short (18)	70.19	70.37	67.41	2.41	69.07	64.63
				medium (21)	73.97	67.78	66.83	4.13	65.71	68.10

Table 2: Average task success rates on RoboTwin 2.0, grouped by task horizon. Each task is evaluated over 30 episodes under paired environment settings. short , medium , and long horizons correspond to average trajectory lengths of

<150

150

279

, and

\geq 280

steps, respectively.

Appendix figures & tables29 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Benchmark	Action objective	Distillation trainable parameters	Action dim.	Chunk length	Teacher steps	One-step parameterization
$\pi_{0.5}$	RoboCasa365	Flow matching	Action expert 693.4M	32	50	10	$z-v(z,t{=}1)$
$\pi_{0.5}$ -LoRA	RoboCasa365	Flow matching	LoRA + input/output proj. + timestep MLP, 9.1M	32	50	10	$z-v(z,t{=}1)$
	LIBERO				10
	RoboTwin 2.0				50
GR00T N1.5	RoboCasa365	Flow matching	Action-generation head 1,068.8M	32	16	4	$z+v(z,t{=}0)$
GR00T N1.6	RoboCasa365	Flow matching	Action-generation head 1,418.6M	128	50	4	$z+v(z,t{=}0)$

Appendix

Table 4: Model configurations used in our experiments

Model	Benchmark / setting	Starting checkpoint
$\pi_{0.5}$	RoboCasa365	robocasa/robocasa365_checkpoints
	LIBERO	gs://openpi-assets/checkpoints/pi05_libero/
	RoboTwin 2.0	gs://openpi-assets/checkpoints/pi05_base/params (fine-tuned by us for 75k steps with batch size 64 on TianxingChen/RoboTwin2.0 , 50 tasks)
GR00T N1.5	RoboCasa365	robocasa/robocasa365_checkpoints
GR00T N1.6	RoboCasa365	Abhi03/grootn16_robocasa365_multitask_learning
GR00T N1.6	Real world	nvidia/GR00T-N1.6-3B

Appendix

Table 5: Starting checkpoints used for each model and benchmark

Hyperparameter	Setting
Training steps	10,000
Optimizer	AdamW, $\beta_{1}=0.9$ , $\beta_{2}=0.95$
Learning rate	$3\times 10^{-5}\rightarrow 3\times 10^{-7}$
LR schedule	Linear decay; no learning-rate warmup
Weight decay	$0$
Gradient clipping	$1.0$

Appendix

Table 6: Common student optimization settings

Hyperparameter	Setting
Kernel	Laplacian, $K_{R}(a,a^{\prime})\propto\exp(-\lVert a-a^{\prime}\rVert/R)$
Bandwidths	$R\in\{0.02,0.05,0.2\}$
Current-policy samples $M$	64 by default (8 gradient-carrying + 56 detached) 32 (8 gradient-carrying + 24 detached) on RoboCasa365 for $\pi_{0.5}$
Positive sample	One aligned demonstration action, separate from the $M$ policy samples
Default KDE pool	64 policy samples + 1 demonstration sample
Input perturbation	Laplace noise $\epsilon_{R}$ with scale $R$ on generated policy samples

Appendix

Table 7: DriftOPD-specific hyperparameters

Benchmark	Model	$N_{c}$ (# tasks)	Batch size	Gradient-carrying samples / step
RoboCasa365	$\pi_{0.5}$ (full fine-tuning)	4	8	32
	$\pi_{0.5}$ (LoRA $r{=}16$ )	4	8	32
	GR00T N1.5	16	8	128
	GR00T N1.6	16	8	128
RoboTwin 2.0	$\pi_{0.5}$ (LoRA $r{=}16$ )	4	8	32
RoboTwin 2.0	ABot-M0	64	8	512

Appendix

Table 8: Model-specific batch configurations.

N_{c}

denotes the number of task contexts per optimization step, and batch size denotes the number of gradient-carrying policy samples per context. Students evaluated on LIBERO-Plus are the LIBERO students, so they share the LIBERO configuration.

Figure 5: Architecture of the demonstration-trained critic used in DriftOPD. The frozen VLA prefix, the robot state, and the candidate action chunk are encoded jointly, and a learned value token is read out by twin Q-heads.

Component	Setting
Input	Frozen VLA prefix tokens + state token + action-chunk tokens + learned value token
Encoder	2-layer pre-norm bidirectional Transformer encoder
Hidden dimension	512
Attention heads	8
FFN dimension	2048
Dropout	$0$

Appendix

Table 9: Architecture and optimization settings of the demonstration-trained critic.

Method	Implementation
One-step Teacher	Direct one-step generation from the corresponding teacher checkpoint without additional distillation. For ABot-M0, the baseline uses its dedicated one-step generator.
sCD	$P_{\mathrm{mean}}=-1.0$ , $P_{\mathrm{std}}=1.4$ , $\sigma_{d}=1$ , tangent normalization $c=0.1$ , JVP tangent estimation with central finite-difference fallback $\epsilon=10^{-3}$ , and adaptive-weighting learning rate $10^{-4}$ .
MFD	Maximum step size $0.02$ , solver-step cap 32, minimum interval $0.02$ . TTUR ratio $K=10$ with logit-normal time sampling on LIBERO and RoboTwin 2.0; $K=50$ with uniform time sampling on RoboCasa365.
Drift	Identical kernel, bandwidth, perturbation, normalization, sampling and student-optimization settings as DriftOPD, with the critic term disabled.
DriftOPD	Drift objective combined with the frozen demonstration-trained critic using the settings in Tables 7 and 9 .

Appendix

Table 10: Implementation details of the one-step distillation baselines.

Benchmark	Task set	# Tasks	Episodes per task	Initialization	Actions before replanning	Termination
RoboCasa365	pretrain split	50	30	$\pi_{0.5}$ : seed 7	$\pi_{0.5}$ : 5	Success or task horizon
	(leaderboard)			GR00T: no explicit seed	GR00T N1.5: 16
					GR00T N1.6: 8
RoboTwin 2.0	50 tasks	50	30	Accepted-seed	25	Success or
RoboTwin 2.0	demo_clean	50	30	manifest	25	task-specific limit
LIBERO	4 suites	40	50	Seed 7	$\pi_{0.5}$ : 5	Success or

Appendix

Table 11: Simulation benchmark configurations.

Figure 6: Real-world experimental setup. (a) MolmoAct 2 Research Kit on the YAM leader–follower bimanual platform, with top and wrist RealSense cameras providing RGB observations. The right arm is used for unimanual tasks and both arms for bimanual tasks. (b) Objects and containers used for the unimanual pick-and-place and bimanual handover tasks.

Item	Setting
Robot	I2RT YAM leader–follower system
Demonstrations	Unimanual (right arm): 424 episodes, 269,089 frames, 8 tasks Bimanual: 214 episodes, 97,284 frames, 4 tasks
Teleoperation rate	$\approx 200\,\mathrm{Hz}$
Camera rate	$\approx 30\,\mathrm{Hz}$
Policy cameras	Top, right, and left Intel RealSense, RGB only
Raw resolution	$640\times 480$

Appendix

Table 12: Real-world demonstration data and GR00T N1.6 teacher fine-tuning

Object	Size	Role
Small basket (white, shallow)	$20.2\times 14.5\times 6$	Target container
Plate (white, round)	diameter $22.5$	Target container
Pot (stainless steel)	diameter $16$ , height $9.2$	Target container
Plush tennis ball	diameter $\approx 12$	Grasped object
Plush soccer ball	diameter $\approx 12$	Grasped object
Plush basketball	diameter $\approx 5$	Grasped object

Appendix

Table 13: Objects used in the real-world experiments. Dimensions are height

\times

width

\times

depth in centimeters; for spherical and circular items the diameter is reported instead.

Item	Setting
Tasks	Unimanual: four pick-and-place tasks Bimanual: two pick-and-handover-and-place tasks
Evaluation settings	15 fixed scenes per task
Pairing	Teacher session captures the scenes; all methods replay the same scenes in the same order
Control frequency	$30\,\mathrm{Hz}$
Inference interval	$0.3\,\mathrm{s}$ , i.e. about 9 commanded control steps between replans
Inference mode	Asynchronous rate-limited inference

Appendix

Table 14: Real-world evaluation configuration

Workload	Hardware	Usage
Student distillation	NVIDIA B200, 180 GB	1 GPU per run
Critic training	NVIDIA B200, 180 GB	1 GPU per run
Real-world teacher fine-tuning	NVIDIA B200, 180 GB	1 GPU per run
Simulation evaluation	RoboTwin 2.0 & LIBERO-Plus: NVIDIA A100-SXM4-40GB RoboCasa365 & LIBERO: NVIDIA B200	1 evaluation lane per GPU
Real-world inference	NVIDIA GeForce RTX 4090, 24 GB	Local robot workstation

Appendix

Table 15: Hardware used for training and evaluation

Component	RoboTwin 2.0	RoboCasa365	LIBERO / LIBERO-Plus
Benchmark code	RoboTwin, upstream commit c3ddfa8b (23 May 2026); camera images rendered with the rasterization shader instead of the upstream 32-spp ray-tracing path	RoboCasa 1.0.1 (vendored)	LIBERO (vendored)
Simulator	SAPIEN 3.0.0b1	MuJoCo 3.3.1	MuJoCo 2.3.7 ( $\pi_{0.5}$ , ABot-M0); 3.8.1 (SmolVLA)
Physics timestep	$1/250\,\mathrm{s}$	Benchmark default	Benchmark default
robosuite	—	1.5.2	1.4.0
Motion planning	mplib 0.2.1, nvidia-curobo 0.7.8, warp-lang 1.15.0	—	—
Python	3.10.20	3.11.15	3.11.15

Appendix

Table 16: Benchmark-specific software environments. Each benchmark uses its own Python environment because the simulator dependencies conflict.

Model	Teacher steps	Task	Teacher	1-step
Model	Teacher steps	Task	Teacher	Teacher	sCD	MFD	Drift	DriftOPD
atomic_seen (18 tasks)
$\pi_{0.5}$	10	CloseBlenderLid	–	6.67	0.00	3.33	3.33	3.33
		CloseFridge	–	23.33	46.67	46.67	30.00	56.67
		CloseToasterOvenDoor	–	0.00	0.00	6.67	3.33	0.00
		CoffeeSetupMug	–	13.33	3.33	10.00	13.33	6.67

Appendix

Table 17: Per-task success rates of

\pi_{0.5}

(full fine-tuning) and its one-step students on RoboCasa365. Each task is evaluated over 30 episodes. The multi-step Teacher column is taken from the RoboCasa365 leaderboard ( https://robocasa.ai/leaderboard.html ).

Model	Teacher steps	Task	Teacher	1-step
Model	Teacher steps	Task	Teacher	Teacher	sCD	MFD	Drift	DriftOPD
atomic_seen (18 tasks)
$\pi_{0.5}$ (LoRA $r{=}16$ )	10	CloseBlenderLid	–	6.67	0.00	0.00	3.33	3.33
		CloseFridge	–	23.33	10.00	43.33	20.00	43.33
		CloseToasterOvenDoor	–	0.00	0.00	3.33	0.00	6.67
		CoffeeSetupMug	–	13.33	10.00	10.00	16.67	10.00

Appendix

Table 18: Per-task success rates of

\pi_{0.5}

(LoRA

r{=}16

) and its one-step students on RoboCasa365. Each task is evaluated over 30 episodes. The multi-step Teacher column is taken from the RoboCasa365 leaderboard ( https://robocasa.ai/leaderboard.html ).

Model	Teacher steps	Task	Teacher	1-step
Model	Teacher steps	Task	Teacher	Teacher	sCD	MFD	Drift	DriftOPD
atomic_seen (18 tasks)
GR00T N1.5	4	CloseBlenderLid	–	6.67	3.33	10.00	16.67	26.67
		CloseFridge	–	73.33	60.00	90.00	56.67	70.00
		CloseToasterOvenDoor	–	43.33	33.33	53.33	23.33	26.67
		CoffeeSetupMug	–	16.67	16.67	3.33	33.33	16.67

Appendix

Table 19: Per-task success rates of GR00T N1.5 and its one-step students on RoboCasa365. Each task is evaluated over 30 episodes. The multi-step Teacher column is taken from the RoboCasa365 leaderboard ( https://robocasa.ai/leaderboard.html ).

Model	Teacher steps	Task	Teacher	1-step
Model	Teacher steps	Task	Teacher	Teacher	sCD	MFD	Drift	DriftOPD
atomic_seen (18 tasks)
GR00T N1.6	4	CloseBlenderLid	–	3.33	6.67	0.00	10.00	13.33
		CloseFridge	–	66.67	80.00	63.33	90.00	76.67
		CloseToasterOvenDoor	–	56.67	60.00	30.00	66.67	50.00
		CoffeeSetupMug	–	36.67	36.67	3.33	20.00	26.67

Appendix

Table 20: Per-task success rates of GR00T N1.6 and its one-step students on RoboCasa365. Each task is evaluated over 30 episodes. The multi-step Teacher column is taken from the RoboCasa365 leaderboard ( https://robocasa.ai/leaderboard.html ).

Model	Teacher steps	Task	Teacher	1-step
Model	Teacher steps	Task	Teacher	Teacher	sCD	MFD	Drift	DriftOPD
short (18 tasks, 77–147 demo steps)
$\pi_{0.5}$ -LoRA	10	click_bell	30.00	80.00	90.00	100.00	86.67	86.67
		click_alarmclock	50.00	80.00	73.33	100.00	83.33	86.67
		grab_roller	100.00	100.00	100.00	100.00	100.00	100.00
		turn_switch	40.00	50.00	56.67	66.67	43.33	50.00

Appendix

Table 21: Per-task success rates of

\pi_{0.5}

and its LoRA-based (

r=16

) student policies on RoboTwin 2.0 with the ALOHA 14-DoF embodiment. Tasks are ordered by average demonstration trajectory length and grouped into short , medium and long horizons. Each task is evaluated over 30 episodes from a shared accepted-seed manifest.

Model	Teacher steps	Task	Teacher	1-step
Model	Teacher steps	Task	Teacher	Teacher	sCD	MFD	Drift	DriftOPD
short (18 tasks, 77–147 demo steps)
ABot-M0	10	click_bell	96.67	100.00	90.00	0.00	80.00	93.33
		click_alarmclock	100.00	100.00	96.67	3.33	93.33	90.00
		grab_roller	100.00	93.33	83.33	0.00	93.33	90.00
		turn_switch	80.00	63.33	70.00	23.33	73.33	53.33

Appendix

Table 22: Per-task success rates of ABot-M0 and its student policies on RoboTwin 2.0 with the ALOHA 14-DoF embodiment. Tasks are ordered by average demonstration trajectory length and grouped into short , medium and long horizons. Each task is evaluated over 30 episodes from a shared accepted-seed manifest.

Model	Teacher steps	Suite	Teacher	1-step
Model	Teacher steps	Suite	Teacher	Teacher	sCD	MFD	Drift	DriftOPD
$\pi_{0.5}$ -LoRA	10	libero_spatial (10)	98.8%	99.2%	98.6%	99.2%	98.2%	98.4%
		libero_object (10)	98.2%	98.4%	97.2%	97.2%	98.4%	97.4%
		libero_goal (10)	98.0%	98.2%	97.4%	98.2%	98.0%	97.0%
		libero_10 (10)	92.4%	93.0%	89.2%	87.8%	91.8%	91.8%
		Average (40)	96.85%	97.20%	95.60%	95.60%	96.60%	96.15%

Appendix

Table 23: Average task success rates on LIBERO, reported separately for each evaluation suite. Each task is evaluated over 50 episodes under paired environment settings.

Model	Teacher steps	Suite	Teacher	1-step
Model	Teacher steps	Suite	Teacher	Teacher	sCD	MFD	Drift	DriftOPD
$\pi_{0.5}$ -LoRA	10	Sensor Noise (1,601)	86.70%	84.51%	85.38%	86.20%	85.51%	86.38%
		Camera Viewpoints (1,599)	71.36%	71.79%	70.61%	72.61%	70.86%	71.23%
		Robot Initial States (1,550)	75.16%	71.74%	73.16%	72.97%	74.39%	74.58%
		Language Instructions (1,537)	84.65%	86.21%	84.65%	85.75%	85.88%	85.49%
		Objects Layout (1,525)	85.90%	85.77%	86.69%	86.23%	86.75%	86.56%

Appendix

Table 24: Average task success rates on LIBERO-Plus, reported separately for each evaluation suite. Each of the 10,030 perturbed task instances is evaluated over a single episode under paired environment settings.

Figure 7: Training curves of one-step distillation on RoboCasa365. Faint lines are logged values and bold lines are EMAs. For DriftOPD, the total loss is shown with its drift term (dashed) and critic term $-Q$ (red).

Figure 8: Bandwidth-wise RMS of the drifting loss term before normalization, for the drifting distillation (Drift) and DriftOPD on RoboCasa365.

Task	Success Rate (%)						Oscillation ( ∘ )
	4-step	1-step					4-step	1-step
	Teacher	Teacher	sCD	MFD	Drift	DriftOPD	Teacher	Teacher	sCD	MFD	Drift	DriftOPD
unimanual (right arm)
Easy	93.3	40.0	0.0 (60.0)	0.0 (0.0)	86.7	93.3	1.70	1.77	4.64	7.55	1.97	2.01
Moderate	73.3	46.7	6.7 (20.0)	0.0 (0.0)	60.0	60.0	1.61	1.90	4.81	7.77	1.83	1.89
Hard	66.7	13.3	0.0 (0.0)	0.0 (0.0)	60.0	66.7	1.65	1.79	4.87	7.69	1.70	1.89

Appendix

Table 25: Real-world comparison of task success rate and action oscillation across the multi-step teacher and one-step policies. Values in parentheses indicate partial success rates (e.g., picking the object but failing to place or handover it), and oscillation is computed according to Eq. ( 97 ).

Figure 9: LoRA rank sweep for $\pi_{0.5}$ on RoboCasa365 ( atomic_seen ). Hollow markers: full fine-tuning; dashed line: 1-step teacher (31.30%).

Figure 10: Seed-to-seed action diversity at a fixed observation (GR00T N1.6, RoboCasa365, 10k-step students). For 288 training observations (18 tasks $\times$ 16), each student samples $K=32$ one-step action chunks that differ only in input noise. The y-axis is the pairwise RMS among the 32 chunks (12 action dims $\times$ 16 steps, normalized space); each dot is one observation, boxes show the median and IQR, and the mean is given under each label. MSE only collapses to a near-deterministic output (0.016), while Drift (0.120) and DriftOPD (0.117) keep about 7 $\times$ more spread.

Figure 11: Effect of the success-field weight $\beta_{\mathrm{Success}}$ on GR00T N1.6 (RoboCasa365 atomic_seen , 18 tasks $\times$ 30 episodes). Dashed lines indicate Drift (47.22) and DriftOPD (49.44). Success rate remains flat for $\beta_{\mathrm{Success}}\in[0.1,3.0]$ , drops at $\beta_{\mathrm{Success}}=0.03$ , and collapses at $\beta_{\mathrm{Success}}=10$ .

Vision-Language-Action Models

Vision-language-action models turn visual observations and natural-language instructions into actions for robots and other embodied agents.

See all 1,194 papers

VLA-OPD: Bridging Offline SFT and Online RL for Vision-Language-Action Models via On-Policy Distillation
Mar 27, 2026Zhide Zhong, Haodong Yan, Junfeng Li +3Vision-Language-Action Models On-Policy Distillation
HKUST (GZ)
DriftingVLA: Native One-Step Vision-Language-Action Generation via Per-Dimension Temporal Drifting
Aug 30, 2026Yuxuan Gao, Shiqi Zhang, Yedong Shen +6Efficient VLA Model Inference Visuomotor Policy Learning
University of Science and Technology of China, Hefei 230026, China. · LYNSENSE, Shanghai, China.
FastOPD: On-Policy Distillation for Lightweight VLA Deployment
Oct 2, 2026Yoojin Oh, Jeongsol Kim, Yeonwoo Seo +7Efficient VLA Model Inference Vision-Language-Action Models
KAIST · Sungkyunkwan University

Mar 27, 2026J/K move · Enter open · S save

VLA-OPD: Bridging Offline SFT and Online RL for Vision-Language-Action Models via On-Policy Distillation

Zhide Zhong, Haodong Yan, Junfeng Li, Junjie He+2

HKUST (GZ)

Vision-Language-Action Models On-Policy Distillation RL for VLMs

Open paper AskPDF arXiv

Although pre-trained Vision-Language-Action (VLA) models exhibit impressive generalization in robotic manipulation, post-training remains crucial to ensure reliable performance during deployment. However, standard offline Supervised Fine-Tuning (SFT) suffers from distribution shifts and catastrophic forgetting of pre-trained capabilities, while online Reinforcement Learning (RL) struggles with sparse rewards and poor sample efficiency. In this paper, we propose On-Policy VLA Distillation (VLA-OPD), a framework bridging the efficiency of SFT with the robustness of RL. Instead of relying on sparse environmental rewards, VLA-OPD leverages an expert teacher to provide dense, token-level supervision on the student's self-generated trajectories. This enables active error correction on policy-induced states while preserving pre-trained general capabilities through gentle alignment. Crucially, we formulate VLA-OPD via a Reverse-KL objective. Unlike standard Forward-KL that induces mode-covering entropy explosion, or Hard-CE that causes premature entropy collapse, our bounded mode-seeking objective ensures stable policy learning by filtering out the teacher's epistemic uncertainty while maintaining action diversity. Experiments on LIBERO and RoboTwin2.0 benchmarks demonstrate that VLA-OPD significantly improves sample efficiency over RL and robustness over SFT, while effectively mitigating catastrophic forgetting during post-training.

Abstract

Vision-Language-Action Models

Explore similar work

VLA-OPD: Bridging Offline SFT and Online RL for Vision-Language-Action Models via On-Policy Distillation

DriftingVLA: Native One-Step Vision-Language-Action Generation via Per-Dimension Temporal Drifting

FastOPD: On-Policy Distillation for Lightweight VLA Deployment