Many Ways to Succeed: Diversity-Driven RL Fine-Tuning for VLA Generalization

Haoru Li; Jinmei Liu; Zhiyong Wang; Xiaoming Li; Zhenhong Sun; Daoyi Dong; Chunlin Chen; Zhi Wang

cs.ROOct 7, 2026

Many Ways to Succeed: Diversity-Driven RL Fine-Tuning for VLA Generalization

Authors: Haoru Li, Jinmei Liu, Zhiyong Wang, Xiaoming Li, Zhenhong Sun, Daoyi Dong, Chunlin Chen, Zhi Wang

Organizations: Nanjing University · Harbin Institute of Technology (Shenzhen) · Australian National University · University of Technology Sydney

Abstract

Reinforcement learning (RL) fine-tuning improves vision-language-action (VLA) policies through closed-loop experience, yet generalization beyond the fine-tuning distribution remains limited. Our analysis reveals a selective reshaping of exploration: RL contracts behavior globally, yet diversifies successful trajectories, elicits success with fewer rollouts, and covers more of the latent task-valid solution space than supervised fine-tuning. Broader successful-mode coverage may provide alternative strategies under distribution shifts. Inspired by this, we introduce DRIVE (Diversity-driven RL fIne-tuning for VLA gEneralization), which turns successful-behavior diversity into an explicit RL objective. DRIVE groups rollouts under matched task conditions, compares their trajectories with temporal alignment, and derives a success-conditioned intrinsic reward from relative behavioral diversity. This design encourages broader coverage of feasible solutions without rewarding diverse failures or superficial timing differences. Across LIBERO-Plus, ManiSkill3, and RoboTwin 2.0, DRIVE improves the average out-of-domain (OOD) performance over vanilla RL fine-tuning by 5.3 points on $π_0$ and 2.0 points on $π_{0.5}$ . On a dual-arm AgileX PiPER-X platform, DRIVE further increases average OOD success from 64.1% to 73.3% (+9.2 points), demonstrating gains that persist under physical deployment.

Figures & tables

Figure 1: Beyond task success, diverse successful trajectories reveal broader solution modes. This diversity provides a new opportunity for improving VLA generalization.

Figure 2: Exploration dynamics and successful-behavior accessibility in VLA reinforcement fine-tuning. (a–b) During RFT, similarity increases across all rollouts while decreasing among successful trajectories; SR denotes success rate. (c) Pass@

K

shows that the final RFT policy elicits success with fewer rollouts than SFT. (d) Coverage@

K

, measured against SFT successful trajectories at

K=256

, shows broader reference-repertoire coverage under limited sampling.

Figure 3: Overview of DRIVE. (a) Rollouts are encoded as per-decision feature trajectories. (b) GAK computes behavioral similarity by softly aligning feature sequences over monotonic temporal paths, handling execution-rate mismatch. (c) Pairwise similarities are converted into success-conditioned diversity rewards and added to environment rewards to promote exploration diversity.

Method	LIBERO-Plus Spatial		LIBERO-Plus Object		LIBERO-Plus Goal		LIBERO-Plus Long		ManiSkill3		RoboTwin 2.0		Avg.
Method	IND	OOD	IND	OOD	IND	OOD	IND	OOD	IND	OOD	IND	OOD	IND	OOD
$\pi_{0}$
SFT	22.4	14.8	42.0	27.6	18.4	14.1	30.0	24.5	39.7	21.1	42.0	46.1	36.6	29.2
Vanilla RFT	78.8	48.9	70.8	53.4	55.6	37.9	66.0	46.1	66.2	35.5	65.9	62.1	66.6	48.1
DRIVE	79.6	50.2	74.4	53.5	54.8	39.8	66.4	50.8	68.4	40.3	70.6	71.4	69.3	53.4
$\pi_{0.5}$

Table 1: Main results on simulation benchmarks. Values are task success rates (%). RoboTwin 2.0 averages over Click Bell and Press Stapler. Avg. is the macro-average over benchmark families. Best and second-best results are bold and underlined.

Figure 4: Generalization and behavioral dynamics during RFT for

\pi_{0.5}

on ManiSkill3. (a) IND success rate across training steps. (b) Average OOD success rate across visual-language, semantic, and execution shifts at selected training steps for Vanilla RFT and DRIVE with two shaping strengths. (c) Normalized GAK similarity over all trajectories for Vanilla RFT and DRIVE. (d) Normalized GAK similarity over successful trajectories for Vanilla RFT and DRIVE. Behavioral analyses in (c–d) use DRIVE with

\alpha=0.1

. Lower similarity indicates greater behavioral diversity.

Figure 5: Effect of success conditioning on IND performance. IND evaluation success rates of DRIVE with and without success conditioning throughout RFT.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Suite	Clean	Confounding	Target Pose	Scene Theme	Surface	Total
Spatial	10	78	66	44	52	250
Object	10	77	72	37	54	250
Goal	10	81	63	42	54	250
Long	10	65	50	52	73	250

Appendix

Table 3: Composition of the LIBERO-Plus IND task pools. Each suite contains the ten original clean LIBERO tasks and 240 selected Plus variants.

Perturbation family	Spatial	Object	Goal	Long
Objects Layout	385	403	425	312
Camera Viewpoints	376	396	408	419
Robot Initial States	350	398	409	393
Language Instructions	390	354	410	383
Light Conditions	292	297	279	274
Background Textures	258	248	281	289

Appendix

Table 4: LIBERO-Plus perturbation statistics for the four evaluated suites. Counts denote variants in the complete Plus manifests and are distinct from the selected IND subsets in Table 3 .

Split / Category	Evaluation focus	Settings	Episodes
IND	Training objects, scenes, and state distribution	1	320
Visual-Language OOD	Visual appearance and language variation	6	1,920
Semantic OOD	Object/receptacle grounding under semantic ambiguity	4	1,280
Execution OOD	Initial-state shifts and execution-time changes	3	960
Total	IND + three OOD categories	14	4,480

Appendix

Table 5: ManiSkill3 evaluation protocol. Each setting is evaluated over 320 episodes. The OOD benchmark contains 13 settings grouped into visual-language, semantic, and execution shifts.

Perturbation family	IND	OOD	Description
Clean	50	0	Nominal task configurations
Objects	134	34	Object additions and pose variations
Background	750	150	Scene/background appearance variations
Lighting	0	50	Held-out illumination conditions
Camera	0	50	Held-out camera configurations
Robot State	66	16	Robot initial-state variations

Appendix

Table 6: RoboTwin 2.0 IND/OOD construction for each evaluated task. The same split structure is independently instantiated for Click Bell and Press Stapler .

	$\pi_{0}$				$\pi_{0.5}$
Parameter	Spatial	Object	Goal	Long	Spatial	Object	Goal	Long
Runner epochs	200	300	400	200	300	250	100	300
Global batch size	2048	2048	2048	2048	2048	2048	2048	2048
PPO update epochs	4	4	4	4	3	3	3	4
Actor learning rate	$5{\times}10^{-6}$	$5{\times}10^{-6}$	$5{\times}10^{-6}$	$5{\times}10^{-6}$	$5{\times}10^{-6}$	$5{\times}10^{-6}$	$5{\times}10^{-6}$	$5{\times}10^{-6}$
Critic learning rate	$1{\times}10^{-4}$	$1{\times}10^{-4}$	$1{\times}10^{-4}$	$1{\times}10^{-4}$	$1{\times}10^{-4}$	$1{\times}10^{-4}$	$1{\times}10^{-4}$	$1{\times}10^{-4}$

Appendix

Table 7: Training hyperparameters for LIBERO-Plus. Runner epochs and DRIVE shaping coefficients correspond to the final experimental configurations. All compared methods within each column use the same runner horizon.

	ManiSkill3		Click Bell		Press Stapler
Parameter	$\pi_{0}$	$\pi_{0.5}$	$\pi_{0}$	$\pi_{0.5}$	$\pi_{0}$	$\pi_{0.5}$
Runner epochs	300	200	100	100	100	50
Global batch size	5120	5120	2048	2048	2048	2048
PPO update epochs	4	5	5	5	5	5
Actor learning rate	$7.91{\times}10^{-6}$	$7.91{\times}10^{-6}$	$5.6{\times}10^{-6}$	$5{\times}10^{-6}$	$5.6{\times}10^{-6}$	$5{\times}10^{-6}$
Critic learning rate	$1.55{\times}10^{-4}$	$1.55{\times}10^{-4}$	$1.1{\times}10^{-4}$	$1{\times}10^{-4}$	$1.1{\times}10^{-4}$	$1{\times}10^{-4}$

Appendix

Table 8: Training hyperparameters for ManiSkill3 and RoboTwin 2.0. Training horizons are shared by all compared methods within each benchmark–backbone setting.

Method	Modification
Vanilla RFT	Standard PPO + Flow-SDE
Higher Noise	Flow-SDE noise $\sigma=0.7$
KL Regularization	KL penalty $\beta=0.5$
Clip-Higher	PPO upper clip $0.20\rightarrow 0.28$
DRIVE	Success-conditioned GAK shaping

Appendix

Table 9: Method-specific modifications relative to Vanilla RFT.

Explore similar work

RL $^2$ -VLA: Adaptive RL Latent Compositional Steering with Test-Time Scaling for Vision-Language-Action Models
Jul 29, 2026Derek Ming Siang Tan, Shailesh Shailesh, Srikrishna Iyer +4Vision-Language-Action Models Test-Time Adaptation of VLA Models
1National University of Singapore · 3Singapore Technologies Engineering · University of Toronto
What to Ignore, What to React: Visually Robust RL Fine-Tuning of VLA Models
May 13, 2026Yuanfang Peng, Jingjing Fu, Chuheng Zhang +6VLM Robustness Domain Generalization
1Hong Kong University of Science and Technology · 2Microsoft Research Asia · 3Zhejiang University
FORCE: Efficient VLA Reinforcement Fine-Tuning via Value-Calibrated Warm-up and Self-Distillation
Jun 24, 2026Shuyi Zhang, Yunfan Lou, Hongyang Cheng +8RL for Robotics Vision-Language-Action Models
Institute of Automation, Chinese Academy of Sciences, Beijing, China · Beijing Academy of Artificial Intelligence, Beijing, China · State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University, Beijing, China

Jul 29, 2026, cs.ROJ/K move · Enter open · S save

RL $^2$ -VLA: Adaptive RL Latent Compositional Steering with Test-Time Scaling for Vision-Language-Action Models

Derek Ming Siang Tan, Shailesh Shailesh, Srikrishna Iyer, William Wei Jie Teo+3

1National University of Singapore · 3Singapore Technologies Engineering · University of Toronto

Vision-Language-Action Models Test-Time Adaptation of VLA Models Robustness of VLA Models RL for VLMs

Open paper AskPDF arXiv

Despite the impressive visuomotor capabilities enabled by Vision-Language-Action (VLA) models, their performance often degrades on challenging and out-of-domain tasks. Recent test-time steering and scaling methods improve performance without extensive data collection and retraining, but action samples often remain concentrated around similar behaviors and therefore inherit correlated failure modes. Moreover, existing methods apply the same intervention strategy at every timestep, regardless of whether the base policy is already likely to succeed. To address these limitations, we introduce

RL^2

, an adaptive inference-time steering framework that leverages Reinforcement Learning on VLA Latents. First, we train a lightweight offline RL policy conditioned on expressive latents extracted from the VLA action expert and compose its flow velocity with that of the frozen VLA during inference. This compositional steering strategy combines the behavioral priors of large-scale imitation learning with the action diversity induced by offline RL beyond dominant demonstration modes. We further discover that inference-time steering follows fundamentally different scaling laws under success and failure states, revealing that action diversity is most beneficial when the base VLA is likely to fail, but can unnecessarily perturb already-accurate actions when success is likely. Building on this insight,

RL^2

activates compositional steering only when failure is predicted. Across the SIMPLER and PolaRiS benchmarks,

RL^2

improves success rates by up to +17.3% in out-of-domain settings, while ablations and scaling studies demonstrate the importance of latent representations and RL training. Finally, real-world experiments demonstrate that these gains transfer beyond simulation, establishing

RL^2

as a practical and modular steering framework for VLA deployment.

Abstract

Explore similar work

RL2^22-VLA: Adaptive RL Latent Compositional Steering with Test-Time Scaling for Vision-Language-Action Models

What to Ignore, What to React: Visually Robust RL Fine-Tuning of VLA Models

FORCE: Efficient VLA Reinforcement Fine-Tuning via Value-Calibrated Warm-up and Self-Distillation

RL $^2$ -VLA: Adaptive RL Latent Compositional Steering with Test-Time Scaling for Vision-Language-Action Models