Vision-Language-Action Jump-Starting for Reinforcement Learning Robotic Agents

Angelo Moroncelli; Roberto Zanetti; Marco Maccarini; Loris Roveda

cs.LGApr 15, 2026

Vision-Language-Action Jump-Starting for Reinforcement Learning Robotic Agents

Authors: Angelo Moroncelli, Roberto Zanetti, Marco Maccarini, Loris Roveda

Organizations: University of Applied Science and Arts of Southern Switzerland, Department of Innovative Technologies, IDSIA-SUPSI, Lugano, Switzerland · Università della Svizzera Italiana, Faculty of Informatics, Lugano, Switzerland · Politecnico di Milano, Mechanical Department, Milano, Italy

Abstract

Reinforcement learning (RL) enables high-frequency, closed-loop control for robotic manipulation, but scaling to long-horizon tasks with sparse or imperfect rewards remains difficult due to inefficient exploration and poor credit assignment. Vision-Language-Action (VLA) models leverage large-scale multimodal pretraining to provide generalist, task-level reasoning, but current limitations hinder their direct use in fast and precise manipulation. In this paper, we propose Vision-Language-Action Jump-Starting (VLAJS), a method that bridges sparse VLA guidance with on-policy RL to improve exploration and learning efficiency. VLAJS treats VLAs as transient sources of high-level action suggestions that bias early exploration and improve credit assignment, while preserving the high-frequency, state-based control of RL. Our approach augments Proximal Policy Optimization (PPO) with a directional action-consistency regularization that softly aligns the RL agent's actions with VLA guidance during early training, without enforcing strict imitation, requiring demonstrations, or relying on continuous teacher queries. VLA guidance is applied sparsely and annealed over time, allowing the agent to adapt online and ultimately surpass the guiding policy. We evaluate VLAJS on six challenging manipulation tasks: lifting, pick-and-place, peg reorientation, peg insertion, poking, and pushing in simulation, and validate a subset on a real Franka Panda robot. VLAJS consistently outperforms PPO and distillation-style baselines in sample efficiency, reducing required environment interactions by over 50% in several tasks. Real-world experiments demonstrate zero-shot sim-to-real transfer and robust execution under clutter, object variation, and external perturbations.

Figures & tables

Fig. 2 : Comparison of guidance strategies in RL. Methods are categorized by guidance type (behavioral vs. auxiliary) and imitation persistence (none, transient, and persistent). Vanilla RL uses no guidance, DAgger-like methods apply persistent behavioral imitation, and policy distillation/RPD rely on persistent auxiliary losses. JSRL provides transient behavioral guidance, while VLAJS introduces transient auxiliary guidance via a directional action-consistency loss, accelerating early learning without limiting asymptotic performance.

Panel of Figure 2 — Fig. 3 : Guidance mechanisms for exploration in RL. (a) Relies on random exploration. (b) Executes an imitation-learned policy for an initial phase (solid path). (c) Continuously biases learning via a teacher-provided signal (dashed red path) without directly executing actions.

Panel of Figure 3 — Fig. 4 : Auxiliary guidance during rollouts. (a) The policy generates actions solely through on-policy exploration at a fixed control frequency, learning both direction and action scale incrementally from reward. (b) A teacher provides continuous action targets throughout the rollout, constraining both direction and magnitude and forcing the policy to match the teacher’s action scale (distillation/RPD style). (c) Guidance is applied sparsely within a rollout and progressively annealed across rollouts, biasing action direction while allowing the policy to learn its own action magnitude and eventually explore freely.

Panel of Figure 4 — Fig. 5 : Auxiliary losses for VLA-guided RL. (a) Distillation-based methods ( e.g. , RPD) use an MSE loss that penalizes the full Euclidean distance between policy and teacher actions, constraining both action direction and magnitude. (b) VLAJS instead employs a directional action-consistency loss that penalizes angular misalignment between policy and VLA actions, while remaining invariant to action scale. (c–d) Plots show the superimposition of all training runs across tasks, which consistently follow the same trend: VLA guidance produces a pronounced jump-start, visible as a sharp early bump in rollout reward improvement (the metric used for guidance deactivation), after which the signal remains low as guidance is adaptively turned off.

Fig. 6 : Simulation and real-world manipulation tasks used in our evaluation. Left: six ManiSkill simulation tasks (PickCube, PickPlaceCube, LiftPegUpright, PegInsertionSide, PokeCube, PushCube). Right: zero-shot real-world deployment on a Franka Panda robot across diverse language-specified tasks.

Fig. 7 : Learning curves for long-horizon tasks. Sparse RPD makes distillation feasible in long-horizon tasks significantly accelerating convergence compared to PPO baselines. Standard RPD is omitted due to prohibitive training time.

Fig. 8 : Learning curves and sample-efficiency comparison for suboptimal reward tasks. VLAJS consistently outperforms PPO and distillation-based baselines—VLAJS (RPD). The initial increase in return is driven by the VLA-based jump-start. Once the VLA guidance is deactivated, the agent transitions to a purely learned policy, which results in a temporary reduction in return while maintaining or improving task success (see the additional material for return plots).

Task	$t^{\ast}$	PPO (SR_ $t^{\ast}$ /AUC $\uparrow$ ) $(\%)$	VLAJS (RPD) (SR_ $t^{\ast}$ /AUC $\uparrow$ ) $(\%)$	VLAJS (SR_ $t^{\ast}$ /AUC $\uparrow$ ) $(\%)$
PickCube-v1	2.5M	88.7 [83.3,94.4] / 72.4 [71.5,73.8]	59.0 [33.3,76.9] / 71.0 [68.7,73.5]	91.7 [90.0,93.3] / 72.9 [72.2,74.0]
PickCube-v2	9.9M	0.0 / 59.6 [52.8,69.2]	1.1 [0.0,2.9] / 76.7 [56.1,89.3]	95.1 [92.3,100.0] / 88.7 [87.4,90.2]
PickPlaceCube-v1	11.4M	45.4 [0.0,93.3] / 70.3 [67.5,74.0]	86.7 [60.0,100.0] / 72.8 [70.2,76.7]	39.1 [0.0,93.8] / 72.3 [65.1,81.3]
PickPlaceCube-v2	37.7M	0.0 / 0.0	0.0 / 0.0	65.9 [0.0,100.0] / 66.0 [50.7,86.2]
LiftPegUpright-v1	2.0M	84.8 [63.6,100.0] / 85.9 [84.1,87.3]	0.8 [0.0,1.3] / 81.1 [76.7,85.1]	80.3 [50.0,100.0] / 87.3 [86.3,88.4]
LiftPegUpright-v2	7.3M	16.9 [0.0,36.4] / 76.6 [70.4,80.5]	0.3 [0.0,1.0] / 45.3 [17.8,60.0]	91.5 [85.7,100.0] / 80.5 [76.6,83.0]

TABLE I : Per-task performance reported as Success Rate at

t^{\ast}

(SR_

t^{\ast}

) / Area Under the Success Curve (AUC 0..B ) , both in percent. SR_

t^{\ast}

measures the fraction of evaluation episodes that successfully complete the task within the task-specific time budget

t^{\ast}

(in environment steps). AUC 0..B integrates the success rate over the full episode horizon

[0,B]

, capturing both final performance and learning speed. Bracketed values denote bootstrap 95% confidence intervals over random seeds (

n=6

). Bold numbers indicate the best mean performance among the baselines.

Fig. 9 : Policy robustness under external perturbations and clutter. VLAJS maintains performance where VLA-only policies fail.

Policy	Lift Cube	Pick & Place	Peg Reorientation
OpenVLA-best	47%	40%	–
VLAJS (zero-shot)	70%	80%	20%

TABLE II : Real robot deployment success rates (20 trials per task).

Algorithm	SR_ $t^{\ast}$ $\uparrow$ $(\%)$	Wins	AUC $\uparrow$ $(\%)$	Wins
PPO	0.0	0	7.2	0
Sparse RPD (OpenVLA)	8.3	1	12.8	0
Sparse RPD (OpenVLA-best)	40.3	3	37.6	4

TABLE III : Macro averages across the 4 long-horizon tasks. Sparse RPD (OpenVLA-best) achieves both earlier usability (SR_

t^{\ast}

) and substantially higher overall sample efficiency (AUC), while PPO fails to achieve meaningful success within the training horizon.

Panel of Figure 12 — Fig. 10 : Comparisons on VLA teachers.

Explore similar work

Jun 30, 2026

Z-1: Efficient Reinforcement Learning for Vision-Language-Action Models

Vision-Language-Action (VLA) models offer a promising framework for robotic manipulation by connecting language instructions, visual observations, and continuous control. However, most existing policies remain limited by behavior cloning or supervised fine-tuning (SFT) from fixed demonstrations, which provides limited opportunity to improve from the policy's own failures. In this paper, we present Z-1, a reinforcement learning (RL) post-training framework for flow-based VLA models. Built on top of

π_{0.5}

, Z-1 uses only publicly released RoboCasa demonstrations for SFT and then applies a task-wise Group Relative Policy Optimization (GRPO) strategy across

24

standard RoboCasa tasks. To improve the efficiency and stability of online optimization, Z-1 combines shared-prefix rollout construction, tree-structured trajectory branching, completion-aware reward calibration, and selective joint training of VLM and Action Expert. Across all

24

RoboCasa tasks, Z-1 achieves an average success rate of

80.6\%

, improving over its SFT initialization by

13.2\%

points and outperforms the published sota models. These results show that systematic GRPO post-training can substantially improve flow-based VLA policies without additional private demonstrations.

Lang Cao, Renhong Chen, Luyi Li +3

Zioneer Robot Team

Robot Skill Learning Efficient VLA Models Robotic Manipulation Vision-Language-Action Models RL for VLMs

Ask PDF

May 25, 2026

EXPO-FT: Sample-Efficient Reinforcement Learning Finetuning for Vision-Language-Action Models

The ability to efficiently and reliably learn new tasks has been a foundational challenge in robotics. Vision-Language-Action (VLA) models have demonstrated strong generalization across diverse manipulation tasks, yet pretrained policies consistently fall short of the reliability required for real-world deployment. Reinforcement learning (RL) fine-tuning offers a promising path to bridge this gap, but existing approaches either train from scratch without fully leveraging pretrained priors, or fine-tune VLAs without achieving the sample efficiency and success rates that practical deployment demands. We present EXPO-FT, a system for stable, sample-efficient RL finetuning of pretrained VLA policies that closes this gap. Our system solves a suite of challenging manipulation tasks, including routing string lights and inserting the plug to light it up, striking a pool ball into a pocket, and inserting a flower into a wine bottle, each requiring combinations of high precision, dynamic actions, and robustness to varied initial states. Our system achieves perfect task performance (30/30 successes) across all evaluated tasks within an average of 19.1 minutes of online robot data, outperforming both prior RL-from-scratch and VLA finetuning approaches. We release an open-source codebase with the aim of facilitating broader adoption of RL finetuning of VLA models in robotics.

Perry Dong, Kuo-Han Hung, Tian Gao +2

Stanford University

Robot Skill Learning Robotic RL Efficient VLA Models Robotic Manipulation Data-Efficient Robot Learning Vision-Language-Action Models+2

Ask PDF

Sep 25, 2026

Find Something You Can't Do: Agentic Real-World Reinforcement Learning for Self-Improving VLA Models

Vision--language--action (VLA) models provide strong priors for robotic manipulation but are typically deployed as frozen policies, unable to improve from their own failures. Real-world reinforcement learning (RL) offers a path to continued improvement, yet manual environment resets and task-success supervision hinder autonomous learning. We introduce FIND, an agentic real-world RL framework that closes the loop between scene understanding, weakness-aware practice, self-evaluation, and policy improvement in a persistent workspace. FIND reframes autonomous practice as a scene-conditioned, performance-aware task-selection problem: instead of restoring a predefined scene after each rollout, it uses the resulting scene to determine what to practice next. A vision--language agent identifies feasible tasks from a predefined library, prioritizes those with lower recent success rates, and evaluates outcomes using paired pre- and post-execution observations. We instantiate FIND with a frozen

π_{0.5}

VLA and residual off-policy RL. Across eight real-world manipulation tasks, the independent human-assessed success rate improves from

55\%

71.9\%

. A representative run completes 456 autonomous episodes within 6 hours of interaction, requiring 30 scene-recovery interventions and no human-provided reward labels during online learning. Ablations and systematic evaluations further examine key design choices, agent evaluation accuracy, and human intervention requirements. Our website is made publicly available at: FIND.github.io.

Yuan Fang, Zechu Li, Haolei Tong +2

TU Darmstadt · University of Augsburg · Tongji University +1

Robotic RL Agentic RL RL for VLMs Robotic Manipulation

Ask PDF