Fine-Tuning VLAs with Self-Demonstrated Generative Control for Multi-Task Manipulation

Prachi Garg; Steve Xing; Prahit Yaugand; Saurabh Gupta; Derek Hoiem

cs.ROAug 19, 2026

Fine-Tuning VLAs with Self-Demonstrated Generative Control for Multi-Task Manipulation

Authors: Prachi Garg, Steve Xing, Prahit Yaugand, Saurabh Gupta, Derek Hoiem

Organizations: University of Illinois Urbana-Champaign

Abstract

State-of-the-art vision-language-action (VLA) models such as $π_{0.5}$ exhibit strong semantic understanding, instruction following and task behavior. However, when deployed on new robots, even minor mismatches in hardware configuration relative to pretraining can cause severe performance drops. Finetuning the VLA on in-domain expert data from the new embodiment improves performance on the expert task but leads to a loss in its original instruction following and behavioral priors. In this paper, we propose a self-supervised method that generates online interaction rollouts from the zero-shot VLA as additional training data for finetuning. Our experiments show this finetuning scheme yields strong multi-task policies that, on the target robot, (1) inherit prior tasks distilled from the zero-shot model, (2) enable generalist instruction following, while (3) learning new skills from expert data with improved sample efficiency. We demonstrate the success of our approach across test sets probing generalization on a real ALOHA robot and a new simulation benchmark in RoboTwin. Video results are available at https://self-supervised-control.pages.dev/

Figures & tables

Figure 1: Left : Zero-shot $\pi_{0.5}$ policy follows instructions but cannot complete grasps on our robot. Center : Fine-tuning on expert “pick-up” data alone loses instruction following, reaching for the wrong object (rows 1–2), and loses the place motion after pick (rows 2–3). Right (Ours) : A single multi-task policy, jointly trained on 14 minutes of expert-supervised teleoperation for “pick-up” and self-supervised generative rollouts from $\pi_{0.5}$ for “pick-and-place” succeeds across all task families.

Figure 2: $\pi_{0.5}$ VLA zero-shot is able to semantically localize and follow instructions to reach a green octagon and marker placed close-by on our ALOHA platform.

Figure 3: A new way to fine-tune pretrained VLAs. We combine two multi-task data sources: (1) Bottom row: expert-supervised demonstrations on a new task (E.g. “Pick up the red cube”), and (2) Top row: real-world interaction data generated by rolling out the frozen base policy on self-supervised tasks drawn from its pretraining distribution on the target robot (E.g. “Pick up the spoon and place it in the blue bowl”). Rehearsing the expert demonstrations alongside the base policy’s self-demonstrations preserves its semantic understanding, instruction following, and behavioral priors; yielding a multi-task policy that can be deployed to perform both new and pretrained tasks on the target robot. Red arrows ( ${\color[rgb]{1,0,0}\dashrightarrow}$ ) denote action chunks generated by the base VLA policy.

Figure 4: Our simulation benchmark in RoboTwin is designed to test generalization to new object layouts in task families that are (1) expert-supervised, $T_{ES}$ (stacking two blocks) and (2) self-supervised, $T_{SS}$ (broader set of pick-place, lifting, opening tasks). We also test generalization to (3) novel objects within expert tasks, $T_{NO}$ and (4) new compositions of expert skills, $T_{NC}$ .

Benchmark 1: Pick up object (real ALOHA robot, right arm only)
	Split	Task prompts	Scene / objects	Size (Seeds)
TRAIN	Expert demos	Pick up the red/green cube	Red & green cube; 8 unique cube positions	30
	Self-demos	Pick up X and place in Y; Pick up laundry place in basket	Varied objects & scenes	14
TEST	$T_{\text{ES}}$	Pick up the red/green cube	10 new positions; tested in pairs	20
	$T_{\text{SS}}^{1}$	Pick up X and place in Y	Diverse objects X and containers Y	20
	$T_{\text{SS}}^{2}$	Pick up laundry place in basket	9 clothes, 2 baskets	20

Table 1: Summary of training data and test sets across all three benchmarks. Self-demos are typically on task prompts the base policy is familiar with. Our test sets.

T_{\text{ES}}

: expert supervised tasks,

T_{\text{SS}}

: self-supervised tasks,

T_{\text{NO}}

: novel object generalization,

T_{\text{NC}}

: novel composition of

T_{\text{ES}}

skills.

	$T_{ES}$ : Expert Supervised Tasks		$T_{NO}$ : Novel Object Tasks		$T_{SS}$ Self-Supervised Tasks
	(“Pick up red/green cube”)		(“Pick up X”)		(“Pick up X and place it in Y”)			(“Pick up laundry and place in basket”)
Method	IF (pre-grasp)	SR	IF (pre-grasp)	SR	IF (pre-grasp)	IF (post-grasp)	SR	IF (pre-grasp)	IF (post-grasp)	Partial SR
$\pi_{0.5}$ Zero-Shot	80	0	70	0	70	25	0	65	30	5
$\pi_{0.5}$ Single Task ES	90	50	40	25	45	25	0	—	—	—
$\pi_{0.5}$ Multi-Task ES, Single Object/Scene	65	65	45	30	55	35	10	—	—	—
$\pi_{0.5}$ Multi-Task ES	100	65	50	50	75	0	0	100	50	10

Table 2: ALOHA Robot. Multi-task policy results across a variety of pick up-only and pick-and-place tasks. Our method not only outperforms

\pi_{0.5}

zero-shot and fine-tuning baselines for tasks that are expert demonstrated

T_{ES}

like “Pick up red/green cube”, but also enables task success on various pick and place tasks purely distilled through the zero-shot policy’s self-supervised demos. Our approach enables fine-tuning to get a MT policy ready to deploy on our robot with only 14 minutes of expert teleoperation data, while retaining strong generalization priors from the base VLA. Each row’s results are from a single model. All values are percentages over 20 trials, except Single Task ES on

T_{ES}

which uses 10 trials.

	$T_{ES}$ : Expert Supervised Tasks			$T_{NO}$ : Novel Object Tasks
	(“Pick up gear X and insert it into gear shaft X”)			(“Pick up gear X and insert it into gear shaft X”, novel gears)
Method	IF (pre-grasp)	IF (post-grasp)	SR	IF (pre-grasp)	IF (post-grasp)	SR
$\pi_{0.5}$ Zero-Shot	0	0	0	20	0	0
$\pi_{0.5}$ Multi-Task ES	100	40	30	70	40	0
$\pi_{0.5}$ Multi-Task ES+SS (Ours)	100	90	90	100	90	30

Table 3: ALOHA Robot Caterpillar. This is a complex, long-horizon bimanual task of colored peg insertion on a non-stationary board. This task is out-of-distribution w.r.t. base policy’s pretraining. Using expert- and self-supervised fine-tuning ( Multi-task ES+SS ) improves all metrics across in-distribution gear colors (

T_{ES}

) and out-of-distribution gear colors (

T_{NO}

). (%, 10 trials)

Figure 5: A rollout from caterpillar test set showing successful alignment of orange gear on peg.

Table 9

	$T_{ES}$ : Expert Supervised		$T_{NO}$ : Novel Object		$T_{SS}$ : Self-Supervised Tasks
	(“Pick up red/green cube”)		(“Pick up X”)		(“Pick up X and place it in Y”)			(“Pick up laundry and place in basket”)
Method	IF (pre-grasp)	SR	IF (pre-grasp)	SR	IF (pre-grasp)	IF (post-grasp)	SR	IF (pre-grasp)	IF (post-grasp)	Partial SR
$\pi_{0.5}$ MT ES+ES (all tasks)	100	95	65	60	90	90	65	100	100	90
$\pi_{0.5}$ MT ES+SS (Ours)	100	90	55	55	90	90	55	100	95	40

Table 6: ALOHA Robot: comparison with full expert supervision. ES+ES is an upper-bound reference that collects expert human teleoperation on our robot for all tasks, both

T_{ES}

and

T_{SS}

, replacing our self-supervised demonstrations with expert data on the same scenes and task list. MT ES+SS (Ours) is the same checkpoint from Table 2 . (%, 20 trials)

Method	IF (correct obj.)	Push motion	Picks instead $\downarrow$	SR (push correct obj.)
$\pi_{0.5}$ Zero-Shot	45	35	0	30
$\pi_{0.5}$ MT ES+ES (all tasks)	85	10	35	5
$\pi_{0.5}$ MT ES+SS (Ours)	85	70	10	60

Table 7: ALOHA Robot: Zero-shot evaluation on a held-out skill family (“Push objects”). The same two checkpoints from Table 6 , evaluated on a skill absent from both fine-tuning mixtures. Push motion : executed a push on any object; Picks instead : picked the object up. (%, 20 trials)

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: This figure demonstrates rollouts comparing our method with the Multi-Task expert-only baseline and zero-shot $\pi_{0.5}$ . As mentioned in main paper section 5.2 , even when the base policy shows correct instruction following and semantic understanding, it can’t grasp objects due to a potential calibration gap on our embodiment. In these rollouts 3 and 5, the Multi-Task ES baseline picks up the wrong object and stops, whereas our method has stronger recovery behavior to pick up the correct object and successfully places objects.

Figure 7: Our method is able to better localize and grasp the fork (rollout 15), does not stop after pick up (rollout 9). In rollout 11, it successfully places the green cube but also goes for the purple afterwards which is an artifact we observed.

Figure 8: In these rollouts, we show the last frame of the test episode in the last column. This is a particularly challenging task to even distill because the base policy rollouts are very noisy. The common failure mode for the Multi-Task ES baseline with expert-only ‘Pick up X’ data is that even after successful grasps, it overfits to the ‘Pick up’ task and stops (episodes 1 and 14). In some cases, it moves towards the basket (episodes 4 and 10) but even if its directly above the basket, it does not release the cloth. This is consistent with the ‘Pick up X and place in container Y’ test set and the baseline cannot tell the difference between pick up and pick up and place. Our method on the other hand, goes above the edge of the basket and releases the cloth. Due to the difficulty of the task and our focus on instruction following, we consider partial success (such as any part of the cloth placed inside the basket when the episode ends) as a success.

Figure 9: Caterpillar Benchmark, $T_{ES}$ test set rollouts. [Rollout 6] The purple and blue colors are hard to tell and our method is able to do that (notice the purple gear is placed on purple shaft in ours and on the blue shaft in the baseline. [Rollout 7] Several of the failures of the baseline is due to unstable grasps, resulting in the gears slipping out and being thrown far away with no recovery behavior. [Rollout 5] Our method placed the purple gear on the purple shaft as compared to the baseline which placed it on top of the green shaft.

Figure 10: Our stationary ALOHA-1 [ 43 ] platform with two ViperX 300 follower arms and two WidowX 250 leader arms for collecting expert human demonstrations via teleoperation. It has parallel jaw grippers. We use RGB images from the top scene camera, left wrist camera, and the right wrist camera as input. All of these are Intel RealSense D435 cameras, with raw image resolution of $640\times 480$ pixels.

Episode	Task Prompt
0	Pick up the laundry and place it in the basket.
1	Pick up the blue cloth and place it in the basket.
2	Pick up the laundry and place it in the basket.
3	Pick up the laundry and place it in the basket.
4	Pick up the laundry and place it in the basket.
5	Pick up the red triangular object and place it in the box.

Appendix

Table 8: Task prompts

\mathcal{P}_{\text{SS}}

for collecting rehearsal data

\mathcal{D}_{\text{SS}}

Configuration	Replay ratio $\rho$	SS sampling	Avg. $T_{SS}$	Avg. $T_{ES}$
$\pi_{0.5}$ ES+SS (Ours) , $\|\mathcal{D}_{SS}\|=10$ /task	1:1	Uniform	69.2	97.0
Effect of replay ratio $\rho$
Natural ratio (proportional sampling)	$\sim$ 1:2	Uniform	56.4	97.0
Effect of SS sampling distribution (with $\rho{=}1{:}1$ )
Task-balanced (weighted) sampling	1:1	Task-balanced	66.2	99.0
Effect of buffer size $\|\mathcal{D}_{SS}\|$ (with $\rho{=}1{:}1$ )

Appendix

Table 9: Ablation of self-supervised rehearsal dataset

\mathcal{D}_{SS}

composition and sampling on the RoboTwin validation set. We vary (i) the replay ratio

\rho

, defined as the ratio of SS-to-ES samples per minibatch; (ii) the SS sampling distribution, which determines how samples are drawn from

\mathcal{D}_{SS}

at the task level— Uniform draws each

T_{SS}

task with equal probability, while Task-balanced weights tasks inversely to their frame counts to up-weight shorter trajectories; and (iii) the Rehearsal buffer size

|\mathcal{D}_{SS}|

, the number of self-demo episodes per task. Best performance is with

\rho=1:1

, uniform sampling of SS tasks and rehearsal dataset size

\geq

expert data size.

	Stage 1 Training Tasks
Experiment	Place Dual Shoes	Beat Block Hammer	Pick Dual Bottles	Place Container Plate	Handover Mic	Place Empty Cup	Place Bread Basket	Lift Pot	Open Microwave	Put Bottles Dustbin	Avg. Stage 1
$\pi_{0.5}$ Zero-Shot	–	–	–	–	–	–	–	–	–	–	0
$\pi_{0.5}$ Flow (default openpi)	72	100	100	100	100	98	82	98	52	94	89.6
$\pi_{0.5}$ FAST+Flow+Knowledge Insulation	2	34	42	90	30	78	28	66	12	25	40.7
$\pi_{0.5}$ FAST+Flow (Base Policy)	76	94	98	100	100	98	82	98	72	90	90.8

Appendix

Table 10: RoboTwin Stage 1. Test set results.

\pi_{0.5}

base policies trained on the 10 RoboTwin training tasks; all Stage 2 experiments are initialized from

\pi_{0.5}

FAST+Flow.

	$T_{SS}$ : Self-Supervised Tasks											$T_{ES}$ : Expert Supervised Tasks
Experiment	Place Dual Shoes	Beat Block Hammer	Pick Dual Bottles	Place Container Plate	Handover Mic	Place Empty Cup	Place Bread Basket	Lift Pot	Open Microwave	Put Bottles Dustbin	Avg. $T_{SS}$	Stack $R\rightarrow G$	Stack $G\rightarrow R$	Avg. $T_{ES}$
$\pi_{0.5}$ Zero-Shot	–	–	–	–	–	–	–	–	–	–	0	–	–	–
Mid-Training Experiments
$\pi_{0.5}$ RoboTwin Base Policy	76	94	98	100	100	98	82	98	72	90	90.8	–	–	–
Post-Training Experiments: All models initialized from RoboTwin Base Policy
Rehearsal, frame-wise (Oracle)	58	92	100	96	100	94	86	92	56	82	85.6	96	98	97

Appendix

Table 11: Full RoboTwin Simulation results, all 10

T_{SS}

tasks. Post-training fine-tunes on new expert tasks (

T_{ES}

) using self-demos from the mid-training stage (

T_{SS}

) to prevent forgetting. Success rates (%,

\uparrow

) averaged across 50 seeds per task. Bold marks our method; oracle rows (shaded) assume access to mid-training data and are upper-bound references. This is the full version of Table 5 .

	$T_{NO}$ : Novel Object Tasks					$T_{NC}$ : Novel Composition Tasks
Experiment	Cylinder on Cube	Sphere on Cube	Sphere + Distractor	Cylinder + Distractor	Avg. OOD objects	Stack Blocks Three	Stack Bowls Two	Stack Bowls Three	Avg. OOD compose
Post-Training Experiments: All models initialized from RoboTwin Base Policy
Rehearsal (Oracle)	42	46	12	20	30.0	2	44	0	15.33
Parameter Efficient Fine-tuning, Multi-Task ES
LoRA	46	66	22	16	37.5	0	6	0	2.00
Freeze SigLIP+VLM, Tune AE	22	26	10	4	15.5	0	12	0	4.00

Appendix

Table 12: Generalization of the stack two blocks task to (i) new objects

T_{NO}

: cylinder and sphere and (ii) composing new tasks

T_{NC}

: stacking bowls and three cubes. Success rates (%,

\uparrow

Figure 11: Here, we discuss four failure modes of our method. [Row 1] This is an example when the robot repeatedly picks and places the object inside the box, after successfully picking and placing it from the table. This is the failure mode we discussed in Section 6 of main paper, but it can also be seen as correct behavior since we don’t specify that the policy not pick it up if its inside the box. [Row 2] After a successful pick and place of the red cube, the robot pushes and lifts the edge of the box. This is a kind of noisy behavior we observed in the self-demos, which is being distilled into our fine-tuned policy. [Row 3] The laundry self-demos can be unsafe with a majority of them grabbing the edge of the container and sometimes exhibiting behavior unsafe for the wrist camera as it topples the basket and tries to go inside it. [Row 4] The most common failure mode we observe for laundry is that it loosens its grasp or releases the cloth at the edge of the basket resulting in it falling outside. Having seen no expert data for pick and place, these tasks are limited by the base policy’s distilled failure behavior and can be iteratively improved.

Figure 12: $T_{SS}$ Self-supervised tasks in our RoboTwin benchmark. Each row shows five keyframes from a rollout on a Stage 1 task, with the task instruction shown above. These 10 tasks are used exclusively as rehearsal tasks. We collect data through online self-demonstration rollouts in the simulator during Stage 2 fine-tuning and are never seen as expert demonstrations.

Figure 13: $T_{ES}$ Expert-supervised tasks in our RoboTwin benchmark. We collect motion planner expert data for these two stack block tasks in Stage 2.

Figure 14: $T_{NO}$ Novel-Object tasks in our RoboTwin benchmark. These stack novel objects instead of the red and green cubes part of expert fine-tuning data.

Figure 15: $T_{NC}$ Here we compose new tasks within the skill family of stacking. These are challenging tasks since there is no self-supervised or expert-supervised data for it. The text prompt in stack two bowls is different from the text prompt in stack bowls tasks from $T_{ES}$ and $T_{NO}$ .

Figure 16: ALOHA test set $T_{ES}$ , main paper Table 2 .

Figure 17: ALOHA test set $T_{NO}$ , main paper Table 2 .

Figure 18: ALOHA test set $T_{SS}^{1}$ , main paper Table 2 .

Figure 19: ALOHA test set $T_{SS}^{2}$ , main paper Table 2 .

Figure 20: ALOHA test set $T_{ES}$ , main paper Table 3 .

Figure 21: ALOHA test set $T_{NO}$ , main paper Table 3 .

Explore similar work

EXPO-FT: Sample-Efficient Reinforcement Learning Finetuning for Vision-Language-Action Models
May 25, 2026Perry Dong, Kuo-Han Hung, Tian Gao +2Robot Skill Learning Robotic RL
Stanford University
Axiom-0: Hierarchical Self-Refinement for Vision-Language-Action Policies
Oct 9, 2026Renbin Shu, Jing Yang, Xiangyuan You +7Vision-Language-Action Models Robotic Manipulation
4Axiom Robotics Team
RL Token: Bootstrapping Online RL with Vision-Language-Action Models
Apr 24, 2026Charles Xu, Jost Tobias Springenberg, Michael Equi +4RL for Robotics Robot Skill Learning
Physical Intelligence

May 25, 2026J/K move · Enter open · S save

EXPO-FT: Sample-Efficient Reinforcement Learning Finetuning for Vision-Language-Action Models

Perry Dong, Kuo-Han Hung, Tian Gao, Dorsa Sadigh+1

Stanford University

Robot Skill Learning Robotic RL Efficient VLA Models Robotic Manipulation Data-Efficient Robot Learning Vision-Language-Action Models+2

Open paper AskPDF arXiv

The ability to efficiently and reliably learn new tasks has been a foundational challenge in robotics. Vision-Language-Action (VLA) models have demonstrated strong generalization across diverse manipulation tasks, yet pretrained policies consistently fall short of the reliability required for real-world deployment. Reinforcement learning (RL) fine-tuning offers a promising path to bridge this gap, but existing approaches either train from scratch without fully leveraging pretrained priors, or fine-tune VLAs without achieving the sample efficiency and success rates that practical deployment demands. We present EXPO-FT, a system for stable, sample-efficient RL finetuning of pretrained VLA policies that closes this gap. Our system solves a suite of challenging manipulation tasks, including routing string lights and inserting the plug to light it up, striking a pool ball into a pocket, and inserting a flower into a wine bottle, each requiring combinations of high precision, dynamic actions, and robustness to varied initial states. Our system achieves perfect task performance (30/30 successes) across all evaluated tasks within an average of 19.1 minutes of online robot data, outperforming both prior RL-from-scratch and VLA finetuning approaches. We release an open-source codebase with the aim of facilitating broader adoption of RL finetuning of VLA models in robotics.

Abstract

Explore similar work

EXPO-FT: Sample-Efficient Reinforcement Learning Finetuning for Vision-Language-Action Models

Axiom-0: Hierarchical Self-Refinement for Vision-Language-Action Policies

RL Token: Bootstrapping Online RL with Vision-Language-Action Models