Fine-tuning Vision-Language-Action (VLA) models commonly relies on human teleoperation demonstrations, while reinforcement learning (RL) with sparse binary rewards faces an exploration challenge when successful trajectories are rarely sampled. We propose SynthDemo-RL, a teacher-student framework in which an automated teacher converts simulator-privileged state into successful manipulation trajectories, a VLA student is distilled from them by supervised fine-tuning (SFT), and PPO with binary task-success rewards refines the student. We study reward coverage, the fraction of tasks for which at least one success is observed under the fixed evaluation protocol, as a complement to the average success rate. On LIBERO-PRO, a public benchmark of perturbed LIBERO tasks for which no demonstrations exist, 27 of 57 scored tasks are at exactly 0% success for a pi_0.5 policy fine-tuned on the original LIBERO tasks. Direct PPO from this policy, under the same PPO recipe and the same RL compute as SynthDemo-RL's refinement stage, rescues 10 of these 27 tasks and leaves 17 at 0%. SynthDemo-RL, with 50 synthesized trajectories per task and no new human demonstrations, rescues all 27 and reaches average success rates of 97.8% and 97.1% on the Position and Task axes of LIBERO-PRO, respectively. On standard LIBERO, the same pipeline reaches 96.0% with no human demonstrations, within 1.7 points of pi_0.5 trained on 50 human demonstrations per task. We further validate the pipeline on RoboTwin 2.0 and verify that trajectories from a policy trained in a MuJoCo twin execute open-loop on a physical robot.
Figures & tables
Fig. 1 : SynthDemo-RL: VLA task adaptation without new human demonstrations. Human demonstrations exist only for the original task (green marker). LIBERO-PRO asks for the other bowl (red), and a VLA fine-tuned on the original tasks has no observed successes on 27 of the 57 perturbed tasks under the evaluation protocol. An automated teacher solves the perturbed task from simulator-privileged state, SFT distills its successful trajectories into the VLA student, and PPO refines the student past its teacher. Bottom: per-task success (Table III ). Direct PPO leaves 20 of the 57 tasks with no observed success after adaptation (17 unrescued of the 27, plus 3 driven to 0%), while ours leaves none.
Method
Spatial
Object
Goal
Avg.
Zero human demonstrations (ours)
(a) SynthDemo-Teacher (1-attempt)
79.9
98.0
43.0
73.6
(b) SynthDemo-SFT
55.3
69.8
45.1
56.7
(c) SynthDemo-RL (ours)
96.2
98.6
93.3
96.0
Human-demonstration reference
(d) π0.5 + human-demo SFT
97.8
99.0
96.2
97.7
TABLE I: Success rates (%) on LIBERO (50 trials/task). Row (a): teacher single-attempt success during data generation. Retained demonstrations are 100% successful by construction. Rows (b, c): fixed final checkpoints under the prescribed training schedules. Rows (d, d’): our human-demonstration reference (50 demos/task). Goal is the 9-task subset of Sec. IV-A1 .
Position
Task
Method
Spatial
Object
Goal †
Avg.
Coverage
Spatial
Object
Goal †
Avg.
Coverage
Before adaptation
π0.5 -LIBERO
48.4
16.8
24.7
30.0
14/29
51.2
10.8
27.0
29.7
16/28
Existing approaches (quoted) ‡
SPARK [ 27 ]
56.0
43.4
40.0
46.5
–
72.4
36.4
14.0
40.9
–
Pigey [ 28 ]
66.0
54.0
44.0
54.7
–
80.0
54.0
22.0
52.0
–
TABLE II : Task success rates (%) and reward coverage on LIBERO-PRO (50 trials/task for our runs). For evaluated policies, “Coverage” counts tasks per axis with at least one observed success in the reported trials (teacher: at least one successful synthesis; success rates use each task’s final tuned configuration). SFT success rates are averaged over three training seeds, and ± denotes the across-seed standard deviation of the axis average. Every seed achieves at least one success on all 57 tasks in 50 evaluation trials per task.
Suite
Method
0
1
2
3
4
5
6
7
8
9
Position perturbations
Spatial
Original
96
30
100
0
60
98
0
0
100
0
Spatial
Ctrl PPO
98
22
100
0
90
98
0
0
100
0
Spatial
Ours (SFT)
62
42
64
62
38
96
18
66
28
74
Spatial
Ours (+RL)
92
100
98
96
94
100
96
100
92
96
Object
Original
30
98
0
2
0
0
34
0
0
4
TABLE III: Per-task success (%) on LIBERO-PRO (50 trials/task). Original: π0.5 -LIBERO before adaptation. Ctrl: direct PPO at matched RL compute. Ours: SynthDemo SFT, then PPO under the same RL compute as Ctrl (blue rows). Goal omits Task 3. Task-Goal Task 0 is excluded and not scored (ill-posed, see the footnote in Sec. IV-A1 ). Gray cells: the 27 tasks with 0% Original success. Red cells: 0% success. Bold: best result per task.
TABLE IV: Success rates (%) on four RoboTwin 2.0 single-arm tasks. Rows (a)–(c) use 50 trajectories per task synthesized by our teacher. Rows (e, e’) use an OpenVLA-OFT backbone trained on RoboTwin’s expert demonstrations (1,000 per task for SimpleVLA-RL), with results quoted from [ 2 ] . The comparison is contextual, not controlled.
Fig. 2 : Physical executability on a real robot. Top: real camera views (red box) and the twin’s hybrid renders of the same views. Middle: the four conditions in the twin. Bottom: hardware success (%) over 20 trials per condition (target object released and at rest on its goal region). Ours executes twin rollouts open-loop, the reference runs closed-loop, so the rows are not a controlled comparison. Human demonstrations exist for stack only.
Measure
S
S †
H
δ
Action saturation (frac. frames)
0.158
0.000
0.027
1.00
vertical axis only
0.498
–
0.094
1.00
Jerk norm
0.125
0.087
0.069
0.997
Velocity norm
0.118
–
0.099
0.58
Frozen π0.5 base FM loss
0.123
0.090
0.093
0.89
Within-task DTW diversity
0.199
–
0.200
−0.02
TABLE V: How far the synthetic demonstrations are from human ones, and what closing that distance buys. Per-episode medians on LIBERO-Spatial for the original synthetic data (S), the humanized-teacher regeneration (S † ), and human demonstrations (H). Cliff’s δ compares S against H ( n=864 episodes per source; Mann–Whitney p<10−48 for every row except within-task DTW diversity, p=0.28 ). The frozen-base loss scores each dataset under the unmodified π0.5 base checkpoint with identical normalization: a descriptive measure of mismatch with the frozen model.
Fig. 3 : From teacher signature to policy behavior. Top: end-effector paths on one LIBERO-Spatial task: human demonstrations (H), teacher demonstrations (S), and successful rollouts before (SFT) and after RL, overlaid thin. The episode closest to each source’s median detour ratio (path length over start–end displacement) is bold. SFT and RL share the initial state, and all panels share one crop. Bottom: vertical action of the bold episodes; gray bands mark saturation ( ∣a∣≥0.9375 ). RL straightens the geometry (median detour ratio 3.7→3.2 vs 2.5 for human) but keeps the teacher’s saturation signature: median saturation 0.126 (SFT) and 0.133 (RL) against 0.158 (S) and 0.027 (H).