Organizations: Tsinghua University · Shanghai AI Laboratory · Harbin Institute of Technology · Zhongguancun Academy · Peking University · Carnegie Mellon University
Simulation offers a scalable and low-cost way to enrich vision-language-action (VLA) training, reducing reliance on expensive real-robot demonstrations. However, most sim-real co-training methods rely on supervised fine-tuning (SFT), which treats simulation as a static source of demonstrations and does not exploit large-scale closed-loop interaction. Consequently, real-world gains and generalization are often limited. In this paper, we propose an RL-based sim-real Co-training (RL-Co) framework that leverages interactive simulation while preserving real-world capabilities. Our method follows a generic two-stage design: we first warm-start the policy with SFT on a mixture of real and simulated demonstrations, then fine-tune it with reinforcement learning in simulation while adding an auxiliary supervised loss on real-world data to anchor the policy and mitigate catastrophic forgetting. We evaluate our framework on four real-world tabletop manipulation tasks using two representative VLA architectures, OpenVLA and π0.5, and observe consistent improvements over real-only fine-tuning and SFT-based co-training, including +24% real-world success on OpenVLA and +20% on π0.5. Beyond higher success rates, RL co-training yields stronger generalization to unseen task variations and substantially improved real-world data efficiency, providing a practical and scalable pathway for leveraging simulation to enhance real-robot deployment.
Figures & tables
Figure 1: Training paradigms. RL-Co uses simulation for RL interaction while anchoring updates with real-world SFT.
Figure 2: Overview of RL-Co. Stage I initializes the VLA policy with mixed real and simulated demonstrations. Stage II fine-tunes it with simulation RL while using real-world SFT as a regularizer to preserve deployable behaviors.
Figure 3: Environment visualizations. (a) Real and simulated tabletop tasks used in our sim-real co-training experiments. (b) Examples of visual domain-randomized simulation and data-augmented real observations used to test whether visual diversity alone explains the gains.
VLA Model
Experiment Setting
Pick and Place
Push Cube
Open Drawer
Close Drawer
Avg
OpenVLA
Real-Only Training
6.3 ± 0.0
20.0 ± 13.3
0.0 ± 0.0
10.0 ± 10.0
9.1 ± 5.8
SFT Co-Training
23.4 ± 4.7
51.7 ± 5.0
0.0 ± 0.0
85.0 ± 5.0
40.0 ± 3.7
RL-Co (Ours)
57.8 ± 10.9
68.3 ± 11.7
35.0 ± 15.0
95.0 ± 5.0
64.0 ± 0.7
π0.5
Real-Only Training
71.9 ± 9.4
0.0 ± 0.0
0.0 ± 0.0
35.0 ± 15.0
26.7 ± 1.4
SFT Co-Training
68.8 ± 9.4
10.0 ± 3.3
10.0 ± 0.0
95.0 ± 5.0
45.9 ± 4.4
RL-Co (Ours)
81.3 ± 9.4
18.3 ± 1.7
65.0 ± 5.0
100.0 ± 0.0
66.1 ± 4.0
Table 1: Comparison of real-world success rates under different training paradigms. We compare our RL-Co approach with real-only SFT and SFT co-training across four tabletop manipulation tasks, evaluated on both OpenVLA and π0.5 . Results are reported in terms of success rate (SR, % ). Values are the mean ± population standard deviation over two evaluation sessions of one trained checkpoint. Avg macro-averages tasks within each session before computing the mean and standard deviation.
Figure 4: Comparison of generalization under unseen settings. Solid bars show unseen success rate; pale bars show in-distribution success rate.
Method
SR (%)
Heavy DR zero-shot
10.9±7.8
Cosmos augmentation
67.2±1.6
RL-Co (Ours)
81.3±9.4
Table 2: Targeted visual-diversity comparison. Success rates on Pick and Place with π0.5 under the same real-world evaluation protocol.
Figure 5: Mechanistic ablations of RL-Co. (a) Simulation SFT initialization enables efficient RL training. (b) Real-world supervision in both stages improves deployment. (c) RL co-training remains effective across co-training ratios α and regularization weights β . Shaded regions indicate standard deviation where applicable.
Figure 6: Real-data efficiency. Shaded regions indicate standard deviation.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Real-world Setup. The real-world evaluation platform includes a tabletop workspace, a Franka Panda robotic manipulator fixed to the table, and a RGB camera for visual perception. All objects are positioned on the table surface.
Figure 8: Visualization of Four Tabletop Manipulation Tasks. For each task, we present one successful trajectory and uniformly sample seven frames along the execution. Each row corresponds to a single trajectory shown from start to completion.
Figure 9: Manipulated Objects in Simulation and the Real World. The left panel shows the objects used in simulation, while the right panel presents the real-world objects. All simulated objects are used during training. The real-world objects are divided into training objects and unseen objects for generalization evaluation.
Figure 10: Initial Regions for Manipulative Objects. For the Pick and Place task, the bowl is placed within the orange region, while the objects are initialized in the blue region. For the Push Cube task, each cube is initialized within its corresponding orange region. For the Open / Close Drawer tasks, the front edge of the drawer is initialized within the orange region.
Perturbation ID
Translation (cm)
Rotation (deg)
P1
(+5,0,0) (forward)
+30∘
P2
(−5,0,0) (backward)
−30∘
P3
(0,+5,0) (left)
+30∘
P4
(0,−5,0) (right)
−30∘
P5
(0,0,+5) (upward)
+30∘
Appendix
Table 3: Robot initial state perturbations applied to the TCP in the Pick and Place task. Translations are defined in the world frame, and rotations are applied around the vertical ( z ) axis.
Figure 11: Simulation Training Results. We report the simulation success rates across all settings during RL training.
Figure 12: Execution sequence of the Peg Insertion task. The robot must align the grasped peg with the square hole and insert it into the base.
Method
Real SR (%)
SFT-based sim–real co-training
32.5
RL-Co (Ours)
42.5
Appendix
Table 4: Real-world success rate on the Peg Insertion task. The task uses the π0.5 backbone, 50 real-world demonstrations, and 500 simulated trajectories.
Figure 13: Correlation between simulation and real-world success during RL co-training. We evaluate intermediate checkpoints of π0.5 on the Pick and Place and Open Drawer tasks. Real-world performance generally improves together with simulation performance.
Figure 14: Simulation success rate of one-stage RL co-training. Without Stage I task-specific SFT initialization, the policy maintains near-zero success in simulation even after several million interaction steps.
Method
Real SR (%)
One-stage RL co-training
0.0
RialTo-style distillation [ 51 ]
75.0
RL-Co (Ours)
81.3
Appendix
Table 5: Additional baselines on Pick and Place with π0.5 . Results are reported as real-world success rate without standard deviation because each additional baseline was evaluated once.
Training condition ( π0.5 , Open Drawer )
Real SR (%)
SFT co-training with MimicGen demonstrations
10.0
SFT co-training with strong RL rollouts
20.0
RL-Co
65.0
Appendix
Table 6: Strong-rollout SFT control using the same 20 real demonstrations.
Parameter Names
Setting
Pick and Place
Push Cube
Open Drawer
Close Drawer
General
Number of Real Demos
50
50
20
30
OpenVLA
Co-training ratio α
0.5
0.5
0.5
0.5
SFT learning rate
5×10−4
5×10−4
5×10−4
5×10−4
Regularization weight β
0.1
0.01
0.01
0.01
Actor learning rate
10−4
10−4
10−4
10−4
Critic learning rate
3×10−3
3×10−3
3×10−3
3×10−3
Appendix
Table 7: Hyperparameter settings for different VLA models and tasks. We report task-specific hyperparameters used for OpenVLA and π0.5 across four real-world manipulation tasks.
Operation
Time
Note
Simulator setup
∼ 1 human working day
Shared across four tasks
Collect 200 real demonstrations
∼ 2 human working days
Working hours only
MimicGen generation
1.5 h (RTX 4090)
Unattended computation
RL co-training
9.43 h ( 2× H100)
Unattended computation
Appendix
Table 8: Time costs for Open Drawer . Simulator setup is shared across the four tasks. Computation times are wall-clock durations on the listed hardware.
Figure 15: Simulation success during π0.5 RL co-training versus wall-clock time across four tasks.
Vision-Language-Action (VLA) models can generalize across diverse manipulation tasks, but their imitation-learning-based policies remain brittle in precise physical interactions due to compounding execution errors; Can a reinforcement learning policy trained purely in simulation improve the robustness of real-world VLAs zero-shot? Residual RL, which learns a corrective policy on top of a frozen VLA, offers a natural framework, but existing approaches face a fundamental sim-to-real dilemma: privileged-state methods require lossy distillation for deployment; image-based methods suffer from the visual domain gap; and real-world RL is costly and unsafe. We propose an object-centric residual RL framework that refines VLA actions using object poses, enabling a compact observation space that transfers consistently between simulation and reality. To align the two domains, we additionally replay the same teleoperation demonstrations in simulation to train a sim counterpart of the real-world VLA. The residual RL policy is trained only in simulation with pose noise injection and dropout, and transfers zero-shot to the real robot. Across five manipulation tasks on a real Franka Research 3 (FR3) robot, our method improves the success rate from 42% to 76% zero-shot, and the improved rollouts can be further reused to retrain the base VLA for self-improvement without additional teleoperation. Project page: https://www.microsoft.com/en-us/research/articles/object-centric-residual-rl/
Kinam Kim, Namiko Saito, Heecheol Kim +3
KAIST, South Korea · Microsoft Research Asia - Tokyo, Japan · The University of Tokyo, Japan
The strong performance of large vision-language models (VLMs) trained with reinforcement learning (RL) has motivated similar approaches for fine-tuning vision-language-action (VLA) models in robotics. Many recent works fine-tune VLAs directly in the real world to avoid addressing the sim-to-real gap. While real-world RL circumvents sim-to-real issues, it inherently limits the generality of the resulting VLA, as scaling scene and object diversity in the physical world is prohibitively difficult. This leads to the paradoxical outcome of transforming a broadly pretrained model into an overfitted, scene-specific policy. Training in simulation can instead provide access to diverse scenes, but designing those scenes is also costly. In this work, we show that VLAs can be RL fine-tuned across broad scene and object distributions and with reduced labor by leveraging 3D world generative models. Using these models together with a language-driven scene designer, we generate 100 diverse interactive scenes containing unique objects and backgrounds, enabling scalable and highly parallel policy learning. Starting from a pretrained imitation baseline, our approach increases simulation success from 9.7% up to 79.8% while achieving a 1.25× speedup in task completion time. We further demonstrate successful sim-to-real transfer enabled by the quality of the generated scenes together with domain randomization, improving real-world success from 21.7% to 75% and achieving a 1.13× speedup. Finally, we further highlight the benefits of leveraging the effectively unlimited data from 3D world generative models through an ablation study showing that increasing scene diversity directly improves zero-shot generalization.
Vision-Language-Action (VLA) models have become a prominent paradigm for mapping multimodal inputs, including semantic instructions, visual observations of the scene, and proprioceptive observations, to robot actions. Most state-of-the-art models predict actions in the end-effector pose space as sequences of action chunks. Training and evaluating these models requires large-scale collections of real-world demonstrations, pairing robot actions with the corresponding visual and proprioceptive observations. Collecting such data on real hardware typically relies on human teleoperation, making the process costly, time-consuming, and difficult to scale. We present an open-source sim-to-real experimental protocol that addresses this bottleneck: expert trajectories generated in simulation are replayed open-loop on a real Franka FR3 setup, where the corresponding real visual and proprioceptive observations are recorded and converted into a format compatible with VLA training. The same deployment stack is then reused, in closed-loop, to evaluate a trained policy on that setup, so that data collection and evaluation share an identical hardware configuration. Because each real recording is paired with the simulated trajectory that produced it, the protocol also yields a direct measurement of the sim-to-real gap. We release the collected datasets on Hugging Face together with the pipeline source code https://gitlab.isir.upmc.fr/kappel/sim2real_public_chunk_control.