Vision-language-action (VLA) models provide broad, instruction-conditioned manipulation behaviors, but their physical execution can remain imprecise during contact-rich interaction. Residual reinforcement learning (RL) can correct such errors while keeping the VLA frozen, but real-robot RL is costly and safety-critical. We propose VLA Latent-Conditioned RL (VLaRL), which enables residual RL for frozen VLAs to be trained in simulation and deployed on real robots without real-world RL or online adaptation. The key challenge is transferring the learned residual policy despite the visual gap between simulation and reality. Rather than requiring pixel-level visual correspondence, VLaRL uses the VLA's internal vision-language latent representation to condition residual control and as the sim-to-real transfer interface, and learns a lightweight mapper that transforms simulation-derived latents toward the real latent distribution. Across four contact-rich manipulation tasks and two VLA backbones, VLaRL improves real-world success in all task-backbone combinations, while controlled ablations demonstrate the importance of both latent conditioning and latent alignment for transferring simulation-trained residual control.
Figures & tables
Fig. 1: Concept of VLaRL. A simulation-trained residual policy locally corrects the nominal behavior of a frozen VLA for precise physical execution. Sim-to-real alignment of the VLM latent representation enables the learned corrections to transfer to the real robot without real-world RL or online adaptation.
Fig. 2: Overview of VLaRL and its four-stage workflow: (1) fine-tune the VLA, (2) train the latent mapper, (3) train the residual policy in simulation with the VLA and mapper frozen, and (4) deploy the frozen VLA and residual policy on the real robot. During deployment, real-image latents are supplied directly and the mapper is not used.
Fig. 3: The four manipulation tasks and their digital-twin observations. Within each task panel, the top row shows the real left- and hand-camera images, and the bottom row shows the corresponding simulated left- and hand-camera images extracted at the same time step.
Task
Local-control challenge
Instruction
Success criterion
Press button
Precise target contact
“Press the red / blue / yellow / green button”
Beep + light
Push block
Sustained contact
“Push the blue / red block”
Displacement >10 cm
Stack cup
Grasp + spatial alignment
“Put the red / white cup inside the white / red cup”
Overlap >4 cm
Close drawer
Articulated contact
“Close the upper / bottom drawer”
Opening <1 cm
TABLE I: Evaluation tasks and success criteria.
Fig. 4: Simulation success rates across the four task categories and two VLA backbones. VLA + Mapper + Residual RL denotes VLaRL (Ours). Error bars report the variation over three seeds.
Fig. 5: Real-robot success rates over 40 trials for each task and VLA backbone. VLA + Residual RL denotes VLaRL (Ours), which improves the frozen base VLA in all eight evaluated combinations.
Fig. 6: Representative Flower real-robot rollout progressions for the base VLA and VLaRL across the four tasks. Columns show temporal progression from early to final states; rows show the base and proposed methods for the left and hand cameras.
Fig. 7: Flower ablations of the VLM latent input and sim-to-real latent mapper. “VLaRL (Ours)” is the complete system; “w/o VLM latent” retains the nominal VLA action, proprioception, and force, while “w/o Mapper” uses raw simulated VLM latents. Hatched bars denote simulation and solid bars denote real-robot evaluation. Simulation error bars report variation over three seeds; real-robot results use 40 trials per task.
Fig. 8: Transfer to objects not used during residual-policy training. The cup experiment changes appearance and language-assigned roles, while the pushing experiment changes object geometry.
Fig. 9: Integrated-gradients attribution of the residual-action norm to the VLM latent, reference VLA action, proprioception, and force. Absolute group attributions are normalized to sum to 100%.
Vision-Language-Action (VLA) models can generalize across diverse manipulation tasks, but their imitation-learning-based policies remain brittle in precise physical interactions due to compounding execution errors; Can a reinforcement learning policy trained purely in simulation improve the robustness of real-world VLAs zero-shot? Residual RL, which learns a corrective policy on top of a frozen VLA, offers a natural framework, but existing approaches face a fundamental sim-to-real dilemma: privileged-state methods require lossy distillation for deployment; image-based methods suffer from the visual domain gap; and real-world RL is costly and unsafe. We propose an object-centric residual RL framework that refines VLA actions using object poses, enabling a compact observation space that transfers consistently between simulation and reality. To align the two domains, we additionally replay the same teleoperation demonstrations in simulation to train a sim counterpart of the real-world VLA. The residual RL policy is trained only in simulation with pose noise injection and dropout, and transfers zero-shot to the real robot. Across five manipulation tasks on a real Franka Research 3 (FR3) robot, our method improves the success rate from 42% to 76% zero-shot, and the improved rollouts can be further reused to retrain the base VLA for self-improvement without additional teleoperation. Project page: https://www.microsoft.com/en-us/research/articles/object-centric-residual-rl/
Kinam Kim, Namiko Saito, Heecheol Kim +3
KAIST, South Korea · Microsoft Research Asia - Tokyo, Japan · The University of Tokyo, Japan
Vision-language-action (VLA) policies provide strong priors for language-conditioned manipulation, but remain brittle in off-nominal states requiring targeted recovery. We propose ReCoVLA -- a failure-conditioned residual recovery framework that keeps a pretrained VLA policy frozen, uses an external vision-language model (VLM) to infer the failure mode and recovery stage, and compiles a structured reward from task-relevant components. Rather than using the VLM to generate actions or rewards directly, ReCoVLA uses it as a semantic reward selector: it predicts a recovery descriptor and reward mask for in-simulation residual-policy training, followed by zero-shot sim-to-real deployment of the trained recovery policies. This decouples high-level failure understanding from low-level corrective control to support different VLAs. Experiments across short-horizon, long-horizon, and contact-rich manipulation tasks show that ReCoVLA outperforms the tested baselines on average. In simulation, our reward compiler improves average success from 36.7% for the fine-tuned π0.5 baseline to 66.7%. In physical zero-shot sim-to-real experiments, ReCoVLA achieves the best average performance, with 61.7% success.
Haodi Hu, Chung-Ta Huang, Jing Liu +4
University of Southern California, USA · Mitsubishi Electric Research Laboratories (MERL), USA · Harvard University, USA
Vision-language-action (VLA) models connect visual perception and language understanding with robot control, providing a foundation for instruction-conditioned manipulation. However, conventional pretrained VLA policies lack a mechanism for gradual adaptation through accumulated interaction experience during deployment. We therefore propose VLRLA, a general framework that equips existing VLA models with in-context reinforcement learning capabilities for deployment-time adaptation. Our approach augments a VLA model with a reinforcement learning module that learns from historical observations, actions, rewards, and termination signals. During deployment, the policy gradually refines its action generation and progressively improves task success while keeping model parameters fixed. Our approach is also designed for general applicability and compatibility with diverse learning history construction strategies and model training pipelines. Finally, through extensive experiments, we systematically investigate the learning properties of VLRLA in simulation and demonstrate its ability to adapt to unseen scenarios during real-world deployment.