cs.ROSep 19, 2026

Transferring the Intelligence of VLMs to Robotic Control

Authors: Meng-Hao Guo, Zhe-Han Mo, Jia-Jun Wang, Yi Zhang, Kejin Wang, Yi-Xuan Deng, Jia-Peng Zhang, Yongming Rao, 

Organizations: Tsinghua University · Tencent Hunyuan

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Humans can seamlessly adapt to both physical and digital worlds, suggesting that while a digital-to-real gap exists in embodiment, environment and task, human intelligence itself may transfer across this gap. This naturally raises a fundamental question: can the intelligence of vision-language models (VLMs) similarly generalize from the digital world to the physical world for robotic control? We investigate this question through RoboDawn, a human-intuitive interface that exposes robotic control to an agentic VLM through a compact set of discrete translation, rotation, and gripper commands. Using this interface, the VLM controls a robot in a closed loop: it observes the current visual state, reasons about the next action, executes it, and adapts subsequent decisions to the resulting state. Furthermore, we introduce an in-context learning (ICL) scheme that uses a few demonstrations to ground the VLM in both interface usage and task-solving strategies. Experiments on RoboTwin 2.0 C2R and RoboDojo demonstrate that RoboDawn achieves strong performance without task-specific robot training. In the zero-shot setting, RoboDawn outperforms several strong policies trained on benchmarkspecific robot data, while a single in-context demonstration further yields substantial performance gains and establishes state-of-the-art (SOTA) results. On RoboTwin 2.0 C2R, the success rate increases from 53.2% zero-shot to 73.6% one-shot, exceeding the solid baseline π0.5 (46.0%). Similar gains are observed on RoboDojo, where success rate improves from 35.67% zero-shot to 47.17% one-shot. The same framework also transfers to real-world robots, performing block-in-basket and block stacking on Franka.

Figures & tables

Figure 1: Robo Dawn transfers general-purpose VLM intelligence from the digital world to physical robot control through a human-intuitive action interface and in-context demonstrations.
Figure 2: Overview of Robo Dawn . Robo Dawn conditions a pretrained VLM on (a) a human-intuitive semantic action interface and (b) in-context demonstrations. At each interaction round, the VLM receives (c) multi-view visual observations and robot state together with task context, execution feedback, and memory, performs (d) reasoning, and generates (e) semantic action commands. The commands are grounded into (f) planned robot motions and executed in the environment; the resulting physical outcomes are then converted into (g) grounded feedback and memory updates for the next round. This loop enables continual observation, reasoning, execution, and adaptation without task-specific parameter updates.
MethodNumber of ShotsSuccess rate (%)
FastWAM Yuan et al. (2026)Full Set1.9
StarVLA Community (2026)Full Set10.6
GalaxeaVLA Jiang et al. (2025)Full Set12.7
Xiaomi-Robotics-0 Cai et al. (2026)Full Set18.2
GR00T-1.7 Bjorck et al. (2025)Full Set20.7
X-VLA Zheng et al. (2025)Full Set20.9
Table 1: Evaluation on RoboTwin 2.0 C2R Chen et al. (2025) . Full Set follows the official training protocol, where a single policy is jointly post-trained on all 50 benchmark tasks using 50 clean demonstrations per task. 1-Shot denotes using one randomly sampled demonstration from the clean set. Success rates (%) are reported under the domain-randomized evaluation setting. CC denotes Claude Code. HarnessVLA uses the pretrained LingBot-VLA as the default VLA.
Figure 4
Table 3: Ablation studies of Robo Dawn on RoboTwin 2.0.
Method# ShotsScoreSuccess rate (%)
LingBot-VLA Wu et al. (2026)Full Set5.502.96
X-VLA Zheng et al. (2025)Full Set10.136.52
π0.5\pi_{0.5} Intelligence et al. (2025)Full Set11.416.91
Hy-Embodied-0.5-VLA Zhang et al. (2026a)Full Set13.078.80
Xiaomi-Robotics-1 Team et al. (2026b)Full Set20.0713.93
GalaxeaVLA (G0.5) Liu et al. (2026a)Full Set20.2314.88
Table 4: Evaluation on RoboDojo Chen et al. (2026a) . Full Set follows the official RoboDojo training protocol, in which a single policy is post-trained on the full RoboDojo-specific training set. Score denotes the task progress score. Compared with the full set of 42 tasks, the 34 tasks setting excludes 8 open tasks. For these 8 tasks, other methods may not have corresponding training data.
Figure 4: Representative failure cases of Robo Dawn on RoboDojo.
 Block in Basket (Franka)Block Stacking (Franka)Cloth Folding (Piper)
Success Rate9/105/100/10
Table 5: Real-world evaluation. Success rates on three real-world manipulation tasks.
More in this topic

VLMs for Robotics

Vision-language models for robotics connect visual and linguistic understanding to grounded navigation, manipulation, and robot control.

See all 657 papers

Explore similar work

CardsList
Sep 9, 2026

Show-Harness: Just a VLM Agent Can Play Robots

Foundation vision-language models (VLMs) exhibit broad intelligence about the world, yet translating this intelligence into robot control remains challenging. We present Show-Harness, an Embodied Harness that enables VLMs to "play" robots through a compact semantic interface linking intent to action. Show-Harness exposes discrete semantic action units that VLMs can naturally reason over, while embodiment-specific interpreters deterministically ground them into local robot actions, keeping the VLM directly responsible for fine-grained physical decisions. Through the same interface, Show-Harness demonstrates the feasibility of (1) directly unlocking closed-source frontier VLMs for zero-shot robot control, and (2) adapting small-scale open-source VLMs for low-cost deployment with just a few GPU-hours of fine-tuning. We further develop GUMI (GUI Manipulation Interface), which extends the same semantic action space to GUI-based demonstration collection, allowing humans and agents to "play" robots across embodiments without specialized teleoperation hardware. Extensive experiments show that Show-Harness-equipped VLM agents generalize robustly across tasks, embodiments, and environments, outperforming representative agentic and VLA paradigms. These results suggest that the right interface can unlock substantial embodied capability from foundation VLMs, without requiring additional model capacity or costly embodiment-specific pretraining.

Yanzhe Chen, Zechen Bai, Zhijun Cao +7

Show Lab, National University of Singapore

Cross-Embodiment Robot LearningLanguage-Conditioned Robot ControlVLMs for RoboticsVision-Language AgentsAgentic AI for Robotics
AskPDF
Sep 28, 2026

RoboICL: Embodied In-Context Learning with GPT-6 Astra

General-purpose vision-language models offer a promising way to zero-shot robot control: \gptastra{} excels at open-ended and language- or image-conditioned manipulation but remains substantially weaker on high-precision and long-horizon tasks. We introduce RoboICL, an in-context robot-control framework that narrows these gaps without robot-specific parameter updates or a learned VLA. RoboICL separates demonstration context, which provides recorded examples when available, from interaction memory, which accumulates the model's own actions and observed outcomes. Both use a shared observation--action--receipt--observation grammar. To preserve experience across task stages, RoboICL combines sampled demonstration blocks with bounded anchored memory. Fixed anchors keep earlier rollout interactions available for in-context learning, while the latest interaction supports immediate error correction. Across 30 RoboDojo tasks, using zero shot for Open and one demonstration elsewhere, RoboICL improves on official zero-shot \gptastra{} by 20--27 progress-score points in every category. It leads the leaderboard baselines on Memory and Open, achieves comparable performance to the strongest Precision baseline, and remains competitive on Long-Horizon. Its 30-task Overall score is 50.64, versus 33.68 for the strongest baseline. On a separate ten-task subset, RoboICL scores 60.60, within 2.00 points of the π0.5π_{0.5} + \gptastra{} hybrid approach. On three real-robot tasks, mean progress rises from 14.45 at zero shot to 63.33 at one shot and 78.89 at three shots. On two development tasks, optional Jev-gated action reuse reduces \gptastra{} calls by 33--48%. Code is available at https://github.com/Mosi-AI/RoboICL.

Fangcheng Liu, Yeqing Shen, Anda Cheng +7

Samsung Research, Beijing, China · Samsung Robotics eXperience · Shanghai Jiao Tong University

Robotic ControlICL for RoboticsLong-Horizon Robotic ManipulationLanguage-Conditioned Robot ControlRobot Learning
AskPDF
Feb 17, 2026

World Action Models are Zero-shot Policies

State-of-the-art Vision-Language-Action (VLA) models excel at semantic generalization but struggle to generalize to unseen physical motions in novel environments. We introduce DreamZero, a World Action Model (WAM) built upon a pretrained video diffusion backbone. Unlike VLAs, WAMs learn physical dynamics by predicting future world states and actions, using video as a dense representation of how the world evolves. By jointly modeling video and action, DreamZero learns diverse skills effectively from heterogeneous robot data without relying on repetitive demonstrations. This results in over 2x improvement in generalization to new tasks and environments compared to state-of-the-art VLAs in real robot experiments. Crucially, through model and system optimizations, we enable a 14B autoregressive video diffusion model to perform real-time closed-loop control at 7Hz. Finally, we demonstrate two forms of cross-embodiment transfer: video-only demonstrations from other robots or humans yield a relative improvement of over 42% on unseen task performance with just 10-20 minutes of data. More surprisingly, DreamZero enables few-shot embodiment adaptation, transferring to a new embodiment with only 30 minutes of play data while retaining zero-shot generalization.

Seonghyeon Ye, Yunhao Ge, Kaiyuan Zheng +33

NVIDIA

World Action ModelsZero-Shot Robotic ManipulationRobot Learning from VideoCross-Embodiment Robot LearningVideo Diffusion ModelsEfficient Inference for World Action Models
AskPDF