A robot policy needs to understand what a task is asking for in a scene, and how to move the robot to accomplish that task. To achieve these two skills, we introduce Zero-real Agent Policy (ZAP), a recipe that teaches a small off-the-shelf vision-language model to control a robot via a simple tool harness. We use a pretrained 27B parameter model to bootstrap world understanding and reasoning, followed by supervised fine-tuning and GRPO on simple simulated tasks to teach it how to move the robot. Without observing real-world robot data during training, ZAP reaches 96.7% success on real-world tasks, matching GPT-6 Astra in our harness and outperforming the MolmoAct2 VLA (33.3%), while running 3x faster and 17x cheaper than GPT-6 Astra. Furthermore, ZAP also generalizes to an unseen embodiment as well as new objects, goals and strategies never seen during training. On a single-arm Franka, ZAP reaches 55.9% on RoboLab-120, 4.9 percentage points ahead of the leading VLA/WAM baseline.
Figures & tables
Figure 1: Motion learned in simulation carries over to new real tasks that need reasoning. Left : two of the simulated training tasks. These simple pick and place, stacking, and reorientation tasks with everyday objects teach the model only how to move. Right : the same model, trained exclusively in simulation, tested zero-shot on a real bi-manual robot. Without using any real robot data in training, ZAP generalizes to completely new objects (test tube), new goals (choosing a different hole in the rack), new strategies (mid-air handoff), and complex skills and reasoning (drawing an X on a tic-tac-toe board, playing chess). These tasks show the model composing reasoning from pretraining with motion learned in simulation.
Approach
Speed
Cost
Generalization
VLAs / WAMs
✓
✓
–
Code as Policies
✓
✓
–
Large agent as policy (frontier LLM + tools)
–
–
✓
ZAP (ours)
✓
✓
✓
Table 1: Comparing robot policy paradigms. VLAs/WAMs and Code as Policies offer fast, inexpensive execution, while frontier agents provide broader generalization at higher latency and cost. ZAP combines fast, low-cost execution with generalization to new instructions, objects, scenes, and goals.
Figure 2: Three scenes, one shared interface. Left to right: separate banana-in-bowl episodes on the simulated bi-manual YAM, the real YAM, and a simulated single-arm Franka, an embodiment never seen in training. From the agent’s perspective, the only thing that changes is the system prompt specifying the details of the robot and camera layouts. Across all three settings, ZAP uses the same interface: camera images annotated with SAM 3.1 masks and numbered boxes, 3D object readouts, end-effector pose commands, and controller feedback. The box below shows one step from the Franka episode: a 3D object readout, the agent’s move_to command, and the motion controller’s response. Both YAM settings use these same formats.
Model
Success (%)
Base Qwen 3.6 27B
8.3
ZAP (SFT)
58.3
ZAP (SFT + RL)
74.0
GPT-6 Astra
72.2
Opus 5.5
61.2
Table 2: Success on the bimanual YAM adaptation of RoboLab-120 [ 30 ] . ZAP exceeds both GPT-6 Astra and Opus 5.5.
Model
Success (%)
ZAP (ours, 27B)
96.7
GPT-6 Astra (our harness)
96.7
GPT-6 Astra (Robocurve)
36.7
MolmoAct2
33.3
Base Qwen 3.6 27B
16.7
Table 3: Real-world results. Left: success rate (%) over 30 trials per model. Right: mean ± std per task. ZAP is served with vLLM [ 33 ] on one B300 priced at $7.10/hour, with a 2-token multi-token-prediction head.
RoboLab-120, single-arm Franka
Success (%)
ZAP (transferred from bimanual YAM)
55.9
Lakeside
51.0
FLUX 3 Action
42.9
Table 4: The same motion skill on a new robot. Without any Franka training, ZAP exceeds Lakeside and FLUX 3 Action, the leading VLA/WAM policies on the current RoboLab-120 leaderboard [ 34 ] .
SFT data
Num Traces
RoboLab-120 Success (%)
Early mining
455
17.2
Longer mining
3.9k
58.3
Table 5: Ablations. Left: more extensive trace mining improves SFT generalization on RoboLab-120 with the bimanual YAM. Right: a 9B model trained directly on separate single-arm Isaac Sim tasks also transfers to RoboLab-120. Lakeside and FLUX 3 Action results are from the benchmark leaderboard [ 34 ] .
Figure 3: A new object and a new goal. Frames from the single test tube episode (a–c: overhead camera, zoomed; d: wrist camera). The model picks up the tube lying on its side, rotates the gripper so the tube stands upright, carries it over the rack and lowers it into a hole, cap up.
Figure 4: Steered by the instruction. Final scenes of two episodes from the same starting scene: (a) the default prompt; (b) a request for a hole at the top left of the rack.
Figure 5: Mid-air bimanual handoff. Frames from the handoff episode, described step by step in the text. Training contains no bimanual interaction close to a handoff.
Figure 6: Reasoning about what the goal means. Top: tic-tac-toe, prompted to draw an X in the center of the board. Bottom: chess, prompted with black to move, mate in one.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Slot yaw, tilt and tilt direction. The gripper orientation parameterization used by move_to (§ 3.1 ). Left: slot yaw ψ turns the jaw slot (blue), the axis a held object lies along, about the vertical. Middle: tilt τ leans the tool axis (teal) away from straight down; 0∘ is top-down and 90∘ horizontal. Right: tilt direction ϕ sets the compass direction the fingertips tip toward. Each angle changes exactly one thing about the gripper that can be seen. All angles are measured counterclockwise from world +x .
Figure 8: Prefix-tree search (§ 3.3 ). A node is created after each physical action (a move, grasp or release) settles. The search keeps returning to the most promising partial episode and branching from there, instead of starting every attempt from scratch; branches that keep failing are pruned. We allow up to about 1,000 leaves per task.
A prevailing view in robot learning is that simulation alone is not enough; effective sim-to-real transfer is widely believed to require at least some real-world data collection or task-specific fine-tuning to bridge the gap between simulated and physical environments. We challenge that assumption. With sufficiently large-scale and diverse simulated synthetic training data, we show that zero-shot transfer to the real world is not only possible, but effective for both static and mobile manipulation. We introduce MolmoBot-Engine, a fully open-source pipeline for procedural data generation across robots, tasks, and diverse simulated environments in MolmoSpaces. With it, we release MolmoBot-Data, a dataset of 1.8 million expert trajectories for articulated object manipulation and pick-and-place tasks. We train three policy classes: MolmoBot, a Molmo2-based multi-frame vision-language model with a flow-matching action head; MolmoBot-Pi0, which replicates the π0 architecture to enable direct comparison; and MolmoBot-SPOC, a lightweight policy suitable for edge deployment and amenable to RL fine-tuning. We evaluate on two robotic platforms: the Franka FR3 for tabletop manipulation tasks and the Rainbow Robotics RB-Y1 mobile manipulator for door opening, drawer manipulation, cabinet interaction, and mobile pick-and-place. Without any real-world fine-tuning, our policies achieve zero-shot transfer to unseen objects and environments. On tabletop pick-and-place, MolmoBot achieves a success rate of 79.2% in real world evaluations across 4 settings, outperforming π0.5 at 39.2%. Our results demonstrate that procedural environment generation combined with diverse articulated assets can produce robust manipulation policies that generalize broadly to the real world. Technical website: https://allenai.github.io/MolmoBot
Abhay Deshpande, Maya Guru, Rose Hendrix +23
Allen Institute for AI · Technische Universität Darmstadt · University of Washington +2
We present ZeroBot, a real2sim framework for learning a robot manipulation task from scratch in minutes under challenging conditions: zero human demonstrations, zero policy pre-training, and zero known object models. Given only a single view of an object and a goal pose for that object, ZeroBot uses image-to-3D generative models to obtain a complete object mesh, which is used in simulation for large-scale parallel reinforcement learning. To accelerate training, we introduce an action space which leverages the generated geometry and learned value function to sample states involving robot-object contact. When evaluated on real-world tasks including grasping, pushing, articulated object interaction, and multi-stage manipulation, ZeroBot achieves an 87% success rate with an average training time of 119 seconds. These results show the value of using image-to-3D models in a real2sim framework for rapid, autonomous robot learning.
Ivan Kapelyukh, Xiaohan Zhang, Stephen James +2
Imperial College London · Robotics and AI Institute
Designing or learning robot policies that generalize zero-shot across a range of language instructions and objects is a core problem in robotics. Vision-Language-Action models (VLAs) learn such policies end-to-end by repurposing existing Vision-Language Models (VLMs), but generalization to new instructions and objects remains challenging. An alternative is to implement a modular policy by leveraging an explicit VLM-based 3D scene representation and motion planning. While modular policies show strong zero-shot potential, they typically retrieve objects based on semantics without explicit spatial reasoning, severely restricting their overall grounding capabilities. They also interact with objects using basic grasping and navigation skills. In this work, we address these limitations by unifying grounding capabilities and robot skills in a single agentic action space through a scene-agent tool interface. By leveraging part-level affordances, our skills generalize across diverse objects and enable zero-shot interactions such as unplugging chargers and opening drawers. We name the resulting framework Agentic Scene Policies (ASP). Through extensive real-world experiments, we show how ASP consistently outperforms leading VLAs in the zero-shot setting. We also demonstrate the extensibility of our framework by introducing a mobile version of ASP to tackle room-level queries. See our project page (https://montrealrobotics.ca/agentic-scene-policies.github.io/) for more results.
Sacha Morin, Kumaraditya Gupta, Mahtab Sandhu +4
Université de Montréal · Mila - Quebec AI Institute · Sapienza University of Rome