Pretraining Knows What, Simulation Teaches How: Compositional Generalization in a Sim-Trained Robot Agent

Benjamin Pikus; Daniel Saunders; Mircea Mironenco; Salman Mohammadi; Mathew Mathew; Sirou Zhu; Divya Makkar; Xuhua Huang

cs.ROOct 26, 2026

Pretraining Knows What, Simulation Teaches How: Compositional Generalization in a Sim-Trained Robot Agent

Authors: Benjamin Pikus, Daniel Saunders, Mircea Mironenco, Salman Mohammadi, Mathew Mathew, Sirou Zhu, Divya Makkar, Xuhua Huang,

Organizations: Luma AI

Abstract

A robot policy needs to understand what a task is asking for in a scene, and how to move the robot to accomplish that task. To achieve these two skills, we introduce Zero-real Agent Policy (ZAP), a recipe that teaches a small off-the-shelf vision-language model to control a robot via a simple tool harness. We use a pretrained 27B parameter model to bootstrap world understanding and reasoning, followed by supervised fine-tuning and GRPO on simple simulated tasks to teach it how to move the robot. Without observing real-world robot data during training, ZAP reaches 96.7% success on real-world tasks, matching GPT-6 Astra in our harness and outperforming the MolmoAct2 VLA (33.3%), while running 3x faster and 17x cheaper than GPT-6 Astra. Furthermore, ZAP also generalizes to an unseen embodiment as well as new objects, goals and strategies never seen during training. On a single-arm Franka, ZAP reaches 55.9% on RoboLab-120, 4.9 percentage points ahead of the leading VLA/WAM baseline.

Figures & tables

Panel of Figure 1 — Figure 1: Motion learned in simulation carries over to new real tasks that need reasoning. Left : two of the simulated training tasks. These simple pick and place, stacking, and reorientation tasks with everyday objects teach the model only how to move. Right : the same model, trained exclusively in simulation, tested zero-shot on a real bi-manual robot. Without using any real robot data in training, ZAP generalizes to completely new objects (test tube), new goals (choosing a different hole in the rack), new strategies (mid-air handoff), and complex skills and reasoning (drawing an X on a tic-tac-toe board, playing chess). These tasks show the model composing reasoning from pretraining with motion learned in simulation.

Approach	Speed	Cost	Generalization
VLAs / WAMs	✓	✓	–
Code as Policies	✓	✓	–
Large agent as policy (frontier LLM + tools)	–	–	✓
ZAP (ours)	✓	✓	✓

Table 1: Comparing robot policy paradigms. VLAs/WAMs and Code as Policies offer fast, inexpensive execution, while frontier agents provide broader generalization at higher latency and cost. ZAP combines fast, low-cost execution with generalization to new instructions, objects, scenes, and goals.

Panel of Figure 2 — Figure 2: Three scenes, one shared interface. Left to right: separate banana-in-bowl episodes on the simulated bi-manual YAM, the real YAM, and a simulated single-arm Franka, an embodiment never seen in training. From the agent’s perspective, the only thing that changes is the system prompt specifying the details of the robot and camera layouts. Across all three settings, ZAP uses the same interface: camera images annotated with SAM 3.1 masks and numbered boxes, 3D object readouts, end-effector pose commands, and controller feedback. The box below shows one step from the Franka episode: a 3D object readout, the agent’s move_to command, and the motion controller’s response. Both YAM settings use these same formats.

Model	Success (%)
Base Qwen 3.6 27B	8.3
ZAP (SFT)	58.3
ZAP (SFT + RL)	74.0
GPT-6 Astra	72.2
Opus 5.5	61.2

Table 2: Success on the bimanual YAM adaptation of RoboLab-120 [ 30 ] . ZAP exceeds both GPT-6 Astra and Opus 5.5.

Model	Success (%)
ZAP (ours, 27B)	96.7
GPT-6 Astra (our harness)	96.7
GPT-6 Astra (Robocurve)	36.7
MolmoAct2	33.3
Base Qwen 3.6 27B	16.7

Table 3: Real-world results. Left: success rate (%) over 30 trials per model. Right: mean

\pm

std per task. ZAP is served with vLLM [ 33 ] on one B300 priced at $7.10/hour, with a 2-token multi-token-prediction head.

RoboLab-120, single-arm Franka	Success (%)
ZAP (transferred from bimanual YAM)	55.9
Lakeside	51.0
FLUX 3 Action	42.9

Table 4: The same motion skill on a new robot. Without any Franka training, ZAP exceeds Lakeside and FLUX 3 Action, the leading VLA/WAM policies on the current RoboLab-120 leaderboard [ 34 ] .

SFT data	Num Traces	RoboLab-120 Success (%)
Early mining	455	17.2
Longer mining	3.9k	58.3

Table 5: Ablations. Left: more extensive trace mining improves SFT generalization on RoboLab-120 with the bimanual YAM. Right: a 9B model trained directly on separate single-arm Isaac Sim tasks also transfers to RoboLab-120. Lakeside and FLUX 3 Action results are from the benchmark leaderboard [ 34 ] .

Panel of Figure 3 — Figure 3: A new object and a new goal. Frames from the single test tube episode (a–c: overhead camera, zoomed; d: wrist camera). The model picks up the tube lying on its side, rotates the gripper so the tube stands upright, carries it over the rack and lowers it into a hole, cap up.

Panel of Figure 4 — Figure 4: Steered by the instruction. Final scenes of two episodes from the same starting scene: (a) the default prompt; (b) a request for a hole at the top left of the rack.

Panel of Figure 5 — Figure 5: Mid-air bimanual handoff. Frames from the handoff episode, described step by step in the text. Training contains no bimanual interaction close to a handoff.

Panel of Figure 6 — Figure 6: Reasoning about what the goal means. Top: tic-tac-toe, prompted to draw an X in the center of the board. Bottom: chess, prompted with black to move, mate in one.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Slot yaw, tilt and tilt direction. The gripper orientation parameterization used by move_to (§ 3.1 ). Left: slot yaw $\psi$ turns the jaw slot (blue), the axis a held object lies along, about the vertical. Middle: tilt $\tau$ leans the tool axis (teal) away from straight down; $0^{\circ}$ is top-down and $90^{\circ}$ horizontal. Right: tilt direction $\phi$ sets the compass direction the fingertips tip toward. Each angle changes exactly one thing about the gripper that can be seen. All angles are measured counterclockwise from world $+x$ .

Panel of Figure 8 — Figure 8: Prefix-tree search (§ 3.3 ). A node is created after each physical action (a move, grasp or release) settles. The search keeps returning to the most promising partial episode and branching from there, instead of starting every attempt from scratch; branches that keep failing are pruned. We allow up to about 1,000 leaves per task.

Task prompt	ZAP	GPT-6 Astra	GPT-6 Astra (RC)	MolmoAct2	Base Qwen
Put the mustard in the left bin	3/3	3/3	2/3	1/3	0/3
Pack canned foods into the bin	3/3	2/3	1/3	1/3	1/3
Put the whiteboard marker in the mug	2/3	3/3	1/3	0/3	0/3
Put the cube and the banana in the bowl	3/3	3/3	0/3	2/3	0/3
Put the Rubiks cube in the bowl	3/3	3/3	0/3	3/3	0/3
Put the white cup in the center of the table	3/3	3/3	1/3	0/3	2/3

Appendix

Table 6: Per-task successes. RC: Robocurve harness.

Generalization in Robotic Manipulation

Generalization in robotic manipulation studies how robots transfer grasping and interaction skills to new objects, scenes, tasks, and embodiments.

See all 355 papers

MolmoB0T: Large-Scale Simulation Enables Zero-Shot Manipulation
Mar 17, 2026Abhay Deshpande, Maya Guru, Rose Hendrix +23Sim-to-Real Transfer Zero-Shot Robotic Manipulation
Allen Institute for AI · Technische Universität Darmstadt · University of Washington +2
ZeroBot: Learning from Scratch in Minutes with Generative Real2Sim
Sep 27, 2026Ivan Kapelyukh, Xiaohan Zhang, Stephen James +2Robotic RL Robotic Manipulation
Imperial College London · Robotics and AI Institute
Agentic Scene Policies
Sep 23, 2025Sacha Morin, Kumaraditya Gupta, Mahtab Sandhu +4Robot Policy Generalization Robot Learning
Université de Montréal · Mila - Quebec AI Institute · Sapienza University of Rome

Mar 17, 2026J/K move · Enter open · S save

MolmoB0T: Large-Scale Simulation Enables Zero-Shot Manipulation

Abhay Deshpande, Maya Guru, Rose Hendrix, Snehal Jauhri+22

Allen Institute for AI · Technische Universität Darmstadt · University of Washington+2

Sim-to-Real Transfer Zero-Shot Robotic Manipulation Generalization in Robotic Manipulation Synthetic Data Generation for Robotics VLMs for Robotics Articulated Object Manipulation+1

Open paper AskPDF arXiv

A prevailing view in robot learning is that simulation alone is not enough; effective sim-to-real transfer is widely believed to require at least some real-world data collection or task-specific fine-tuning to bridge the gap between simulated and physical environments. We challenge that assumption. With sufficiently large-scale and diverse simulated synthetic training data, we show that zero-shot transfer to the real world is not only possible, but effective for both static and mobile manipulation. We introduce MolmoBot-Engine, a fully open-source pipeline for procedural data generation across robots, tasks, and diverse simulated environments in MolmoSpaces. With it, we release MolmoBot-Data, a dataset of 1.8 million expert trajectories for articulated object manipulation and pick-and-place tasks. We train three policy classes: MolmoBot, a Molmo2-based multi-frame vision-language model with a flow-matching action head; MolmoBot-Pi0, which replicates the

π_0

architecture to enable direct comparison; and MolmoBot-SPOC, a lightweight policy suitable for edge deployment and amenable to RL fine-tuning. We evaluate on two robotic platforms: the Franka FR3 for tabletop manipulation tasks and the Rainbow Robotics RB-Y1 mobile manipulator for door opening, drawer manipulation, cabinet interaction, and mobile pick-and-place. Without any real-world fine-tuning, our policies achieve zero-shot transfer to unseen objects and environments. On tabletop pick-and-place, MolmoBot achieves a success rate of 79.2% in real world evaluations across 4 settings, outperforming

π_{0.5}

at 39.2%. Our results demonstrate that procedural environment generation combined with diverse articulated assets can produce robust manipulation policies that generalize broadly to the real world. Technical website: https://allenai.github.io/MolmoBot

Abstract

Generalization in Robotic Manipulation

Explore similar work

MolmoB0T: Large-Scale Simulation Enables Zero-Shot Manipulation

ZeroBot: Learning from Scratch in Minutes with Generative Real2Sim

Agentic Scene Policies