MolmoB0T: Large-Scale Simulation Enables Zero-Shot Manipulation

Abhay Deshpande; Maya Guru; Rose Hendrix; Snehal Jauhri; Ainaz Eftekhar; Rohun Tripathi; Max Argus; Jordi Salvador

cs.ROMar 17, 2026

MolmoB0T: Large-Scale Simulation Enables Zero-Shot Manipulation

Authors: Abhay Deshpande, Maya Guru, Rose Hendrix, Snehal Jauhri, Ainaz Eftekhar, Rohun Tripathi, Max Argus, Jordi Salvador,

Organizations: Allen Institute for AI · Technische Universität Darmstadt · University of Washington · University of California, Los Angeles ·

Abstract

A prevailing view in robot learning is that simulation alone is not enough; effective sim-to-real transfer is widely believed to require at least some real-world data collection or task-specific fine-tuning to bridge the gap between simulated and physical environments. We challenge that assumption. With sufficiently large-scale and diverse simulated synthetic training data, we show that zero-shot transfer to the real world is not only possible, but effective for both static and mobile manipulation. We introduce MolmoBot-Engine, a fully open-source pipeline for procedural data generation across robots, tasks, and diverse simulated environments in MolmoSpaces. With it, we release MolmoBot-Data, a dataset of 1.8 million expert trajectories for articulated object manipulation and pick-and-place tasks. We train three policy classes: MolmoBot, a Molmo2-based multi-frame vision-language model with a flow-matching action head; MolmoBot-Pi0, which replicates the $π_0$ architecture to enable direct comparison; and MolmoBot-SPOC, a lightweight policy suitable for edge deployment and amenable to RL fine-tuning. We evaluate on two robotic platforms: the Franka FR3 for tabletop manipulation tasks and the Rainbow Robotics RB-Y1 mobile manipulator for door opening, drawer manipulation, cabinet interaction, and mobile pick-and-place. Without any real-world fine-tuning, our policies achieve zero-shot transfer to unseen objects and environments. On tabletop pick-and-place, MolmoBot achieves a success rate of 79.2% in real world evaluations across 4 settings, outperforming $π_{0.5}$ at 39.2%. Our results demonstrate that procedural environment generation combined with diverse articulated assets can produce robust manipulation policies that generalize broadly to the real world. Technical website: https://allenai.github.io/MolmoBot

Figures & tables

Figure 1: MolmoBot leverages diverse simulation data to achieve zero-shot sim-to-real transfer on multiple robotic tasks such as pick-and-place and door opening. This unlocks the ability to dramatically scale up the training data for generalist robotic foundation models.

Figure 2: MolmoBot-Engine . Starting from a pre-built MolmoSpaces [ 17 ] house, we sample task-relevant objects, randomize visual and physical parameters, and iteratively replan as necessary until a successful trajectory is found.

Figure 3: Expert demonstrations across multiple robots and manipulation tasks. Each row shows a trajectory conditioned on a language instruction. The top two rows illustrate Franka tabletop tasks (pick and pick-and-place), while the bottom rows show RB-Y1 mobile manipulation tasks (door opening and drawer opening). Columns visualize sequential frames from each trajectory.

Task	Robot	Episodes	Frames	Assets (Obj. + Rec.)	Envs.	Avg. Length	Total Length
Door-open	RBY1	79.0k	15.4M	–	15.1k	19.6 s	429 h
Open	RBY1	46.6k	6.9M	217	10.5k	14.8 s	192 h
Pick	RBY1	62.3k	7.4M	4.2k	28.0k	10.7 s	184 h
Pick	Franka	781.8k	56.9M	10.7k	73.2k	4.8 s	1,042 h
Pick-and-Place	RBY1	14.8k	2.4M	2.4k + 183	9.3k	14.0 s	58 h
Pick-and-Place	Franka	554.2k	143.9M	7.0k + 494	61.6k	17.1 s	2,638 h

Table 1: MolmoBot-Data statistics by task. All episodes include RGB observations, proprioceptive state, action labels, and privileged information such as object visibility that the use of simulation affords. The number of assets reflects pickup objects and receptacles for pick-and-place and variants.

Dataset	Source	Episodes	Hours	Unique Envs.	Tasks	Embod.	Mobile Manip.
DROID [ 16 ]	Real	76k	350	564	86	1	✗
Open X-Embodiment [ 27 ]	Real	1M+	–	–	527	22	✓
AgiBot-World [ 1 ]	Real	1M+	2,976	100+	217	1	✓
RoboMimic [ 24 ]	Sim	$\sim$ 1k	–	1	5	1	✗
MimicGen [ 23 ]	Sim	50k+	–	–	18	1	✗
InternData-A1 [ 34 ]	Sim	630k	7,433	227	70	4	✗

Table 2: Comparison to prior manipulation datasets. MolmoBot-Data provides substantially more episodes and environment diversity through procedural generation.

Figure 4: Policy architectures. We train three policy classes on MolmoBot-Data. Left: Input observations include RGB images from multiple camera views at the current (and optionally initial) timesteps, proprioceptive state, a language task instruction, and optional 2D point conditioning for specifying target objects or locations. Top right: MolmoBot uses a Molmo2 vision-language backbone with a DiTX-based flow matching action head that attends to visual features via cross-attention and predicts action chunks of 16 timesteps. Bottom right: MolmoBot-SPOC uses SigLIP2 vision and text encoders with a bidirectional transformer decoder that processes learned action query embeddings to predict actions in parallel. Both architectures support optional point conditioning. MolmoBot-Pi0 (not shown) exactly follows the $\pi_{0}$ architecture [ 3 ] to enable controlled comparison.

Task	Sampling Ratio
Pick	20%
Pick-and-place Fixed Height	10%
Pick-and-place Random Height	35%
Pick-and-place-next-to	20%
Pick-and-place-color	15%

Table 3: Multitask data mixture for all MolmoBot Franka FR3 policies. The data mix is selected to ensure all coverage of each of the individual task’s training set.

Model	Open	Door-open	Pick	Pick-and-place
MolmoBot Multitask	20%	20%	30%	30%
MolmoBot Door Specialist	-	100%	-	-
MolmoBot-SPOC Rigid	-	-	50%	50%
MolmoBot-SPOC Articulated	45%	55%	-	-

Table 4: Multitask data mixture for all MolmoBot and MolmoBot-SPOC RB-Y1 policies.

Panel of Figure 5 — Figure 5: Real-world environments for our DROID evaluations. From left to right: kitchen, workroom, bedroom, office. Additional details in the Appendix.

Panel of Figure 6 — Figure 6: Real-world environments for our RBY1 articulated and rigid object mobile manipulation evaluations.

Figure 7: MolmoBot policies exhibit strong zero-shot sim2real performance across our real-world DROID evaluations, outperforming SOTA policies trained on large-scale real-world demonstrations. Bar heights reflect mean success rate and error bars represent 95% confidence intervals, estimated via stratified bootstrapping. Here, MolmoBot denotes the MolmoBot (F=2) variant.

Door	Trial	Grasp	Opened	Failure Mode
1	1	✓ †	✗	HW fault during grasp phase
	2	✗	✗	Base collision
	3	✗	✗	Joint limit reached
2	1	✓ †	✗	HW fault during grasp phase
	2	✓	✓ †	HW fault during opening phase
	3	✓	✓ †	HW fault during opening phase

Table 5: Door opening task results. Each door has a distinct visual texture. Trials differ in robot base position. † Hardware fault occurred during trial.

Model	Pick MSProc	Pick Classic	Pick	Pick Rand.-Cam.	Pick&Place	PnP Next-To	PnP Color	Avg.	Real
$\pi_{0.5}$ [ 2 ]	18.1 $\,\pm$ 2.4	6.4 $\,\pm$ 1.5	7.0 $\,\pm$ 1.6	8.0 $\,\pm$ 1.9	11.7 $\,\pm$ 2.1 /7.6 $\,\pm$ 1.7	8.2 $\,\pm$ 2.2 /6.2 $\,\pm$ 1.9	10.4 $\,\pm$ 1.9 /6.7 $\,\pm$ 1.6	10.0	31.3
$\pi_{0.5}$ -Finetune	48.0 $\,\pm$ 3.1	28.3 $\,\pm$ 2.8	25.8 $\,\pm$ 2.7	29.7 $\,\pm$ 2.9	43.5 $\,\pm$ 6.0 /37.4 $\,\pm$ 5.8	$\mathbf{28.4}$ $\,\pm$ 3.2 /14.7 $\,\pm$ 2.5	48.3 $\,\pm$ 3.1 /38.9 $\,\pm$ 3.0	36.0	–
StereoVLA [ 10 ]	6.6 $\,\pm$ 2.6	4.3 $\,\pm$ 1.5	1.1 $\,\pm$ 1.0	N/A	0	N/A	0	-	–
LAP-VLA [ 42 ]	19.4 $\,\pm$ 2.4	2.4 $\,\pm$ 1.0	3.1 $\,\pm$ 1.1	2.7 $\,\pm$ 1.0	3.81 $\,\pm$ 1.5 /1.59 $\,\pm$ 1.0	6.48 $\,\pm$ 2.8 /3.41 $\,\pm$ 2.1	3.1 $\,\pm$ 1.1 /1.5 $\,\pm$ 0.8	4.8	–
X-VLA [ 46 ]	3.3 $\,\pm$ 1.0	0.5 $\,\pm$ 0.5	0.7 $\,\pm$ 0.5	0.8 $\,\pm$ 0.5	0.1 $\,\pm$ 0.2 /0.1 $\,\pm$ 0.2	1.9 $\,\pm$ 0.9 /1.0 $\,\pm$ 0.7	0.9 $\,\pm$ 0.6 /0.5 $\,\pm$ 0.5	1.2	–
MolmoBot-Pi0	66.2 $\,\pm$ 2.9	35.7 $\,\pm$ 3.0	33.3 $\,\pm$ 2.9	39.8 $\,\pm$ 3.1	44.7 $\,\pm$ 3.1 /38.2 $\,\pm$ 3.1	24.7 $\,\pm$ 3.2 /13.3 $\,\pm$ 2.5	46.2 $\,\pm$ 3.1 /40.0 $\,\pm$ 3.1	41.5	46.7

Table 6: Evaluation on simulation held-out environments and real robot episodes. Simulation success rates are evaluated over 1000 episodes per task. Real robot evaluations are done over 120 episodes. All models evaluated zero-shot in real without task-specific finetuning. For pick-and-place tasks, we report both oracle success (first number, which is the success conditions being fulfilled at any timestep) and success at end (second number, the success conditions being fulfilled at the final timestep). The delta between captures both unstable/unsuitable placement and the inability of policies to determine when a specified task is already completed, e.g. by repeatedly picking up an object which has already been placed correctly. We additionally report the half-width of the 95% confidence interval bounds for each result.

Model	Pick MSProc (sim)	Pick Kitchen (real)
StereoVLA [ 10 ]	6.6	–
LAP-VLA [ 42 ]	19.4	–
$\pi_{0}$ [ 3 ]	13.5	20.0
$\pi_{0.5}$ [ 2 ]	18.1	63.3
$\pi_{0.5}$ -Finetune	48.0	–
DreamZero [ 40 ]	44.3	–

Table 7: Simulation and real evaluation with restricted camera setup. Success rate averaged over 1000 episodes in simulation and 30 tasks in a real-world kitchen. All models evaluated zero-shot without task-specific finetuning.

Model	Pick	Pick & Place	Open	Door Open
MolmoBot Multitask	44.8	22.5	25.2	70.2
MolmoBot Door Specialist	–	–	–	77.7
MolmoBot-SPOC Rigid	10.5	1.8	–	–
MolmoBot-SPOC Articulated	–	–	21.8	58.8

Table 8: Simulation evaluation for RB-Y1 policies on held-out environments. All models evaluated zero-shot without task-specific finetuning.

Figure 8: Effect of scaling various dimensions of simulated training data, evaluated on DROID (real-world, top) and pick classic (simulation, bottom). (a) We vary trajectory count sampled from 5,000 houses; performance improves predictably with scale, particularly in real-world evaluations. (b) We control the number of unique object classes across 50,000 trajectories from 5,000 houses. Object diversity improves simulation performance but not real-world, possibly due to the evaluation using fewer, more general objects. (c) We vary the number of simulated house environments across 50,000 trajectories. Environment diversity does not improve performance in either setting, suggesting the local nature of pick makes background diversity unnecessary. Error bars: 95% binomial CIs.

Figure 9: Ablations on training and action parameterization. We evaluate MolmoBot-Img on both DROID (real-world, top row) and pick classic (simulation, bottom row). (a) We train while sampling multiple denoising timesteps

T

per example in parallel during training to improve convergence and final performance. We ablate

T\in\{1,2,4,8\}

and find that simulation performance improves steadily as

T

increases and peaks at

T=8

, while real-world performance is less monotonic and peaks at

T=4

. (b) We train using either absolute or delta action representations for 200K steps on Franka FR3 policies. The absolute action representation substantially improves real-world performance over delta actions, while the two representations perform similarly in simulation. Error bars denote 95% binomial proportion confidence intervals.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Task ID(s)	Task Prompt
Workroom	spoon_tray	“put the wooden spoon on the light blue tray”
	spoon_box	“put the wooden spoon in the wooden box”
	tape_tray	“put the blue tape on the light blue tray”
	tape_box	“put the blue tape in the wooden box”
	blue_mug_tray	“put the blue mug on the light blue tray”
	blue_mug_box	“put the blue mug in the wooden box”

Appendix

Table 9: Task prompts for each task in each benchmark for our real-world DROID pick-and-place evaluations.

Benchmark

Policy

Spoon Tray

Spoon Box

Tape Tray

Tape Box

Blue Mug Tray

Blue Mug Box

Copper Mug Tray

Copper Mug Box

Timer Tray

Timer Box

Avg

Workroom

MolmoBot

3/3

2/3

3/3

1/3

3/3

90%

MolmoBot-Img

3/3

2/3

0/3

3/3

1/3

77%

MolmoBot-Pi0

1/3

0/3

3/3

2/3

1/3

0/3

2/3

60%

\pi_{0.5}\text{-DROID}

2/3

0/3

1/3

0/3

27%

\pi_{0}

0/3

1/3

0/3

Apple Easy

Apple Hard

Mug Easy

Mug Hard

Banana Easy

Banana Hard

Mouse Easy

Mouse Hard

Clutter Brown

Clutter Black

Avg

Appendix

Table 10: DROID real-world evaluation results on each task in each environment. Each cell shows successes out of 3 trials.

Model	Carrot $\rightarrow$ Plate	Eggplant $\rightarrow$ Basket	Stack Cubes	Spoon $\rightarrow$ Cloth	Avg.
MolmoBot-Img	45.8%	0%	0%	0%	11.5%
MolmoBot-Pi0	0%	0%	0%	0%	0%
$\pi_{0.5}$ -DROID	0%	0%	0%	0%	0%
$\pi_{0}$ -DROID	0%	0%	0%	0%	0%

Appendix

Table 11: Zero-shot evaluation on SIMPLER tasks reimplemented for DROID Franka FR3.

	MolmoBot-Img		MolmoBot-Pi0		$\pi_{0.5}$ -DROID		$\pi_{0}$ -DROID
Task	std.	no dist.	std.	no dist.	std.	no dist.	std.	no dist.
Alphabet soup	0%	16%	0%	2%	0%	2%	0%	0%
Cream cheese	0%	24%	0%	2%	2%	0%	0%	0%
Salad dressing	20%	40%	0%	2%	0%	0%	0%	0%
BBQ sauce	0%	42%	0%	14%	0%	2%	0%	0%
Ketchup	0%	42%	0%	6%	0%	4%	0%	0%

Appendix

Table 12: Zero-shot evaluation on LIBERO-Object reimplemented for DROID Franka FR3. We evaluate with the standard set of distractors (std.) and without distractors (no dist.) to isolate failure sources.

Sim-to-Real Transfer

Simulation-to-reality transfer adapts models and policies trained in simulation for reliable performance in physical or otherwise real-world settings.

See all 300 papers

ZeroBot: Learning from Scratch in Minutes with Generative Real2Sim
Sep 27, 2026Ivan Kapelyukh, Xiaohan Zhang, Stephen James +2Robotic RL Robotic Manipulation
Imperial College London · Robotics and AI Institute
Genie Sim 3.0 : A High-Fidelity Comprehensive Simulation Platform for Humanoid Robot
Jan 5, 2026Chenghao Yin, Da Huang, Di Yang +16Robotic Data Collection Robotics Simulation
Efficient Sim-to-Real Transfer of World-Action Models from Synthetic Priors
Jun 30, 2026Zixing Wang, Kausik Sivakumar, Jinghuan Shang +5Sim-to-Real Transfer Action-Conditioned World Models
1Purdue University · 2Robotics and AI Institute

Sep 27, 2026J/K move · Enter open · S save

ZeroBot: Learning from Scratch in Minutes with Generative Real2Sim

Ivan Kapelyukh, Xiaohan Zhang, Stephen James, Laura Herlant+1

Imperial College London · Robotics and AI Institute

Robotic RL Robotic Manipulation Data-Efficient Robot Learning Real-to-Sim Transfer

Open paper AskPDF arXiv

We present ZeroBot, a real2sim framework for learning a robot manipulation task from scratch in minutes under challenging conditions: zero human demonstrations, zero policy pre-training, and zero known object models. Given only a single view of an object and a goal pose for that object, ZeroBot uses image-to-3D generative models to obtain a complete object mesh, which is used in simulation for large-scale parallel reinforcement learning. To accelerate training, we introduce an action space which leverages the generated geometry and learned value function to sample states involving robot-object contact. When evaluated on real-world tasks including grasping, pushing, articulated object interaction, and multi-stage manipulation, ZeroBot achieves an 87% success rate with an average training time of 119 seconds. These results show the value of using image-to-3D models in a real2sim framework for rapid, autonomous robot learning.

Abstract

Sim-to-Real Transfer

Explore similar work

ZeroBot: Learning from Scratch in Minutes with Generative Real2Sim

Genie Sim 3.0 : A High-Fidelity Comprehensive Simulation Platform for Humanoid Robot

Efficient Sim-to-Real Transfer of World-Action Models from Synthetic Priors