Scaling robot manipulation policies requires large amounts of diverse, action-labeled data, yet collecting such data in the real world remains expensive and difficult to scale. Existing methods either depend on human teleoperation, rely on simulation with limited visual realism, or use human videos that lack embodiment-specific annotations. In this paper, we introduce Grounded-Dreamer, a framework for synthesizing physically grounded robot manipulation data with video world models while avoiding human teleoperation demonstrations. Grounded-Dreamer combines the visual realism of generative video models with simulator physics engines. Starting from a small set of real images depicting robot task initial states, our method automatically reconstructs task-relevant objects and camera geometry, aligns them in simulation, and collects physically valid trajectories. These trajectories are used to post-train both a video world model and an inverse dynamics model, enabling the generation of visually realistic robot manipulation videos and corresponding robot actions from real images. To filter physically implausible rollouts, Grounded-Dreamer replays generated trajectories in the reconstructed simulation and retains only those that successfully complete the tasks. The real-world robot policies are trained solely on the grounded synthetic trajectories without any human demonstrations. Empirical results in real-world tasks demonstrate that Grounded-Dreamer can synthesize physically grounded data and substantially facilitate policy learning without requiring human teleoperation effort.
Figures & tables
Figure 1: Overview of Grounded-Dreamer. Our framework automates the generation of massive, high-fidelity robot data from a small number of real-world reference images. By combining generative world models with simulation-based verification, Grounded-Dreamer produces data that is both visually realistic and physically consistent. This simulation verification loop effectively filters out the physical hallucinations common in purely generative approaches, allowing robot policies to be trained directly on scalable synthetic data that transfers robustly to complex real-world scenarios.
Figure 2: The Grounded-Dreamer Pipeline. Given several real-world reference images, our pipeline first reconstructs physically aligned simulation scenes and objects. Within these environments, task data is automatically collected and processed with interaction-aware masking to fine-tune Cosmos-Predict 2.5 as the core data generation model. An image refinement model then synthesizes diverse initial frames by combining reconstructed assets with real scene backgrounds and object appearances. Conditioned on these frames, the generation model produces manipulation videos, which are converted into action trajectories using an Inverse Dynamics Model (IDM). Finally, all synthesized trajectories are replayed in simulation for closed-loop verification, filtering out physically inconsistent samples before policy training and real-world deployment.
Figure 3: Qualitative results of (A) simulation data acquisition and interaction-aware masking, and (B) Grounded-Dreamer video generation.
Figure 4: Visualization of Simulation Validation. The physical simulation correctly captures a task failure due to incorrect gripper placement, whereas the world model exhibits spatial hallucination by generating a hallucinated successful outcome.
Figure 5: Real-world Setup and Task Execution. (Left) Real-robot evaluation environment; (Right) Visualizations of our policy executing manipulation tasks.
Method
Pick
Button
Close Drawer
Pick and Place
Sweep Trash
Slice Sausage
Put in Drawer
Sim-only (DP)
10%
10%
0%
5%
5%
0%
0%
Real-only (DP)
50%
60%
60%
50%
50%
70%
35%
GD data (DP)
45%
50%
45%
35%
45%
55%
25%
Low-data Real (DP)
10%
10%
15%
0%
10%
10%
5%
Co-training (DP)
55%
65%
50%
45%
55%
60%
40%
Sim-only ( π0.5 )
10%
15%
15%
5%
10%
5%
0%
Table 1: Main experimental results across seven tasks. We compare the success rates of DP and π0.5 under different data regimes, with RIGVid as a representative baseline.
Method
Pick
Button
Close Drawer
Pick and Place
Sweep Trash
Slice Sausage
Put in Drawer
Real-only (DP)
10%
5%
5%
0%
5%
5%
0%
Verified WM data (DP)
40%
50%
35%
30%
40%
45%
15%
Real-only ( π0.5 )
25%
20%
10%
20%
20%
15%
5%
Verified WM data ( π0.5 )
45%
60%
55%
40%
60%
50%
30%
Table 2: Success Rates in OOD Scenarios. Performance comparison of models trained on two types of data across seven manipulation tasks under out-of-distribution (OOD) conditions.
Figure 6: Scaling Success Rates. Success rates on two representative tasks improve as the number of Grounded-Dreamer (GD) generated trajectories increases.
Method
Pick
Button
Close Drawer
Pick and Place
Sweep Trash
Slice Sausage
Put in Drawer
w/ Masking (DP)
45%
50%
45%
35%
45%
55%
25%
w/o Masking (DP)
0%
5%
0%
0%
0%
0%
0%
w/ Masking ( π0.5 )
55%
75%
55%
50%
60%
60%
30%
w/o Masking ( π0.5 )
5%
0%
10%
0%
10%
5%
0%
Table 3: Ablation on Interaction-Aware Masking.
Method
Pick
Button
Close Drawer
Pick and Place
Sweep Trash
Slice Sausage
Put in Drawer
100% (DP)
45%
50%
45%
35%
45%
55%
25%
80% (DP)
40%
40%
35%
30%
30%
45%
10%
50% (DP)
25%
30%
25%
25%
20%
20%
5%
100% ( π0.5 )
55%
75%
55%
50%
60%
60%
30%
80% ( π0.5 )
35%
60%
50%
35%
45%
50%
20%
50% ( π0.5 )
25%
35%
25%
20%
25%
30%
10%
Table 4: Ablation on Replay Verification.
Figure 7: Ablation without Masking. Visualizations of video rollouts generated by the world model fine-tuned on raw simulation data without the interaction-aware masking strategy.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Additional visualizations of automated data collection in simulation.
Figure 9: Additional visualizations of Grounded-Dreamer video generation.
Figure 10: Qualitative Visualizations of Refined Initial Frames.