Scaling robot manipulation policies requires large amounts of diverse, action-labeled data, yet collecting such data in the real world remains expensive and difficult to scale. Existing methods either depend on human teleoperation, rely on simulation with limited visual realism, or use human videos that lack embodiment-specific annotations. In this paper, we introduce Grounded-Dreamer, a framework for synthesizing physically grounded robot manipulation data with video world models while avoiding human teleoperation demonstrations. Grounded-Dreamer combines the visual realism of generative video models with simulator physics engines. Starting from a small set of real images depicting robot task initial states, our method automatically reconstructs task-relevant objects and camera geometry, aligns them in simulation, and collects physically valid trajectories. These trajectories are used to post-train both a video world model and an inverse dynamics model, enabling the generation of visually realistic robot manipulation videos and corresponding robot actions from real images. To filter physically implausible rollouts, Grounded-Dreamer replays generated trajectories in the reconstructed simulation and retains only those that successfully complete the tasks. The real-world robot policies are trained solely on the grounded synthetic trajectories without any human demonstrations. Empirical results in real-world tasks demonstrate that Grounded-Dreamer can synthesize physically grounded data and substantially facilitate policy learning without requiring human teleoperation effort.
Figures & tables
Figure 1: Overview of Grounded-Dreamer. Our framework automates the generation of massive, high-fidelity robot data from a small number of real-world reference images. By combining generative world models with simulation-based verification, Grounded-Dreamer produces data that is both visually realistic and physically consistent. This simulation verification loop effectively filters out the physical hallucinations common in purely generative approaches, allowing robot policies to be trained directly on scalable synthetic data that transfers robustly to complex real-world scenarios.
Figure 2: The Grounded-Dreamer Pipeline. Given several real-world reference images, our pipeline first reconstructs physically aligned simulation scenes and objects. Within these environments, task data is automatically collected and processed with interaction-aware masking to fine-tune Cosmos-Predict 2.5 as the core data generation model. An image refinement model then synthesizes diverse initial frames by combining reconstructed assets with real scene backgrounds and object appearances. Conditioned on these frames, the generation model produces manipulation videos, which are converted into action trajectories using an Inverse Dynamics Model (IDM). Finally, all synthesized trajectories are replayed in simulation for closed-loop verification, filtering out physically inconsistent samples before policy training and real-world deployment.
Figure 3: Qualitative results of (A) simulation data acquisition and interaction-aware masking, and (B) Grounded-Dreamer video generation.
Figure 4: Visualization of Simulation Validation. The physical simulation correctly captures a task failure due to incorrect gripper placement, whereas the world model exhibits spatial hallucination by generating a hallucinated successful outcome.
Figure 5: Real-world Setup and Task Execution. (Left) Real-robot evaluation environment; (Right) Visualizations of our policy executing manipulation tasks.
Method
Pick
Button
Close Drawer
Pick and Place
Sweep Trash
Slice Sausage
Put in Drawer
Sim-only (DP)
10%
10%
0%
5%
5%
0%
0%
Real-only (DP)
50%
60%
60%
50%
50%
70%
35%
GD data (DP)
45%
50%
45%
35%
45%
55%
25%
Low-data Real (DP)
10%
10%
15%
0%
10%
10%
5%
Co-training (DP)
55%
65%
50%
45%
55%
60%
40%
Sim-only ( π0.5 )
10%
15%
15%
5%
10%
5%
0%
Table 1: Main experimental results across seven tasks. We compare the success rates of DP and π0.5 under different data regimes, with RIGVid as a representative baseline.
Method
Pick
Button
Close Drawer
Pick and Place
Sweep Trash
Slice Sausage
Put in Drawer
Real-only (DP)
10%
5%
5%
0%
5%
5%
0%
Verified WM data (DP)
40%
50%
35%
30%
40%
45%
15%
Real-only ( π0.5 )
25%
20%
10%
20%
20%
15%
5%
Verified WM data ( π0.5 )
45%
60%
55%
40%
60%
50%
30%
Table 2: Success Rates in OOD Scenarios. Performance comparison of models trained on two types of data across seven manipulation tasks under out-of-distribution (OOD) conditions.
Figure 6: Scaling Success Rates. Success rates on two representative tasks improve as the number of Grounded-Dreamer (GD) generated trajectories increases.
Method
Pick
Button
Close Drawer
Pick and Place
Sweep Trash
Slice Sausage
Put in Drawer
w/ Masking (DP)
45%
50%
45%
35%
45%
55%
25%
w/o Masking (DP)
0%
5%
0%
0%
0%
0%
0%
w/ Masking ( π0.5 )
55%
75%
55%
50%
60%
60%
30%
w/o Masking ( π0.5 )
5%
0%
10%
0%
10%
5%
0%
Table 3: Ablation on Interaction-Aware Masking.
Method
Pick
Button
Close Drawer
Pick and Place
Sweep Trash
Slice Sausage
Put in Drawer
100% (DP)
45%
50%
45%
35%
45%
55%
25%
80% (DP)
40%
40%
35%
30%
30%
45%
10%
50% (DP)
25%
30%
25%
25%
20%
20%
5%
100% ( π0.5 )
55%
75%
55%
50%
60%
60%
30%
80% ( π0.5 )
35%
60%
50%
35%
45%
50%
20%
50% ( π0.5 )
25%
35%
25%
20%
25%
30%
10%
Table 4: Ablation on Replay Verification.
Figure 7: Ablation without Masking. Visualizations of video rollouts generated by the world model fine-tuned on raw simulation data without the interaction-aware masking strategy.
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Additional visualizations of automated data collection in simulation.
Figure 9: Additional visualizations of Grounded-Dreamer video generation.
Figure 10: Qualitative Visualizations of Refined Initial Frames.
Scaling robot learning requires large-scale, diverse demonstrations, yet real-world data collection via teleoperation remains prohibitively expensive and time-consuming. While video diffusion models offer a promising avenue for data scaling, existing generative approaches are often limited to superficial visual augmentation, or suffer from embodiment hallucinations that yield physically infeasible motions. We present a generalizable embodiment-centric world model that achieves scalable data generation by synthesizing photorealistic demonstrations with novel objects, in novel scenes, and from novel viewpoints. Our approach anchors generation to rendered robot motion while conditioning on explicit scene and object priors, effectively decoupling trajectory execution from environment synthesis. This formulation has the potential to unlock two powerful data scaling capabilities: (1) retrieval and rebirth, which repurposes existing trajectories into entirely new contexts without new motion data; and (2) prop-free teleoperation, where operators manipulate empty air and the model hallucinates the target objects and scene afterwards, eliminating reset time. We demonstrate with real-world experiments that our generated data consistently improves downstream policy performance and significantly reduces real-world data requirements across diverse manipulation tasks.
Junjie Ye, Rong Xue, Basile Van Hoorick +6
1USC Physical Superintelligence (PSI) Lab · 2Toyota Research Institute
Training generalist robots demands large-scale, diverse manipulation data, yet real-world collection is prohibitively expensive, and existing simulators are often constrained by fixed asset libraries and manual heuristics. To bridge this gap, we present V-Dreamer, a fully automated framework that generates open-vocabulary, simulation-ready manipulation environments and executable expert trajectories directly from natural language instructions. V-Dreamer employs a novel generative pipeline that constructs physically grounded 3D scenes using large language models and 3D generative models, validated by geometric constraints to ensure stable, collision-free layouts. Crucially, for behavior synthesis, we leverage video generation models as rich motion priors. These visual predictions are then mapped into executable robot trajectories via a robust Sim-to-Gen visual-kinematic alignment module utilizing CoTracker3 and VGGT. This pipeline supports high visual diversity and physical fidelity without manual intervention. To evaluate the generated data, we train imitation learning policies on synthesized trajectories encompassing diverse object and environment variations. Extensive evaluations on tabletop manipulation tasks using the Piper robotic arm demonstrate that our policies robustly generalize to unseen objects in simulation and achieve effective sim-to-real transfer, successfully manipulating novel real-world objects.
The collection of large-scale and diverse robot demonstrations remains a major bottleneck for imitation learning, as real-world data acquisition is costly and simulators offer limited diversity and fidelity with pronounced sim-to-real gaps. While generative models present an attractive solution, existing methods often alter only visual appearances without creating new behaviors, or suffer from embodiment inconsistencies that yield implausible motions. To address these limitations, we introduce AnchorDream, an embodiment-aware world model that repurposes pretrained video diffusion models for robot data synthesis. AnchorDream conditions the diffusion process on robot motion renderings, anchoring the embodiment to prevent hallucination while synthesizing objects and environments consistent with the robot's kinematics. Starting from only a handful of human teleoperation demonstrations, our method scales them into large, diverse, high-quality datasets without requiring explicit environment modeling. Experiments show that the generated data leads to consistent improvements in downstream policy learning, with relative gains of 36.4% in simulator benchmarks and nearly double performance in real-world studies. These results suggest that grounding generative world models in robot motion provides a practical path toward scaling imitation learning.
Junjie Ye, Rong Xue, Basile Van Hoorick +4
Toyota Research Institute · USC Physical Superintelligence (PSI) Lab