Training generalist robots demands large-scale, diverse manipulation data, yet real-world collection is prohibitively expensive, and existing simulators are often constrained by fixed asset libraries and manual heuristics. To bridge this gap, we present V-Dreamer, a fully automated framework that generates open-vocabulary, simulation-ready manipulation environments and executable expert trajectories directly from natural language instructions. V-Dreamer employs a novel generative pipeline that constructs physically grounded 3D scenes using large language models and 3D generative models, validated by geometric constraints to ensure stable, collision-free layouts. Crucially, for behavior synthesis, we leverage video generation models as rich motion priors. These visual predictions are then mapped into executable robot trajectories via a robust Sim-to-Gen visual-kinematic alignment module utilizing CoTracker3 and VGGT. This pipeline supports high visual diversity and physical fidelity without manual intervention. To evaluate the generated data, we train imitation learning policies on synthesized trajectories encompassing diverse object and environment variations. Extensive evaluations on tabletop manipulation tasks using the Piper robotic arm demonstrate that our policies robustly generalize to unseen objects in simulation and achieve effective sim-to-real transfer, successfully manipulating novel real-world objects.
Figures & tables
Fig. 2: Overview of the V-Dreamer Pipeline. The framework consists of three integrated stages: (1) Semantic-to-Physics Scene Synthesis. An LLM-based semantic planner decomposes prompts into asset manifests, which are then transformed into 3D meshes via generative visual synthesis and memory-efficient reconstruction before being assembled into a physics-validated layout. (2) Video-Prior-Based Trajectory Generation. Using the stabilized scene as a prior, we employ video foundation models with targeted negative prompting to dream up physically plausible manipulation motions. (3) Sim-to-Gen Alignment. Actionable 3D trajectories are extracted from the 2D pixel domain through mask-restricted tracking, metric depth estimation, and 3D motion lifting, ultimately mapping visual dreams into executable robot end-effector poses.
Fig. 3: Qualitative examples of V-Dreamer synthesized tasks of increasing difficulty. Left: Input Prompts and generated 3D scenes. Middle: Generated manipulation videos and the corresponding robot manipulation results. Right: Extracted end-effector trajectories for demonstrations.
Fig. 4: Qualitative examples of V-Dreamer synthesized scenes with varying object instances, textures, and layouts.
Fig. 5: Simulation Evaluation. (Left top) Representative synthesized training scenes with object augmentation, illustrating diverse spatial layouts used to generate demonstrations. (Left bottom) Zero-shot inference trajectory of the learned policy on an unseen mug, showing the manipulation sequence (Init → Pick → Grasp → Place). (Right) Success rate of the learned policy on 10 unseen mug instances under different training dataset sizes (500 trials each). Performance improves with increasing data scale and peaks at 2.5k synthesized demonstrations.
Fig. 6: V-Dreamer enables a Real2Sim–Sim2Gen–Sim2Real pipeline for real-world robot manipulation. Real-world observations are first converted into simulation environments (Real2Sim). V-Dreamer then synthesizes executable manipulation trajectories in simulation (Sim2Gen) for policy training. Finally, the learned policy is deployed on a physical robot arm (Sim2Real), successfully executing a pick-and-place task that transfers a novel yellow cube to the red target without any real-world fine-tuning.