cs.ROMar 6, 2026

DreamToNav: Generalizable Navigation for Robots via Generative Video Planning

Authors: Valerii Serpiva, Jeffrin Sam, Chidera Simon, Hajira Amjad, Iana Zhura, Artem Lykov, Dzmitry Tsetserukou

Organizations: The authors are with the Intelligent Space Robotics Laboratory, Skolkovo Institute of Science and Technology Moscow, Bolshoy Boulevard 30, bld. 1, 121205, Moscow, Russia.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We present DreamToNav, a novel autonomous robot framework that uses generative video models to enable intuitive, human-in-the-loop control. Instead of relying on rigid waypoint navigation, users provide natural language prompts (e.g. ``Follow the person carefully''), which the system translates into executable motion. Our pipeline first employs Qwen 2.5-VL-7B-Instruct to refine vague user instructions into precise visual descriptions. These descriptions condition NVIDIA Cosmos 2.5, a state-of-the-art video foundation model, to synthesize a physically consistent video sequence of the robot performing the task. From this synthetic video, we extract a valid kinematic path using visual pose estimation, robot detection and trajectory recovery. By treating video generation as a planning engine, DreamToNav allows robots to visually "dream" complex behaviors before executing them, providing a unified framework for obstacle avoidance and goal-directed navigation without task-specific engineering. We evaluate the approach on both a wheeled mobile robot and a quadruped robot in indoor navigation tasks. DreamToNav achieves a success rate of 76.7%, with final goal errors typically within 0.05-0.10 m and trajectory tracking errors below 0.15 m. These results demonstrate that trajectories extracted from generative video predictions can be reliably executed on physical robots across different locomotion platforms.

Figures & tables

Fig. 1: Pipeline for trajectory extraction from a single image and text prompt. A user provides an input image of a robot and a textual prompt describing the desired motion. The image prompt pair is processed by a VLM and the COSMOS video generation module to synthesize a plausible motion sequence. Visual odometry and pose estimation are then applied to the generated video to recover robot poses and track visual features over time. The final output is the estimated trajectory of the robot derived solely from the initial image and text prompt.
Fig. 2: Robot detection results using the trained YOLO11n model. The detector identifies both a wheeled UGV platform and a quadruped robot dog on diffused and real frames.
Fig. 3: Robot pose estimation pipeline. Left: robot detection in the camera image. Right: estimated robot trajectory obtained from visual odometry and pose estimation compared with the ground-truth robot pose recorded by the VICON motion capture system.
Fig. 4: Trajectory extraction from generated video frames. The robot is detected in each generated frame (right), and its pose is estimated using PnP to recover the robot trajectory (blue). The corresponding camera trajectory obtained from visual odometry is shown in red.
Fig. 5: Experimental evaluation of the proposed navigation framework. A user first captures an initial frame of the environment. Based on this image, two navigation tasks are generated: (1) move to the red square object and (2) move to the blue square object. The figure shows the initial frame, the final generated frames corresponding to each task, and the resulting UGV trajectories. The red and blue dots represent the robot pose estimated using visual odometry during execution, while the black dashed trajectory corresponds to the ground-truth robot motion recorded by the VICON motion capture system. The axes xy indicate the camera pose of the initial captured frame.
Fig. 6: Comparison between the video sequence generated by DreamToNav (top row) and the corresponding real-world execution (bottom row). Given a scene image and a natural language instruction, the system generates a future video predicting the robot’s motion while avoiding obstacles. A trajectory extracted from the generated frames is then executed on the physical robot. The real robot follows a motion pattern consistent with the generated video, demonstrating that visually generated plans can be translated into executable robot behavior.
Fig. 7: Quadruped obstacle-avoidance experiment. The trajectory extracted from the generated video (blue) closely follows the real trajectory recorded by VICON (black), while avoiding obstacles in the scene. The axes xy indicate the camera pose of the initial captured frame.
TaskPath (m)Final Err (m)Track Err (m)Succ.
UGV →\rightarrow Red2.80.05–0.100.05–0.107/10
UGV →\rightarrow Blue2.90.05–0.100.05–0.108/10
Quadruped2.30.05–0.100.03–0.088/10
Range2.3–2.9<0.10<0.10<0.15<0.1523/30
TABLE I: Navigation performance of DreamToNav.
More in this topic

Robot Navigation

Robot navigation studies how robots perceive, localize, plan, and move safely toward goals in static or changing environments.

See all 532 papers

Explore similar work

CardsList

  1. Action Agent: Agentic Video Generation Meets Flow-Constrained Diffusion

    May 2, 2026Jeffrin Sam, Nguyen Khang, Yara Mahmoud +2Trajectory GenerationDiffusion-Based Planning

    Intelligent Space Robotics Laboratory, Skoltech, Bolshoy Boulevard, 30, Moscow 121205, Russia

  2. NavDreamer: Video Models as Zero-Shot 3D Navigators

    Feb 10, 2026Xijie Huang, Weiqi Gai, Tianyue Wu +5Vision-Language NavigationModel-Based Planning

    Zhejiang University · Differential Robotics · Beihang University

  3. Visual Cue Guided Video Planning for Generalizable Robot Navigation

    Sep 15, 2026Hojin Lee, Sizhe Lester Li, Maximilian Hilger +4Embodied NavigationRobot Navigation

    Munich Institute of Robotics and Machine Intelligence (MIRMI), Technical University of Munich, 80797 Munich, Germany. · Computer Science and Artificial Intelligence Laboratory (CSAIL), Massachusetts Institute of Technology, Cambridge, MA 02139, USA. · Centre for Applied Autonomous Sensor Systems (AASS), Örebro University, 72081 Örebro, Sweden.