Organizations: The authors are with the Intelligent Space Robotics Laboratory, Skolkovo Institute of Science and Technology Moscow, Bolshoy Boulevard 30, bld. 1, 121205, Moscow, Russia.
We present DreamToNav, a novel autonomous robot framework that uses generative video models to enable intuitive, human-in-the-loop control. Instead of relying on rigid waypoint navigation, users provide natural language prompts (e.g. ``Follow the person carefully''), which the system translates into executable motion. Our pipeline first employs Qwen 2.5-VL-7B-Instruct to refine vague user instructions into precise visual descriptions. These descriptions condition NVIDIA Cosmos 2.5, a state-of-the-art video foundation model, to synthesize a physically consistent video sequence of the robot performing the task. From this synthetic video, we extract a valid kinematic path using visual pose estimation, robot detection and trajectory recovery. By treating video generation as a planning engine, DreamToNav allows robots to visually "dream" complex behaviors before executing them, providing a unified framework for obstacle avoidance and goal-directed navigation without task-specific engineering. We evaluate the approach on both a wheeled mobile robot and a quadruped robot in indoor navigation tasks. DreamToNav achieves a success rate of 76.7%, with final goal errors typically within 0.05-0.10 m and trajectory tracking errors below 0.15 m. These results demonstrate that trajectories extracted from generative video predictions can be reliably executed on physical robots across different locomotion platforms.
Figures & tables
Fig. 1: Pipeline for trajectory extraction from a single image and text prompt. A user provides an input image of a robot and a textual prompt describing the desired motion. The image prompt pair is processed by a VLM and the COSMOS video generation module to synthesize a plausible motion sequence. Visual odometry and pose estimation are then applied to the generated video to recover robot poses and track visual features over time. The final output is the estimated trajectory of the robot derived solely from the initial image and text prompt.
Fig. 2: Robot detection results using the trained YOLO11n model. The detector identifies both a wheeled UGV platform and a quadruped robot dog on diffused and real frames.
Fig. 3: Robot pose estimation pipeline. Left: robot detection in the camera image. Right: estimated robot trajectory obtained from visual odometry and pose estimation compared with the ground-truth robot pose recorded by the VICON motion capture system.
Fig. 4: Trajectory extraction from generated video frames. The robot is detected in each generated frame (right), and its pose is estimated using PnP to recover the robot trajectory (blue). The corresponding camera trajectory obtained from visual odometry is shown in red.
Fig. 5: Experimental evaluation of the proposed navigation framework. A user first captures an initial frame of the environment. Based on this image, two navigation tasks are generated: (1) move to the red square object and (2) move to the blue square object. The figure shows the initial frame, the final generated frames corresponding to each task, and the resulting UGV trajectories. The red and blue dots represent the robot pose estimated using visual odometry during execution, while the black dashed trajectory corresponds to the ground-truth robot motion recorded by the VICON motion capture system. The axes xy indicate the camera pose of the initial captured frame.
Fig. 6: Comparison between the video sequence generated by DreamToNav (top row) and the corresponding real-world execution (bottom row). Given a scene image and a natural language instruction, the system generates a future video predicting the robot’s motion while avoiding obstacles. A trajectory extracted from the generated frames is then executed on the physical robot. The real robot follows a motion pattern consistent with the generated video, demonstrating that visually generated plans can be translated into executable robot behavior.
Fig. 7: Quadruped obstacle-avoidance experiment. The trajectory extracted from the generated video (blue) closely follows the real trajectory recorded by VICON (black), while avoiding obstacles in the scene. The axes xy indicate the camera pose of the initial captured frame.
We present Action Agent, a two-stage framework that unifies agentic navigation video generation with flow-constrained diffusion control for multi-embodiment robot navigation. In Stage I, a large language model (LLM) acts as an orchestration module that selects video diffusion models, refines prompts through iterative validation, and accumulates cross-task memory to synthesize physically plausible first-person navigation videos from language and image inputs. This increases video generation success from 35% (single-shot) to 86% across 50 navigation tasks. In Stage II, we introduce FlowDiT, a Flow-Constrained Diffusion Transformer that converts optimized goal videos and language instructions into continuous velocity commands using action-space denoising diffusion. FlowDiT integrates DINOv2 visual features, learned optical flow for ego-motion representation, and CLIP language embeddings for semantic stopping. We pretrain on the RECON outdoor navigation dataset and fine-tune on 203 Unitree G1 humanoid episodes collected in Isaac Sim to calibrate velocity dynamics. A single 43M-parameter checkpoint achieves 73.2% navigation success in simulation and 64.7% task completion on a real Unitree G1 in unseen indoor environments under open-loop execution, while operating at 40--47 Hz. We evaluate Action Agent across three embodiments: a Unitree G1 humanoid (real hardware), a drone, and a wheeled mobile robot (Isaac Sim), demonstrating that decoupling trajectory imagination from execution yields a scalable and embodiment-aware paradigm for language-guided navigation.
Jeffrin Sam, Nguyen Khang, Yara Mahmoud +2
Intelligent Space Robotics Laboratory, Skoltech, Bolshoy Boulevard, 30, Moscow 121205, Russia
Previous Vision-Language-Action models face critical limitations in navigation: scarce, diverse data from labor-intensive collection and static representations that fail to capture temporal dynamics and physical laws. We propose NavDreamer, a video-based framework for 3D navigation that leverages generative video models as a universal interface between language instructions and navigation trajectories. Our main hypothesis is that video's ability to encode spatiotemporal information and physical dynamics, combined with internet-scale availability, enables strong zero-shot generalization in navigation. To mitigate the stochasticity of generative predictions, we introduce a sampling-based optimization method that utilizes a VLM for trajectory scoring and selection. An inverse dynamics model is employed to decode executable waypoints from generated video plans for navigation. To systematically evaluate this paradigm in several video model backbones, we introduce a comprehensive benchmark covering object navigation, precise navigation, spatial grounding, language control, and scene reasoning. Extensive experiments demonstrate robust generalization across novel objects and unseen environments, with ablation studies revealing that navigation's high-level decision-making nature makes it particularly suited for video-based planning.
Xijie Huang, Weiqi Gai, Tianyue Wu +5
Zhejiang University · Differential Robotics · Beihang University
Generative video models can serve as a promising backbone for robot navigation by predicting future observations as video plans. Recent approaches often condition video planning on short-horizon guidance and recover geometric waypoints through scene reconstruction, leaving longer-horizon planning and precise video-to-action translation less explored. We present CueNav, a video model-based navigation framework combining visual cue guided video planning with an embodiment-specific Inverse-Dynamics Model (IDM). As visual cues, we use a Bird's-Eye View (BEV) map to convey global task context and retain part of the robot body in the egocentric observation to expose embodiment context. These cues guide the video planner, while the IDM translates dense flow fields extracted from the video plan into robot actions. With the visual cue encoding global task context, CueNav achieves nearly 2x higher success in maze navigation than planning without the cue. The body-aware view with the IDM enables precise navigation with 70% success in a narrow passage where comparison methods largely fail to complete the task. We further demonstrate zero-shot semantic-conditioned navigation and deployment of the same video planner across different robot platforms. Our results show that visual cue-guided video planning with embodiment-specific action grounding paves the way toward a generalizable navigation framework for longer-horizon planning and embodiment-aware control. Additional results and code are available on our project website: https://cuenav.github.io.
Hojin Lee, Sizhe Lester Li, Maximilian Hilger +4
Munich Institute of Robotics and Machine Intelligence (MIRMI), Technical University of Munich, 80797 Munich, Germany. · Computer Science and Artificial Intelligence Laboratory (CSAIL), Massachusetts Institute of Technology, Cambridge, MA 02139, USA. · Centre for Applied Autonomous Sensor Systems (AASS), Örebro University, 72081 Örebro, Sweden.