Organizations: The authors are with the Intelligent Space Robotics Laboratory, Center for Digital Engineering, Skolkovo Institute of Science and Technology.
Efficiently predicting motion plans directly from vision remains a fundamental challenge in robotics, where planning typically requires explicit goal specification and task-specific design. Recent vision-language-action (VLA) models infer actions directly from visual input but demand massive computational resources, extensive training data, and fail zero-shot in novel scenes. We present a unified image-space diffusion policy handling both meter-scale navigation and centimeter-scale manipulation via multi-scale feature modulation, with only 5 minutes of self-supervised data per task. Three key innovations drive the framework: (1) Multi-scale FiLM conditioning on task mode, depth scale, and spatial attention enables task-appropriate behavior in a single model; (2) trajectory-aligned depth prediction focuses metric 3D reasoning along generated waypoints; (3) self-supervised attention from AnyTraverse enables goal-directed inference without vision-language models and depth sensors. Operating purely from RGB input (2.0 GB memory, 10 Hz), the model achieves robust zero-shot generalization to novel scenes while remaining suitable for onboard deployment.
Figures & tables
Fig. 1: A single unified diffusion policy handles both centimeter-scale pre-grasping planning (left: object approach with ± 2 cm precision, cyan inset shows object attention) and meter-scale motion planning (right: hallway navigation with ± 10 cm obstacle avoidance, orange overlay shows traversable floor). Knowledge transfer between tasks is achieved through context-aware conditioning with shared model weights.
Fig. 2: Context-aware cross-task diffusion policy architecture. Top (Supervision): RGB input is processed by a frozen visual encoder, modulated via FiLM conditioning using task-specific context (task mode, depth scale, spatial attention). The state-modulated decoder outputs traversability T^t for supervision. Attention maps differ by task: navigation uses floor traversability (left heatmap), while manipulation focuses on target objects (right heatmap). Purple dashed lines indicate single-task paths; blue dashed lines show cross-task conditioning shared between operational modes. Bottom (Diffusion): The UNet performs iterative denoising conditioned on context c and noisy trajectory xt+Sg , progressively refining predictions over N steps (right: xt→xt−1→x0 ).
Fig. 3: Ground truth generation pipeline for start and goal prediction. For each sample: RGB input (64 × 64), traversability map, object attention heatmap, ground-truth trajectory, and trajectory overlaid on the traversability map.
Fig. 4: Qualitative results showing attention maps (top) and trajectories (bottom) across three tasks. (a) Exploration : Floor attention identifies traversable regions for obstacle-free navigation. (b) Navigation to Goal : Attention highlights target table while generating goal-directed trajectory. (c) Pre-Grasping : Object-centric attention enables centimeter-precise approach planning. Green circles: start; red stars: predicted goals. All results demonstrate zero-shot generalization on novel test scenes.
Fig. 5: Task performance on novel test scenes. Navigation: 100% goal and collision success. Pre-grasping: 70.6% goal success, 100% collision-free. Exploration: 100% obstacle avoidance. Zero-shot generalization across all modes.
Fig. 6: 3D trajectory prediction with obstacle avoidance. The model generates collision-free paths (green) maintaining mean 2.98 mclearance from obstacles (minimum 2.20 m). Vertical and lateral trajectory variations validate depth-based 3D spatial reasoning.
Scenario
Method
SR (%)
Dist (m)
Exploration
NoMaD
93
2.24
Ours
100
2.98
Autonomous Goal
NoMaD
N/A ∗
N/A
Ours
100
0.29
Provide goal
NoMaD
60
0.5
Ours
52
0.35
TABLE I: Navigation Performance. SR: Success Rate; Dist: mean clearance (exploration) / goal distance (navigation). ∗ N/A: NoMaD not trained for autonomous semantic goals. Goal success: ≤ 0.5 m for interaction.
Fig. 7: Goal prediction accuracy (bird’s-eye view, n=8 ). Pre-grasping achieves 4.71 cm mean error versus 29.68 cm for navigation, demonstrating task-appropriate precision through depth-scale conditioning: centimeter-level for manipulation, decimeter-level for navigation.
Image-goal visual navigation is a fundamental capability for embodied agents. Existing navigation policies efficiently predict waypoint trajectories but lack visual foresight, while navigation world models can anticipate future observations but often require costly planning rollouts. We present UniNav, a unified world-action model that generates future visual observations and continuous waypoint trajectories through a single diffusion process. Given history frames and a goal image, UniNav jointly denoises visual and waypoint tokens within a single transformer, unifying future prediction and action generation in a shared framework. To improve spatial grounding, we incorporate geometry-aware camera tokens. We also train on both trajectory-labeled navigation data and video-only data, enabling the model to benefit from diverse videos without waypoint annotations. Based on this unified framework, we introduce two variants: UniNav-Full jointly predicts interpretable future observations and their corresponding trajectories, while UniNav-Fast removes future-image tokens at inference for efficient trajectory prediction. Experiments on navigation benchmarks show that UniNav outperforms the strongest baseline in ATE across all datasets. With one-step inference, UniNav-Fast achieves a latency of 0.1s without a substantial accuracy drop. Code will be released.
Changqing Zhou, Yueru Luo, Zeyu Jiang +1
1The Hong Kong University of Science and Technology (Guangzhou) · 2The Chinese University of Hong Kong, Shenzhen
Visual traversability estimation is central to autonomous navigation, yet most approaches either rely on prompt-driven Vision-Language Model (VLM) or decouple traversability from trajectory planning, requiring separate planners with heavy mapping, manual tuning, and extended deployment time. We propose EmbodiedDiffusion, a diffusion-based framework that simultaneously predicts traversability maps and generates feasible trajectories from RGB images using planner-free synthetic supervision and embodiment conditioning for cross-platform transfer. The framework distills category-level traversability semantics from a VLM teacher into a lightweight student model during training, enabling prompt-free, real-time inference at deployment. A modular FiLM-based conditioning mechanism isolates embodiment-specific reasoning into a compact trainable subset of the network, allowing rapid adaptation to new robot platforms without retraining the visual backbone or the trajectory diffusion model. Across indoor environments with quadruped and aerial robots, EmbodiedDiffusion achieves 80-100% navigation success in the full-data regime with real-time inference (90 ms) and adapts to new platforms using only 10 min of visual data collection, demonstrating scalable, unified traversability reasoning and trajectory generation for heterogeneous robots.
Iana Zhura, Sausar Karaf, Faryal Batool +7
Intelligent Space Robotics Laboratory, Center for Digital Engineering, Skolkovo Institute of Science and Technology.
Vision Navigation Foundation Models (VNMs) promise end-to-end learned navigation policies capable of zero-shot deployment across diverse embodiments and environments. To maintain generality, many vision-based navigation models predict normalized actions. However, this normalization introduces a critical deployment vulnerability: applying different scaling factors to the same normalized trajectory alters its physical geometry, which degrades navigation performance and increases collision risks. We address this vulnerability by conditioning the model on normalized action histories alongside image observations, providing explicit context on the relationship between the model's predictions and the robot's actual physical displacement. Furthermore, current VNMs often struggle in visually repetitive environments that lack distinct features. To resolve this issue, we integrate a DINOv3 encoder, whose richer representations enable our model to capture both spatial and geometric dimensions between observations. VISTA generalizes robustly to out-of-distribution environments, achieving 100% goal prediction accuracy in zero-shot, real-world deployment in Outdoor, Forest and Office settings, and an average of 95% checkpoints crossed, demonstrating consistent path following in unseen environments.
Maeva Guerrier, Koki Kobayashi, Simon Roy +2
Polytechnique Montreal, MILA · Institute of Science Tokyo · Polytechnique Montreal +2