We present EgoNav, a system that enables a humanoid robot to traverse diverse, unseen environments by learning entirely from 5 hours of human walking data, with no robot data or finetuning. A diffusion model predicts distributions of plausible future trajectories conditioned on past trajectory, a 360 deg visual memory fusing color, depth, and semantics, and video features from a frozen DINOv3 backbone that capture appearance cues invisible to depth sensors. A hybrid sampling scheme achieves real-time inference in 10 denoising steps, and a receding-horizon controller selects paths from the predicted distribution. We validate EgoNav through offline evaluations, where it outperforms baselines in collision avoidance and multi-modal coverage, and through zero-shot deployment on a Unitree G1 humanoid across unseen indoor and outdoor environments. Behaviors such as waiting for doors to open, navigating around crowds, and avoiding glass walls emerge naturally from the learned prior. We will release the dataset and trained models. Our website: https://egonav.weizhuowang.com
Figures & tables
Fig. 1: EgoNav learns a navigation prior from human walking data: given past trajectory (purple) and a 360° visual context of the scene, a diffusion model generates a distribution of plausible future paths (red ribbons), with ribbon width indicating likelihood. The learned prior transfers directly to a Unitree G1 humanoid with zero robot data.
Fig. 2: Overview of the proposed method: A rolling buffer of 32 segmented RGB frames and cleaned depth frames are combined to construct a single visual memory (VM). The VM is encoded into a 64-dimensional embedding, fused with DINOv3 video features, and concatenated with 6D pose as input to the diffusion model. Then the future trajectories are denoised. All input and output of the prediction module are in the egocentric frame.
Fig. 3: Dataset: The dataset has a mix of weather, road, lighting, and traffic.
Fig. 4: Comparing depth frame with visual memory: A raw depth frame has only ∼ 90° of FOV and misses important scene information. The depth frame sees only the open space ahead and does not capture the stairs, the right turn path, or the wall to the left. Black regions are areas not yet observed.
Fig. 5: Channels in Visual Memory: The visual memory integrates past frames into a single panorama. It consists of a depth, color, and intensity-encoded 8-class semantic channel. 4 out of 8 channels are shown in the figure.
Fig. 6: Metrics overview: Collision-free score (CFS) uses selected semantics. All subsequent trajectories will be marked as collided; higher CFS and Smoothness are better, lower Best of N is better.
Full System (Full Data)
Collision ↑
Smoothness ↑
Best of 1 ↓
Best of 15 ↓
GT
97.6
∞
0.0
0.0
Ours
91.4
4.82
0.76
0.39
w/o DINOv3 features
90.6
4.76
0.81
0.41
Ours - Pilot
89.2
2.04
0.87
0.47
Component Ablations (Pilot Data)
w/o attention
86.6
2.78
1.00
0.49
TABLE I: Offline evaluation across ablations and baselines. Best , 2nd .
Fig. 7: Hybrid generation step search: BoN, Collision, Smoothness score for different DDIM/DDPM step combinations. The optimal configuration is 5 DDIM + 5 DDPM. All sweeps are performed on the same checkpoint. Combination 0+0 is pure noise therefore omitted.
Fig. 8: Real-world deployment on Unitree G1. The humanoid navigates diverse unseen environments using EgoNav with zero robot training data. Insets show predicted trajectory distributions overlaid on the semantic point cloud. (A,B) Static indoor scenes with different clutter levels. (C) T-junction with multi-modal trajectory predictions. (D) The robot waits at a closed door and proceeds when it opens, demonstrating semantic understanding. (E) Navigating through a gap between pedestrians. (F) Avoiding a glass wall invisible to depth sensors, enabled by DINOv3 video features.
Static
Corridor
Glass
Dynamic
Duration (min)
16
7.5
6.5 (5)
7.5
Distance (m)
471
240
225 (120)
201
Interventions
9
1
5 (11)
6
Interventions/min
0.56
0.13
0.77 (2.2)
0.80
Autonomous time%
97.2
99.3
96.2 (89.0)
96.0
TABLE II: Real-world deployment statistics on the Unitree G1.
Modeling the cognitive and experiential factors of human navigation is central to deepening our understanding of human-environment interaction and to enabling safe social navigation and effective assistive wayfinding. Most existing methods focus on forecasting motions in fully observed scenes and often neglect human factors that capture how people feel and respond to space. To address this gap, we propose EgoCogNav, a multimodal egocentric navigation framework that jointly forecasts perceived path uncertainty, trajectories and head motion from egocentric video, gaze, and motion history. To facilitate research in the field, we introduce the Cognition-aware Egocentric Navigation (CEN) dataset consisting of 6 hours real-world egocentric recordings capturing diverse navigation behaviors in real-world scenarios. Experiments show that EgoCogNav learns the perceived uncertainty that strongly correlates with human-like behaviors such as scanning, hesitation, and backtracking while improving trajectory and head-motion forecasting on held-out navigation recordings.
Zhiwen Qiu, Ziang Liu, Wenqian Niu +2
Cornell University, Ithaca, NY, USA · Georgia Institute of Technology, Atlanta, GA, USA
Deploying humanoid robots in unstructured terrain remains an open problem. While classic reinforcement learning struggles with the sheer complexity of real-world interactions, more promising methods leveraging human priors remain limited to models lacking contextual awareness. The restricted motion synthesis is a direct consequence of existing dataset pipelines failing to capture human-scene sequences in challenging environments. To bridge this gap between humanoid learning and scene reconstruction, we introduce the Egocentric Human-Terrain Reconstruction (EgoHTR) dataset. We develop and open-source a reconstruction pipeline capturing 55 scene-aligned 4D human motion sequences in diverse, complex environments using a multi-sensor setup of egocentric wearables and a portable 3D scanner. The resulting dataset comprises over 150k frames, which we evaluate against motion-capture ground truth, demonstrating state-of-the-art accuracy and establishing a rigorous benchmark for human motion analysis and synthesis. Further, we leverage this data to train perceptive locomotion policies, demonstrating hardware deployment on a Unitree G1 for reconstructed reference motions. Our pipeline enables community-driven dataset extensions and factors the problem to help researchers build foundational, context-aware robots that reliably traverse uneven terrain.
Alex Brandes, Haig Conti Georges Sajelian, Manthan Patel +11
Humanoid robots have achieved strong locomotion capabilities, but reliable navigation on versatile terrains remains challenging because obstacle avoidance must be coordinated with dynamically feasible motion. In this work, we present GuideWalk, a unified end-to-end framework that integrates traversability-aware navigation guidance with terrain-adaptive locomotion teacher for humanoid navigation. Specifically, we introduce a navigation module that provides explicit velocity guidance, decoupling obstacle avoidance from terrain conditions to enable robust planning across diverse environments. We propose a composite teacher distillation scheme, where goal-directed commands and dynamically consistent actions are aggregated and distilled into a single policy. To further improve robustness, the distilled policy is refined with reinforcement learning and an auxiliary behavior cloning objective, which promotes exploration while preserving desirable teacher behaviors. Experiments demonstrate that GuideWalk achieves stable and effective navigation while maintaining stable humanoid locomotion.