DR-VLA: Learning Human-Like Quadruped Navigation from Egocentric Web Videos via Domain-Invariant Scene Representations

Jun Tian; Yingjie Niu; Ruo Chen; Hongyi Lin; Chao Li

cs.ROOct 13, 2026

DR-VLA: Learning Human-Like Quadruped Navigation from Egocentric Web Videos via Domain-Invariant Scene Representations

Authors: Jun Tian, Yingjie Niu, Ruo Chen, Hongyi Lin, Chao Li

Organizations: Deep Robotics · University of Illinois at Urbana-Champaign · Zhejiang University · Tsinghua University · Massachusetts Institute of Technology

Abstract

Quadruped robots performing outdoor deliveries must interpret complex scenes, anticipate changes, and navigate in socially compliant ways. Robot demonstrations are costly to collect at the scale needed for generalization. Egocentric web videos offer abundant human navigation experience, but differences in viewpoint and motion make this experience difficult to transfer to robots. We introduce DR-VLA, a Mixture-of-Transformers framework that transfers human navigation experience through a domain-invariant scene representation (DISR). The framework combines vision-language understanding and latent world modeling with a flow-matching action expert. We introduce EgoLynx, comprising 10,000 hours of curated egocentric web video (EgoLynx-Web) and 2,000 clips of teleoperated robot navigation collected during real-world deliveries (EgoLynx-Robot). We pretrain the understanding and world experts on EgoLynx-Web without action labels, then adapt the model to robot observations and train the action expert on EgoLynx-Robot to predict executable trajectories. We also propose the Quadruped Navigation Score (QNS), a pseudo-closed-loop metric combining safety, instruction compliance, comfort, and progress. On the held-out EgoLynx-Robot test split, DR-VLA improves QNS by approximately 5% and reduces open-loop final heading error (FHE) by 14.6%, each relative to the best competing result for that metric. Ablations show that web-video pretraining improves generalization and that the model's key components contribute to performance. As a future-prediction target, DISR outperforms five generic visual representations and yields performance close to task-specific 3D perception without dedicated geometric labels or point clouds for target construction. Trained separately on NAVSIM v1 without web pretraining, DR-VLA achieves 92.1 PDMS, suggesting that the framework also applies to autonomous driving.

Figures & tables

Figure 1: Overview. DR-VLA learns to understand and anticipate scenes from egocentric web videos (left) and to act from teleoperated quadruped clips (center bottom). Three experts (center top) map four fisheye views and a navigation instruction to a trajectory and to future domain-invariant scene representations (DISR), which encode semantic, spatial, and relational structure (right).

Figure 2: DR-VLA and the DISR Pretraining. Top: Combined Spatial and Semantic information are encoded by $E_{\phi}$ into the DISR latent $\mathbf{z}$ and reconstructed by $D_{\phi}$ . Bottom left: the understanding, world, and action experts share attention and are trained with $\mathcal{L}_{\text{cap}}$ , $\mathcal{L}_{\text{world}}$ (targets from the frozen $E_{\phi}$ ), and $\mathcal{L}_{\text{act}}$ . Bottom right: block-wise attention mask.

VLN-based policies
	Open-loop				Pseudo-closed-loop
Model	ADE $\downarrow$	FDE $\downarrow$	AHE $\downarrow$	FHE $\downarrow$	NC $\uparrow$	WAC $\uparrow$	TTC $\uparrow$	CL $\uparrow$	EP $\uparrow$	IC $\uparrow$	QNS $\uparrow$
InternNav ( Wei et al., 2026a )	0.079	0.098	0.038	0.062	0.945	0.955	0.871	0.815	0.893	0.818	0.764
NaVILA ( Cheng et al., 2025 )	0.030	0.059	0.028	0.048	0.968	0.975	0.912	0.838	0.941	0.893	0.843
OmniNav ( Xue et al., 2026 )	0.090	0.152	0.033	0.054	0.939	0.949	0.863	0.811	0.866	0.803	0.743
StreamVLN ( Wei et al., 2026b )	0.036	0.067	0.032	0.053	0.964	0.972	0.905	0.843	0.933	0.884	0.833

Table 1: Quantitative navigation results on the EgoLynx-Robot test split. ADE/FDE and AHE/FHE denote average/final displacement and heading errors, respectively. The pseudo-closed-loop metrics comprise no collision (NC), walkable-area compliance (WAC), time-to-collision (TTC), clearance (CL), ego progress (EP), instruction compliance (IC), and the Quadruped Navigation Score (QNS).

Figure 3: Predicted trajectories in unseen scenes. Predictions are shown on replayed observations from four scenes (columns). The rows correspond to variants trained on EgoLynx-Web only (using DPVO pseudo-trajectories), EgoLynx-Robot only, or both.

Representation	Use	Sensors	Box/Line labels	Front end	QNS $\uparrow$	Latency (ms) $\downarrow$
DISR	Target	Camera	No	None	0.884	120
OCC+BEV	Target	Camera	Yes	None	0.885	127
DISR	Input	Camera	No	DISR extraction	0.897	220
OCC+BEV	Input	Camera + PC	Yes	OCC+BEV	0.906	347

Table 3: DISR and OCC+BEV as targets or inputs on EgoLynx-Robot. Sensors are inference-time inputs. PC denotes sensor point clouds, used only offline in OCC+BEV-target. Box/Line labels are additional geometric supervision. Front end denotes online representation extraction. Latency includes the full perception and policy pipeline. The DISR-target QNS is from Table 1 .

Method	Input	Backbone	NC	DAC	TTC	Comf.	EP	PDMS
TransFuser ( Chitta et al., 2023 )	C+L	ResNet-34	97.7	92.8	92.8	100	79.2	84.0
DiffusionDrive ( Liao et al., 2025 )	C+L	ResNet-34	98.2	96.2	94.7	100	82.2	88.1
WoTE ( Li et al., 2025b )	C+L	ResNet-34	98.5	96.8	94.9	99.9	81.9	88.3
DiffusionDriveV2 ( Zou et al., 2025 )	C+L	ResNet-34	98.3	97.9	94.8	99.9	87.5	91.2
PRIX ( Wozniak et al., 2026 )	C	ResNet-34	98.1	96.3	94.1	100	82.3	87.8
iPad ( Guo et al., 2026 )	C	ResNet-34	98.6	98.3	94.9	100	88.0	91.7

Table 4: Results on NAVSIM v1 navtest ( Dauner et al., 2024 ) . Sub-scores and PDMS of prior work are taken from the cited papers. “C” cameras, “L” LiDAR.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Table 7

Explore similar work

Jun 1, 2026

Co-training with Ego-centric Video and Demonstration for Robot Navigation Task

Vision-language-action (VLA) models are promising for diverse robotic tasks, but their performance heavily depends on large-scale high-quality training data, whose collection on real robots is costly and time-consuming. While prior work has explored augmenting manipulation datasets with egocentric human videos, applying such approaches to mobile robot navigation remains challenging due to viewpoint changes during locomotion. In this paper, we propose a framework that converts egocentric walking videos into datasets for mobile robot imitation learning. The proposed method estimates camera motion from human videos and transforms it into action representations compatible with ground mobile robots. By jointly training a VLA model on human-derived and robot-collected datasets, the model achieves improved language understanding and more robust action generation than training with either data source alone. Experiments on a fruit-search navigation task demonstrate that human egocentric videos provide an effective and scalable data source for mobile robot learning.

Shoya Kuno, Yumo Ouchi, Kanata Suzuki

Spatial Robotics Research Center, Fujitsu Limited., Kanagawa 211-8588, Japan.

Mobile Robotics Robot Navigation Cross-Embodiment Robot Learning Learning from Demonstration Vision-Language-Action Models Robot Imitation Learning

Ask PDF

Jun 15, 2026

ACE-Ego-0: Unifying Egocentric Human and Robotic Data for VLA Pretraining

Vision-Language-Action (VLA) models benefit from large-scale and diverse embodied data, yet scaling robot trajectory collection is costly and labor-intensive. Recent advances show that large-scale egocentric human videos provide complementary real-world supervision in pretraining. However, joint training on human and robot data remains challenging due to divergences in action spaces, embodiment structures, temporal dynamics, and supervision quality. We introduce ACE-EGO-0, a unified VLA pretraining framework jointly leveraging heterogeneous data sources. To extract large-scale pretraining supervision from egocentric human videos, we build a scalable egocentric video-to-action pipeline that converts raw human videos into robot-format pseudo-action trajectories. To make these labels comparable with robot demonstrations, ACE-EGO-0 uses a unified action representation based on camera-space actions, morphology conditioning, and time-aligned action chunking. To robustly leverage noisy pseudo-action supervision from egocentric human videos, we formulate a reliability-aware training objective with a human auxiliary loss that concentrates supervision on reliable signals. We instantiate ACE-EGO-0 on 4.53K hours of robot and simulation data, together with 1.48K hours of pseudo-action-labeled egocentric human data. Experiments show that incorporating large-scale human supervision under reliability-aware weighting consistently improves both unified joint pretraining and supervised fine-tuning. ACE-EGO-0 achieves state-of-the-art performance on RoboCasa GR1 TableTop and RoboTwin 2.0, while demonstrating strong transfer to real-world bimanual manipulation.

Hao Li, Ganlong Zhao, Yufei Liu +8

ACE Robotics · CUHK MMLab · SJTU +2

Cross-Embodiment Robot Learning Vision-Language-Action Models Robot Imitation Learning Vision-Language Pretraining

Ask PDF

Mar 10, 2026

Implicit Geometry Representations for Vision-and-Language Navigation from Web Videos

Vision-and-Language Navigation (VLN) has long been constrained by the limited diversity and scalability of simulator-curated datasets, which fail to capture the complexity of real-world environments. To overcome this limitation, we introduce a large-scale video-instruction framework derived from web-based room tour videos, enabling agents to learn from natural human walking demonstrations in diverse, realistic indoor settings. Unlike existing datasets, our framework integrates both open-ended description-enriched trajectories and action-enriched trajectories reconstructed in 3D, providing richer spatial and semantic supervision. A key extension in this work is the incorporation of implicit geometry representations, which extract spatial cues directly from RGB frames without requiring fragile 3D reconstruction. This approach substantially improves data utilization, alleviates reconstruction failures, and unlocks large portions of previously unusable video data. Comprehensive experiments across multiple VLN benchmarks (CVDN, SOON, R2R, and REVERIE) demonstrate that our method not only sets new state-of-the-art performance but also enables the development of robust zero-shot navigation agents. By bridging large-scale web videos with implicit spatial reasoning, this work advances embodied navigation towards more scalable, generalizable, and real-world applicable solutions.

Mingfei Han, Haihong Hao, Liang Ma +6

Department of Computer Vision, Mohamed Bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates. · School of Information Science and Technology, University of Science and Technology of China. · Shenzhen Campus of Sun Yat-Sen University.

Vision-Language Navigation 3D Scene Representation

Ask PDF