Implicit Geometry Representations for Vision-and-Language Navigation from Web Videos

Mingfei Han; Haihong Hao; Liang Ma; Kamila Zhumakhanova; Ekaterina Radionova; Jingyi Zhang; Xiaojun Chang; Xiaodan Liang

cs.CVMar 10, 2026

Implicit Geometry Representations for Vision-and-Language Navigation from Web Videos

Authors: Mingfei Han, Haihong Hao, Liang Ma, Kamila Zhumakhanova, Ekaterina Radionova, Jingyi Zhang, Xiaojun Chang, Xiaodan Liang, +1 more

Organizations: Department of Computer Vision, Mohamed Bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates. · School of Information Science and Technology, University of Science and Technology of China. · Shenzhen Campus of Sun Yat-Sen University.

Abstract

Vision-and-Language Navigation (VLN) has long been constrained by the limited diversity and scalability of simulator-curated datasets, which fail to capture the complexity of real-world environments. To overcome this limitation, we introduce a large-scale video-instruction framework derived from web-based room tour videos, enabling agents to learn from natural human walking demonstrations in diverse, realistic indoor settings. Unlike existing datasets, our framework integrates both open-ended description-enriched trajectories and action-enriched trajectories reconstructed in 3D, providing richer spatial and semantic supervision. A key extension in this work is the incorporation of implicit geometry representations, which extract spatial cues directly from RGB frames without requiring fragile 3D reconstruction. This approach substantially improves data utilization, alleviates reconstruction failures, and unlocks large portions of previously unusable video data. Comprehensive experiments across multiple VLN benchmarks (CVDN, SOON, R2R, and REVERIE) demonstrate that our method not only sets new state-of-the-art performance but also enables the development of robust zero-shot navigation agents. By bridging large-scale web videos with implicit spatial reasoning, this work advances embodied navigation towards more scalable, generalizable, and real-world applicable solutions.

Figures & tables

Fig. 1: Overview of our origin RoomTour3D with COLMAP reconstruction and explicit geometry information. Starting from a room tour video, we first apply BLIP-2 [ 91 ] on frame sequence to predict the room locations. Next, we use RAM [ 18 ] and Grounding-DINO [ 19 ] to identify objects within the frames and employ Depth-Anything [ 20 ] for depth prediction. Subsequently, COLMAP is used to reconstruct the 3D scene with complete geometry information, and we sample human walking trajectories from the continuous frames. The trajectory captures open-world objects, their positions, and depths relative to the camera. Finally, we use advanced LLM, i.e ., GPT-4 to generate the free-form descriptions for pretraining, namely description-enriched trajectories. Specifically, for the trajectory shown in the figure, which involves instant turning points, we specially treat <0> to <6> as walking trajectory, <A> <B> and <C> as side-watching points and use them as negative candidates for navigation finetuning task, namely action-enriched trajectories. For more details, please refer to Section III .

Fig. 2: Instruction generation in a controllable way. (a) Using open-sourced expert models, we identify what objects are in the frames , and assess how far an object is and determine where an object is located . The information is then textualized to create richly detailed frame captions. (b) BLIP-2 is adopted to predict and smooth room location across sequential frames. (c) Combining room locations and object information, we use GPT-4 for controllable and open-vocabulary instruction generation. The prompt consists of a task instruction that defines the generation task, and in-context examples that constrain the output style.

Fig. 3: Model training diagram using RoomTour3D. Two complementary tasks are designed to enhance NaviLLM: (a) Pretraining. Sampled frames along the trajectory serve as candidate observations, and the model is optimized to summarize object progression along the path. (b) Finetuning. Each frame acts as a navigable step. Given the historical observations <0>–<2> and a navigation instruction, the model predicts the next action by choosing among candidate views (A–D).

Fig. 4: Overview of implicit-geometry training. Our RoomTour3D-IGR processes both explicit geometry from simulators and implicit geometry from RoomTour videos, alongside task instructions. (a) For simulator data, the pipeline follows the original RoomTour3D setup: RGB observations are encoded by the scene encoder, while explicit geometric features (e.g., distance and heading) are incorporated. For instance, <2>–<A>: 1.6 m, 88° denotes a spatial relation guiding the agent’s next action. (b) For RoomTour videos, frames 1–2 form the trajectory history, and frame 3 acts as a navigation candidate. RGB frames are encoded by the scene encoder, while a VGGT-based spatial encoder extracts implicit geometric features. These embeddings are projected via a spatial projector into the LLM’s latent space to guide accurate action prediction.

Methods	CVDN		SOON		R2R		REVERIE
Methods	Val-U	Test	Val-U	Test	Val-U	Test	Val-U	Test
Models Focusing on Single Task
PREVALENT [ 47 ]	3.15	2.44	-	-	53	51	-	-
HOP [ 48 ]	4.41	3.24	-	-	57	59	26.1	24.3
HAMT [ 36 ]	5.13	5.58	-	-	61	60	30.2	26.7
DUET [ 37 ]	-	-	22.6	21.4	60	58	33.7	36.0

TABLE I: Overall comparison with the baseline methods. Our RoomTour3D data can boost NaviLLM by a margin on SOON, R2R and REVERIE on SPL metric and on CVDN GP metric. ⋆ denotes reproduced results. † denotes involving all RoomTour3D data into geometry-aware training, refering to Table VI for detailed comparison.

Methods	Data type		CVDN		SOON		R2R		REVERIE
Methods	Desc	Action	Val-U	Test	Val-U	Test	Val-U	Test	Val-U	Test
NaviLLM (w. Pretrain) ⋆	$\times$	$\times$	6.09	-	28.0	-	57	-	31.4	-
RoomTour3D ${}_{\text{Desc}}$	$\checkmark$	$\times$	6.96	7.55	30.2	26.5	62	62	37.1	35.1
RoomTour3D ${}_{\text{Action}}$	$\checkmark$	$\checkmark$ (E)	6.33	7.22	31.7	27.8	62	62	37.4	36.4
RoomTour3D–IGR	$\checkmark$	$\checkmark$ (E+I)	7.10	7.73	32.0	28.3	64	63	38.0	36.5

TABLE II: RoomTour3D

{}_{\text{Desc}}

and RoomTour3D

{}_{\text{Action}}

stand for description-enriched trajectories only and action-enriched trajectories. Involving description-enriched trajectories and action-enriched trajectories achieves improvement over multiple benchmarks. The inclusion of implicit geometric representation achieves a further performance boost on both Val-U and Test sets. (E) denotes explicit geometric trajectories, and (I) denotes implicit geometry representation.

Object tags	Depth / Box	Room type	CVDN	SOON		R2R		REVERIE
Object tags	Depth / Box	Room type	GP $\uparrow$	SR $\uparrow$	SPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$	SR $\uparrow$	SPL $\uparrow$
$\times$	$\times$	$\times$	6.09	33.6	28.0	66	57	38.3	31.4
✓	$\times$	$\times$	5.41	32.5	26.5	64	56	42.5	34.4
✓	✓	$\times$	6.49	37.6	30.4	68	62	41.7	36.0
✓	✓	✓	6.96	38.8	30.2	69	62	43.3	37.1

TABLE III: Ablation study on the input modalities for the trajectory summarization task.

Methods	SR $\uparrow$	SPL $\uparrow$
Random Walk [ 100 ]	3	2
Commercial Models
NavGPT (GPT-3.5) ⋆ [ 101 ]	13.89	9.12
NavGPT (GPT-4) [ 101 ]	34	29
MapGPT (GPT-4) [ 102 ]	38.8	25.8
MapGPT (GPT-4V) [ 102 ]	43.7	34.8

TABLE IV: Overall comparison with SOTA zero-shot methods on R2R. † denotes training exclusive navigable actions. ⋆ denotes using 36 views setting.

Geometry encoding	CVDN	SOON	R2R	REVERIE
Explicit	6.33	31.7	62	37.4
Implicit	7.07	30.9	64	36.8
Explicit + Implicit	7.38	32.7	66	39.2

TABLE V: Ablation study on geometric encoding. Compared with using only explicit or only implicit encoding, our RoomTour3D–IGR combines both and serves as a better choice. We report SPL for R2R, REVERIE and SOON, GP for CVDN on Val-U.

Methods	CVDN	SOON	R2R	REVERIE
RoomTour3D ${}_{\text{Desc}}$	6.96	30.2	62	37.1
RoomTour3D–IGR	7.10	32.0	64	38.0
$\textbf{RoomTour3D--IGR }^{\dagger}$	7.38	32.7	66	39.2

TABLE VI: Ablation on training data usage. As implicit geometry relaxes reconstruction constraints, more trajectories can be included for geometric training. Consistent performance improvement across CVDN, SOON, R2R, and REVERIE (GP reported for CVDN, SPL for others) can be observed.

\dagger

means involvement of all RoomTour3D data.

Fig. 5: Visual robustness evaluation under common degradations on R2R Val Unseen. The figure illustrates four types of perturbations (Gaussian noise, motion blur, JPEG compression, defocus blur, and brightness) with image examples, together with SPL and SR results. Compared to NaviLLM, our RoomTour3D-trained agent suffers smaller performance drops under all degradations, highlighting improved tolerance in real-world.

Fig. 6: Visualization of navigation paths of NaviLLM [ 24 ] and our RoomTour3D model on R2R-unseen. ① denotes the starting location. The red circle marks an incorrect endpoint, while the green circle indicates the correct one. (a) In a scenario requiring directional judgment, our method makes the correct decision, whereas the baseline is misled by a similar-looking entrance at the waypoint and mistakenly turns right. (b) In an unusual room layout, where two different sinks appear in close proximity, the baseline is distracted by the first incorrect sink it encounters. In contrast, our method follows the instruction “go straight until you pass” more reliably and successfully reaches the correct sink.

Jingyi Zhang is a master student with Shenzhen Campus of Sun Yat-Sen University. His research interests include computer vision and machine learning, with a focus on representation learning and visual understanding.

Table 18

Explore similar work

AgentVLN: Towards Agentic Vision-and-Language Navigation
Mar 18, 2026Zihao Xin, Wentong Li, Yixuan Jiang +6Vision-Language Navigation Robot Navigation
Nanjing University of Aeronautics and Astronautics · Shandong University · Zhejiang University
Bridging the 2D-3D Gap: A Hierarchical Semantic-Geometric Map for Vision Language Navigation
May 25, 2026Kailing Li, Tianwen Qian, Lijin Yang +43D Spatial Reasoning Vision-Language Navigation
School of Computer Science and Technology, East China Normal University · 2Bosch Corporate Research, Shanghai, China · 3King Abdullah University of Science and Technology
StageVLN: Spatial and Trajectory Auxiliary Guidance for Efficient Vision-Language Navigation
Oct 5, 2026Anh Dao, Quan-Dung Pham, Le Danh Vinh +6Vision-Language Navigation
VinMotion, Inc., Vietnam · University of Southern California, USA

Mar 18, 2026J/K move · Enter open · S save

AgentVLN: Towards Agentic Vision-and-Language Navigation

Zihao Xin, Wentong Li, Yixuan Jiang, Ziyuan Huang+5

Nanjing University of Aeronautics and Astronautics · Shandong University · Zhejiang University

Vision-Language Navigation Robot Navigation VLMs for Robotics

Open paper AskPDF arXiv

Vision-and-Language Navigation (VLN) requires an embodied agent to ground complex natural-language instructions into long-horizon navigation in unseen environments. While Vision-Language Models (VLMs) offer strong 2D semantic understanding, current VLN systems remain constrained by limited spatial perception, 2D-3D representation mismatch, and monocular scale ambiguity. In this paper, we propose AgentVLN, a novel and efficient embodied navigation framework that can be deployed on edge computing platforms. We formulate VLN as a Partially Observable Semi-Markov Decision Process (POSMDP) and introduce a VLM-as-Brain paradigm that decouples high-level semantic reasoning from perception and planning via a plug-and-play skill library. To resolve multi-level representation inconsistency, we design a cross-space representation mapping that projects perception-layer 3D topological waypoints into the image plane, yielding pixel-aligned visual prompts for the VLM. Building on this bridge, we integrate a context-aware self-correction and active exploration strategy to recover from occlusions and suppress error accumulation over long trajectories. To further address the spatial ambiguity of instructions in unstructured environments, we propose a Query-Driven Perceptual Chain-of-Thought (QD-PCoT) scheme, enabling the agent with the metacognitive ability to actively seek geometric depth information. Finally, we construct AgentVLN-Instruct, a large-scale instruction-tuning dataset with dynamic stage routing conditioned on target visibility. Extensive experiments show that AgentVLN consistently outperforms prior state-of-the-art methods (SOTA) on long-horizon VLN benchmarks, offering a practical paradigm for lightweight deployment of next-generation embodied navigation models. Code: https://github.com/Allenxinn/AgentVLN.

Abstract

Explore similar work

AgentVLN: Towards Agentic Vision-and-Language Navigation

Bridging the 2D-3D Gap: A Hierarchical Semantic-Geometric Map for Vision Language Navigation

StageVLN: Spatial and Trajectory Auxiliary Guidance for Efficient Vision-Language Navigation