Enhancing Vision-Language Navigation with Multimodal Event Knowledge from Real-World Indoor Tour Videos
Authors: Haoxuan Xu, Tianfu Li, Wenbo Chen, Yi Liu, Xingxing Zuo, Yaoxian Song, Haoang Li
Organizations: The Hong Kong University of Science and Technology (Guangzhou), Guangzhou 511453, China · Tsinghua University, Shenzhen 518055, China · Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi 31122, UAE · Hangzhou City University, Hangzhou 310015, China
Vision-Language Navigation (VLN) agents often struggle with long-horizon reasoning in unseen environments, particularly when facing ambiguous, coarse-grained instructions. While recent advances use knowledge graph to enhance reasoning, the potential of multimodal event knowledge inspired by human episodic memory remains underexplored. In this work, we propose an event-centric knowledge enhancement strategy for automated process knowledge mining and feature fusion to solve coarse-grained instruction and long-horizon reasoning in VLN task. First, we construct YE-KG, the first large-scale multimodal spatiotemporal knowledge graph, with over 86k nodes and 83k edges, derived from real-world indoor videos. By leveraging multimodal large language models (i.e., LLaVa, GPT4), we extract unstructured video streams into structured semantic-action-effect events to serve as explicit episodic memory. Second, we introduce STE-VLN, which integrates the above graph into VLN models via a Coarse-to-Fine Hierarchical Retrieval mechanism. This allows agents to retrieve causal event sequences and dynamically fuse them with egocentric visual observations. Experiments on REVERIE, R2R, and R2R-CE benchmarks demonstrate the efficiency of our event-centric strategy, outperforming state-of-the-art approaches across diverse action spaces. Our data and code are available on the project website https://sites.google.com/view/y-event-kg/.
Figures & tables
Fig. 1: Various VLN modeling paradigms. (a) Conventional VLN methods design a planner based directly on instructions and visual observations [ 7 ] , often lacking a deep understanding of complex indoor environmental knowledge. (b) Previous knowledge-enhanced VLN models introduce entity knowledge embeddings [ 8 ] as priors but fail to effectively associate objects and scenes with dynamic navigation actions. (c) Our method constructs video-based event knowledge for the VLN planner. It empowers the robot with explicit knowledge priors that uniquely capture the spatiotemporal features linking objects, scenes, and navigation actions.
Fig. 2: Overview of YouTube‑Event‑KG (YE-KG) (a) Multimodal Event Knowledge Graph Construction. We first collect large-scale real-world indoor tour videos from YouTube, followed by visual frame extraction and scene segmentation. Then, we employ LLaVA-NeXT-Video together with GPT-4 to generate semantically grounded event descriptions as nodes, connecting them via directed edges representing temporal adjacency (causal navigational links) to yield the YE-KG . (b) Data Statistics. The video duration distribution in the knowledge graph is tightly concentrated between 6 and 13 seconds, highlighting the corpus’s high quality and temporal consistency. (c) YE‑KG Visualization. An example event is visualized together with its associated video features, illustrating how event nodes are grounded in spatiotemporal visual context.
Fig. 3: The overview of STE-VLN framework enhanced by the multimodal event knowledge. (a) The texture instruction and egocentric visual observation are fed into the VLN planner, enriched by external knowledge. The red dashed line frame illustrates the coarse-to-fine retrieval process: {T^event(i)}i=1n and {F^i}i=1n denote the top- n retrieved textual event descriptions and their corresponding video features from the YE-KG, which are fused with the original instruction I and current visual observation ftv , respectively. (b) Knowledge-Guided Transformer is designed to implement ASTFF, where the egocentric observation serves as the Query (Q) and retrieved video knowledge provides the Key (K) and Value (V).
Models
Validation Seen
Validation Unseen
Test Unseen
SR ↑
SPL ↑
RGS ↑
RGSPL ↑
SR ↑
SPL ↑
RGS ↑
RGSPL ↑
SR ↑
SPL ↑
RGS ↑
RGSPL ↑
HAMT [ 11 ]
43.29
40.19
27.2
25.18
32.95
30.2
18.92
17.28
30.40
26.67
14.88
13.08
HOP+ [ 27 ]
55.87
49.55
40.76
36.22
36.07
31.13
22.49
19.33
33.82
28.24
20.20
16.86
DSRG [ 28 ]
75.69
68.09
61.07
54.72
47.83
34.02
32.69
23.37
54.04
37.09
32.49
22.18
GridMM [ 31 ]
–
–
–
–
51.37
36.47
34.57
24.56
53.13
36.60
34.87
23.45
DUET [ 12 ]
71.75
63.94
57.41
51.14
46.98
33.73
32.15
23.03
52.51
36.06
31.88
22.06
TABLE I: Performance comparison on the coarse-grained REVERIE [ 3 ] benchmark. Our STE-VLN is implemented on top of the GOAT [ 35 ] backbone. We report the validation and test metrics compared to state-of-the-art methods.
Models
Validation Seen
Validation Unseen
SR ↑
OSR ↑
SR ↑
OSR ↑
HAMT [ 11 ]
76
82
66
73
DUET [ 12 ]
79
86
72
81
TD-STP [ 29 ]
77
83
70
76
GeoVLN [ 32 ]
79
–
68
–
DSRG [ 28 ]
81
88
73
81
TABLE II: Performance comparison on the fine-grained R2R [ 1 ] benchmark. Our STE-VLN is implemented on top of the GOAT [ 35 ] backbone, demonstrating the method’s generalization capability on fine-grained instructions.
Methods
Validation Seen
Validation Unseen
SR ↑
SPL ↑
NE ↓
OSR ↑
SR ↑
SPL ↑
NE ↓
OSR ↑
Seq2Seq [ 23 ]
37
35
7.12
46
32
30
7.37
40
SASRA [ 39 ]
36
34
7.71
–
24
22
8.32
–
CMTP [ 37 ]
36
31
7.10
56
26
23
7.90
38
LAW [ 36 ]
40
37
–
–
35
31
–
–
HPN [ 38 ]
46
43
5.48
53
36
34
6.31
40
TABLE III: Performance comparison on the continuous environment R2R-CE [ 23 ] benchmark. Our STE-VLN is implemented on top of the ETPNav [ 42 ] backbone, validating its effectiveness in continuous spaces.
TOP- n
EVENT
SCENE
Validation Seen
Validation Unseen
SR
SPL
RGS
RGSPL
SR
SPL
RGS
RGSPL
0
0
0
78.64
71.40
63.74
57.85
53.37
36.70
38.43
26.09
1
1
0
80.15
71.46
64.07
57.81
54.55
36.77
39.08
26.07
0
1
78.14
71.24
63.48
57.90
52.87
36.50
38.04
26.11
2
2
0
80.60
71.43
64.55
57.82
55.10
36.75
40.01
26.07
1
1
80.55
71.27
64.46
57.87
54.98
36.53
39.69
26.10
TABLE IV: Ablation study on different combinations of event and scene knowledge. “TOP- n ” is the number of fine-grained knowledge items. “EVENT” counts descriptions labeled Event-0 , while “SCENE” counts those labeled Scene-1 .
Fig. 4: Visualization of event-level knowledge–guided decision correction in unseen VLN scenarios. By retrieving related event sequences and visual cues via a coarse-to-fine mechanism, our method compensates for missing procedural priors and guides the agent to the target. The light orange arrows represent the trajectories of the GOAT [ 35 ] baseline, which erroneously leads to a bathtub instead of the target sink.
Fig. 5: Real-world deployment. The agent robustly follows natural language instructions in real office settings. (a) “I am thirsty, find the water for me”: navigation to the water dispenser. (b) “Go to the sofa, then find the green box”: localization of the sofa followed by retrieval of the green box. Top rows: first-person views; bottom rows: third-person views, illustrating seamless traversal of diverse functional areas.
Fusion Strategy
Validation Seen
Validation Unseen
Text
Visual
SR ↑
SPL ↑
RGS ↑
RGSP ↑
SR ↑
SPL ↑
RGS ↑
RGSP ↑
✗
✗
78.64
71.40
63.74
57.85
53.37
36.70
38.43
26.09
✓
✗
80.32
71.28
64.47
57.91
54.90
36.54
39.80
26.12
✗
✓
79.34
71.32
63.98
57.81
53.95
36.59
38.82
26.07
✓
✓
80.65
71.20
64.52
57.92
55.33
36.46
39.92
26.12
TABLE V: Ablation study results on different fusion strategies.
Component
Metric
Value
Model Overhead
ASTFF Params
4.73 M
KG Features Storage
487 MB
Inference Latency
Coarse Retrieval (Once)
3.92 ms
Fine Retrieval (Per Step)
0.02 ms
TABLE VI: Efficiency Analysis. The computational cost introduced by STE-VLN is negligible compared to the backbone model. Latency is measured on a single NVIDIA 4090 GPU.