Organizations: State Key Laboratory of Novel Software Technology, Nanjing University · School of Intelligence Science and Technology, Nanjing University · School of Artificial Intelligence, Nanjing University
Multimodal lifelong navigation requires an agent to autonomously explore unseen environments while sequentially completing navigation tasks specified by object categories, language descriptions, or reference images. Existing methods primarily accomplish these tasks by constructing state-centric semantic scene graphs. By treating scene graphs as persistent repositories of semantic observations, these methods struggle to distinguish similar instances, jointly represent semantic targets and exploration frontiers, and effectively exploit navigation memory and trajectory experience. To address these limitations, we propose Spatio-Temporal Event Graph Navigation (STEGNav), a training-free framework that extends conventional scene graphs into spatio-temporal event graphs along complementary spatial and temporal axes. The spatial axis performs query-conditioned instance grounding and jointly represents semantic targets and occupancy-aware exploration frontiers characterized by reachability, path cost, and exploration utility. The temporal axis employs trajectory-aware dual-window memory to retain recent decision--trajectory events and verified cross-subtask navigation outcomes. A VLM-based navigation agent reasons over the resulting spatio-temporal event graph and selects either a target instance or an exploration frontier as its next navigation goal. STEGNav achieves 66.3% SR and 39.7 SPL on GOAT-Bench, as well as SR scores of 64.0% and 69.4% on HM3Dv1 and HM3Dv2, respectively. Ablation studies and error analyses validate the complementary effects of the two axes, demonstrating that event-driven spatio-temporal representations improve navigation reliability and cross-subtask experience reuse.
Figures & tables
Figure 1: Limitations of state-centric scene graphs. Spatially, they suffer from instance ambiguity and semantic–spatial separation; temporally, the lack of explicit navigation history leads to memory loss and repeated exploration.
Figure 2: Overview of STEGNav. The spatial axis unifies query-conditioned instances with occupancy-aware frontiers, while the temporal axis organizes recent trajectories and verified cross-subtask outcomes. Their integration forms a spatio-temporal event graph for VLM-based navigation, verification, and continual update.
Method
Training-free
SR ↑
SPL ↑
SenseAct-NN Monolithic [CVPR24]
✗
12.3
6.8
SenseAct-NN Skill Chain [CVPR24]
✗
29.5
11.3
MTU3D [ICCV25]
✗
47.2
27.7
Modular CLIP on Wheels [CVPR24]
✓
16.1
10.4
Modular GOAT [CVPR24]
✓
24.9
17.2
VLMNav [NeSy25]
✓
20.1
9.6
Table 1: Main results on the GOAT-Bench Val-Unseen split. MSGNav does not report SPL under our evaluation protocol.
Method
Training-Free
HM3Dv1
HM3Dv2
SR ↑
SPL ↑
SR ↑
SPL ↑
ZSON [NeurIPS22]
✗
25.5
12.6
–
–
PixNav [ICRA24]
✗
37.9
20.5
–
–
SPNet [RA-L23]
✗
31.2
10.1
–
–
CompassNav [ICLR26]
✗
56.6
27.6
59.6
26.9
SGM [CVPR24]
✗
60.2
30.8
–
–
Table 2: Object navigation results on the HM3D benchmarks.
Module
Object
Language
Image
Overall
3D Scene Graph
QIG
OFG
DWM
SR
SPL
SR
SPL
SR
SPL
SR
SPL
✗
✗
✗
✗
50.5
24.0
41.8
22.5
44.3
27.0
45.7
24.5
✓
✗
✗
✗
61.6
30.5
51.6
32.0
54.5
38.0
55.8
33.4
✓
✓
✗
✗
64.1
32.0
55.6
35.3
58.0
41.7
59.4
36.2
✓
✓
✓
✗
70.7
37.8
61.5
38.6
68.2
47.5
66.9
41.1
✓
✓
✓
✓
74.8
40.8
63.7
38.7
73.9
49.1
70.9
42.7
Table 3: Component ablation results on the GOAT-Bench Val-Unseen subset. QIG denotes query-conditioned instance grounding, OFG denotes occupancy-aware frontier grounding, and DWM denotes trajectory-aware dual-window memory. The best result in each column is highlighted in bold .
Figure 3: Failure counts of the 3D scene-graph baseline and STEGNav on the GOAT-Bench Val-Unseen subset (278).
Figure 4: Successful language-guided navigation. Given the query “refrigerator next to the kitchen cabinet” , STEGNav progressively explores informative regions and grounds the target instance using semantic and spatial cues. The trajectory achieves an SPL of 0.587 with only 4 navigation steps, reaching the target within 0.40 m.
Figure 5: Successful object navigation. Given the query “Can you find the piano?” , STEGNav selects informative frontiers, discovers the target region, and successfully reaches the target. The trajectory achieves an SPL of 0.913 with 6 navigation steps, reaching the target within 0.97 m.
Vision-and-Language Navigation (VLN) requires agents to reason over accumulated observations while continuously exploring unseen regions. However, existing environment representations often struggle to jointly support explicit semantic memory and continuous exploration guidance. To address this challenge, we propose Cognitive Graph-Field Memory (CGFM), a persistent multimodal scene representation that couples explicit relational memory with continuous spatial intuition. CGFM organizes objects, spatial relations, and visual observations into a multimodal scene graph, enabling target retrieval and long-horizon reasoning across navigation tasks. When no reliable target match is identified, graph-based evidence is projected into a goal-conditioned semantic-frontier field to guide exploration toward semantically promising frontiers and regions. Building upon CGFM, we introduce CGFM-Nav, a foundation-model-based framework for lifelong multimodal navigation that integrates task-relevant subgraph selection, VLM reasoning, and verification feedback into a closed decision loop. Preliminary experiments on GOAT-Bench show that, under the same Qwen3-VL-8B backbone, CGFM-Nav improves the overall success rate from 53.2% to 63.0% and SPL from 30.0% to 39.6%, demonstrating the effectiveness of combining explicit semantic memory with semantic-guided exploration.
Yuxiang Xiao, Xibei Chen, Xin Zhou +3
Department of Mechanical Engineering, National University of Singapore, Singapore
Service robots operating for months in the same homes, offices, and facilities should become more reliable with experience instead of searching familiar space from scratch for every request. Yet ObjectNav is predominantly formulated as one-shot exploration, leaving a central deployment challenge unresolved: recognizing an object does not identify a reachable place to stop, and one confident map error can terminate the task. We introduce SSTG-Nav, a reusable metric-semantic memory that turns a one-time survey into actionable object goals, consolidates evidence across viewpoints, and retains spatially distinct recovery standoffs. On 1,000 HM3D-v2 episodes across 36 scenes, our goal-independent topology achieves a 99.4% geometric success ceiling. Holding semantic responses fixed, metric grounding raises SR/SPL from 0.835/0.560 to 0.920/0.603, and source-aware fusion reaches 0.926/0.586. Fusion-aware Top-3 recovery raises Success@1/2/3 to 0.928/0.965/0.975 and reaches 0.601 SPL@3. Model, field-of-view, density, and corruption controls identify where these gains originate, and a ROS2/Nav2 realization demonstrates the complete reusable query-to-execution pipeline. Together, the results establish pre-exploration as a powerful practical regime for dependable, repeated semantic navigation.
Daojie Peng, Bingtao Wang, Jun Ma
The Hong Kong University of Science and Technology (Guangzhou) · Shandong University
Vision-language navigation requires embodied agents to navigate environments using natural language instructions and visual observations. Existing approaches typically decompose navigation into sequential language-guided decisions or rely on online exploration without prior environmental knowledge. Scene graph representations offer compact semantic memory but remain decoupled from downstream navigation, which still depends on dense metric maps. To close this gap, we propose an object--path graph that unifies open-vocabulary semantic reasoning with topological navigation. The proposed representation jointly supports semantic grounding, graph-based localization, and navigation within a single lightweight topological framework. Building on this graph, we introduce a navigation strategy that combines global path planning with local inter-node execution through lightweight node localization and semantic visual servoing, enabling navigation directly over the graph without dense metric reconstruction. Experiments on HM3D and Replica demonstrate competitive performance in open-vocabulary object grounding through the proposed hierarchical graph structure, while achieving effective navigation performance. Real-world robot experiments further validate the practicality of the proposed framework.
Linwei Zheng, Daojie Peng, Bingtao Wang +2
The Hong Kong University of Science and Technology (Guangzhou) · The Hong Kong University of Science and Technology