What Should a Navigator Remember? Spatiotemporal Memory for Zero-Shot Vision-Language Navigation

Yuhang Peng; Zhikai Wang; Ziyi Wang; Jiaru Zhong; Xiaoji Zheng; Jingzhi Huang; Yi Wang; Jiangtao Gong

cs.ROOct 28, 2026

What Should a Navigator Remember? Spatiotemporal Memory for Zero-Shot Vision-Language Navigation

Authors: Yuhang Peng, Zhikai Wang, Ziyi Wang, Jiaru Zhong, Xiaoji Zheng, Jingzhi Huang, Yi Wang, Jiangtao Gong

Organizations: The Hong Kong Polytechnic University · Tsinghua University · Shanghai Jiao Tong University

Abstract

Zero-shot vision-and-language navigation (VLN) requires an agent to use past observations without navigation-specific training. We study what information an 8B-scale navigation system should retain and how it should be organized for subsequent decisions. Our agent combines instruction progress, hierarchically compressed observation history anchored to poses, and accumulated geometry in complementary task, temporal, and spatial memories. An 8B observer summarizes observations, and an 8B reasoner uses the memories to select waypoint intents or request additional views, a geometric controller executes the selected targets. A semantic information perspective motivates the distinction between retaining evidence, making it accessible to the reasoner, and acquiring further observations. On R2R-CE val-unseen, the complete system achieves the highest among the evaluated Qwen3-based zero-shot baselines. Removing temporal memory reduces SR and sharply increases average path length from 16.9 to 39.1m, removing the spatial-map and planning component reduces SR as well. Real-world experiments on a Unitree Go2 show that our zero-shot method achieves higher success rates than the post-trained baseline in both corridor and office environments, suggesting better transfer to the evaluated real-world settings. As a complementary evaluation, LAW2 uses paired instruction interventions to examine behavioral responses, instruction following ability beyond endpoint success. These results support preserving task progress, observation history, and geometry in structured memory for zero-shot navigation with smaller open models.

Figures & tables

Figure 1: Spatiotemporal memory for zero-shot VLN. Task memory tracks instruction progress, temporal memory retains pose-anchored observations and summaries, and spatial memory accumulates geometry and the committed target. The reasoner uses these representations to select a waypoint or request additional views through PREVIEW , geometric control executes the waypoint and subsequent observations update memory.

Figure 2: Qualitative results on R2R (top) and in real-world experiments (bottom).

Method	Navigation Backbone	Obs.	SR $\uparrow$	SPL $\uparrow$	nDTW $\uparrow$	OSR $\uparrow$	NE $\downarrow$	PL	Steps	RSR $\uparrow$
Post-training
Aware VLN c Guo et al. (2026)	AwareVLN	Mono	0.647	0.563	0.656	0.744	4.10	12.79	107	0.869
Aware VLN c Guo et al. (2026)	AwareVLN (4-bit)	Mono	0.652	0.569	0.660	0.760	3.89	12.69	110	0.858
InternNav c Wei et al. (2026)	InternVLA	Mono	0.585	0.538	–	0.659	4.70	–	99	0.888
LightNav c Wang et al. (2026)	LightNav	Mono	0.680	0.625	0.633	0.722	3.99	–	84	0.942
Zero-shot

Table 1: Results on R2R-CE val-unseen. Bold: best within the post-training group, and best among Qwen3(-VL)-8B-based methods in the zero-shot group.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Navigation Performance
Model Variant	Full	w/o Spatial	w/o Temporal	w/o S&T	w/o Preview
SR $\uparrow$	0.263	0.205	0.190	0.185	0.190
SPL $\uparrow$	0.196	0.142	0.112	0.139	0.151
OSR $\uparrow$	0.328	0.345	0.475	0.492	0.295
NE $\downarrow$	7.56	8.48	9.31	8.46	8.05
Path & Trajectory Efficiency

Appendix

Table 2: Ablation study evaluation of model variants across navigation performance, path efficiency, step budget, and average token consumption per episode.

	VLM Calls	Per Call / Step		Per Episode
Method	/ Episode	Input	Output	Input	Output	Total
HSGM	11.7	$\approx$ 23.0k	285	$\approx$ 270k	$\approx$ 3.3k	$\approx$ 273.3k
Ours	131	$\approx$ 6.0k	$\approx$ 114	864k	16.3k	880.3k
– Qwen3-VL-8B	131	$\approx$ 4.9k	$\approx$ 83	640k	10.9k	651.9k
– JoyAI	131	$\approx$ 1.7k	$\approx$ 41	224k	5.4k	229.4k

Appendix

Table 3: Comparison of VLM call frequency and token consumption between HSGM and Ours, reported per call/step and per episode.

	Corridor				Office
	Simple Instruction		Complex Instruction		Simple Instruction		Complex Instruction
Method	NE $\downarrow$	SR $\uparrow$	NE $\downarrow$	SR $\uparrow$	NE $\downarrow$	SR $\uparrow$	NE $\downarrow$	SR $\uparrow$
AwareVLN	2.85	0.3	3.20	0.1	3.10	0.2	3.46	0.0
Ours	0.68	0.8	0.84	0.8	0.91	0.6	1.07	0.4

Appendix

Table 4: Real-world evaluation in the corridor and the office. Simple Instruction : follow a short instruction and stop. Complex Instruction : a multi-step instruction that requires a turn or passing a doorway before reaching the target. NE (m) is the navigation error; SR is the success rate within

2.0

Method	Dataset	Follow $\uparrow$	Opposite $\downarrow$	Straight	SR $\uparrow$	MeanFollow $\uparrow$	WordEffect $\uparrow$
Ours	ORIG	0.829	0.134	0.037	0.286	0.7680	0.2955
Ours	FLIP	0.707	0.220	0.073	0.143	0.7680	0.2955
AwareVLN	ORIG	0.915	0.073	0.012	0.646	0.6770	0.1830
AwareVLN	FLIP	0.439	0.549	0.012	0.463	0.6770	0.1830
CA-Nav	ORIG	0.646	0.268	0.085	0.256	0.4570	-0.0063
CA-Nav	FLIP	0.268	0.671	0.061	0.171	0.4570	-0.0063

Appendix

Table 5: Initial-turn obedience on 82 selected episodes of R2R Val Unseen. ORIG uses the original instructions, while FLIP reverses only the selected direction word. Both conditions use the same v1-3 start orientation. Follow, Opposite, and Straight are fractions of all evaluated episodes; Straight includes straight or insufficient departures. MeanFollow and WordEffect aggregate the paired conditions. SR measures success at the original goal and is auxiliary. Aggregate values are approximately computed from the rounded rates shown.

Method	ORIG			GOAL-ONLY
Method	SR $\uparrow$	nDTW $\uparrow$	PL (m)	SR $\uparrow$	nDTW $\uparrow$	PL (m)
Ours	0.293	0.354	19.37	0.181	0.390	11.25
AwareVLN	0.624	0.653	13.21	0.353	0.481	11.89
CA-Nav	0.241	0.162	28.34	0.184	0.168	25.75

Appendix

Table 6: GOAL-ONLY experiment on 174 episodes of R2R Val Unseen, retaining only the final annotated sub-instruction. SR is evaluated against the unchanged target location, nDTW uses the original reference trajectory.

Method	Instruction	SR $\uparrow$	OSR $\uparrow$	RSR $\uparrow$	SPL $\uparrow$	NE $\downarrow$	nDTW $\uparrow$
Ours	ORIG	0.270	0.350	0.771	0.155	8.15	0.387
	A1	0.260	0.340	0.765	0.149	7.98	0.357
	A2	0.240	0.330	0.727	0.123	8.19	0.365
	A3	0.260	0.340	0.765	0.136	8.06	0.350
	A4	0.280	0.360	0.778	0.145	8.67	0.351
AwareVLN	ORIG	0.590	0.760	0.776	0.499	4.11	0.621

Appendix

Table 7: PARAPHRASE experiment on 200 episodes of R2R Val Unseen. A1: same register; A2: terse; A3: colloquial; A4: landmark re-description. NE is in meters. CA-Nav uses Qwen for instruction parsing.

Figure 3: Qualitative results for the reconstructed office scene: (a) reconstructed point cloud; (b) image rendered using 3D Gaussian Splatting; and (c) original image.

Transformation	Episodes	Instruction change
FLIP	82	Reverse the initial turn direction
GOAL-ONLY	174	Retain the final annotated segment
PARAPHRASE A1	200	Same-register rewriting
PARAPHRASE A2	200	Terse imperatives
PARAPHRASE A3	200	Colloquial rewriting
PARAPHRASE A4	200	Landmark re-description

Appendix

Table 8: law2 cohorts. The four paraphrase styles share 200 source episodes and yield 800 rewrites; every condition has a matched ORIG control.

Vision-Language Navigation

Vision-Language Navigation studies how embodied agents use natural-language instructions and visual observations to plan and execute routes.

See all 313 papers

SpaceVLN: A Zero-Shot Vision-and-Language Navigation Agent with Online Spatial Cognitive Memory and Reasoning
Jun 8, 2026Yucheng Deng, Pingrui Lai, Xinhai Li +5Visual Spatial Reasoning Embodied Navigation
School of Information Science and Electronic Engineering & School of Integrated Circuits, Shanghai Jiao Tong University · Institute of Artificial Intelligence, China Telecom · 3Central South University +1
Structure over Reasoning: Unified Zero-Shot Navigation with a Small Vision-Language Model
Oct 18, 2026Zeqi Li, Xunyi Zhao, Zerui Li +3Visual Robot Navigation Vision-Language Navigation
Technical University of Munich, Germany. · Adelaide University, Responsible AI Research Centre, Australian Institute for Machine Learning, Australia. · Institute for Infocomm Research (I2R), A*STAR, Singapore.
MemoryVLN: From Ego-Aware Memory Representation to Cross-Context Memory Reuse in Vision-Language Navigation
Oct 13, 2026Xueyao Wei, Yifei Wang, Delun Li +4Vision-Language Navigation Memory-Augmented VLMs
Beijing Institute of Technology · Zhongguancun Academy · Peking University

Jun 8, 2026J/K move · Enter open · S save

SpaceVLN: A Zero-Shot Vision-and-Language Navigation Agent with Online Spatial Cognitive Memory and Reasoning

Yucheng Deng, Pingrui Lai, Xinhai Li, Chenjia Bai+4

School of Information Science and Electronic Engineering & School of Integrated Circuits, Shanghai Jiao Tong University · Institute of Artificial Intelligence, China Telecom · 3Central South University+1

Visual Spatial Reasoning Embodied Navigation Vision-Language Navigation Object-Goal Navigation Memory-Augmented Agents

Open paper AskPDF arXiv

Vision-and-Language Navigation in continuous environments requires agents to understand the spatial structure of previously unseen environments in order to follow language instructions. Although foundation models have opened a promising path toward zero-shot navigation without task-specific policy training, many navigators still rely on local visual cues and linear history-based reasoning, overlooking the spatial nature of navigation across explored regions, traversed paths, landmarks, and their spatial relations. In this paper, we propose SpaceVLN, a navigation agent built around Spatial Cognitive Memory and Task-Guided Spatial Reasoning. Specifically, SpaceVLN introduces an efficient stagewise closed-loop framework where planning and execution are organized around verifiable space--landmark stages. During navigation, the agent progressively abstracts explored regions into Spatial Waypoints and dynamically maintains subtask-grounded landmark evidence, forming a hierarchical Spatial Cognitive Memory for progress localization and spatial-relation understanding. Built on this memory, Spatial-CoT integrates task-progress reasoning with spatial perception, analysis, and prediction, enabling Task-Guided Spatial Reasoning for embodied navigation. The unified stage interface enables SpaceVLN to address both Vision-and-Language Navigation and Object-Goal Navigation under a unified zero-shot setting, without task-specific policy training. Across R2R-CE, RxR-CE, GN-Bench, and HM3D-OVON, SpaceVLN achieves state-of-the-art zero-shot performance, and real-robot deployment further validates its applicability. These results highlight Spatial Cognitive Memory and Task-Guided Spatial Reasoning as a practical foundation for stronger embodied navigation agents.

Abstract

Vision-Language Navigation

Explore similar work

SpaceVLN: A Zero-Shot Vision-and-Language Navigation Agent with Online Spatial Cognitive Memory and Reasoning

Structure over Reasoning: Unified Zero-Shot Navigation with a Small Vision-Language Model

MemoryVLN: From Ego-Aware Memory Representation to Cross-Context Memory Reuse in Vision-Language Navigation