What Should a Navigator Remember? Spatiotemporal Memory for Zero-Shot Vision-Language Navigation
Organizations: The Hong Kong Polytechnic University · Tsinghua University · Shanghai Jiao Tong University
Abstract
Zero-shot vision-and-language navigation (VLN) requires an agent to use past observations without navigation-specific training. We study what information an 8B-scale navigation system should retain and how it should be organized for subsequent decisions. Our agent combines instruction progress, hierarchically compressed observation history anchored to poses, and accumulated geometry in complementary task, temporal, and spatial memories. An 8B observer summarizes observations, and an 8B reasoner uses the memories to select waypoint intents or request additional views, a geometric controller executes the selected targets. A semantic information perspective motivates the distinction between retaining evidence, making it accessible to the reasoner, and acquiring further observations. On R2R-CE val-unseen, the complete system achieves the highest among the evaluated Qwen3-based zero-shot baselines. Removing temporal memory reduces SR and sharply increases average path length from 16.9 to 39.1m, removing the spatial-map and planning component reduces SR as well. Real-world experiments on a Unitree Go2 show that our zero-shot method achieves higher success rates than the post-trained baseline in both corridor and office environments, suggesting better transfer to the evaluated real-world settings. As a complementary evaluation, LAW2 uses paired instruction interventions to examine behavioral responses, instruction following ability beyond endpoint success. These results support preserving task progress, observation history, and geometry in structured memory for zero-shot navigation with smaller open models.
Figures & tables
| Method | Navigation Backbone | Obs. | SR | SPL | nDTW | OSR | NE | PL | Steps | RSR |
| Post-training | ||||||||||
| Aware VLN c Guo et al. (2026) | AwareVLN | Mono | 0.647 | 0.563 | 0.656 | 0.744 | 4.10 | 12.79 | 107 | 0.869 |
| Aware VLN c Guo et al. (2026) | AwareVLN (4-bit) | Mono | 0.652 | 0.569 | 0.660 | 0.760 | 3.89 | 12.69 | 110 | 0.858 |
| InternNav c Wei et al. (2026) | InternVLA | Mono | 0.585 | 0.538 | – | 0.659 | 4.70 | – | 99 | 0.888 |
| LightNav c Wang et al. (2026) | LightNav | Mono | 0.680 | 0.625 | 0.633 | 0.722 | 3.99 | – | 84 | 0.942 |
| Zero-shot | ||||||||||
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Model Variant | Full | w/o Spatial | w/o Temporal | w/o S&T | w/o Preview |
|---|---|---|---|---|---|
| Navigation Performance | |||||
| SR | 0.263 | 0.205 | 0.190 | 0.185 | 0.190 |
| SPL | 0.196 | 0.142 | 0.112 | 0.139 | 0.151 |
| OSR | 0.328 | 0.345 | 0.475 | 0.492 | 0.295 |
| NE | 7.56 | 8.48 | 9.31 | 8.46 | 8.05 |
| Path & Trajectory Efficiency | |||||
| VLM Calls | Per Call / Step | Per Episode | ||||
|---|---|---|---|---|---|---|
| Method | / Episode | Input | Output | Input | Output | Total |
| HSGM | 11.7 | 23.0k | 285 | 270k | 3.3k | 273.3k |
| Ours | 131 | 6.0k | 114 | 864k | 16.3k | 880.3k |
| – Qwen3-VL-8B | 131 | 4.9k | 83 | 640k | 10.9k | 651.9k |
| – JoyAI | 131 | 1.7k | 41 | 224k | 5.4k | 229.4k |
| Corridor | Office | |||||||
| Simple Instruction | Complex Instruction | Simple Instruction | Complex Instruction | |||||
| Method | NE | SR | NE | SR | NE | SR | NE | SR |
| AwareVLN | 2.85 | 0.3 | 3.20 | 0.1 | 3.10 | 0.2 | 3.46 | 0.0 |
| Ours | 0.68 | 0.8 | 0.84 | 0.8 | 0.91 | 0.6 | 1.07 | 0.4 |
| Method | Dataset | Follow | Opposite | Straight | SR | MeanFollow | WordEffect |
|---|---|---|---|---|---|---|---|
| Ours | ORIG | 0.829 | 0.134 | 0.037 | 0.286 | 0.7680 | 0.2955 |
| FLIP | 0.707 | 0.220 | 0.073 | 0.143 | |||
| AwareVLN | ORIG | 0.915 | 0.073 | 0.012 | 0.646 | 0.6770 | 0.1830 |
| FLIP | 0.439 | 0.549 | 0.012 | 0.463 | |||
| CA-Nav | ORIG | 0.646 | 0.268 | 0.085 | 0.256 | 0.4570 | -0.0063 |
| FLIP | 0.268 | 0.671 | 0.061 | 0.171 |
| Method | ORIG | GOAL-ONLY | ||||
|---|---|---|---|---|---|---|
| SR | nDTW | PL (m) | SR | nDTW | PL (m) | |
| Ours | 0.293 | 0.354 | 19.37 | 0.181 | 0.390 | 11.25 |
| AwareVLN | 0.624 | 0.653 | 13.21 | 0.353 | 0.481 | 11.89 |
| CA-Nav | 0.241 | 0.162 | 28.34 | 0.184 | 0.168 | 25.75 |
| Method | Instruction | SR | OSR | RSR | SPL | NE | nDTW |
|---|---|---|---|---|---|---|---|
| Ours | ORIG | 0.270 | 0.350 | 0.771 | 0.155 | 8.15 | 0.387 |
| A1 | 0.260 | 0.340 | 0.765 | 0.149 | 7.98 | 0.357 | |
| A2 | 0.240 | 0.330 | 0.727 | 0.123 | 8.19 | 0.365 | |
| A3 | 0.260 | 0.340 | 0.765 | 0.136 | 8.06 | 0.350 | |
| A4 | 0.280 | 0.360 | 0.778 | 0.145 | 8.67 | 0.351 | |
| AwareVLN | ORIG | 0.590 | 0.760 | 0.776 | 0.499 | 4.11 | 0.621 |
| Transformation | Episodes | Instruction change |
|---|---|---|
| FLIP | 82 | Reverse the initial turn direction |
| GOAL-ONLY | 174 | Retain the final annotated segment |
| PARAPHRASE A1 | 200 | Same-register rewriting |
| PARAPHRASE A2 | 200 | Terse imperatives |
| PARAPHRASE A3 | 200 | Colloquial rewriting |
| PARAPHRASE A4 | 200 | Landmark re-description |
Vision-Language Navigation
Vision-Language Navigation studies how embodied agents use natural-language instructions and visual observations to plan and execute routes.
See all 313 papers