cs.ROOct 13, 2026

MemoryVLN: From Ego-Aware Memory Representation to Cross-Context Memory Reuse in Vision-Language Navigation

Authors: Xueyao Wei, Yifei Wang, Delun Li, Shaoan Wang, Junzhi Yu, Wenjie Song, Xingyu Chen

Organizations: Beijing Institute of Technology · Zhongguancun Academy · Peking University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large vision-language models (LVLMs) have shown strong potential for embodied vision-language navigation (VLN), yet their performance degrades on long-horizon trajectories due to context-length limits, redundant historical observations, and unstable spatial alignment across time. These issues make it difficult to both compress long navigation histories and reliably reuse past spatial experience. To address these limitations, we propose MemoryVLN, a unified memory-driven navigation system built around memory construction and management. During memory construction, Long Sequence Compression employs forgetting-curve-inspired temporal sampling to preserve dense recent evidence and sparse yet representative distant observations under a limited token budget. Ego-Aware Memory Representation further integrates 2D semantic tokens, 3D geometric tokens, and 6D agent poses, allowing each memory to encode both what was observed and where it was observed. During memory management, Hierarchical Memory Bank maintains in-episode and context-level memories through cross-context warm-start and confidence-controlled write-back. Memory Enhancement then retrieves the Top-KK relevant memories according to the current navigation state and fuses them with the current visual tokens. Together, these mechanisms transform raw navigation history into compact, spatially grounded, selectively retrievable, and reusable navigation experience. Extensive experiments on R2R-CE and RxR-CE demonstrate consistent improvements over the state-of-the-art baselines by a substantial margin (i.e., 2% and 6.5% in successful rate).

Figures & tables

Figure 2: Detailed pipeline of MemoryVLN. (a) Long Sequence Compression performs inter-chunk budget allocation and intra-chunk key-frame selection. (b) Ego-Aware Visuospatial Representation integrates ViT-based 2D tokens, VGGT-based 3D tokens, and 6D agent poses. (c) Memory Bank maintains context-level and in-episode memories through warm start, write-back, and short-/long-term memory updates. (d) Memory Enhancement retrieves Top- KK relevant memories and residually fuse them with the current-frame tokens for action prediction.
MethodObservationR2R Val-UnseenRxR Val-Unseen
Pano.Odo.DepthS.RGBNE ↓\downarrowOS ↑\uparrowSR ↑\uparrowSPL ↑\uparrowNE ↓\downarrowSR ↑\uparrowSPL ↑\uparrownDTW ↑\uparrow
HPN+DN ∗ [ 24 ]✓\checkmark✓\checkmark✓\checkmark–6.3140.036.034.0––––
CMA ∗ [ 27 ]✓\checkmark✓\checkmark✓\checkmark–6.2052.041.036.08.7626.522.147.0
Sim2Sim ∗ [ 26 ]✓\checkmark✓\checkmark✓\checkmark–6.0752.043.036.0––––
VLN-BERT ∗ [ 13 ]✓\checkmark✓\checkmark✓\checkmark–5.7453.044.039.08.9827.022.646.7
Ego 2 -Map ∗ [ 41 ]✓\checkmark✓\checkmark✓\checkmark–5.5456.047.041.0––––
Table 1: Comparison with SOTA methods on R2R-CE and RxR-CE val-unseen split. ∗ denotes methods using a waypoint predictor, while †\dagger denotes models trained only with VLN-CE expert trajectories without additional navigation data. Pano, Odo, Depth, and S.RGB respectively represent panoramic view, odometry, depth, and single RGB.
 R2R-CERxR-CE
VariantNE ↓\downarrowSR ↑\uparrowSPL ↑\uparrowNE ↓\downarrowSR ↑\uparrowSPL ↑\uparrow
w/o Long-Seq. Comp.OOMOOM
w/o Pos-Encoding3.6652.7549.253.3557.2448.74
w/o In-Episode3.5753.3449.063.3056.6948.13
w/o Context3.5554.1649.653.2957.3749.09
Full MemoryVLN3.5254.7650.263.3357.8450.42
Table 2: Component ablations on the R2R-CE and RxR-CE val-unseen splits.
Figure 3: Qualitative evaluation in Habitat simulation and a real indoor environment. Top: JanusVLN gradually deviates from the instructed route and fails, whereas MemoryVLN preserves historical and spatial context and reaches the target. Bottom: MemoryVLN successfully executes two real-world instructions with different routes and target locations.
Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

ComponentValueRole
Action backboneQwen2.5-VLLanguage-conditioned action prediction.
Geometry backbonefrozen VGGTProduces geometric tokens; no VGGT parameters are updated.
Training / evaluation seed42Training and evaluation use this seed; evaluation seeds Python, NumPy, PyTorch CPU, and all CUDA devices.
Reported training runs1All reported results come from one training run with seed 42; no across-run variation is reported.
Training global batch646464 GPUs ×\times per-device batch size 11 ×\times gradient accumulation 11 .
Compute platform64 ×\times A100 80GB8 ml.hpcpni2.28xlarge instances; each has 112 vCPUs, 1960 GiB host memory, and 2.5.1-cu12.4.1-py3.9-ubuntu22.04 .
Appendix
Table 3: Implementation values used for the reported results. The submitted config.json records the model and memory settings; an evaluation command may explicitly override these values.
Figure 4: Selected R2R val-unseen comparisons for SR and SPL as functions of training data. The dashed line marks 46.4% training data, the last data-matched comparison point; the reported-final point to its right is marked with an asterisk because it is not training-data matched. Annotations report MemoryVLN minus JanusVLN in percentage points.
DatasetMetricNNJanusVLNMemoryVLNΔ\Delta (pp)95% CITestpp
R2RSR1,83952.854.8+2.0[-0.7, 4.6]exact McNemar0.158
R2RSPL1,83949.050.3+1.2[-1.3, 3.7]Wilcoxon signed-rank0.327
RxRSR3,66951.057.2+6.2[4.5, 8.0]exact McNemar2.71×10−122.71\times 10^{-12}
RxRSPL3,57043.548.7+5.2[3.8, 6.7]Wilcoxon signed-rank1.27×10−121.27\times 10^{-12}
Appendix
Table 4: Paired significance tests on the same evaluation episodes. Differences are MemoryVLN minus JanusVLN in percentage points; bootstrap intervals quantify episode-level uncertainty and do not represent across-run variability. For RxR SPL, 99 pairs with non-finite values are excluded, leaving 3,570 valid episode pairs.
Figure 5: Representative MemoryVLN failure trajectories. From top to bottom, the rows illustrate room-level branch confusion, ordinal-sensitive final-doorway grounding, a landmark-turn error not corrected by context memory, and long-instruction progress loss. Each row pairs egocentric observations with a top-down map showing the goal, shortest path, and executed agent trajectory.
DatasetScene / EpisodeStepsNEOracle NEColl.Failure type
R2RQUCTc6BB5sX / 5777828.6114.130Semantic branch confusion
R2Rx8F5xyUWy9e / 2316010.766.083Final-doorway grounding
R2REU6Fwq7SyZv / 12065610.285.820Failed-context reuse
RxR2azQ1b91cZZ / 56914416.4012.493Progress loss / early stop
Appendix
Table 5: Four representative failure cases in the same top-to-bottom order as Fig. 5 . All four trajectories have zero Success, SPL, and Oracle Success.
Figure 6: Qualitative comparison between JanusVLN and MemoryVLN on four Habitat episodes under identical instructions. Each example combines representative egocentric observations with top-down trajectories, where the yellow star marks the goal, the green curve marks the shortest path, and the blue curve marks the executed agent trajectory. JanusVLN fails in these examples, whereas MemoryVLN reaches the target region and stops successfully.
Figure 7: Qualitative real-world navigation results for five natural-language instructions using a Unitree GO2 quadruped in an indoor office environment. Navigation-policy inference is performed on a single NVIDIA GeForce RTX 4090 GPU. The egocentric sequences show the robot following corridor and doorway transitions, passing the specified landmarks, and stopping at the requested destination. The targets include a printer, a coffee machine, a designated room, a water dispenser, and a conference-room podium.
More in this topic

Vision-Language Navigation

Vision-Language Navigation studies how embodied agents use natural-language instructions and visual observations to plan and execute routes.

See all 296 papers

Explore similar work

CardsList
Jul 26, 2026

MemVLN: Episodic and Procedural Memory for Vision-and-Language Navigation

Vision-and-Language Navigation in Continuous Environments (VLN-CE) requires agents to maintain long-horizon visual history for trajectory consistency while executing actions with low latency. Existing video-based VLN approaches typically struggle to satisfy both demands simultaneously. To address these challenges, we propose MemVLN, a novel VLN framework that achieves state-of-the-art performance with real-time inference efficiency (14 FPS). MemVLN utilizes a visual encoder to process continuous observations and a Large Language Model (LLM) to interpret instructions and generate actions. Central to our approach is an Episodic Memory management that applies pyramidal resolutions. This mechanism concentrates computation on immediate percepts while retaining compressed long-term history. Complementing to this design, we introduce Procedural Memory for fast action with a compact vocabulary of atomic mid-level actions to bypass auto-regressive decoding latency. Experiments on VLN-CE show that MemVLN-4B surpasses the baseline Qwen3-VL-4B architecture by 5.8% SR in R2R and 9.7% SR in RxR, while achieving a 7×\times speedup in inference latency.

Yuqi Liu, Shengju Qian, Tianyuan Qu +5

1The Chinese University of Hong Kong · 2LIGHTSPEED · 3The University of Hong Kong +1

Memory-Augmented VLMsEfficient VLM InferenceEfficient VLA ModelsVision-Language Navigation
AskPDF
Oct 19, 2026

ReInsVLN: Revisiting Historical Evidence for Training-Free Vision-and-Language Navigation

Vision-and-language navigation (VLN) relies on retaining and using relevant observations. Yet growing histories introduce redundancy, and retained information may remain underused. We introduce ReInsBench, a diagnostic benchmark that examines history utilization through controlled continuation trials. Its results show that history-review reinstruction can recover navigation failures using the existing task context. Motivated by this finding, we propose ReInsVLN, a training-free framework supporting memory construction and utilization in frozen navigators. Coverage- and Instruction-Aware Compression constructs compact visual context through temporal redundancy reduction, coverage-aware protection, and instruction-guided token selection. Forgetting-Inspired History Reminders guide navigators to revisit retained observations according to a predefined schedule, without additional model calls. On R2R Val-Unseen, ReInsVLN improves success rates by 8.48 and 3.37 percentage points over the native StreamVLN and InternVLA-N1, reaching 64.27% and 64.44%, respectively. Visual compression largely preserves navigation performance while reducing prefill latency and KV cache memory by up to 21.77% and 32.25%, respectively, on matched observation workloads. Real-world trials on humanoid and quadruped robots further support the applicability of ReInsVLN to sequential instruction following and ordinal navigation.

Siwei Cheng, Hao Wu, Zitong Qi +7

University of Science and Technology of China · Eastern Institute of Technology, Ningbo · Shanghai Jiao Tong University

Vision-Language NavigationMemory-Augmented VLMsVLN Benchmarks
AskPDF
Jul 31, 2026

HAM-VLN: Harnessing Hierarchical Agentic Memory for Zero-Shot Vision-and-Language Navigation

Vision-and-language navigation (VLN) enables robots to follow instructions in previously unseen environments. Recently, a training-free paradigm has emerged: the robot queries a multimodal LLM to understand its observations and plan the next action. However, long-horizon navigation based on either image streams or dense map inevitably introduces a growing memory and reasoning bottleneck. We present HAM-VLN, a decision-coupled, agent-authored memory that equips the robot with a persistent, depth-grounded world graph. In the same model call used to select the next action, HAM-VLN also records semantic and reflective information---including room type, objects, navigation progress, and failure notes. Recent waypoints remain verbatim within a bounded window, while older history re-enters the context only through retrieval scored by relevance, recency, and salience, together with one-hop topological expansion. This design requires no additional LLM calls beyond the per-waypoint decision. Compared to previous methods, HAM-VLN not only improves various navigation metrics but also reduces the context length by more than 65%. Specifically, HAM-VLN achieves 61.0% Success Rate (SR) on VLN-CE R2R, 52.7% SR on VLN-CE RxR, and 79.7% SR on HM3D-v2 ObjectNav without any training.

An Liu, Bingxi Liu, Hongyu Ding +6

Institute of Automation, Chinese Academy of Sciences, Beijing, China · Southern University of Science and Technology, Shenzhen, China · Pengcheng Laboratory, Shenzhen, China +3

Memory-Augmented VLMsVision-Language NavigationLong-Horizon LLM AgentsMemory-Augmented Agents
AskPDF