MemVLN: Streaming Episodic Memory for Vision-Language Navigation

Yan Wang; Jiangyong Huang; Siyuan Huang; Baoxiong Jia

cs.CVOct 26, 2026

MemVLN: Streaming Episodic Memory for Vision-Language Navigation

Authors: Yan Wang, Jiangyong Huang, Siyuan Huang, Baoxiong Jia

Organizations: State Key Laboratory of General Artificial Intelligence, BIGAI · Peking University

Abstract

Vision-language navigation (VLN) requires an embodied agent to follow language instructions through 3D environments. Maintaining persistent episodic memory remains a challenge for VLM-based streaming agents in long-horizon navigation. We present MEMVLN, a streaming episodic memory framework that maintains a bounded bank of compressed perceptual features annotated with timesteps and action-derived relative poses, and produces compact memory tokens via cross-attention. We pair the proposed memory with a frozen pretrained geometry encoder. To learn the memory module, we design a masking-based trajectory summarization task with multi-granularity sub-trajectory decomposition, encouraging route reconstruction exclusively from episodic memory. Under navigation training restricted to the official R2R/RxR trajectories, MEMVLN achieves state-of-the-art performance on most reported metrics on standard VLN-CE benchmarks. Long-horizon analyses further characterize when episodic memory is useful. For physical evaluation, we collect 30 instruction-conditioned episodes in one physical scene, use 24 as demonstrations for post-training, and hold out six for testing. On the held-out episodes, MEMVLN improves success over the VLM baseline from 17% to 40%, and post-training further raises it to 47%.

Figures & tables

Figure 1: MemVLN overview. We design a streaming episodic memory framework that endows vlm with navigation memory for streaming video. MemVLN employs a memory module that progressively stores compressed perceptual features and produces compact memory tokens via cross-attention. MemVLN elevates vision-language navigation performance, including long-horizon and real-world scenarios.

Figure 2: MemVLN model overview. MemVLN combines Qwen3-VL’s native SigLIP-2-based vision encoder and LLM, a frozen StreamVGGT geometry encoder, and our memory module. It receives a short-term context for local perception and an episodic memory for long-horizon reasoning.

Method	Observation				R2R Val-Unseen				Training
Method	Pano.	Odo.	Depth	S.RGB	NE $\downarrow$	OS $\uparrow$	SR $\uparrow$	SPL $\uparrow$	External Data
CMA ( Hong et al., 2022 ) [CVPR22]	✓	✓	✓		6.20	52.0	41.0	36.0	-
VLN $\circlearrowright$ BERT ( Hong et al., 2021 ) [CVPR21]	✓	✓	✓		5.74	53.0	44.0	39.0	-
Ego 2 -Map ( Hong et al., 2023 ) [ICCV23]	✓	✓	✓		5.54	56.0	47.0	41.0	-
DreamWalker ( Wang et al., 2023a ) [ICCV23]	✓	✓	✓		5.53	59.0	49.0	44.0	-
AO-Planner ( Chen et al., 2025 ) [AAAI25]	✓		✓		5.55	59.0	47.0	33.0	-

Table 1: Evaluation results on VLN-CE R2R Val-Unseen split. “Pano.”, “Odo.”, and “S.RGB” denote panoramic view, odometry, and single RGB; at policy inference, MemVLN instead derives relative pose from executed actions. External Data counts fine-tuning examples beyond those derived from official R2R/RxR trajectories ( e.g . , EnvDrop ( Tan et al., 2019 ) ). Bold marks the best streaming result among methods restricted to those official navigation trajectories. Results are partially referenced from JanusVLN ( Zeng et al., 2026 ) .

Method	Observation				RxR Val-Unseen				Training
Method	Pano.	Odo.	Depth	S.RGB	NE $\downarrow$	SR $\uparrow$	SPL $\uparrow$	nDTW $\uparrow$	External Data
CMA ( Hong et al., 2022 ) [CVPR22]	✓	✓	✓		8.76	26.5	22.1	47.0	-
VLN $\circlearrowright$ BERT ( Hong et al., 2021 ) [CVPR21]	✓	✓	✓		8.98	27.0	22.6	46.7	-
AO-Planner ( Chen et al., 2025 ) [AAAI25]	✓		✓		7.06	43.3	30.5	50.1	-
NavMorph ( Yao et al., 2025 ) [ICCV25]			✓	✓	8.85	30.8	22.8	44.2	-
Sim2Real ( Wang et al., 2024b ) [CoRL24]				✓	8.79	36.7	25.5	18.1	0 $K$

Table 2: Evaluation results on VLN-CE RxR Val-Unseen split. Settings follow Tab. 1 and JanusVLN ( Zeng et al., 2026 ) .

Setting	R2R Val-Unseen				RxR Val-Unseen
Setting	NE $\downarrow$	OS $\uparrow$	SR $\uparrow$	SPL $\uparrow$	NE $\downarrow$	SR $\uparrow$	SPL $\uparrow$	nDTW $\uparrow$
Baseline	6.56	51.1	43.9	40.9	6.90	48.8	43.9	59.7
+ mem.	6.38	56.7	45.6	41.6	6.28	53.3	45.9	59.6
+ geo.	6.04	58.1	49.2	45.2	5.87	56.3	49.7	62.6
+ mem. + geo.	5.85	56.3	49.7	45.8	5.74	56.1	49.4	63.0
+ mem. + geo. + aux.	5.19	61.2	55.0	50.5	5.59	55.4	48.4	64.6

Table 3: Ablations of memory (mem.), geometry (geo.), and auxiliary supervision (aux.).

$N_{b}$	NE $\downarrow$	OSR $\uparrow$	SR $\uparrow$	SPL $\uparrow$
0	5.87	57.4	50.9	47.2
100	5.19	61.2	55.0	50.5
200	6.24	58.7	50.2	45.7

Table 4: Memory capacity ablations on R2R Val-Unseen: bank size

N_{b}

and token count

N_{m}

Figure 3: Long-horizon analysis on RxR-CE Val-Unseen. Success rate (top) and navigation error (bottom) are grouped by instruction complexity (left) and instruction type (right). Base denotes the vlm baseline, and +Mem adds the memory module. Back, Stop, and Pass denote instructions involving backtracking, stop conditions and long hallways, and passing landmarks, respectively.

Figure 5: Qualitative results in simulation and the real world. Selected observations, actions, and instructions for RxR-CE (top, with top-down trajectories) and G1 (bottom).

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Dataset	Training Data		RGB Input
Setting	Dataset	Episodes	Nav. Samples	Resolution	HFOV	Camera
Simulation	R2R Train	10,819	$\sim$ 334K	$640\times 480$	$90^{\circ}$	Habitat-Sim
Simulation	RxR Train	19,996	$\sim$ 1.02M	$640\times 480$	$90^{\circ}$	Habitat-Sim
Real-world PT	G1 demonstrations	24	1,053	$640\times 480$	$90^{\circ}$	Intel RealSense D455

Appendix

Table 5: Training data statistics. Navigation samples count step-wise action-prediction targets after action merging. Simulation sample counts are rounded to the nearest thousand.

Table 10

Metric	Base	Full MemVLN
Timed observations, $N$	204	204
Mean latency (ms) $\downarrow$	503.2	691.0
P50 latency (ms) $\downarrow$	535.2	704.3
P95 latency (ms) $\downarrow$	550.0	779.8
Policy rate (Hz) $\uparrow$	1.99	1.45
16-frame observations, $N$	168	168

Appendix

Table 8: Fixed-replay inference latency and peak resource usage on one RTX 4090. Latency statistics pool two fresh-process repeats of the same 102 fixed decisions;

N

denotes timed observations rather than independent episodes. The 16-frame recent-context subset contains 168 timed observations per variant. Policy rate is

1000/\text{mean latency (ms)}

. Resource rows report the maximum across valid repeats. Process GPU is the peak PID-scoped memory reported by nvidia-smi , and RSS is the peak host resident set size of the evaluation process.

$N_{b}$	R2R Val-Unseen				RxR Val-Unseen
$N_{b}$	NE $\downarrow$	OSR $\uparrow$	SR $\uparrow$	SPL $\uparrow$	NE $\downarrow$	SR $\uparrow$	SPL $\uparrow$	nDTW $\uparrow$
0	5.87	57.4	50.9	47.2	5.72	54.9	49.0	64.5
100	5.19	61.2	55.0	50.5	5.59	55.4	48.4	64.6
200	6.24	58.7	50.2	45.7	6.15	54.3	48.0	61.5

Appendix

Table 9: Memory capacity ablations on R2R-CE and RxR-CE.

ID	Instruction
I1	Now you are facing a brown curtain. Turn right and walk pass an armchair towards the fire extinguisher. Turn left and walk toward the glass door in front of you to the right, that’s your destination.
I2	Now you are facing the reception. On your left you should see some brown sofas. Walk past the sofas to the handwriting on the wall. Stop at the door on your right.
I3	Now you are in a corridor. Keep moving forward. Once you pass the bookshelf on the right side, turn right and go to the sink in the corner.
I4	Go straight towards the glass windows and past the sofas on your right. After you reach the windows, turn right. Keep moving forward and stop at the second door on your right.
I5	Now you are in a corridor. On your left, there are some white cabinets. Go past them and then turn left. You should see a red fire extinguisher on the ground. That box is your destination.
I6	Now you are in a lab. On your right there are some wooden stakes. Move forward and pass the chairs in front of you. You should see a robot sitting on a chair. Walk past the robot and stop at the bucket next to the lamp.

Appendix

Table 10: Real-world evaluation instructions. We use the instruction IDs in subsequent results.

ID	Base	+Mem	MemVLN	MemVLN-PT
I1	20 / 40	40 / 60	80 / 20	60 / 80
I2	0 / 60	0 / 60	20 / 60	20 / 40
I3	40 / 60	80 / 60	40 / 40	80 / 60
I4	0 / 100	0 / 100	0 / 60	0 / 60
I5	20 / 20	0 / 40	20 / 40	60 / 40
I6	20 / 60	60 / 80	80 / 60	60 / 0

Appendix

Table 11: Per-instruction real-world results. Each cell reports SR / CR (%) over five repeated trials for the corresponding instruction and method. Bold marks the best nonzero SR and lowest CR in each row.

Figure 6: Qualitative results of MemVLN on R2R-CE.

Figure 7: Qualitative results of MemVLN on RxR-CE.

Figure 8: Qualitative results of MemVLN in the real world. Selected observations and actions from two G1 navigation rollouts, together with their navigation instructions.

Vision-Language Navigation

Vision-Language Navigation studies how embodied agents use natural-language instructions and visual observations to plan and execute routes.

See all 310 papers

MemoryVLN: From Ego-Aware Memory Representation to Cross-Context Memory Reuse in Vision-Language Navigation
Oct 13, 2026Xueyao Wei, Yifei Wang, Delun Li +4Vision-Language Navigation Memory-Augmented VLMs
Beijing Institute of Technology · Zhongguancun Academy · Peking University
MemVLN: Episodic and Procedural Memory for Vision-and-Language Navigation
Jul 26, 2026Yuqi Liu, Shengju Qian, Tianyuan Qu +5Memory-Augmented VLMs Efficient VLM Inference
1The Chinese University of Hong Kong · 2LIGHTSPEED · 3The University of Hong Kong +1
HAM-VLN: Harnessing Hierarchical Agentic Memory for Zero-Shot Vision-and-Language Navigation
Jul 31, 2026An Liu, Bingxi Liu, Hongyu Ding +6Memory-Augmented VLMs Vision-Language Navigation
Institute of Automation, Chinese Academy of Sciences, Beijing, China · Southern University of Science and Technology, Shenzhen, China · Pengcheng Laboratory, Shenzhen, China +3

Oct 13, 2026J/K move · Enter open · S save

MemoryVLN: From Ego-Aware Memory Representation to Cross-Context Memory Reuse in Vision-Language Navigation

Xueyao Wei, Yifei Wang, Delun Li, Shaoan Wang+3

Beijing Institute of Technology · Zhongguancun Academy · Peking University

Vision-Language Navigation Memory-Augmented VLMs Long-Context VLMs

Open paper AskPDF arXiv

Large vision-language models (LVLMs) have shown strong potential for embodied vision-language navigation (VLN), yet their performance degrades on long-horizon trajectories due to context-length limits, redundant historical observations, and unstable spatial alignment across time. These issues make it difficult to both compress long navigation histories and reliably reuse past spatial experience. To address these limitations, we propose MemoryVLN, a unified memory-driven navigation system built around memory construction and management. During memory construction, Long Sequence Compression employs forgetting-curve-inspired temporal sampling to preserve dense recent evidence and sparse yet representative distant observations under a limited token budget. Ego-Aware Memory Representation further integrates 2D semantic tokens, 3D geometric tokens, and 6D agent poses, allowing each memory to encode both what was observed and where it was observed. During memory management, Hierarchical Memory Bank maintains in-episode and context-level memories through cross-context warm-start and confidence-controlled write-back. Memory Enhancement then retrieves the Top-

K

relevant memories according to the current navigation state and fuses them with the current visual tokens. Together, these mechanisms transform raw navigation history into compact, spatially grounded, selectively retrievable, and reusable navigation experience. Extensive experiments on R2R-CE and RxR-CE demonstrate consistent improvements over the state-of-the-art baselines by a substantial margin (i.e., 2% and 6.5% in successful rate).

Abstract

Vision-Language Navigation

Explore similar work

MemoryVLN: From Ego-Aware Memory Representation to Cross-Context Memory Reuse in Vision-Language Navigation

MemVLN: Episodic and Procedural Memory for Vision-and-Language Navigation

HAM-VLN: Harnessing Hierarchical Agentic Memory for Zero-Shot Vision-and-Language Navigation