PROSPECT: Unified Streaming Vision-Language Navigation via Semantic--Spatial Fusion and Latent Predictive Representation

Zehua Fan; Wenqi Lyu; Wenxuan Song; Linge Zhao; Yifei Yang; Xi Wang; Junjie He; Lida Huang

cs.CVMar 4, 2026

PROSPECT: Unified Streaming Vision-Language Navigation via Semantic--Spatial Fusion and Latent Predictive Representation

Authors: Zehua Fan, Wenqi Lyu, Wenxuan Song, Linge Zhao, Yifei Yang, Xi Wang, Junjie He, Lida Huang, +7 more

Organizations: Z. Fan and F. Gao are with Shanghai Jiao Tong University, Shanghai, China. · W. Lyu is with the University of Adelaide, Adelaide, Australia. · W. Song and J. He are with the Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China. · L. Zhao is with Wuhan University, Wuhan, China. · Y. Yang is with Beijing Jiaotong University, Beijing, China. · X. Wang is with AIR Wuxi Innovation Center, Tsinghua University, Wuxi, China. · L. Huang and Y. Wang are with Tsinghua University, Beijing, China. · H. Liu, B. Sun, G. Bao, X. Mao, and L. Xu are with Lenovo, Beijing, China.

Abstract

Multimodal large language models (MLLMs) have advanced zero-shot end-to-end Vision-Language Navigation (VLN), yet robust navigation requires not only semantic understanding but also predictive modeling of environment dynamics and spatial structure. We propose PROSPECT, a unified streaming navigation agent that couples a streaming Vision-Language-Action (VLA) policy with latent predictive representation learning. PROSPECT uses CUT3R as a streaming 3D foundation spatial encoder to produce long-context, absolute-scale spatial features, and fuses them with SigLIP semantic features via cross-attention. During training, we introduce learnable stream query tokens that query the streaming context and predict next-step 2D and 3D latent features (rather than pixels or explicit modalities), supervised in the latent spaces of frozen SigLIP and CUT3R teachers. The predictive branch shapes internal representations without inference overhead. Experiments on VLN-CE benchmarks and real-robot deployment demonstrate state-of-the-art performance and improved long-horizon robustness under diverse lighting. We will release code for the community soon.

Figures & tables

Fig. 1: Overview of PROSPECT. (a) Streaming setup: A streaming attention mask enforces temporal causality and isolates 2D/3D query tokens to prevent cross-modal leakage. SigLIP and CUT3R provide 2D semantic and absolute-scale 3D spatial feature streams, fused by cross-attention for the policy. (b) Unified model: In training, stream query tokens predict next-step 2D/3D latent features under frozen SigLIP/CUT3R supervision (no inference cost). At inference, only the VLA policy runs at $\sim$ 4 Hz. (c) Results: First-tier VLN-CE performance and zero-shot Habitat navigation; larger gains on the long-horizon RxR benchmark than on R2R, indicating stronger robustness for complex instruction following. Real-robot deployment is robust under diverse lighting.

Fig. 2: Architecture of PROSPECT. Instruction and observations (historical keyframes and current frame) share one pipeline: frozen SigLIP and CUT3R with cross-attention fusion; keyframes are condensed into long-term memory $M$ . The model uses a KV cache for context and autoregressively outputs navigation actions. Training only: 2D/3D query tokens reverse-query the stream; lightweight decoders predict next-step latents under cosine (2D) and MSE (3D) with frozen teachers. Predictive branch removed at inference.

Fig. 3: Streaming attention mask used by PROSPECT. Upper (gray): Causal mask for navigation context (ctxt) and actions (act): each

\text{act}_{i}

may attend only to

\text{ctxt}_{0:i}

and

\text{act}_{0:i-1}

, ensuring no future leakage. Middle (red): Each 2D query token

\langle\text{Query2d}_{i}\rangle

attends only to its own round and prior rounds’ ctxt/act; it cannot attend to any other Query2d, any Query3d, or future rounds—enforcing both turn isolation and modality disentanglement. Lower (blue): Same for 3D query tokens

\langle\text{Query3d}_{i}\rangle

Setting	NE $\downarrow$	OSR $\uparrow$	SR $\uparrow$	SPL $\uparrow$
Baseline (SigLIP only)	6.05	53.8	45.5	41.6
Ours (SigLIP + CUT3R)	5.91	55.0	46.7	41.8
Ours (+ WM-2D only)	5.89	56.0	47.0	42.0
Ours (+ WM-3D only)	5.90	55.4	47.2	41.9
Ours (+ WM-2D + WM-3D)	5.82	57.6	48.7	42.9

TABLE II: Module ablation on R2R val-unseen .

Encoder	Time (s)	SR $\uparrow$	SPL $\uparrow$	OSR $\uparrow$	NE $\downarrow$
VGGT [ 19 ]	OOM	OOM	OOM	OOM	OOM
InfiniteVGGT [ 21 ]	0.284	43.2	38.0	54.4	6.61
Ours (CUT3R) [ 22 ]	0.245	48.7	42.9	57.6	5.82

TABLE III: Spatial encoder ablation on R2R val-unseen .

Horizon	Model	#Ep	SR $\uparrow$	SPL $\uparrow$	OSR $\uparrow$	NE $\downarrow$
Short (1–50)	Baseline	459	51.20	48.18	55.34	5.08
	Ours	486	51.23	48.84	54.53	4.86
	Difference	+27	+0.03	+0.66	-0.81	-0.22
Medium (50–100)	Baseline	1038	49.61	43.79	61.27	5.64
	Ours	1061	54.29	48.04	63.71	5.46
	Difference	+23	+4.68	+4.25	+2.44	-0.18

TABLE IV: Performance by task horizon on R2R val-unseen .

Mask Design	NE $\downarrow$	OSR $\uparrow$	SR $\uparrow$	SPL $\uparrow$
Leaky	6.81	51.3	40.2	35.7
w/o Isolation	6.98	51.1	39.9	35.3
Ours	5.82	57.6	48.7	42.9

TABLE V: Ablation on attention mask design on R2R val-unseen .

Fig. 4: First-person views from ARX-Lift2 under diverse indoor/outdoor lighting.

Scene	Lighting	NaVid [ 1 ]	StreamVLN [ 4 ]	Ours
Indoor
Office	Bright	7/30	12/30	20/30
Warehouse	Bright	6/30	12/30	18/30
Corridor	Moderate	11/30	16/30	22/30
Outdoor
Afternoon	Bright	6/30	10/30	18/30

TABLE VI: Real-robot success rates (completed/total) by scene and lighting.

Explore similar work

StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modeling
Jul 7, 2025Meng Wei, Chenyang Wan, Xiqian Yu +9Efficient VLM Inference Efficient VLA Models
1Shanghai AI Lab · 2The University of Hong Kong · 3Zhejiang University +1
SEDualVLN: A Spatially-Enhanced Dual-System Framework for Vision-Language Navigation
May 17, 2026Jingzhi Huang, Junkai Huang, Wenxuan Song +5Visual Spatial Reasoning Vision-Language Navigation
Hong Kong Polytechnic University · Institute of Automation, Chinese Academy of Sciences · Hong Kong University of Science and Technology (Guangzhou)
AgentVLN: Towards Agentic Vision-and-Language Navigation
Mar 18, 2026Zihao Xin, Wentong Li, Yixuan Jiang +6Vision-Language Navigation Robot Navigation
Nanjing University of Aeronautics and Astronautics · Shandong University · Zhejiang University

Jul 7, 2025J/K move · Enter open · S save

StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modeling

Meng Wei, Chenyang Wan, Xiqian Yu, Tai Wang+8

1Shanghai AI Lab · 2The University of Hong Kong · 3Zhejiang University+1

Efficient VLM Inference Efficient VLA Models Vision-Language Navigation

Open paper AskPDF arXiv

Vision-and-Language Navigation (VLN) in real-world settings requires agents to process continuous visual streams and generate actions with low latency grounded in language instructions. While Video-based Large Language Models (Video-LLMs) have driven recent progress, current VLN methods based on Video-LLM often face trade-offs among fine-grained visual understanding, long-term context modeling and computational efficiency. We introduce StreamVLN, a streaming VLN framework that employs a hybrid slow-fast context modeling strategy to support multi-modal reasoning over interleaved vision, language and action inputs. The fast-streaming dialogue context facilitates responsive action generation through a sliding-window of multi-turn dialogues, while the slow-updating memory context compresses historical visual states using a 3D-aware token pruning strategy. With this slow-fast design, StreamVLN achieves real-time dialogues through KV cache reuse, supporting long video streams with bounded context size and inference cost. Experiments on VLN-CE benchmarks show state-of-the-art performance with low latency, ensuring robustness and efficiency in real-world deployment. The project page is: https://streamvln.github.io/.

Abstract

Explore similar work

StreamVLN: Streaming Vision-and-Language Navigation via SlowFast Context Modeling

SEDualVLN: A Spatially-Enhanced Dual-System Framework for Vision-Language Navigation

AgentVLN: Towards Agentic Vision-and-Language Navigation