All-day Multi-scenes Lifelong Vision-and-Language Navigation with Tucker Adaptation

Xudong Wang; Gan Li; Zhiyu Liu; Yao Wang; Lianqing Liu; Zhi Han

cs.CVMar 15, 2026

All-day Multi-scenes Lifelong Vision-and-Language Navigation with Tucker Adaptation

Authors: Xudong Wang, Gan Li, Zhiyu Liu, Yao Wang, Lianqing Liu, Zhi Han

Organizations: State Key Laboratory of Robotics and Intelligent Systems, Shenyang Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences · North University of China · Xi’an Jiaotong University

Abstract

Deploying vision-and-language navigation (VLN) agents requires adaptation across diverse scenes and environments, but fine-tuning on a specific scenario often causes catastrophic forgetting in others, which severely limits flexible long-term deployment. We formalize this challenge as the all-day multi-scenes lifelong VLN (AML-VLN) problem. Existing parameter-efficient adapters (e.g., LoRA and its variants) are limited by their two-dimensional matrix form, which fails to capture the multi-hierarchical navigation knowledge spanning multiple scenes and environments. To address this, we propose Tucker Adaptation (TuKA), which represents the multi-hierarchical navigation knowledge as a high-order tensor and leverages Tucker decomposition to decouple the knowledge into shared subspaces and scenario-specific experts. We further introduce a decoupled knowledge incremental learning strategy to consolidate shared subspaces while constraining specific experts for decoupled lifelong learning. Building on TuKA, we also develop a VLN agent named AlldayWalker, which continually learns across multiple navigation scenarios, achieving all-day multi-scenes navigation. Extensive experiments show that AlldayWalker consistently outperforms state-of-the-art baselines.

Figures & tables

Figure 1: Illustration of the proposed all-day multi-scenes lifelong vision-and-language navigation learning and Tucker Adaptation (TuKA). It requires VLN agents to continually learn across multiple scenes and diverse environments (low-light, overexposure, and scattering), progressively consolidating navigation knowledge to achieve all-day multi-scenes navigation. Different from LoRA and its variants, which only perform continual learning with single-dimensional task knowledge, our proposed TuKA decouples and represents multi-hierarchical task knowledge in a high-order tensor.

Figure 2: Illustration of catastrophic forgetting in lifelong navigation learning. The new scenario adaptation leads to catastrophic forgetting of old scenarios.

Figure 3: Illustration for comparison of existing LoRAs and our TuKA architecture. Different from the LoRA or MoE-LoRA variants, which represent simple knowledge within a two-hierarchical matrix, TuKA decoupling represents the multi-hierarchical knowledge within a high-order tensor.

Figure 4: Illustration of the proposed decoupled knowledge incremental learning. Our TuKA performs decoupled incremental learning for multi-hierarchical knowledge in a high-dimensional space.

Figure 5: Illustration of navigation scenario examples of the Allday-Habitat simulation platform. It includes four common environments: (a) normal, (b) low-light, (c) overexposure, and (d) scattering.

Figure 6: Illustration of our all-day multi-scenes lifelong VLN Benchmark. Agents are required to perform continual learning across two dimensions: scene and environment. The order of tasks is randomized. Further details please refer to Appendix § E .

Comparisons	T1	T2	T3	T4	T5	T6	T7	T8	T9	T10	T11	T12	T13	T14	T15	T16	T17	T18	T19	T20	T21	T22	T23	T24	Avg.
Seq-FT	0	0	0	0	0	0	5	0	10	5	4	10	7	6	7	14	7	8	4	24	28	24	37	64	11
LwF-LoRA	5	0	5	0	0	0	4	3	13	5	2	14	8	4	8	14	10	9	5	25	31	28	40	65	12
EWC-LoRA	10	7	6	8	3	4	10	4	15	14	5	14	10	6	12	19	16	14	8	24	28	29	38	67	15
Dense MoLE	14	8	14	21	13	12	18	7	19	14	17	19	13	10	18	20	23	17	10	29	32	33	41	66	20
Sparse MoLE	29	10	24	29	17	28	29	11	37	24	23	30	18	24	25	34	31	10	14	38	36	36	45	69	28
MoLA	34	13	32	34	22	21	37	13	48	28	29	33	24	32	29	39	37	22	14	43	41	39	49	68	33

Table 1: Test Results ( SR

\uparrow

in %) of Comparison Experiments under the AML-VLN Settings.

Comparisons	T1	T2	T3	T4	T5	T6	T7	T8	T9	T10	T11	T12	T13	T14	T15	T16	T17	T18	T19	T20	T21	T22	T23	Avg.
Seq-FT	100	100	100	100	100	100	93	100	88	93	94	83	91	92	92	83	87	89	94	73	84	81	78	87
LwF-LoRA	92	100	91	100	100	100	92	89	81	93	95	78	88	93	89	83	85	84	93	71	82	76	73	84
EWC-LoRA	85	92	89	90	94	91	86	87	82	82	92	78	84	78	86	77	75	75	88	72	77	68	66	79
Dense MoLE	80	91	82	73	78	78	81	86	77	81	78	72	73	65	79	72	63	69	85	63	74	64	61	72
Sparse MoLE	68	89	69	64	72	49	58	82	56	68	64	56	68	51	70	58	51	82	82	54	63	59	56	62
MoLA	62	79	59	58	63	62	46	78	43	63	55	42	65	48	68	54	46	60	79	44	55	51	43	55

Table 2: Test Results ( F-SR

\downarrow

in %) of Comparison Experiments under the AML-VLN Settings.

Figure 7: Test results ((a) SPL $\uparrow$ , (b) F-SPL $\downarrow$ , (c) OSR $\uparrow$ , (d) F-OSR $\downarrow$ ) of comparison experiment under the AML-VLN settings. For detailed quantitative results, please refer to our Appendix § K .

Figure 8: The comparisons of third-order tensors for representation multi-hierarchical knowledge.

Sd- $\bm{\mathcal{G}}$	Sd- $\bm{U}^{1}$	Sd- $\bm{U}^{2}$	SR	F-SR	SPL	F-SPL	OSR	F-OSR
✗	✗	✗	53	10	47	17	56	8
✗	✓	✓	55	10	49	17	57	9
✓	✗	✓	65	11	58	18	69	9
✓	✓	✗	62	11	54	18	66	9
✓	✗	✗	63	11	55	18	67	9
✓	✓	✓	65	11	58	18	68	9

Table 3: Ablation Results (%) for Shared Components.

Tasks	T1	T2	T3	T4	T5	T6	T7	T8	T9	T10	T11	T12	T13	T14	T15
SR (24 tasks)	79	23	71	81	33	50	87	38	79	75	79	67	50	86	71
SR (30 tasks)	77	23	71	70	33	50	86	38	79	75	80	66	50	85	70

Table 4: Comparison of SR (%) under 24-task and 30-task learning.

Task	Scene	Environment	Test Number	StreamVLN SR	BranchLoRA SR	SD-LoRA SR	AlldayWalker SR
G1	JeFG25nYj2p	Normal	105	45	52	53	65
G2	ur6pFq6Qu1A	Low-light	120	41	44	45	63
G3	r47D5H71a5s	Scattering	105	36	40	41	54
G4	Vvot9Ly1tCj	Overexposed	108	31	42	39	51
G5	Real-World 4	Normal	100	36	38	37	55
G6	Real-World 5	Low-light	100	18	21	21	43

Table 5: Generalization performance SR (%) across unseen environments (G1–G6).

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Notations	Definitions	Notations	Definitions
$\mathcal{I}$	The user language instruction	$\odot$	The Hadamard product
$\mathcal{S}$	Navigation Scene	$F_{\bm{U}^{1},t-1}$	The Fisher information weights for $\bm{U}^{1}$
$\mathcal{E}$	Scene Environment	$F_{\bm{U}^{2},t-1}$	The Fisher information weights for $\bm{U}^{2}$
$\mathcal{F}$	The backbone agent	$\mathcal{V}(\mathcal{O}_{s})$	The vision features for each scene
$\mathbf{O}$	The video stream observation	$\mathcal{V}(\mathcal{O}_{e})$	The vision features for each environment
$\mathcal{V}(\mathcal{O})$	The CLIP vision encoder	$\mathcal{Y}$	The output navigation actions annotations

Appendix

Table 6: Some Important Notations and Definitions Used in This Paper.

Hyperparameters Name	Value
Number of scene-specific experts $S$	7
Number of environment-specific experts $E$	4
4D Tucker decomposition ranks $\mathbf{r}=(r_{1},r_{2},r_{3},r_{4})$	(8,8,64,64)
Elastic Weight Consolidation regularization weight $\lambda_{1}$	0.2
Stability constraint weight for previously trained experts $\lambda_{2}$	0.2
Orthogonal regularization weight for $\bm{U}3$ / $\bm{U}4$ $\lambda_{3}$	0.1

Appendix

Table 7: Some Important Hyper-parameters in AlldayWalker.

Pattern	Hyperparameters Name	Value
Low-night	Brightness factor $B$	0.15
	Exposure time $T$	0.15
	Sensor gain / ISO $G$	8.0
	Shot noise factor $\alpha_{\text{shot}}$	0.4
	Read noise standard deviation $\sigma_{\text{read}}$	3.0
	Gamma correction value $\gamma$	2.2

Appendix

Table 8: Some Important Degradation Imaging Model Parameters.

Task ID	Scene Index Number	Scene ID	Environment ID	Train Number	Test Number
1	ac26ZMwG7aT	Scene-2	Normal-1	455	120
2	5LpN3gDmAk7	Scene-1	Scattering-2	405	150
3	S9hNv5qa7GM	Scene-4	Normal-1	410	105
4	mJXqzFtmKg4	Scene-3	Low-light-3	445	105
5	mJXqzFtmKg4	Scene-4	Overexposure-4	410	105
6	ULsKaCPVFJR	Scene-5	Overexposure-4	390	120

Appendix

Table 9: Statistics of the 24 Sequential Tasks in Allday-Habitat Benchmark. Each Task Corresponds to One Scene under One Environment Type.

Figure 9: Illustration of the proposed robotic navigation system.

Figure 10: The analysis results for TuKA parameter growth for scaling of ranks.

Rank $r_{1},r_{2}$	Rank $r_{3},r_{4}$	Scale-Type	SR $\uparrow$	F-SR $\downarrow$	SPL $\uparrow$	F-SPL $\downarrow$	OSR $\uparrow$	F-OSR $\downarrow$
4,4	32,32	all	60	11	51	18	62	9
4,4	64,64	sos	64	11	57	18	68	9
8,8	32,32	eos	61	11	52	18	63	9
8,8	64,64	equal	65	11	58	18	68	9

Appendix

Table 10: The Analysis Results for TuKA Parameter Growth for Scaling of Ranks (scale factors

s=0.5

) .

Rank $r_{1},r_{2}$	Rank $r_{3},r_{4}$	Scale-Type	SR $\uparrow$	F-SR $\downarrow$	SPL $\uparrow$	F-SPL $\downarrow$	OSR $\uparrow$	F-OSR $\downarrow$
16,16	128,128	all	68	11	61	18	71	9
16,16	64,64	sos	65	11	60	18	68	9
8,8	128,128	eos	67	11	61	18	71	9
8,8	64,64	equal	65	11	58	18	68	9

Appendix

Table 11: The Analysis Results for TuKA Parameter Growth for Scaling of Ranks (scale factors

s=2

) .

Rank $r_{1},r_{2}$	Rank $r_{3},r_{4}$	Scale-Type	SR $\uparrow$	F-SR $\downarrow$	SPL $\uparrow$	F-SPL $\downarrow$	OSR $\uparrow$	F-OSR $\downarrow$
24,24	192,192	all	69	11	62	18	71	9
24,24	64,64	sos	66	11	60	18	68	9
8,8	192,192	eos	68	11	61	18	71	9
8,8	64,64	equal	65	11	58	18	68	9

Appendix

Table 12: The Analysis Results for TuKA Parameter Growth for Scaling of Ranks (scale factors

s=3

) .

Rank $r_{1},r_{2}$	Rank $r_{3},r_{4}$	Scale-Type	SR $\uparrow$	F-SR $\downarrow$	SPL $\uparrow$	F-SPL $\downarrow$	OSR $\uparrow$	F-OSR $\downarrow$
32,32	256,256	all	69	11	62	18	71	9
32,32	64,64	sos	66	11	60	18	68	9
8,8	256,256	eos	68	11	61	18	71	9
8,8	64,64	equal	65	11	58	18	68	9

Appendix

Table 13: The Analysis Results for TuKA Parameter Growth for Scaling of Ranks (scale factors

s=4

) .

Comparisons	T1	T2	T3	T4	T5	T6	T7	T8	T9	T10	T11	T12	T13	T14	T15	T16	T17	T18	T19	T20	Avg.
3-order SR $\uparrow$	65	21	49	61	30	46	70	30	70	71	60	61	41	73	54	71	48	49	35	68	54
4-order SR $\uparrow$	79	23	71	81	33	50	87	38	79	75	79	67	50	86	71	76	67	63	43	81	65
3-order F-SR $\downarrow$	13	36	33	12	35	15	6	35	11	2	5	10	36	2	33	16	28	16	36	0	19
4-order F-SR $\downarrow$	2	27	23	8	21	1	0	28	9	0	2	6	30	$-3$	24	10	1	4	21	0	11
3-order SPL $\uparrow$	60	16	51	60	28	43	70	26	60	65	50	51	38	61	48	62	48	42	35	51	48
4-order SPL $\uparrow$	70	20	67	73	25	32	80	34	73	71	68	56	48	75	65	70	62	60	41	67	58

Appendix

Table 14: The Analysis Results (SR/F-SR/SPL/F-SPL/OSR/F-OSR in %) of Third-order Tensors for Representation multi-hierarchical Knowledge.

Figure 11: The comparison results for ABC-LoRA with hierarchical architecture.

Comparisons	T1	T2	T3	T4	T5	T6	T7	T8	T9	T10	T11	T12	T13	T14	T15	T16	T17	T18	T19	T20	Avg.
O-LoRA SR $\uparrow$	67	19	47	58	31	42	67	27	67	62	58	49	38	68	52	62	46	52	34	71	51
ABC-LoRA SR $\uparrow$	71	20	55	64	30	43	68	30	68	66	59	52	43	72	61	68	55	52	35	77	55
AlldayWalker SR $\uparrow$	79	23	71	81	33	50	87	38	79	75	79	67	50	86	71	76	67	63	43	81	65
O-LoRA F-SR $\downarrow$	19	41	41	28	38	24	4	38	15	17	11	13	43	2	40	24	32	5	35	0	24
ABC-LoRA F-SR $\downarrow$	15	36	33	21	32	25	4	30	15	15	8	7	36	0	33	15	26	5	25	0	19
AlldayWalker F-SR $\downarrow$	2	27	23	8	21	1	0	28	9	0	2	6	30	-3	24	10	1	4	21	0	11

Appendix

Table 15: The Comparison Results (SR/F-SR/SPL/F-SPL/OSR/F-OSR in %) of ABC-LoRA for Representation Multi-hierarchical Knowledge.

Figure 12: Illustration of the fifth-order tensor TuKA architecture.

Task Statistics				HydraLoRA		SD-LoRA		AlldayWalker
Task ID	Scene ID	Environment	Instruction	SR $\uparrow$	F-SR $\downarrow$	SR $\uparrow$	F-SR $\downarrow$	SR $\uparrow$	F-SR $\downarrow$
1	ac26ZMwG7aT	Normal	VLN	46	44	69	13	79	2
2	5LpN3gDmAk7	Low-light	DUN	29	60	41	35	51	29
3	ac26ZMwG7aT	Overexposure	VLN	30	68	29	34	39	29
4	S9hNv5qa7GM	Scattering	DUN	51	32	60	21	69	15
5	5LpN3gDmAk7	Normal	VLN	33	28	69	8	87	0

Appendix

Table 16: Statistics of the 10 Sequential Tasks for Verifying Fifth-order Tensor TuKA, and Some Comparison Results (SR/F-SR in %).

Method	Avg SR $\uparrow$	Avg F-SR $\downarrow$	Avg SPL $\uparrow$	Avg F-SPL $\downarrow$	Avg OSR $\uparrow$	Avg F-OSR $\downarrow$	Parameter
Seq-FT	11	87	4	95	6	91	14.68 M
LwF-LoRA	12	84	6	92	9	86	14.68 M
EWC-LoRA	15	79	9	87	13	81	14.68 M
Dense MoLE	20	72	14	80	18	74	16.52 M
Sparse MoLE	28	62	21	70	26	64	16.52 M
MoLA	33	55	26	63	31	57	16.52 M

Appendix

Table 17: Comparison of Baselines: Trainable Parameters and Averaged Metrics on the 24-task AML-VLN Benchmark Learning. Metrics (SR/F-SR in %) Are Averages across 24 Tasks, and Metrics (SPL/F-SPL/OSR/F-OSR in %) are Averages across First 20 Tasks Reported in the Paper.

Comparisons	T1	T2	T3	T4	T5	T6	T7	T8	T9	T10	T11	T12	T13	T14	T15	T16	T17	T18	T19	T20	Avg.
Seq-FT	0	0	0	0	0	0	4	0	9	0	2	8	5	3	4	12	3	6	2	18	4
LwF-LoRA	5	0	5	0	0	0	4	3	9	5	2	12	8	2	6	14	7	8	3	21	6
EWC-LoRA	5	6	6	8	2	4	7	4	10	12	5	14	8	4	10	15	14	11	5	23	9
Dense MoLE	9	5	14	16	11	8	16	6	16	14	15	15	10	7	17	18	22	16	9	27	14
Sparse MoLE	24	7	24	15	17	14	27	9	31	22	22	21	17	21	21	29	28	10	12	33	21
MoLA	29	8	32	32	18	21	36	13	45	27	25	26	22	29	24	36	31	17	14	42	26

Appendix

Table 18: Test Results ( SPL

\uparrow

in %) of Comparison Experiments under the AML-VLN Settings.

Comparisons	T1	T2	T3	T4	T5	T6	T7	T8	T9	T10	T11	T12	T13	T14	T15	T16	T17	T18	T19	T20	Avg.
Seq-FT	100	100	100	100	100	100	93	100	91	93	95	89	93	94	94	89	90	89	100	82	95
LwF-LoRA	94	100	94	100	100	100	92	93	90	94	95	86	88	85	92	87	88	85	100	79	92
EWC-LoRA	91	93	89	90	94	92	88	93	87	85	92	83	93	94	89	80	79	84	94	75	87
Dense MoLE	85	91	86	77	78	78	85	89	80	83	82	82	79	69	83	73	67	72	86	68	80
Sparse MoLE	74	90	74	68	76	64	63	85	58	71	67	57	75	53	75	67	58	84	85	56	70
MoLA	68	81	63	62	64	65	48	80	48	67	59	49	69	49	72	62	54	69	83	47	63

Appendix

Table 19: Test Results ( F-SPL

\downarrow

in %) of Comparison Experiments under the AML-VLN Settings.

Comparisons	T1	T2	T3	T4	T5	T6	T7	T8	T9	T10	T11	T12	T13	T14	T15	T16	T17	T18	T19	T20	Avg.
Seq-FT	0	5	0	0	0	3	5	0	10	5	4	10	8	12	7	14	7	8	6	24	6
LwF-LoRA	5	8	7	4	0	7	4	3	17	8	8	15	8	8	10	15	12	11	7	27	9
EWC-LoRA	14	10	6	13	3	12	12	4	23	14	13	19	11	13	14	19	16	17	9	24	13
Dense MoLE	19	11	14	21	15	16	18	8	34	17	17	19	13	14	18	20	25	17	10	33	18
Sparse MoLE	29	13	27	33	18	28	31	11	59	24	23	30	20	29	25	36	31	10	14	38	26
MoLA	34	12	33	38	22	24	37	13	52	32	29	34	26	38	31	39	39	22	14	43	31

Appendix

Table 20: Test Results ( OSR

\uparrow

in %) of Comparison Experiments under the AML-VLN Settings.

Comparisons	T1	T2	T3	T4	T5	T6	T7	T8	T9	T10	T11	T12	T13	T14	T15	T16	T17	T18	T19	T20	Avg.
Seq-FT	100	91	100	100	100	94	93	100	88	93	94	83	87	89	92	83	87	89	93	73	91
LwF-LoRA	92	92	88	89	100	90	92	89	77	91	91	77	88	78	87	79	82	80	94	69	86
EWC-LoRA	81	89	89	77	94	85	84	87	73	82	88	74	81	73	81	77	75	71	91	72	81
Dense MoLE	73	90	84	70	71	77	81	84	69	76	78	72	73	61	79	72	60	69	89	61	74
Sparse MoLE	68	84	67	50	72	49	53	82	55	68	64	56	67	46	70	57	51	82	82	54	64
MoLA	62	79	63	45	63	58	46	78	42	61	55	41	74	39	66	54	42	60	84	44	57

Appendix

Table 21: Test Results ( F-OSR

\downarrow

in %) of Comparison Experiments under the AML-VLN Settings.

Task	Scene Description	Environment	Train Number	Test Number
T25	8WUmhLawc2A	Normal Environment	460	120
T26	8WUmhLawc2A	Low-light Environment	460	120
T27	8WUmhLawc2A	Scattering Environment	460	120
T28	8WUmhLawc2A	Overexposed Environment	460	120
T29	Real-World Scene 3	Normal Environment	400	100
T30	Real-World Scene 3	Low-light Environment	400	100

Appendix

Table 22: Descriptions and dataset statistics of added Tasks T25–T30.

Figure 13: Illustration of the visualization examples for T25-T28.

Figure 14: Illustration of the visualization examples for T29-T30.

Figure 15: Illustration of the visualization examples for G1-G4.

Figure 16: Illustration of the visualization examples for G5-G6.

Tucker Decomposition

See all 15 papers

Oct 13, 2026

MemoryVLN: From Ego-Aware Memory Representation to Cross-Context Memory Reuse in Vision-Language Navigation

Large vision-language models (LVLMs) have shown strong potential for embodied vision-language navigation (VLN), yet their performance degrades on long-horizon trajectories due to context-length limits, redundant historical observations, and unstable spatial alignment across time. These issues make it difficult to both compress long navigation histories and reliably reuse past spatial experience. To address these limitations, we propose MemoryVLN, a unified memory-driven navigation system built around memory construction and management. During memory construction, Long Sequence Compression employs forgetting-curve-inspired temporal sampling to preserve dense recent evidence and sparse yet representative distant observations under a limited token budget. Ego-Aware Memory Representation further integrates 2D semantic tokens, 3D geometric tokens, and 6D agent poses, allowing each memory to encode both what was observed and where it was observed. During memory management, Hierarchical Memory Bank maintains in-episode and context-level memories through cross-context warm-start and confidence-controlled write-back. Memory Enhancement then retrieves the Top-

K

relevant memories according to the current navigation state and fuses them with the current visual tokens. Together, these mechanisms transform raw navigation history into compact, spatially grounded, selectively retrievable, and reusable navigation experience. Extensive experiments on R2R-CE and RxR-CE demonstrate consistent improvements over the state-of-the-art baselines by a substantial margin (i.e., 2% and 6.5% in successful rate).

Xueyao Wei, Yifei Wang, Delun Li +4

Beijing Institute of Technology · Zhongguancun Academy · Peking University

Vision-Language Navigation Memory-Augmented VLMs Long-Context VLMs

Ask PDF

Sep 30, 2026

NavHarness: Adaptive Goals for Agentic Vision-Language Navigation

Vision-Language Navigation (VLN) requires embodied agents to generate actions based on instructions and observations. General-purpose multimodal agents offer a promising basis for this task, but selecting plausible local actions does not ensure that execution remains consistent with the intended route, particularly in long-horizon tasks. Moreover, the accumulated interaction history increases the input required for subsequent decisions, resulting in a significant inference overhead. To this end, we introduce \method, an Agentic VLN framework that includes a Goal Agent that sets adaptive goals for local actions, a Verify Agent that dynamically verifies whether a goal has been completed, a Memory Agent for multimodal context compression, and a Visuomotor Agent to execute adaptive goals. Specifically, the Goal Agent formulates adaptive goals based on the instruction, current observation, and execution history. Then the Visuomotor Agent executes navigation actions to achieve each goal, while the Verify Agent uses a goal-specific verification question to dynamically assess whether the observed outcomes satisfy the intended completion condition. Verified goal completion then marks a boundary for the Memory Agent to compress the corresponding multimodal interaction history while preserving information needed for subsequent navigation. We evaluate navigation on R2R-CE and RxR-CE, examine framework variants across three model backbones, and study context evolution during execution. For Real-World evaluation, \method achieves 83.3% success and 1.51,m navigation error across eight challenging routes evaluated three times each.

Haoxiang Shi, Zaijing Li, Muhe Ding +3

Harbin Institute of Technology (Shenzhen) · Pengcheng Laboratory

Vision-Language Navigation Multimodal Agents

Ask PDF

May 17, 2026

SEDualVLN: A Spatially-Enhanced Dual-System Framework for Vision-Language Navigation

Vision-Language Navigation (VLN) approaches have currently followed two primary paradigms: the end-to-end Vision-Language Model (VLM) policy fine-tuned on navigation trajectories to directly predict actions, and the zero-shot modular pipeline integrating pre-trained Multimodal Large Language Model (MLLM) for training-free generalization to unseen environments. However, end-to-end methods struggle with long-horizon navigation and lack dynamic reasoning ability, whereas zero-shot methods are constrained by limited spatial grounding for reliable planning and require substantial reasoning time. To bridge this gap, we introduce SEDualVLN, a spatially-enhanced dual-system VLN framework. System 1 is a VLM model enhanced with both global and local spatial awareness, tailored for high-frequency action generation. System 2 integrates a general MLLM with a mapping module, wherein the MLLM performs low-frequency waypoint planning by leveraging top-down views of the real-time 3D map alongside streams of rendered path images. Both systems exploit complementary spatial representations to improve the agent's spatial understanding in VLN tasks. Ultimately, they cooperate to complete the navigation task through a fast-slow coordinated approach. SEDualVLN achieves state-of-the-art performance on VLN-CE benchmarks, and further ablation studies demonstrate the effectiveness of each system and module.

Jingzhi Huang, Junkai Huang, Wenxuan Song +5

Hong Kong Polytechnic University · Institute of Automation, Chinese Academy of Sciences · Hong Kong University of Science and Technology (Guangzhou)

Visual Spatial Reasoning Vision-Language Navigation Vision-Language Grounding VLM Reasoning

Ask PDF

Sd- $\bm{\mathcal{G}}$	Sd- $\bm{U}^{1}$	Sd- $\bm{U}^{2}$	SR	F-SR	SPL	F-SPL	OSR	F-OSR
✗	✗	✗	53	10	47	17	56	8
✗	✓	✓	55	10	49	17	57	9
✓	✗	✓	65	11	58	18	69	9
✓	✓	✗	62	11	54	18	66	9
✓	✗	✗	63	11	55	18	67	9
✓	✓	✓	65	11	58	18	68	9

Sd- $\bm{\mathcal{G}}$	Sd- $\bm{U}^{1}$	Sd- $\bm{U}^{2}$	SR	F-SR	SPL	F-SPL	OSR	F-OSR
✗	✗	✗	53	10	47	17	56	8
✗	✓	✓	55	10	49	17	57	9
✓	✗	✓	65	11	58	18	69	9
✓	✓	✗	62	11	54	18	66	9
✓	✗	✗	63	11	55	18	67	9
✓	✓	✓	65	11	58	18	68	9

Abstract

Tucker Decomposition

Explore similar work

MemoryVLN: From Ego-Aware Memory Representation to Cross-Context Memory Reuse in Vision-Language Navigation

NavHarness: Adaptive Goals for Agentic Vision-Language Navigation

SEDualVLN: A Spatially-Enhanced Dual-System Framework for Vision-Language Navigation

Sd- $\bm{\mathcal{G}}$	Sd- $\bm{U}^{1}$	Sd- $\bm{U}^{2}$	SR	F-SR	SPL	F-SPL	OSR	F-OSR
✗	✗	✗	53	10	47	17	56	8
✗	✓	✓	55	10	49	17	57	9
✓	✗	✓	65	11	58	18	69	9
✓	✓	✗	62	11	54	18	66	9
✓	✗	✗	63	11	55	18	67	9
✓	✓	✓	65	11	58	18	68	9