DecoVLN: Decoupling Observation, Reasoning, and Correction for Vision-and-Language Navigation

Zihao Xin; Wentong Li; Yixuan Jiang; Bin Wang; Runmin Cong; Jie Qin; Shengjun Huang

cs.ROMar 13, 2026

DecoVLN: Decoupling Observation, Reasoning, and Correction for Vision-and-Language Navigation

Authors: Zihao Xin, Wentong Li, Yixuan Jiang, Bin Wang, Runmin Cong, Jie Qin, Shengjun Huang

Organizations: Nanjing University of Aeronautics and Astronautics · Shandong University

Abstract

Vision-and-Language Navigation (VLN) requires agents to follow long-horizon instructions and navigate complex 3D environments. However, existing approaches face two major challenges: constructing an effective long-term memory bank and overcoming the compounding errors problem. To address these issues, we propose DecoVLN, an effective framework designed for robust streaming perception and closed-loop control in long-horizon navigation. First, we formulate long-term memory construction as an optimization problem and introduce adaptive refinement mechanism that selects frames from a historical candidate pool by iteratively optimizing a unified scoring function. This function jointly balances three key criteria: semantic relevance to the instruction, visual diversity from the selected memory, and temporal coverage of the historical trajectory. Second, to alleviate compounding errors, we introduce a state-action pair-level corrective finetuning strategy. By leveraging geodesic distance between states to precisely quantify deviation from the expert trajectory, the agent collects high-quality state-action pairs in the trusted region while filtering out the polluted data with low relevance. This improves both the efficiency and stability of error correction. Extensive experiments demonstrate the effectiveness of DecoVLN, and we have deployed it in real-world environments.

Figures & tables

Figure 1: The framework of DecoVLN. DecoVLN decouples the agent’s observation and reasoning processes. The agent can perceive the environment continuously while in motion and, based on the Adaptive Memory Refinement (AMR) mechanism, it filters and stores high-information-density state representations into a memory bank. During the generation phase, the LLM outputs the action chunk which is comprising multiple consecutive actions-based on the input instruction, the current frame, and the memory bank. Subsequently, we construct an error-correction strategy based on state-action pairs. The model autonomously explores according to the instruction and collects State-Action Pairs within a trusted region for error-correction fine-tuning. This process not only enhances data utilization efficiency but also equips the model with introspective and self-correction capabilities.

Method	Observation				R2R				RXR
Method	RGB	Pano.	Depth	Odo.	NE ↓	OS↑	SR↑	SPL↑	NE ↓	SR↑	SPL↑	nDTW↑
HPN+DN [ 16 ]		✓	✓	✓	6.31	40.0	36.0	34.0	-	-	-	-
VLN BERT [ 13 ]		✓	✓	✓	5.74	53.0	44.0	39.0	8.98	27.0	22.6	46.7
CMA [ 13 ]		✓	✓	✓	6.20	52.0	41.0	36.0	8.76	26.5	22.1	47.0
Reborn [ 2 ]		✓	✓	✓	5.40	57.0	50.0	46.0	5.98	48.6	42.0	63.3
Ego 2 -Map [ 14 ]		✓	✓	✓	5.54	56.0	47.0	41.0	-	-	-	-

Table 1: Experimental results on the Val-Unseen dataset of R2R-CE and RxR-CE. * indicates training with additional large-scale datasets. Our method outperforms all approaches on the same benchmarks, even surpassing certain VLN models that were trained with additional large-scale datasets, without using global priors or multi-sensor inputs.

AMR	CF	NE ↓	OS↑	SR↑	SPL↑
		5.89	51.7	47.3	43.9
✓		5.50	57.9	50.9	46.1
✓	✓	5.01	63.5	56.3	50.5

Table 2: Ablation study of different modules on R2R Val-Unseen. AMR denotes Adaptive Memory Refinement, and CF denotes Corrective Fine-tuning.

Figure 2: Evaluate success rates for various

K

values and hyper-parameters on R2R Val-Unseen.

K	R2R			RxR
K	NE ↓	SR↑	SPL↑	NE ↓	SR↑	SPL↑
2	6.01	41.9	39.8	7.09	38.7	36.5
4	5.92	45.5	42.2	6.92	43.8	41.5
8	5.50	50.9	46.1	6.53	49.2	45.2
12	5.56	50.7	45.8	6.50	49.5	45.1

Table 3: Ablation study of different historical memory lengths.

Figure 3: Comparison of Uniform Sampling strategy and Adaptive Memory Refinement mechanism. The history obtained via the uniform sampling strategy captures a large number of instruction-irrelevant images, such as walls and corners. This irrelevant semantic information severely impacts the model’s reasoning performance. In contrast, the Adaptive Memory Refinement mechanism can effectively extract the key navigation points (indicated in red ) from the instruction to achieve efficient and accurate navigation.

$\tau$	R2R			RxR
$\tau$	NE ↓	SR↑	SPL↑	NE ↓	SR↑	SPL↑
1	5.32	53.7	48.1	5.98	52.3	45.5
3	5.01	56.1	50.5	5.73	54.2	46.3
6	5.12	55.9	50.2	5.71	54.3	46.2

Table 4: Ablation study of different trusted region sizes.

Method	Data	NE ↓	OS ↑	SR ↑	SPL ↑
Baseline	-	5.50	57.9	50.9	46.1
DAgger	240K	5.11	63.8	56.0	50.3
DAgger	180K	5.32	62.1	54.8	49.6
Ours	180K	5.01	63.5	56.3	50.5

Table 5: Comparison with vanilla DAgger at different data scales.

Figure 4: The robot accurately follows complex natural language instructions involving spatial reasoning and object grounding, demonstrating robust performance and strong sim-to-real generalization under challenging conditions.

Figure 5: Performance comparison on long-horizon navigation validation set.

Figure 6: Comparison between the pipelines of Streaming VLN and DecoVLN. (a) Streaming VLN paradigm requires storing all historical observation sequences, sampling them during inference, and repeatedly transferring the selected frames between RAM and VRAM. This tight coupling of observation and reasoning leads to high storage overhead and inefficient data transfer. (b) Our DecoVLN, in contrast, introduces an adaptive memory-refinement mechanism during the observation phase. This design selectively preserves high-value semantic information in a VRAM-resident memory bank, which is directly consumed by the VLN model during inference. Our approach significantly reduces storage overhead and improves data transfer efficiency.

Augmentation Strategy	Parameters
Random Brightness	factor = 0.2, prob = 0.5
Random Saturation	factor = 0.2, prob = 0.5
Posterization	bits = 4, prob = 0.5
Random Sharpness	factor = 0.2, prob = 0.5
AutoContrast	prob = 0.3
Gaussian Blur	prob = 0.2

Table 6: Image augmentation strategies and their corresponding parameters used in the SFT stage.

Method	Inputs	R2R				RxR
Method	Inputs	NE↓	OS↑	SR↑	SPL↑	NE↓	SR↑	SPL↑	nDTW↑
StreamVLN [ 43 ]	RGB + Depth	4.98	64.2	56.9	51.9	6.22	52.9	46.0	61.9
StreamVLN [ 43 ]	RGB	5.10	64.0	55.7	50.9	6.16	51.8	45.0	62.1
Ours	RGB	5.01	63.5	56.1	50.5	5.73	54.2	46.3	63.5

Table 7: Comparison with StreamVLN [ 43 ] on the R2R and RxR benchmarks. Our method achieves competitive performance on R2R and significantly outperforms both RGB and RGB+Depth variants of StreamVLN on RxR, despite using only RGB inputs.

Figure 7: Visual comparison between uniform sampling and adaptive memory refinement. Frames selected by the proposed method exhibit higher information density and stronger semantic alignment with the navigation instructions, while effectively filtering redundant observations.

Figure 8: Real-world navigation results of DecoVLN across diverse scenarios. The robot accurately follows complex natural language instructions involving spatial reasoning and object grounding, demonstrating robust performance and strong sim-to-real generalization under challenging conditions.

Vision-Language Navigation

Vision-Language Navigation studies how embodied agents use natural-language instructions and visual observations to plan and execute routes.

See all 310 papers

Dual-Anchoring: Addressing State Drift in Vision-Language Navigation
Apr 19, 2026Kangyi Wu, Pengna Li, Kailin Lyu +5VLM Adaptation Vision-Language Navigation
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, National Engineering Research Center for Visual Information and Applications, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University · Institute of Automation, Chinese Academy of Sciences, School of Artificial Intelligence, University of Chinese Academy of Sciences · Johns Hopkins University +2
AgentVLN: Towards Agentic Vision-and-Language Navigation
Mar 18, 2026Zihao Xin, Wentong Li, Yixuan Jiang +6Vision-Language Navigation Robot Navigation
Nanjing University of Aeronautics and Astronautics · Shandong University · Zhejiang University
MemoryVLN: From Ego-Aware Memory Representation to Cross-Context Memory Reuse in Vision-Language Navigation
Oct 13, 2026Xueyao Wei, Yifei Wang, Delun Li +4Vision-Language Navigation Memory-Augmented VLMs
Beijing Institute of Technology · Zhongguancun Academy · Peking University

Apr 19, 2026J/K move · Enter open · S save

Dual-Anchoring: Addressing State Drift in Vision-Language Navigation

Kangyi Wu, Pengna Li, Kailin Lyu, Xi Lin+4

National Key Laboratory of Human-Machine Hybrid Augmented Intelligence, National Engineering Research Center for Visual Information and Applications, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University · Institute of Automation, Chinese Academy of Sciences, School of Artificial Intelligence, University of Chinese Academy of Sciences · Johns Hopkins University+2

VLM Adaptation Vision-Language Navigation

Open paper AskPDF arXiv

Vision-Language Navigation(VLN) requires an agent to navigate through 3D environments by following natural language instructions. While recent Video Large Language Models(Video-LLMs) have largely advanced VLN, they remain highly susceptible to State Drift in long scenarios. In these cases, the agent's internal state drifts away from the true task execution state, leading to aimless wandering and failure to execute essential maneuvers in the instruction. We attribute this failure to two distinct cognitive deficits: Progress Drift, where the agent fails to distinguish completed sub-goals from remaining ones, and Memory Drift, where the agent's history representations degrade, making it lose track of visited landmarks. In this paper, we propose a Dual-Anchoring Framework that explicitly anchors the instruction progress and history representations. First, to address progress drift, we introduce Instruction Progress Anchoring, which supervises the agent to generate structured text tokens that delineate completed versus remaining sub-goals. Second, to mitigate memory drift, we propose Memory Landmark Anchoring, which utilizes a Landmark-Centric World Model to retrospectively predict object-centric embeddings extracted by the Segment Anything Model, compelling the agent to explicitly verify past observations and preserve distinct representations of visited landmarks. Facilitating this framework, we curate two extensive datasets: 3.6 million samples with explicit progress descriptions, and 937k grounded landmark data for retrospective verification. Extensive experiments in both simulation and real-world environments demonstrate the superiority of our method, achieving a 15.2% improvement in Success Rate and a remarkable 24.7% gain on long-horizon trajectories. To facilitate further research, we will release our code, data generation pipelines, and the collected datasets.

Abstract

Vision-Language Navigation

Explore similar work

Dual-Anchoring: Addressing State Drift in Vision-Language Navigation

AgentVLN: Towards Agentic Vision-and-Language Navigation

MemoryVLN: From Ego-Aware Memory Representation to Cross-Context Memory Reuse in Vision-Language Navigation