Vision-and-Language Navigation (VLN) requires an embodied agent to ground complex natural-language instructions into long-horizon navigation in unseen environments. While Vision-Language Models (VLMs) offer strong 2D semantic understanding, current VLN systems remain constrained by limited spatial perception, 2D-3D representation mismatch, and monocular scale ambiguity. In this paper, we propose AgentVLN, a novel and efficient embodied navigation framework that can be deployed on edge computing platforms. We formulate VLN as a Partially Observable Semi-Markov Decision Process (POSMDP) and introduce a VLM-as-Brain paradigm that decouples high-level semantic reasoning from perception and planning via a plug-and-play skill library. To resolve multi-level representation inconsistency, we design a cross-space representation mapping that projects perception-layer 3D topological waypoints into the image plane, yielding pixel-aligned visual prompts for the VLM. Building on this bridge, we integrate a context-aware self-correction and active exploration strategy to recover from occlusions and suppress error accumulation over long trajectories. To further address the spatial ambiguity of instructions in unstructured environments, we propose a Query-Driven Perceptual Chain-of-Thought (QD-PCoT) scheme, enabling the agent with the metacognitive ability to actively seek geometric depth information. Finally, we construct AgentVLN-Instruct, a large-scale instruction-tuning dataset with dynamic stage routing conditioned on target visibility. Extensive experiments show that AgentVLN consistently outperforms prior state-of-the-art methods (SOTA) on long-horizon VLN benchmarks, offering a practical paradigm for lightweight deployment of next-generation embodied navigation models. Code: https://github.com/Allenxinn/AgentVLN.
Figures & tables
Figure 1: Performance comparison on the Val-Unseen split of RxR-CE dataset [ 23 ] . AgentVLN outperforms existing state-of-the-art models while utilizing a substantially smaller parameter footprint. Furthermore, owing to its lightweight architecture, AgentVLN supports real-time local inference on Jetson embedded edge boards, eliminating the reliance on remote cloud deployment.
Figure 2: Overview of the AgentVLN framework. AgentVLN employs a VLM-as-Brain paradigm, decomposing long-horizon navigation into modular skill executions. Additionally, a context-driven fine-grained strategy and QD-PCoT mitigate localization errors and scale ambiguities, ensuring precise 3D target grounding.
Method
Observation
R2R-CE
S.RGB
Pano.
Depth
Odo.
NE ↓
OS ↑
SR ↑
SPL ↑
HPN+DN [ 21 ]
✓
✓
✓
6.31
40.0
36.0
34.0
VLN BERT [ 18 ]
✓
✓
✓
5.74
53.0
44.0
39.0
CMA [ 18 ]
✓
✓
✓
6.20
52.0
41.0
36.0
Reborn [ 2 ]
✓
✓
✓
5.40
57.0
50.0
46.0
Ego 2 -Map [ 19 ]
✓
✓
✓
5.54
56.0
47.0
41.0
Table 1: Comparison results with SOTA methods on the Val-Unseen dataset for R2R-CE. Our method outperforms other approaches on the same benchmarks, even having fewer parameters.
Method
S.RGB
Pano.
Depth
Odo.
NE ↓
SR ↑
SPL ↑
nDTW ↑
VLN BERT [ 18 ]
✓
✓
✓
8.98
27.0
22.6
46.7
CMA [ 18 ]
✓
✓
✓
8.76
26.5
22.1
47.0
Reborn [ 2 ]
✓
✓
✓
5.98
48.6
42.0
63.3
ETPNav [ 1 ]
✓
✓
✓
5.64
54.7
44.8
61.9
Seq2Seq [ 22 ]
✓
✓
12.10
13.9
11.9
30.8
LAW [ 38 ]
✓
✓
✓
10.90
8.0
8.0
38.0
Table 2: Comparison with SOTA methods on RxR-CE Val-Unseen split.
Figure 3: Visualization of AgentVLN’s navigation. Green points represent the visual prompts generated by the perception-level skills, whereas red circles denote the navigation waypoints selected or predicted by the model. Notably, when traversing narrow passages or confronting severe visual occlusions, the model seamlessly outputs fine-grained atomic actions for trajectory fine-tuning. This demonstrates its robust capability to achieve highly accurate, collision-free navigation relying exclusively on egocentric observations.
Figure 4: Navigation results in real-world indoor and outdoor environments. Experimental results demonstrate that, regardless of whether the agent is navigating through complex, confined indoor spaces or outdoor scenarios with challenging illumination conditions, the proposed model consistently and accurately comprehends natural language instructions, enabling it to rapidly plan and execute precise navigation trajectories.
VLM-as-Brain
CDFG
QD-PCoT
NE ↓
OS↑
SR↑
SPL↑
6.53
48.50
38.60
35.10
✓
4.67
65.10
59.70
55.60
✓
✓
3.90
72.10
65.60
63.40
✓
✓
✓
3.88
73.50
67.20
64.70
Table 3: Ablation study of different modules, where CDFG denotes context-driven fine-grained strategy.
Figure 5: Ablation on the impact of different temporal context length.
Vision-Language Navigation (VLN) approaches have currently followed two primary paradigms: the end-to-end Vision-Language Model (VLM) policy fine-tuned on navigation trajectories to directly predict actions, and the zero-shot modular pipeline integrating pre-trained Multimodal Large Language Model (MLLM) for training-free generalization to unseen environments. However, end-to-end methods struggle with long-horizon navigation and lack dynamic reasoning ability, whereas zero-shot methods are constrained by limited spatial grounding for reliable planning and require substantial reasoning time. To bridge this gap, we introduce SEDualVLN, a spatially-enhanced dual-system VLN framework. System 1 is a VLM model enhanced with both global and local spatial awareness, tailored for high-frequency action generation. System 2 integrates a general MLLM with a mapping module, wherein the MLLM performs low-frequency waypoint planning by leveraging top-down views of the real-time 3D map alongside streams of rendered path images. Both systems exploit complementary spatial representations to improve the agent's spatial understanding in VLN tasks. Ultimately, they cooperate to complete the navigation task through a fast-slow coordinated approach. SEDualVLN achieves state-of-the-art performance on VLN-CE benchmarks, and further ablation studies demonstrate the effectiveness of each system and module.
Jingzhi Huang, Junkai Huang, Wenxuan Song +5
Hong Kong Polytechnic University · Institute of Automation, Chinese Academy of Sciences · Hong Kong University of Science and Technology (Guangzhou)
Vision-and-Language Navigation (VLN) requires an agent to ground language instructions to its own movement within a visual environment. While state-of-the-art methods leverage the reasoning capabilities of Vision-Language Models (VLMs) for end-to-end action prediction, they often lack an explicit and explainable understanding of the relationships between the agent, the instruction, and the scene. Conversely, explicitly building a scene map for heuristic planning is intuitively appealing but relies on additional 3D sensors and hinders large-scale vision-language pre-training. To bridge this gap, we propose AwareVLN, a novel framework that equips the navigation model with a self-aware reasoning mechanism, enabling it to understand the agent's state and task progress in a fully end-to-end and data-driven manner. Our approach features two key innovations: (1) a structural reasoning module that fosters spatial and task-oriented self-awareness, and (2) an automatic data engine with progress division for effective training. Extensive experiments on various datasets in Habitat simulator show our AwareVLN significantly outperforms previous state-of-the-art vision-language navigation methods. Project page: https://gwxuan.github.io/AwareVLN/.
Vision-Language Navigation (VLN) requires embodied agents to generate actions based on instructions and observations. General-purpose multimodal agents offer a promising basis for this task, but selecting plausible local actions does not ensure that execution remains consistent with the intended route, particularly in long-horizon tasks. Moreover, the accumulated interaction history increases the input required for subsequent decisions, resulting in a significant inference overhead. To this end, we introduce \method, an Agentic VLN framework that includes a Goal Agent that sets adaptive goals for local actions, a Verify Agent that dynamically verifies whether a goal has been completed, a Memory Agent for multimodal context compression, and a Visuomotor Agent to execute adaptive goals. Specifically, the Goal Agent formulates adaptive goals based on the instruction, current observation, and execution history. Then the Visuomotor Agent executes navigation actions to achieve each goal, while the Verify Agent uses a goal-specific verification question to dynamically assess whether the observed outcomes satisfy the intended completion condition. Verified goal completion then marks a boundary for the Memory Agent to compress the corresponding multimodal interaction history while preserving information needed for subsequent navigation. We evaluate navigation on R2R-CE and RxR-CE, examine framework variants across three model backbones, and study context evolution during execution. For Real-World evaluation, \method achieves 83.3% success and 1.51,m navigation error across eight challenging routes evaluated three times each.
Haoxiang Shi, Zaijing Li, Muhe Ding +3
Harbin Institute of Technology (Shenzhen) · Pengcheng Laboratory