Organizations: CertaintyX. · the School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore 639798. · the Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213 USA. · the School of Mechanical and Aerospace Engineering, and jointly with the School of Electrical and Electronic Engineering at Nanyang Technological University, Singapore.
Language-guided embodied navigation requires an agent to interpret object-referential instructions, search across multiple rooms, localize the referenced target, and execute reliable motion toward it. Existing systems remain limited in real indoor environments because narrow field-of-view sensing exposes only a partial local scene at each step, often forcing repeated rotations, delaying target discovery, and producing fragmented spatial understanding; meanwhile, directly prompting LLMs with dense 3D maps or exhaustive object lists quickly exceeds the context budget. We present OmniVLN, a zero-shot visual-language navigation framework that couples omnidirectional 3D perception with token-efficient hierarchical reasoning for both aerial and ground robots. OmniVLN fuses a rotating LiDAR and panoramic vision into a hardware-agnostic mapping stack, incrementally constructs a five-layer Dynamic Scene Graph (DSG) from mesh geometry to room- and building-level structure, and stabilizes high-level topology through persistent-homology-based room partitioning and hybrid geometric/VLM relation verification. For navigation, the global DSG is transformed into an agent-centric 3D octant representation with multi-resolution spatial attention prompting, enabling the LLM to progressively filter candidate rooms, infer egocentric orientation, localize target objects, and emit executable navigation primitives while preserving fine local detail and compact long-range memory. Experiments show that the proposed hierarchical interface improves spatial referring accuracy from 77.27% to 93.18%, reduces cumulative prompt tokens by up to 61.7% in cluttered multi-room settings, and improves navigation success by up to 11.68% over a flat-list baseline. We will release the code and an omnidirectional multimodal dataset to support reproducible research.
Figures & tables
Figure 1: OmniVLN, a zero-shot visual-language navigation (VLN) framework coupling 360∘ 3D perception with token-efficient reasoning across aerial & ground platforms.
Figure 2: Overview of the proposed framework. The Multimodal Perception module (left) achieves 360∘ spatio-temporal consistency by fusing data from a rotating LiDAR and panoramic fisheye cameras, enabling the generation of high-fidelity semantic point clouds across robotic platforms. The Hierarchical Representation module (center) incrementally constructs an online five-layer DSG, bridging the gap between low-level geometric places and high-level macro-spatial rooms, while employing a VLM-based hybrid pruning mechanism to refine physical veracity. The LLM-based Reasoning module (right) transforms global graph knowledge into an agent-centric 3D octant observation model, utilizing a closed-loop actor–critic framework and DSG-guided hierarchical prompting to translate natural language queries into executable navigation actions.
Figure 3: Omnidirectional projection model. A 3D point Pt=(x,y,z) in the robot’s egocentric frame is mapped onto a unit sphere and projected to pixel coordinates (u,v) of a 360∘ panoramic image using the equirectangular projection model.
Figure 4: Visualization of the multi-modal perception in the IoT laboratory. The top row shows panoramic RGB observations from the agent’s perspective. The bottom row displays the corresponding 3D semantic point clouds with instantiated objects and their unique IDs, which serve as the raw input for the DSG hierarchy construction.
Representation
VI Accuracy
VD Accuracy
Overall
Non-Hierarchical (Uniform Flat)
86.36%
68.18%
77.27%
Hierarchical (Ours)
95.45%
90.91%
93.18%
Improvement
+9.09%
+22.73%
+15.91%
Table I: Comparative Accuracy of Referring Expression Generation
Figure 5: Multi-resolution spatial attention mechanism for token-efficient navigation. The central red diamond denotes the ego-agent. Semantic object categories are color-coded as follows: chair (blue), table (green), cabinet (orange).
Figure 6: Synthetic datasets across the three spatial-attention tiers. Each scene represents a 10m×10m environment with varying object densities and distributions. Colors denote chair (blue), table (green), cabinet (orange), and screen (purple).
Spatial Condition
Foveal Tier (8-Octant)
Peripheral Tier
Global Memory Tier
Object Density
Low (5)
Mid (10)
High (15)
Low (5)
Mid (10)
High (15)
Low (15)
Mid (30)
High (50)
Dataset ID
D1
D2
D3
D4
D5
D6
D7
D8
D9
Uniform Flat-List
402.60
746.80
1153.07
398.80
806.50
1181.60
1126.33
2089.33
3555.98
Ours
401.60
440.40
604.27
291.80
446.50
584.93
496.27
735.27
1067.62
Reduction (%)
0.25%
41.03%
47.59%
26.83%
44.64%
50.50%
55.94%
64.81%
69.98%
Table II: End-to-End Cumulative Token Consumption Across Different Spatial Conditions and Object Densities.
Figure 7: Navigation Success Rate (%) comparison between Uniform Flat-List and our method across nine datasets ( D1 - D9 ). Background colors indicate the three spatial attention tiers.
Figure 8: Qualitative zero-shot navigation results for the command “go to the nearest chair”. The top row shows real-world execution, and the bottom row shows the synchronized 3D scene graph with target localization and the updated trajectory.
Figure 9: Comparison of semantic mapping completeness under rotating and fixed LiDAR scanning configurations.
Vision-and-Language Navigation (VLN) requires an embodied agent to ground complex natural-language instructions into long-horizon navigation in unseen environments. While Vision-Language Models (VLMs) offer strong 2D semantic understanding, current VLN systems remain constrained by limited spatial perception, 2D-3D representation mismatch, and monocular scale ambiguity. In this paper, we propose AgentVLN, a novel and efficient embodied navigation framework that can be deployed on edge computing platforms. We formulate VLN as a Partially Observable Semi-Markov Decision Process (POSMDP) and introduce a VLM-as-Brain paradigm that decouples high-level semantic reasoning from perception and planning via a plug-and-play skill library. To resolve multi-level representation inconsistency, we design a cross-space representation mapping that projects perception-layer 3D topological waypoints into the image plane, yielding pixel-aligned visual prompts for the VLM. Building on this bridge, we integrate a context-aware self-correction and active exploration strategy to recover from occlusions and suppress error accumulation over long trajectories. To further address the spatial ambiguity of instructions in unstructured environments, we propose a Query-Driven Perceptual Chain-of-Thought (QD-PCoT) scheme, enabling the agent with the metacognitive ability to actively seek geometric depth information. Finally, we construct AgentVLN-Instruct, a large-scale instruction-tuning dataset with dynamic stage routing conditioned on target visibility. Extensive experiments show that AgentVLN consistently outperforms prior state-of-the-art methods (SOTA) on long-horizon VLN benchmarks, offering a practical paradigm for lightweight deployment of next-generation embodied navigation models. Code: https://github.com/Allenxinn/AgentVLN.
Zihao Xin, Wentong Li, Yixuan Jiang +6
Nanjing University of Aeronautics and Astronautics · Shandong University · Zhejiang University
Vision-Language Navigation (VLN) in unseen indoor environments is useful in real-world robotics, where an agent must follow natural-language instructions, locate objects, and answer spatial questions without a pre-built map or fixed object vocabulary. Multimodal vision-language models (VLMs) provide strong open-vocabulary grounding and zero-shot reasoning, but struggle to emit reliable metric quantities such as range, bearing, and comparative spatial relations directly from images. Existing approaches address this by folding geometry into hand-engineered pipelines or asking models to output waypoints, requiring changes to the control stack for different robots, tasks, or vocabularies. We introduce AnchorVLN, an open-vocabulary VLN system built on a simple rule: the VLM proposes semantics; geometry decides metrics. It is realised as EMBODIED-NAV-MCP, a Model Context Protocol (MCP) server driven by a VLM agent through a compact set of callable tools. Since no tool accepts distance in metres or bearing in radians, the schema enforces the semantic-geometry boundary without modifying the downstream autonomy stack. We benchmark both tasks of the CMU Vision-Language Navigation Challenge 2026: 30 instruction-following questions over 15 scenes and a frozen 45-question object-reference set. The full system achieves 64.4 percent on instruction following, dropping by 13.3 percentage points without controller modeling (t = 2.77). On object reference, geometric anchoring clears the challenge overlap threshold on 10 of 45 questions, versus 0 of 45 for direct coordinate estimation, reducing median center error from 3.37 m to 2.48 m.
Vision-Language Navigation (VLN) enables embodied agents to reach target locations in unseen environments by following language instructions. Despite recent progress with vision-language models (VLMs), a critical semantic-geometric gap remains: while VLMs excel at language and 2D visual understanding, they struggle with 3D spatial reasoning and fail to capture the causal dynamics between actions and spatial transitions, resulting in unreliable navigation, particularly in zero-shot settings. To bridge this gap, we propose a Hierarchical Semantic-Geometric Map (HSGM) that transforms 3D geometric information into a structured representation compatible with VLMs, effectively linking them to the physical world. Specifically, HSGM is represented as a multi-channel top-down map organized into three levels: (1) geometric level that records navigable regions and obstacles, (2) semantic level that represents objects and their relations, and (3) decision level that supports high-level task reasoning and goal selection. During navigation, the VLM acts as a high-level semantic planner, interpreting the spatial layout encoded in the HSGM to select geometrically valid waypoints, while low-level, collision-free movements between waypoints are executed by a classical path-planning algorithm, fully decoupling semantic reasoning from action execution. Additionally, complex instructions are decomposed into subtasks to alleviate the problem of progress forgetting or hallucinating in long-horizon navigation. Extensive experiments on R2R-CE and RxR-CE benchmarks demonstrate that our zero-shot framework achieves state-of-the-art performance and even outperforms several supervised methods. Code is available at https://github.com/Teacher-Tom/HSGM_public.
Kailing Li, Tianwen Qian, Lijin Yang +4
School of Computer Science and Technology, East China Normal University · 2Bosch Corporate Research, Shanghai, China · 3King Abdullah University of Science and Technology