OmniVLN: Omnidirectional 3D Perception and Token-Efficient LLM Reasoning for Visual-Language Navigation across Air and Ground Platforms

Zhongyuang Liu; Min He; Shaonan Yu; Xinhang Xu; Muqing Cao; Jianping Li; Jianfei Yang; Lihua Xie

cs.ROMar 18, 2026

OmniVLN: Omnidirectional 3D Perception and Token-Efficient LLM Reasoning for Visual-Language Navigation across Air and Ground Platforms

Authors: Zhongyuang Liu, Min He, Shaonan Yu, Xinhang Xu, Muqing Cao, Jianping Li, Jianfei Yang, Lihua Xie

Organizations: CertaintyX. · the School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore 639798. · the Robotics Institute, Carnegie Mellon University, Pittsburgh, PA 15213 USA. · the School of Mechanical and Aerospace Engineering, and jointly with the School of Electrical and Electronic Engineering at Nanyang Technological University, Singapore.

Abstract

Language-guided embodied navigation requires an agent to interpret object-referential instructions, search across multiple rooms, localize the referenced target, and execute reliable motion toward it. Existing systems remain limited in real indoor environments because narrow field-of-view sensing exposes only a partial local scene at each step, often forcing repeated rotations, delaying target discovery, and producing fragmented spatial understanding; meanwhile, directly prompting LLMs with dense 3D maps or exhaustive object lists quickly exceeds the context budget. We present OmniVLN, a zero-shot visual-language navigation framework that couples omnidirectional 3D perception with token-efficient hierarchical reasoning for both aerial and ground robots. OmniVLN fuses a rotating LiDAR and panoramic vision into a hardware-agnostic mapping stack, incrementally constructs a five-layer Dynamic Scene Graph (DSG) from mesh geometry to room- and building-level structure, and stabilizes high-level topology through persistent-homology-based room partitioning and hybrid geometric/VLM relation verification. For navigation, the global DSG is transformed into an agent-centric 3D octant representation with multi-resolution spatial attention prompting, enabling the LLM to progressively filter candidate rooms, infer egocentric orientation, localize target objects, and emit executable navigation primitives while preserving fine local detail and compact long-range memory. Experiments show that the proposed hierarchical interface improves spatial referring accuracy from 77.27% to 93.18%, reduces cumulative prompt tokens by up to 61.7% in cluttered multi-room settings, and improves navigation success by up to 11.68% over a flat-list baseline. We will release the code and an omnidirectional multimodal dataset to support reproducible research.

Figures & tables

Figure 1: OmniVLN, a zero-shot visual-language navigation (VLN) framework coupling $360^{\circ}$ 3D perception with token-efficient reasoning across aerial & ground platforms.

Figure 2: Overview of the proposed framework. The Multimodal Perception module (left) achieves $360^{\circ}$ spatio-temporal consistency by fusing data from a rotating LiDAR and panoramic fisheye cameras, enabling the generation of high-fidelity semantic point clouds across robotic platforms. The Hierarchical Representation module (center) incrementally constructs an online five-layer DSG, bridging the gap between low-level geometric places and high-level macro-spatial rooms, while employing a VLM-based hybrid pruning mechanism to refine physical veracity. The LLM-based Reasoning module (right) transforms global graph knowledge into an agent-centric 3D octant observation model, utilizing a closed-loop actor–critic framework and DSG-guided hierarchical prompting to translate natural language queries into executable navigation actions.

Figure 3: Omnidirectional projection model. A 3D point $P_{t}=(x,y,z)$ in the robot’s egocentric frame is mapped onto a unit sphere and projected to pixel coordinates $(u,v)$ of a $360^{\circ}$ panoramic image using the equirectangular projection model.

Figure 4: Visualization of the multi-modal perception in the IoT laboratory. The top row shows panoramic RGB observations from the agent’s perspective. The bottom row displays the corresponding 3D semantic point clouds with instantiated objects and their unique IDs, which serve as the raw input for the DSG hierarchy construction.

Representation	VI Accuracy	VD Accuracy	Overall
Non-Hierarchical (Uniform Flat)	86.36%	68.18%	77.27%
Hierarchical (Ours)	95.45%	90.91%	93.18%
Improvement	+9.09%	+22.73%	+15.91%

Table I: Comparative Accuracy of Referring Expression Generation

Figure 5: Multi-resolution spatial attention mechanism for token-efficient navigation. The central red diamond denotes the ego-agent. Semantic object categories are color-coded as follows: chair (blue), table (green), cabinet (orange).

Panel of Figure 6 — Figure 6: Synthetic datasets across the three spatial-attention tiers. Each scene represents a $10\text{m}\times 10\text{m}$ environment with varying object densities and distributions. Colors denote chair (blue), table (green), cabinet (orange), and screen (purple).

Spatial Condition	Foveal Tier (8-Octant)			Peripheral Tier			Global Memory Tier
Object Density	Low (5)	Mid (10)	High (15)	Low (5)	Mid (10)	High (15)	Low (15)	Mid (30)	High (50)
Dataset ID	$D_{1}$	$D_{2}$	$D_{3}$	$D_{4}$	$D_{5}$	$D_{6}$	$D_{7}$	$D_{8}$	$D_{9}$
Uniform Flat-List	402.60	746.80	1153.07	398.80	806.50	1181.60	1126.33	2089.33	3555.98
Ours	401.60	440.40	604.27	291.80	446.50	584.93	496.27	735.27	1067.62
Reduction (%)	0.25%	41.03%	47.59%	26.83%	44.64%	50.50%	55.94%	64.81%	69.98%

Table II: End-to-End Cumulative Token Consumption Across Different Spatial Conditions and Object Densities.

Figure 7: Navigation Success Rate (%) comparison between Uniform Flat-List and our method across nine datasets ( $D_{1}$ - $D_{9}$ ). Background colors indicate the three spatial attention tiers.

Figure 8: Qualitative zero-shot navigation results for the command “go to the nearest chair”. The top row shows real-world execution, and the bottom row shows the synchronized 3D scene graph with target localization and the updated trajectory.

Panel of Figure 9 — Figure 9: Comparison of semantic mapping completeness under rotating and fixed LiDAR scanning configurations.

Vision-Language Navigation

Vision-Language Navigation studies how embodied agents use natural-language instructions and visual observations to plan and execute routes.

See all 310 papers

AgentVLN: Towards Agentic Vision-and-Language Navigation
Mar 18, 2026Zihao Xin, Wentong Li, Yixuan Jiang +6Vision-Language Navigation Robot Navigation
Nanjing University of Aeronautics and Astronautics · Shandong University · Zhejiang University
AnchorVLN: Geometry-Anchored Vision-Language Grounding Reasoning for Open-Vocabulary Navigation
Sep 10, 2026Long Giang Vu, Chengkai Yao, Yuxin Liu +2Vision-Language Navigation Vision-Language Grounding
*These authors contributed equally.
Bridging the 2D-3D Gap: A Hierarchical Semantic-Geometric Map for Vision Language Navigation
May 25, 2026Kailing Li, Tianwen Qian, Lijin Yang +4VLN Benchmarks 3D Spatial Reasoning
School of Computer Science and Technology, East China Normal University · 2Bosch Corporate Research, Shanghai, China · 3King Abdullah University of Science and Technology

Mar 18, 2026J/K move · Enter open · S save

AgentVLN: Towards Agentic Vision-and-Language Navigation

Zihao Xin, Wentong Li, Yixuan Jiang, Ziyuan Huang+5

Nanjing University of Aeronautics and Astronautics · Shandong University · Zhejiang University

Vision-Language Navigation Robot Navigation VLMs for Robotics

Open paper AskPDF arXiv

Vision-and-Language Navigation (VLN) requires an embodied agent to ground complex natural-language instructions into long-horizon navigation in unseen environments. While Vision-Language Models (VLMs) offer strong 2D semantic understanding, current VLN systems remain constrained by limited spatial perception, 2D-3D representation mismatch, and monocular scale ambiguity. In this paper, we propose AgentVLN, a novel and efficient embodied navigation framework that can be deployed on edge computing platforms. We formulate VLN as a Partially Observable Semi-Markov Decision Process (POSMDP) and introduce a VLM-as-Brain paradigm that decouples high-level semantic reasoning from perception and planning via a plug-and-play skill library. To resolve multi-level representation inconsistency, we design a cross-space representation mapping that projects perception-layer 3D topological waypoints into the image plane, yielding pixel-aligned visual prompts for the VLM. Building on this bridge, we integrate a context-aware self-correction and active exploration strategy to recover from occlusions and suppress error accumulation over long trajectories. To further address the spatial ambiguity of instructions in unstructured environments, we propose a Query-Driven Perceptual Chain-of-Thought (QD-PCoT) scheme, enabling the agent with the metacognitive ability to actively seek geometric depth information. Finally, we construct AgentVLN-Instruct, a large-scale instruction-tuning dataset with dynamic stage routing conditioned on target visibility. Extensive experiments show that AgentVLN consistently outperforms prior state-of-the-art methods (SOTA) on long-horizon VLN benchmarks, offering a practical paradigm for lightweight deployment of next-generation embodied navigation models. Code: https://github.com/Allenxinn/AgentVLN.

Abstract

Vision-Language Navigation

Explore similar work

AgentVLN: Towards Agentic Vision-and-Language Navigation

AnchorVLN: Geometry-Anchored Vision-Language Grounding Reasoning for Open-Vocabulary Navigation

Bridging the 2D-3D Gap: A Hierarchical Semantic-Geometric Map for Vision Language Navigation