LightZeroNav: Zero-Shot Vision Language Navigation in Continuous Environments Based on Lightweight VLMs

Kun Luo; Xiangyu Dong; Xiaoguang Ma; Haoran Zhao; Yaoming Zhou

cs.CVMar 16, 2026

LightZeroNav: Zero-Shot Vision Language Navigation in Continuous Environments Based on Lightweight VLMs

Authors: Kun Luo, Xiangyu Dong, Xiaoguang Ma, Haoran Zhao, Yaoming Zhou

Organizations: Foshan Graduate School of Innovation, Northeastern University, Foshan 528311, China · Faculty of Robot Science and Engineering, Northeastern University, Shenyang 110819, China · QingniaoAI, China · School of Aeronautic Science and Engineering, Beihang University, Beijing 100191, China

Abstract

Although vision-language navigation (VLN) has progressed rapidly, zero-shot VLN in continuous environments (VLN-CE) remains highly challenging when using lightweight vision-language models (VLMs), whose limited reasoning capacity makes long-horizon navigation unreliable. In this paper, we propose LightZeroNav to tackle the three major bottlenecks when using lightweight VLMs in zero-shot VLN-CE,i.e.,information redundancy from multi-source inputs, inaccurate progress estimation caused by noisy textual memory, and task entanglement between action execution and stage transition. Using only RGB observations and a lightweight open-source Qwen3-VL-8B backbone, LightZeroNav achieves competitive performance with GPT-4o (~200B) without task-specific training, graph search, or waypoint predictors, demonstrating its effectiveness in zero-shot VLN-CE.

Figures & tables

Fig. 1: Overview of the proposed LightZeroNav framework, which enables zero-shot continuous navigation through collaborative optimization across information redundancy, progress estimation inaccuracy, and task entanglement. At the information redundancy level, Decision-guided Multi-Feature Filtering (DGMF) filters multi-source observations and trajectory cues according to the current decision objective, retaining only decision-relevant evidence for efficient reasoning. At the progress estimation inaccuracy level, Image-level State Grounding Representation (ISGR) combines short-term trajectory observations (STTB) with a visual baseline (VB) to enable implicit progress estimation and positional grounding via image-level comparative reasoning. At the task entanglement level, Stage-wise Execution–Transition Decoupling (SETD) decomposes long-horizon instructions into ordered sub-goals and separates inference into an execution stage for local action rollout under local FOV and a transition stage for sub-goal verification and global progress control under panoramic FOV, mitigating sub-goal coupling and decision ambiguity for lightweight VLMs.

Fig. 2: The prompt-based implementation of the Decision-guided Multi-Feature Filtering(DGMF). It presents three structured prompt designs for execution, rollout summary, and transition, and an actual execution-stage output example showing how decision-relevant evidence is organized into structured reasoning traces.

Method	Backbone / Size	Map / Prior	WP	View	TL	NE $\downarrow$	OSR $\uparrow$	SR $\uparrow$	SPL $\uparrow$
MapGPT-CE [ 17 ]	GPT-4o ( $\sim$ 200B)	Online map	No	Panoramic	12.63	8.16	22.0	7.0	5.04
DiscussNav [ 13 ]	GPT-4 ( $\sim$ 1.76T)	No	No	Panoramic	6.27	7.77	15.0	11.0	10.51
InstructNav [ 11 ]	GPT-4 ( $\sim$ 1.76T)	Value maps	No	Egocentric	7.74	6.89	–	31.0	24.0
Open-Nav [ 13 ]	Llama3.1-70B	No	Yes	Panoramic	8.07	7.25	23.0	16.0	12.90
Open-Nav [ 13 ]	GPT-4 ( $\sim$ 1.76T)	No	Yes	Panoramic	7.68	6.70	23.0	19.0	16.10
CA-Nav [ 12 ]	GPT-4 ( $\sim$ 1.76T)	Constraint prior	No	Egocentric	–	8.32	39.4	23.0	11.0

TABLE I: Comparison with representative zero-shot VLN-CE methods.

Variant	Dual FOV	DGMF	ISGR	SR $\uparrow$	OSR $\uparrow$	SPL $\uparrow$	nDTW $\uparrow$	Steps $\downarrow$	NE $\downarrow$	TL $\downarrow$	Coll. $\downarrow$
Full	✓	✓	✓	$\mathbf{30.0\pm 2.0}$	$45.0\pm 3.0$	$\mathbf{20.0\pm 2.0}$	$\mathbf{18.0\pm 1.0}$	$\mathbf{138.8\pm 6.4}$	$\mathbf{8.32\pm 0.07}$	$\mathbf{18.30\pm 1.06}$	$0.27\pm 0.00$
w/o DGMF	✓		✓	$12.0\pm 6.0$	$33.0\pm 3.0$	$6.0\pm 2.0$	$8.0\pm 2.0$	$170.1\pm 3.2$	$10.19\pm 1.03$	$23.46\pm 1.27$	$\mathbf{0.24\pm 0.01}$
w/o ISGR	✓	✓		$17.0\pm 2.0$	$52.0\pm 2.0$	$6.0\pm 1.0$	$4.0\pm 1.0$	$220.9\pm 6.3$	$10.37\pm 0.70$	$31.39\pm 1.31$	$0.30\pm 0.01$
Dual FOV Only	✓			$14.0\pm 3.0$	$35.0\pm 2.0$	$7.0\pm 1.0$	$6.0\pm 1.0$	$189.6\pm 2.8$	$11.48\pm 0.64$	$25.37\pm 0.52$	$0.27\pm 0.01$
w/o Dual FOV		✓	✓	$23.0\pm 5.0$	$\mathbf{55.0\pm 2.0}$	$8.0\pm 2.0$	$4.0\pm 1.0$	$217.1\pm 5.0$	$9.23\pm 0.39$	$27.94\pm 0.90$	$0.33\pm 0.01$

TABLE II: Module ablation of LightZeroNav with the 8B backbone.

Variant	Dec.	Exe.	Trans.	SR $\uparrow$	OSR $\uparrow$	SPL $\uparrow$	nDTW $\uparrow$	Steps $\downarrow$	NE $\downarrow$	TL $\downarrow$	Coll. $\downarrow$
Full	✓	✓	✓	$\mathbf{30.0\pm 2.0}$	$\mathbf{45.0\pm 3.0}$	$\mathbf{20.0\pm 2.0}$	$18.0\pm 1.0$	$138.8\pm 6.4$	$\mathbf{8.32\pm 0.07}$	$18.30\pm 1.06$	$0.27\pm 0.00$
w/o Transition	✓	✓		$11.0\pm 3.0$	$21.0\pm 3.0$	$10.0\pm 2.0$	$34.0\pm 1.0$	$43.8\pm 6.6$	$8.53\pm 0.06$	$7.28\pm 1.17$	$0.18\pm 0.01$
Execution Only		✓		$8.0\pm 3.0$	$10.0\pm 3.0$	$7.0\pm 3.0$	$\mathbf{36.0\pm 1.0}$	$\mathbf{24.3\pm 4.0}$	$8.42\pm 0.41$	$\mathbf{3.83\pm 0.54}$	$\mathbf{0.16\pm 0.01}$

TABLE III: Ablation of the staged execution structure with the 8B backbone.

Fig. 3: Qualitative comparison of reasoning traces between LightZeroNav and standard CoT in VLN-CE. The grey panels (left) illustrate the structured reasoning of LightZeroNav, where green text highlights its accurate identification of spatial transitions and decision-relevant anchors. In contrast, the pink panels (right) depict the reasoning path of standard CoT under the same scenario, where red text denotes inaccurate grounding of environmental boundaries and erroneous progress estimations. Superscripts denote specific reasoning functions: (a) static self-location, (b) dynamic progress grounding via ISGR, (c) decision-relevant evidence extraction, (d) goal-aligned task anticipation, (e) execution outcome, and (f) reasoning explosion because of information redundancy. The comparison demonstrates how structured framework of LightZeroNav mitigates the logical drift common in standard CoT.

Vision-Language Navigation

Vision-Language Navigation studies how embodied agents use natural-language instructions and visual observations to plan and execute routes.

See all 296 papers

AgenticNav: Zero-Shot Vision-and-Language Navigation as a Tool-Calling Harness
Jun 9, 2026Yijian Li, Changze Li, Hantian Shi +4Vision-Language Navigation Tool-Using Vision-Language Agents
Shanghai Jiao Tong University · Huawei Technologies Ltd.
One MLLM, One Call: Efficient Zero-Shot Vision-and-Language Navigation via Spatial-Aware Waypoints
Sep 6, 2026Shiqi Pan, Qi Zheng, Hanqin Sun +3Vision-Language Navigation VLMs for Robotics
College of Electronics and Information Engineering, Shenzhen University, Shenzhen 518060, China · School of Computer Science, The University of Sydney, Darlington 2008, Australia · College of Computer Science and Software Engineering, Shenzhen University, Shenzhen 518060, China
C $^2$ Nav: Compare Before You Commit for Zero-Shot Vision-and-Language Navigation
Sep 14, 2026Runtian Zheng, Congpeng Zhang, Ying LiuVision-Language Navigation VLM Reasoning
North China University of Technology, Beijing, China

Jun 9, 2026J/K move · Enter open · S save

AgenticNav: Zero-Shot Vision-and-Language Navigation as a Tool-Calling Harness

Yijian Li, Changze Li, Hantian Shi, Jiaying Luo+3

Shanghai Jiao Tong University · Huawei Technologies Ltd.

Vision-Language Navigation Tool-Using Vision-Language Agents Memory-Augmented Agents

Open paper AskPDF arXiv

Zero-shot vision-and-language navigation in continuous environments (VLN-CE) has recently become feasible with large vision-language models (VLMs). However, existing methods typically rely on learned waypoint predictors to propose navigable actions. This severely limits the model's action space and fails to leverage depth inputs effectively. Moreover, memory is commonly handled by accumulating long textual or visual histories with substantial irrelevant context, or by retrieving cross-episode experiences, which weakens the zero-shot setting. In this paper, we rethink zero-shot VLN-CE as an agentic interface between the VLM and the environment, and present AgenticNav, a lightweight navigation harness that exposes action, depth, and memory as callable tools. Instead of choosing from predicted waypoints, the action tool allows the VLM to directly select a target pixel in RGB observations, converting it into executable motion. Depth is exposed through an on-demand pixel-depth tool, enabling the VLM to request precise metric distances only where they matter. For memory, AgenticNav provides a compact map image summarizing the historical trajectory, paired with a recall tool that allows the VLM to selectively revisit past visual observations without overwhelming the prompt context. On the R2R-CE benchmark, AgenticNav establishes new state-of-the-art (SOTA) performance among zero-shot methods given the same VLM backbone. Real-world validation further highlights its zero-shot generalization compared to prior methods. Ablations show that our action tool design outperforms traditional waypoint predictors, and that depth tool and agentic memory further contribute to navigation performance.

Abstract

Vision-Language Navigation

Explore similar work

AgenticNav: Zero-Shot Vision-and-Language Navigation as a Tool-Calling Harness

One MLLM, One Call: Efficient Zero-Shot Vision-and-Language Navigation via Spatial-Aware Waypoints

C2^22Nav: Compare Before You Commit for Zero-Shot Vision-and-Language Navigation

C $^2$ Nav: Compare Before You Commit for Zero-Shot Vision-and-Language Navigation