PASS-VLN: Passability-Aware Local-Command Vision-and-Language Navigation

Cong Zou; Xiangsheng Shi; Xuanhui Wang; Fubin Wang; Guangjun Ge; Chao Wang; Yu Wang

cs.ROOct 16, 2026

PASS-VLN: Passability-Aware Local-Command Vision-and-Language Navigation

Authors: Cong Zou, Xiangsheng Shi, Xuanhui Wang, Fubin Wang, Guangjun Ge, Chao Wang, Yu Wang

Organizations: Qiyuan Lab, Beijing, China. · Department of Electronic Engineering, Tsinghua University, Beijing, China.

Abstract

Reliable vision-and-language navigation (VLN) requires not only understanding instructions but also determining whether and how a robot can physically execute them. However, existing VLN benchmarks typically assume executable instructions and focus on long-horizon tasks. This leaves body-dependent local execution, including passability and collision-free execution, largely unexamined. Evaluating these factors within long-horizon tasks would entangle local physical execution with task planning and long-term spatial memory, making navigation failures difficult to attribute. We therefore focus on grounding local navigation commands in executable motion under environmental and robot body constraints. We present PASS-VLN, a passability-aware local-command VLN benchmark and model. The benchmark holds the scene, instruction, start pose, and reference route fixed while varying robot body width, requiring target completion when the instructed passage is feasible and stopping before blockage otherwise. Our model encodes body width through obstacle inflation in a depth-derived local bird's-eye view, fuses geometry with visual and linguistic context, and predicts discrete latent action codes for continuous closed-loop execution. Experiments show that PASS-VLN improves collision-free success by approximately 12 percentage points over a strong VLN model fine-tuned on our benchmark, while adapting reach-versus-stop behavior to body constraints and generalizing to unseen body widths. When integrated with a high-level planner, PASS-VLN achieves higher success with fewer local commands, while real-world deployment further demonstrates its effectiveness as a low-level executor. The benchmark and model will be released.

Figures & tables

Fig. 1: Construction pipeline and statistics of the PASS-VLN benchmark. Controlled local trajectories are paired with multimodal evidence to generate commands with different spatial-context requirements, followed by body-width-aware passability annotation.

Fig. 2: Overview of the PASS-VLN architecture. Language, visual history, and body-width-conditioned geometric features are fused to predict latent actions, which are decoded into planar target poses for closed-loop velocity control.

		Map-informed						Map-free
Executor	Adaptation	SR $\uparrow$	OSR $\uparrow$	SPL $\uparrow$	NE $\downarrow$	CFR $\uparrow$	CFSR $\uparrow$	SR $\uparrow$	OSR $\uparrow$	SPL $\uparrow$	NE $\downarrow$	CFR $\uparrow$	CFSR $\uparrow$
StreamVLN	ZS	$29.1$	$79.8$	$23.8$	$3.62$	$43.2$	$15.6$	$31.3$	$79.8$	$27.1$	$3.54$	$47.2$	$17.6$
DualVLN	ZS	$29.3$	$68.7$	$23.9$	$3.74$	$66.2$	$23.6$	$35.6$	$73.3$	$30.2$	$3.46$	$69.8$	$27.5$
JanusVLN	ZS	$41.3$	$88.1$	$35.4$	$3.33$	$47.4$	$24.6$	$44.2$	$86.1$	$39.5$	$3.48$	$56.4$	$30.7$
StreamVLN	FT	$77.7$	$89.9$	$72.7$	$1.14$	$81.1$	$64.4$	$82.5$	$89.2$	$81.9$	$1.17$	$76.1$	$63.8$
PASS-VLN	FT	$\mathbf{83.4}$	$\mathbf{93.7}$	$\mathbf{75.2}$	$\mathbf{1.02}$	$\mathbf{90.6}$	$\mathbf{76.1}$	$\mathbf{85.2}$	$\mathbf{91.9}$	$\mathbf{83.0}$	$\mathbf{1.06}$	$\mathbf{89.1}$	$\mathbf{76.0}$

TABLE I: Local-command navigation performance with a fixed

0.35\,\mathrm{m}

body width. ZS evaluates released VLN checkpoints without PASS-VLN-specific training. StreamVLN-FT serves as the controlled task-adapted baseline.

		Map-informed							Map-free
Width	Method	Imp.%	SR $\uparrow$	GSR $\uparrow$	BSR $\uparrow$	CFR $\uparrow$	CFSR $\uparrow$	NE $\downarrow$	Imp.%	SR $\uparrow$	GSR $\uparrow$	BSR $\uparrow$	CFR $\uparrow$	CFSR $\uparrow$	NE $\downarrow$
$0.20$ (U)	PASS-VLN	$1.9$	$83.3$	$83.3$	$83.1$	$\mathbf{97.2}$	$\mathbf{81.1}$	$\mathbf{1.01}$	$1.8$	$\mathbf{85.2}$	$\mathbf{85.2}$	$\mathbf{84.9}$	$\mathbf{97.3}$	$\mathbf{83.6}$	$\mathbf{1.05}$
$0.20$ (U)	GT Passability-Gated VLN	$1.9$	$\mathbf{83.6}$	$\mathbf{83.6}$	$\mathbf{87.7}$	$94.4$	$80.0$	$1.03$	$1.8$	$84.4$	$84.4$	$\mathbf{84.9}$	$96.7$	$82.7$	$1.10$
$0.35$ (S)	PASS-VLN	$3.4$	$\mathbf{83.4}$	$\mathbf{83.4}$	$82.4$	$\mathbf{90.6}$	$\mathbf{76.1}$	$\mathbf{1.02}$	$2.2$	$\mathbf{85.2}$	$\mathbf{85.4}$	$78.5$	$\mathbf{89.1}$	$\mathbf{76.0}$	$\mathbf{1.06}$
$0.35$ (S)	GT Passability-Gated VLN	$3.4$	$82.7$	$82.6$	$\mathbf{83.2}$	$83.7$	$71.5$	$1.07$	$2.2$	$84.7$	$84.8$	$\mathbf{80.0}$	$85.4$	$73.8$	$1.11$
$0.50$ (U)	PASS-VLN	$10.6$	$81.7$	$81.7$	$81.5$	$\mathbf{86.4}$	$\mathbf{68.7}$	$\mathbf{1.03}$	$17.7$	$83.4$	$83.8$	$81.6$	$\mathbf{84.2}$	$\mathbf{69.9}$	$\mathbf{1.06}$

TABLE II: Body-width generalization on seen (S) and unseen (U) widths. GT Passability-Gated VLN uses a narrow (

0.10\,\mathrm{m}

) navigator with simulator-provided passability.

Width Pair	Pair Num.	Narrow GSR $\uparrow$	Wide BSR $\uparrow$	PSR $\uparrow$
$0.20\ \&\ 0.35\,\mathrm{m}$	$87$	$65.5$	$80.5$	$64.4$
$0.20\ \&\ 0.50\,\mathrm{m}$	$787$	$81.3$	$81.8$	$69.8$
$0.20\ \&\ 0.70\,\mathrm{m}$	$2{,}451$	$82.0$	$89.4$	$75.0$
$0.35\ \&\ 0.50\,\mathrm{m}$	$723$	$83.3$	$83.0$	$73.0$
$0.35\ \&\ 0.70\,\mathrm{m}$	$2{,}383$	$84.1$	$89.5$	$76.9$
$0.50\ \&\ 0.70\,\mathrm{m}$	$1{,}683$	$80.8$	$88.7$	$73.3$

TABLE III: Paired body-width evaluation on routes that are passable for the narrower body and impassable for the wider body.

Executor	SR $\uparrow$	OSR $\uparrow$	SPL $\uparrow$	NE $\downarrow$	CFR $\uparrow$	CFSR $\uparrow$	Avg. Cmds. $\downarrow$
StreamVLN-ZS	$90.4$	$91.7$	$53.9$	$2.51$	$46.4$	$45.6$	$5.18$
DualVLN-ZS	$90.4$	$91.4$	$52.4$	$2.46$	$63.6$	$62.6$	$5.33$
JanusVLN-ZS	$94.1$	$94.3$	$69.6$	$2.18$	$57.9$	$57.7$	$4.10$
StreamVLN-FT	$95.0$	$95.0$	$74.4$	$1.94$	$62.2$	$61.8$	$4.32$
PASS-VLN	$\mathbf{99.5}$	$\mathbf{99.6}$	$\mathbf{75.6}$	$\mathbf{1.71}$	$\mathbf{68.8}$	$\mathbf{68.3}$	$\mathbf{3.72}$

TABLE IV: Controlled hierarchical execution on R2R-CE using the same high-level VLM and reference-path planner.

Input / Method	Action	Train Widths	Width Cond.	Seen Width ( $\mathbf{0.35}$ m)			Seen PSR $\uparrow$	Unseen PSR $\uparrow$
Input / Method	Action	Train Widths	Width Cond.	SR $\uparrow$	CFR $\uparrow$	CFSR $\uparrow$	$\mathbf{0.35\&0.70}$ m	$\mathbf{0.20\&0.50}$ m
RGB (StreamVLN-FT)	Atomic	0.35	None	77.7	81.1	64.4	–	–
RGB	Latent	0.35	None	79.4	88.1	71.4	–	–
RGB + Depth	Latent	0.35	None	78.5	90.3	72.7	–	–
RGB + Raw BEV	Latent	0.35	None	79.7	88.9	72.0	–	–
RGB + Raw BEV + Width Prompt	Latent	0.35 + 0.70	Prompt	81.4	89.7	74.0	73.4	11.7

TABLE V: Ablation of action representation, geometric input, and width conditioning. Single-width variants are evaluated under the seen 0.35 m embodiment, while width-conditioned variants are additionally evaluated on seen and unseen body-width pairs.

Fig. 3: Real-world deployment on a Unitree Go2-W, demonstrating width-dependent traversal, blockage stopping, corridor selection, and sequential blockage detection.

Fig. 4: Real-world local-command execution in indoor and outdoor environments, where sequential map-informed commands support long-range navigation and map-free commands test local spatial grounding.

Robot Navigation

Robot navigation studies how robots perceive, localize, plan, and move safely toward goals in static or changing environments.

See all 557 papers

Offline Vision-Language Navigation with Geometric Goal Localization for Outdoor Environments
Jul 24, 2026Ali Salmasi, Xianjia Yu, Tomi WesterlundSmall Language Models Robotics
Turku Intelligent Embedded and Robotic Systems (TIERS) Lab, University of Turku, Turku, Finland.
Revealing and Bridging the Physical Executability Gap in Vision-Language Navigation
Oct 19, 2026Gang He, Zhile Cheng, Kepeng Xu +2Vision-Language Navigation RL for VLMs
Xidian University · Dartmouth College
VLN-NF: Feasibility-Aware Vision-and-Language Navigation with False-Premise Instructions
Apr 12, 2026Hung-Ting Su, Ting-Jun Wang, Jia-Fong Yeh +2VLN Benchmarks Vision-Language Navigation
National Taiwan University · DRIC · National Tsing Hua University

Jul 24, 2026J/K move · Enter open · S save

Offline Vision-Language Navigation with Geometric Goal Localization for Outdoor Environments

Ali Salmasi, Xianjia Yu, Tomi Westerlund

Turku Intelligent Embedded and Robotic Systems (TIERS) Lab, University of Turku, Turku, Finland.

Small Language Models Robotics Vision-Language Navigation

Open paper AskPDF arXiv

Foundation-model-based vision-language navigation (VLN) has advanced autonomous robot navigation by enabling robots to interpret natural-language instructions, identify semantic goals, and follow user-specified behavioral rules. However, existing VLN systems rely heavily on cloud-hosted foundation models for language understanding and semantic grounding, limiting their applicability where network connectivity is unavailable and reliable metric goal localization is required. Although recent small language models (SLMs) enable fully onboard inference, their suitability for navigation instruction decomposition has not been systematically evaluated. This paper makes three contributions toward fully onboard VLN for outdoor environments. First, we present the first systematic benchmark of 17 edge-deployable SLMs against 4 online APIs for robotic navigation instruction decomposition, evaluating accuracy and latency on human-annotated instructions across three computing platforms and providing practical guidance for selecting onboard language models. Second, we propose a lightweight hybrid semantic-geometric goal localization framework that combines open-vocabulary object detection, prompted segmentation, and LiDAR geometry to estimate metric goals, while maintaining visual bearing guidance when reliable geometric observations are unavailable. Third, we integrate these advances into Edge-BehAV, a fully onboard extension of the BehAV architecture that enables cloud-independent behavior-guided navigation. Experimental results show that the best offline SLM matches the instruction decomposition performance of the strongest cloud API while running approximately 9x faster and without network connectivity. The proposed goal localization framework reduces mean goal-distance error from 2.05 m to 0.20 m at lower computational cost, and the complete system succeeds in 31 of 32 closed-loop outdoor trials.

Abstract

Robot Navigation

Explore similar work

Offline Vision-Language Navigation with Geometric Goal Localization for Outdoor Environments

Revealing and Bridging the Physical Executability Gap in Vision-Language Navigation

VLN-NF: Feasibility-Aware Vision-and-Language Navigation with False-Premise Instructions