Revealing and Bridging the Physical Executability Gap in Vision-Language Navigation

Gang He; Zhile Cheng; Kepeng Xu; Zhenyang Liu; Yu-Wing Tai

cs.ROOct 19, 2026

Revealing and Bridging the Physical Executability Gap in Vision-Language Navigation

Authors: Gang He, Zhile Cheng, Kepeng Xu, Zhenyang Liu, Yu-Wing Tai

Organizations: Xidian University · Dartmouth College

Abstract

Continuous vision-and-language navigation (VLN) is typically evaluated by goal-reaching success, assuming it reflects an agent's ability to execute language-grounded behavior. We show that this assumption can fail: permissive contact models in current simulators may mask physically infeasible actions, creating a physical executability gap between semantic competence and embodied realization. To bridge this gap, we introduce Visual-Semantic Traversability Alignment (VeSTA), a framework that aligns VLN policies with physically executable futures. We define visual-semantic traversability as the ability to predict instruction-conditioned trajectories that are semantically valid, geometrically feasible, and physically robust. VeSTA addresses this misalignment through two complementary mechanisms: improving executable support in trajectory generation and risk-aware selection among multimodal futures. Specifically, Trajectory Relative Group Policy Optimization uses strict-contact feedback to reshape the candidate distribution toward executable regions, while the Traversability Risk Critic learns a risk-adjusted scalar utility for candidate-wise future selection. Experiments on R2R and RxR reveal substantial dependence of existing VLN agents on simulator contact semantics. Under strict-contact evaluation, VeSTA substantially reduces collision-induced failures while preserving or improving goal-reaching performance, and controlled real-world robot experiments provide evidence that the learned preference transfers beyond simulation.

Figures & tables

Figure 1: Conceptual illustration: averaging incompatible modes can yield a non-candidate trajectory that collides under strict contact; not all baselines use literal mean aggregation.

Figure 2: Overview of VeSTA. Trajectory Relative Group Policy Optimization uses strict-contact feedback to reshape candidate support toward executable regions. The Traversability Risk Critic learns a scalar utility from progress, contact, and immobilization preferences for candidate-wise receding-horizon selection.

Figure 3: Contact-semantics intervention. Existing agents degrade when obstacle-directed motion is blocked; VeSTA reduces collision burden through distribution alignment and risk-aware selection.

Method	NE $\downarrow$	OS $\uparrow$	SR $\uparrow$	SPL $\uparrow$	Coll. Avg. $\downarrow$	Coll. W. $\downarrow$
NaVid	6.48	30.46	25.49	23.16	18.20	15.80
NaVILA	6.62	38.39	28.55	26.43	35.18	59.62
UniNaVid	6.16	35.43	32.26	30.50	22.46	21.18
StreamVLN	6.19	37.74	30.30	27.59	39.03	67.85
JanusVLN	6.79	30.40	24.80	23.39	54.52	77.64
DualVLN	4.84	57.64	51.50	47.61	13.22	34.00

Table 1: Navigation and execution results on R2R Val-Unseen under strict contact semantics. Standard navigation metrics measure goal-reaching performance, while collision metrics characterize local execution reliability without environment-side contact correction. Collision metrics are percentages.

Method	NE $\downarrow$	OS $\uparrow$	SR $\uparrow$	SPL $\uparrow$	nDTW $\uparrow$	Coll. Avg. $\downarrow$	Coll. W. $\downarrow$
DualVLN (baseline)	6.78	49.06	41.54	35.99	59.13	21.79	39.17
VeSTA (Ours)	6.48	50.50	42.41	36.51	60.45	9.36	15.53

Table 2: Results on RxR Val-Unseen under strict contact semantics. The evaluation includes

N=3669

episodes with English instructions and follows the standard RxR-CE reporting convention. Collision metrics characterize local execution reliability; OS, SR, SPL, nDTW, and collision metrics are percentages.

Figure 5: Effect of Trajectory Relative Group Policy Optimization on the local trajectory distribution. Under the same instruction and scene, the original policy drifts into collision-prone views, whereas the aligned policy follows a geometrically feasible visual corridor, illustrating a shift toward the executable motion manifold.

Method	Contact Sem.	NE $\downarrow$	OS $\uparrow$	SR $\uparrow$	SPL $\uparrow$	Coll. Avg. $\downarrow$	Coll. W. $\downarrow$
StreamVLN	permissive	4.87	64.38	57.10	50.53	5.58	22.64
StreamVLN	strict	6.19	37.74	30.30	27.59	39.03	67.85
JanusVLN	permissive	4.96	63.30	58.02	54.53	8.17	25.04
JanusVLN	strict	6.79	30.40	24.80	23.39	54.52	77.64
DualVLN	permissive	4.03	70.96	65.31	59.08	2.47	9.03
DualVLN	strict	4.84	57.64	51.50	47.61	13.22	34.00

Table 3: Contact-semantics intervention on R2R Val-Unseen. Permissive contact can convert obstacle-directed motion into tangential displacement, whereas strict contact treats it as blocked execution. The performance difference between the two conditions diagnoses the physical executability gap. Collision metrics are percentages.

Variant	NE $\downarrow$	OS $\uparrow$	SR $\uparrow$	SPL $\uparrow$	Coll. Avg. $\downarrow$	Coll. W. $\downarrow$
DualVLN	4.84	57.64	51.50	47.61	13.22	34.00
+TR-GRPO	4.88	56.82	50.84	46.61	12.23	29.71
+TRiC	4.64	59.65	52.15	47.29	7.03	14.31
VeSTA (Ours)	4.63	59.38	53.56	48.93	6.08	12.73

Table 4: Component ablation under strict contact semantics. TR-GRPO shapes executable candidate support, TRiC performs candidate-level utility selection, and VeSTA combines both mechanisms. Collision metrics are percentages.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Comparison	$\Delta$ SR (95% CI)	$\Delta$ SPL (95% CI)
+TR-GRPO vs. DualVLN	$-0.65$ [ $-2.56,1.31$ ]	$-1.00$ [ $-2.73,0.74$ ]
+TRiC vs. DualVLN	$+0.65$ [ $-1.52,2.83$ ]	$-0.32$ [ $-2.23,1.64$ ]
VeSTA vs. DualVLN	$+2.07$ [ $-0.11,4.19$ ]	$+1.33$ [ $-0.63,3.27$ ]
Comparison	$\Delta$ CollAvg (95% CI)	$\Delta$ CollW (95% CI)
+TR-GRPO vs. DualVLN	$-0.99$ [ $-1.93,-0.06$ ]	$-4.29$ [ $-6.00,-2.61$ ]
+TRiC vs. DualVLN	$-6.20$ [ $-7.24,-5.19$ ]	$-19.69$ [ $-21.50,-17.85$ ]

Appendix

Table A1: Paired episode bootstrap on R2R Val-Unseen under strict contact. The upper panel reports navigation outcomes and the lower panel reports collision outcomes. Values are percentage-point method-minus-baseline differences.

Metric	Difference	Paired bootstrap 95% CI
SR	$+0.87$	$[-0.71,2.43]$
SPL	$+0.52$	$[-0.82,1.81]$
NE	$-0.304$	$[-0.496,-0.116]$
nDTW	$+1.32$	$[0.48,2.18]$
CollAvg	$-12.44$	$[-13.24,-11.62]$
CollW	$-23.64$	$[-24.63,-22.62]$

Appendix

Table A2: VeSTA-minus-DualVLN paired bootstrap on the RxR English Val-Unseen subset. SR, SPL, nDTW, and collision values are percentage-point differences; NE is measured in meters. NE uses 3,570 finite paired episodes, while the other metrics use all 3,669 episodes.

R2R group	Episodes	Share	$\Delta$ SR (95% CI)	$\Delta$ CollAvg (95% CI)
Doorway terms present	1,130	61.4%	$+1.95$ [ $-0.80,4.60$ ]	$-6.99$ [ $-8.30,-5.69$ ]
Doorway terms absent	709	38.6%	$+2.26$ [ $-1.27,5.92$ ]	$-7.39$ [ $-9.07,-5.79$ ]
Instruction $>30$ words	558	30.3%	$+2.87$ [ $-1.08,6.81$ ]	$-7.25$ [ $-9.06,-5.44$ ]
Instruction $\leq 30$ words	1,281	69.7%	$+1.72$ [ $-0.78,4.22$ ]	$-7.10$ [ $-8.29,-5.89$ ]
Contains straight	413	22.5%	$+4.84$ [ $0.48,9.20$ ]	$-4.81$ [ $-6.87,-2.75$ ]
Does not contain straight	1,426	77.5%	$+1.26$ [ $-1.26,3.79$ ]	$-7.82$ [ $-8.99,-6.65$ ]

Appendix

Table A3: VeSTA-minus-DualVLN paired differences by instruction-derived R2R groups. Values are percentage points. Categories are overlapping descriptive partitions, and intervals are exploratory without multiple-comparison correction.

Risk signal quartile	Episodes	SR	CollAvg
Collision-risk score Q1 (lowest)	105	84.76	0.70
Collision-risk score Q2	104	74.04	3.69
Collision-risk score Q3	105	49.52	3.87
Collision-risk score Q4 (highest)	105	22.86	19.19
Stuck-risk score Q1 (lowest)	105	90.48	3.31
Stuck-risk score Q2	104	80.77	3.91

Appendix

Table A4: Realized outcomes across quartiles of TRiC risk scores on the complete available rank-0 shard (419 episodes). Metrics are percentages. Endpoint separation supports descriptive risk stratification, but does not establish calibrated probabilities, causal effects, or optimal candidate selection.

Scene	DualVLN count, CFSR [Wilson 95% CI]	VeSTA count, CFSR [Wilson 95% CI]
Hallway	16/20, 80.0 $[58.4,91.9]$	19/20, 95.0 $[76.4,99.1]$
Bedroom	11/20, 55.0 $[34.2,74.2]$	13/20, 65.0 $[43.3,81.9]$
Room-to-Room	6/20, 30.0 $[14.5,51.9]$	9/20, 45.0 $[25.8,65.8]$
Pooled	33/60, 55.0 $[42.5,66.9]$	41/60, 68.3 $[55.8,78.7]$
Scene	$\Delta$ CFSR	Approx. 95% CI for difference
Hallway	$+15.0$	$[-5.0,35.0]$

Appendix

Table A5: Raw collision-free success counts, Wilson intervals for each method, and unpaired normal-approximation intervals for method differences. Percentages and differences are percentage points. The pooled row ignores scene heterogeneity and is descriptive.

Metric	TR-GRPO	TRiC	Additive	Full VeSTA	Residual
CollAvg	0.99	6.19	7.19	7.14	$-0.05$
CollW	4.29	19.69	23.98	21.27	$-2.71$

Appendix

Table A6: Component gains relative to DualVLN and the interaction residual, defined as the full VeSTA gain minus the sum of individual gains. All values are percentage points.

Data	Method	Mean	Median	$\geq 500$	Success	Failure
R2R	DualVLN	172.88	75	21.15	73.29	278.61
R2R	VeSTA	171.16	89	15.39	100.70	252.42
RxR	DualVLN	254.00	152	36.49	92.93	368.44
RxR	VeSTA	243.04	170	26.55	119.17	334.26

Appendix

Table A7: Episode length statistics. Mean, median, successful-episode, and failed-episode columns report steps;

\geq 500

is the percentage of episodes reaching the nominal maximum budget.

Failure mode	Relation to VeSTA	Remaining limitation
Semantic planning failure	Neither component directly corrects an incorrect high-level interpretation.	Requires stronger language grounding or global planning.
Candidate support failure	TR-GRPO is intended to shift generation toward executable local candidates.	Historical probes do not provide a baseline-versus-TR-GRPO coverage comparison.
Candidate selection failure	TRiC ranks available candidates using predicted execution risk and progress.	Logged data do not expose full counterfactual utility for every final-policy decision.
Local controller failure	Both components may reduce risky commands presented to the controller.	Neither changes low-level tracking dynamics or guarantees command feasibility.
Persistent-contact failure	Strict-contact training and risk-aware selection directly discourage repeated contact.	Minor isolated contacts may remain or be introduced on previously safe episodes.
Premature stopping	Positive-progress preferences can discourage non-progress candidates.	Goal-stop calibration is not separately evaluated.

Appendix

Table A8: Conceptual failure taxonomy and the evidenced scope of the two VeSTA mechanisms.

Data	Method	Median	P75	P90	P95	Free	$>10$	$>30$	$>50$
R2R	DualVLN	0.00	16.24	55.93	66.69	66.61	27.24	20.72	13.38
R2R	VeSTA	0.00	7.23	23.11	33.33	57.59	21.48	5.98	0.27
RxR	DualVLN	2.31	45.31	63.27	72.65	47.51	45.24	35.87	19.98
RxR	VeSTA	2.97	15.84	28.95	36.93	39.87	34.26	9.21	0.25

Appendix

Table A9: Episode-level collision-rate distribution. Quantiles, collision-free rate (Free), and threshold exceedance rates are percentages. The corresponding means are 13.22/6.08 on R2R and 21.79/9.36 on RxR for DualVLN/VeSTA.

Data	Baseline stratum	$N$	Range	Base	VeSTA	$\Delta$ SR
R2R	Zero	1225	0–0	0.00	3.97	$-5.22$
R2R	Positive T1	205	0.23–24.77	10.07	5.94	$-1.95$
R2R	Positive T2	205	24.80–53.49	41.23	10.11	$+23.90$
R2R	Positive T3	204	53.49–87.82	67.65	14.85	$+27.94$
RxR	Zero	1743	0–0	0.00	5.23	$-9.41$
RxR	Positive T1	642	0.20–31.48	13.94	9.65	$-6.23$

Appendix

Table A10: Outcomes stratified by baseline episode risk. Range, Base, and VeSTA are CollAvg percentages;

\Delta

SR is VeSTA minus DualVLN in percentage points.

Signal	Spearman	Kendall	Failure AUC	Any-coll. AUC	High-coll. AUC
Collision risk	0.655	0.532	0.794	0.876	0.860
Stuck risk	0.522	0.420	0.867	0.804	0.766

Appendix

Table A11: TRiC score ranking diagnostics on the historical 419-episode rank-0 shard. High collision denotes realized collision rate of at least 10%.

Vision-Language Navigation

Vision-Language Navigation studies how embodied agents use natural-language instructions and visual observations to plan and execute routes.

See all 310 papers

What Limits Vision-and-Language Navigation ?
May 13, 2026Yunheng Wang, Yuetong Fang, Taowen Wang +9Sim-to-Real Transfer Vision-Language Navigation
1HKUST(GZ) · 2JD Explore Academy · 1HKUST(GZ
AgentVLN: Towards Agentic Vision-and-Language Navigation
Mar 18, 2026Zihao Xin, Wentong Li, Yixuan Jiang +6Vision-Language Navigation Robot Navigation
Nanjing University of Aeronautics and Astronautics · Shandong University · Zhejiang University
NavHarness: Adaptive Goals for Agentic Vision-Language Navigation
Sep 30, 2026Haoxiang Shi, Zaijing Li, Muhe Ding +3Vision-Language Navigation Multimodal Agents
Harbin Institute of Technology (Shenzhen) · Pengcheng Laboratory

May 13, 2026J/K move · Enter open · S save

What Limits Vision-and-Language Navigation ?

Yunheng Wang, Yuetong Fang, Taowen Wang, Lusong Li+8

1HKUST(GZ) · 2JD Explore Academy · 1HKUST(GZ

Sim-to-Real Transfer Vision-Language Navigation Vision-Language-Action Models

Open paper AskPDF arXiv

Vision-and-Language Navigation (VLN) is a cornerstone of embodied intelligence. However, current agents often suffer from significant performance degradation when transitioning from simulation to real-world deployment, primarily due to perceptual instability (e.g., lighting variations and motion blur) and under-specified instructions. While existing methods attempt to bridge this gap by scaling up model size and training data, we argue that the bottleneck lies in the lack of robust spatial grounding and cross-domain priors. In this paper, we propose StereoNav, a robust Vision-Language-Action framework designed to enhance real-world navigation consistency. To address the inherent gap between synthetic training and physical execution, we introduce Target-Location Priors as a persistent bridge. These priors provide stable visual guidance that remains invariant across domains, effectively grounding the agent even when instructions are vague. Furthermore, to mitigate visual disturbances like motion blur and illumination shifts, StereoNav leverages stereo vision to construct a unified representation of semantics and geometry, enabling precise action prediction through enhanced depth awareness. Extensive experiments on R2R-CE and RxR-CE demonstrate that StereoNav achieves state-of-the-art egocentric RGB performance, with SR and SPL scores of 81.1% and 68.3%, and 67.5% and 52.0%, respectively, while using significantly fewer parameters and less training data than prior scaling-based approaches. More importantly, real-world robotic deployments confirm that StereoNav substantially improves navigation reliability in complex, unstructured environments. Project page: https://yunheng-wang.github.io/stereonav-public.github.io.

Abstract

Vision-Language Navigation

Explore similar work

What Limits Vision-and-Language Navigation ?

AgentVLN: Towards Agentic Vision-and-Language Navigation

NavHarness: Adaptive Goals for Agentic Vision-Language Navigation