Talk2Escape: Conversational Grounding for Vision-and-Language Navigation

Zerui Li; Sihao Lin; Yanyan Shao; Jiwen Zhang; Xiangyu Shi; Shijie Li; Qi Wu

cs.ROSep 23, 2026

Talk2Escape: Conversational Grounding for Vision-and-Language Navigation

Authors: Zerui Li, Sihao Lin, Yanyan Shao, Jiwen Zhang, Xiangyu Shi, Shijie Li, Qi Wu

Organizations: Australian Institute for Machine Learning, Adelaide University · Responsible AI Research Centre, Australian Institute for Machine Learning, Adelaide University · Zhejiang Wanli University · Fudan University ·

Abstract

While Vision-and-Language Navigation (VLN) has demonstrated remarkable success, the prevailing single-turn paradigm exposes a fundamental vulnerability: agents operate in a strictly open-loop manner. In practice, factors such as perceptual aliasing, sensor noise, and odometry drift can cause minor deviations to accumulate over time, often leading to catastrophic mission failures with no built-in mechanism for error recovery. To address this, we introduce Talk2Escape, a proactive and model-agnostic dialogue intervention framework that reframes navigation as a closed-loop interactive process. At its core, a lightweight vision-language module continuously monitors agent kinematics. Upon detecting localized looping or severe trajectory divergence, it translates raw egocentric observations into concise, grounded queries to solicit targeted corrective feedback from either an algorithmic oracle or a human-in-the-loop. Extensive evaluations in high-fidelity simulators, including R2R-CE, RxR-CE, and VLNVerse, demonstrate that Talk2Escape exhibits consistent improvements across diverse base agents. Empirically, Talk2Escape achieves a 66.0% Success Rate on R2R-CE, outperforming the current supervised and zero-shot state-of-the-art methods. We further validate its sim-to-real transfer on a Unitree Go2 quadruped, proving that proactive dialogue drastically improves navigation robustness in physical environments.

Figures & tables

Fig. 1: Paradigm shift from single-turn to failure-aware dialogue-driven VLN: Left: Single-turn agents operate in an open-loop manner, making them architecturally brittle to physical uncertainties. Right: Our Talk2Escape framework introduces a closed-loop, proactive error-recovery mechanism, which actively solicits corrective feedback from a human guide, effectively rescuing the agent from severe real-world execution failures.

Fig. 2: Overview of the Talk2Escape Framework: ❶ Base navigation agents output spatial kinematics and action during execution. ❷ The Kinematic Monitor continuously evaluates these states. If cascading errors occur, it activates a Repeat ( $\mathcal{C}_{\mathrm{rep}}$ ) or Deviation ( $\mathcal{C}_{\mathrm{away}}$ ) trigger, proactively pausing the robot and sending the error state copy to the Navigation Assistance block. ❸ The Vision-Language Translator grounds current RGB-D observations into a concise, active query. ❹ A Oracle Guide responds with a targeted corrective hint based on relative bearing and distance. ❺ This entire dialogue exchange is injected into the MLLM prompt, empowering the agent to re-plan its actions and successfully escape the unrecoverable state.

Supervised Learning:
#	Methods	R2R-CE					RxR-CE
#	Methods	NE $\downarrow$	OSR $\uparrow$	SR $\uparrow$	SPL $\uparrow$	nDTW $\uparrow$	NE $\downarrow$	SR $\uparrow$	SPL $\uparrow$	nDTW $\uparrow$
1	CMA [ 32 ]	6.30	49.0	38.0	33.0	–	10.4	24.1	19.1	37.4
2	VLN-BERT [ 32 ]	5.74	53.0	44.0	39.0	–	8.98	27.1	22.7	46.7
3	GridMM [ 26 ]	5.11	61.0	49.0	41.0	–	–	–	–	–
4	ETPNav [ 7 ]	4.71	65.0	57.0	49.0	–	5.64	54.8	44.9	61.9

TABLE I: Comparison in continuous environments on R2R-CE and RxR-CE Val-Unseen splits. The best supervised results are highlighted in bold , while the best zero-shot results are underlined . We report metrics on sampled subsets for R2R-CE (100 episodes, following Open-Nav [ 4 ] ) and RxR-CE (260 episodes), following GTA [ 31 ] .

Prompting Strategy	NE ( $\downarrow$ )	SR ( $\uparrow$ )	OSR ( $\uparrow$ )	SPL ( $\uparrow$ )	nDTW ( $\uparrow$ )
NavGPT [ 2 ]	6.10	19.30	61.50	8.13	52.60
NavGPT [ 2 ] + CoT	5.25	28.70	50.50	9.91	44.10
NavGPT [ 2 ] + T2E	2.92	84.60	88.30	35.20	60.50
MapGPT [ 3 ]	5.62	25.53	55.32	7.57	43.38
MapGPT [ 3 ] + CoT	4.51	42.19	72.40	11.32	45.20
MapGPT [ 3 ] + T2E	4.14	67.02	74.47	30.46	53.62

TABLE II: Zero-Shot Performance on VLNVerse. Evaluation on the fine-grained instruction task, comparing standard open-loop agents against their passive Chain-of-Thought (CoT) variants and our proposed interactive T2E module.

Supervised Learning:
Method	SR $\uparrow$ (%)	NE $\downarrow$ (m)
VLN-BERT [ 32 ]	16.0	5.36
RDP [ 16 ]	20.0	5.45
Zero-Shot Learning:
SmartWay [ 28 ]	32.0	4.85
GTA	40.0	3.66

TABLE III: Real-world navigation performance. We compare Talk2Escape against representative supervised (VLN-BERT, RDP) and zero-shot (SmartWay) baselines.

\uparrow

indicates higher is better,

\downarrow

indicates lower is better.

Fig. 5: Real-World Execution of the Talk2Escape Paradigm. (Top) The agent misinterprets the instruction and navigates toward an incorrect door (red cross), prompting a human guide to provide a grounded corrective hint via dialogue. (Bottom) Empowered by this feedback, the agent successfully corrects its route, reaches the intended target (green checkmark), and receives final confirmation.

Vision-Language Navigation

Vision-Language Navigation studies how embodied agents use natural-language instructions and visual observations to plan and execute routes.

See all 313 papers

What Limits Vision-and-Language Navigation ?
May 13, 2026Yunheng Wang, Yuetong Fang, Taowen Wang +9Sim-to-Real Transfer Vision-Language Navigation
1HKUST(GZ) · 2JD Explore Academy · 1HKUST(GZ
NavHarness: Adaptive Goals for Agentic Vision-Language Navigation
Sep 30, 2026Haoxiang Shi, Zaijing Li, Muhe Ding +3Vision-Language Navigation Multimodal Agents
Harbin Institute of Technology (Shenzhen) · Pengcheng Laboratory
Trajectory-Diversity-Driven Robust Vision-and-Language Navigation
Mar 16, 2026Jiangyang Li, Cong Wan, SongLin Dong +4Vision-Language Navigation
Xi’an Jiaotong University · Faculty of Computility Microelectronics, Shenzhen University of Advanced Technology

May 13, 2026J/K move · Enter open · S save

What Limits Vision-and-Language Navigation ?

Yunheng Wang, Yuetong Fang, Taowen Wang, Lusong Li+8

1HKUST(GZ) · 2JD Explore Academy · 1HKUST(GZ

Sim-to-Real Transfer Vision-Language Navigation Vision-Language-Action Models

Open paper AskPDF arXiv

Vision-and-Language Navigation (VLN) is a cornerstone of embodied intelligence. However, current agents often suffer from significant performance degradation when transitioning from simulation to real-world deployment, primarily due to perceptual instability (e.g., lighting variations and motion blur) and under-specified instructions. While existing methods attempt to bridge this gap by scaling up model size and training data, we argue that the bottleneck lies in the lack of robust spatial grounding and cross-domain priors. In this paper, we propose StereoNav, a robust Vision-Language-Action framework designed to enhance real-world navigation consistency. To address the inherent gap between synthetic training and physical execution, we introduce Target-Location Priors as a persistent bridge. These priors provide stable visual guidance that remains invariant across domains, effectively grounding the agent even when instructions are vague. Furthermore, to mitigate visual disturbances like motion blur and illumination shifts, StereoNav leverages stereo vision to construct a unified representation of semantics and geometry, enabling precise action prediction through enhanced depth awareness. Extensive experiments on R2R-CE and RxR-CE demonstrate that StereoNav achieves state-of-the-art egocentric RGB performance, with SR and SPL scores of 81.1% and 68.3%, and 67.5% and 52.0%, respectively, while using significantly fewer parameters and less training data than prior scaling-based approaches. More importantly, real-world robotic deployments confirm that StereoNav substantially improves navigation reliability in complex, unstructured environments. Project page: https://yunheng-wang.github.io/stereonav-public.github.io.

Abstract

Vision-Language Navigation

Explore similar work

What Limits Vision-and-Language Navigation ?

NavHarness: Adaptive Goals for Agentic Vision-Language Navigation

Trajectory-Diversity-Driven Robust Vision-and-Language Navigation