Organizations: Australian Institute for Machine Learning, Adelaide University · Responsible AI Research Centre, Australian Institute for Machine Learning, Adelaide University · Zhejiang Wanli University · Fudan University · Agency for Science, Technology and Research (A*STAR)
While Vision-and-Language Navigation (VLN) has demonstrated remarkable success, the prevailing single-turn paradigm exposes a fundamental vulnerability: agents operate in a strictly open-loop manner. In practice, factors such as perceptual aliasing, sensor noise, and odometry drift can cause minor deviations to accumulate over time, often leading to catastrophic mission failures with no built-in mechanism for error recovery. To address this, we introduce Talk2Escape, a proactive and model-agnostic dialogue intervention framework that reframes navigation as a closed-loop interactive process. At its core, a lightweight vision-language module continuously monitors agent kinematics. Upon detecting localized looping or severe trajectory divergence, it translates raw egocentric observations into concise, grounded queries to solicit targeted corrective feedback from either an algorithmic oracle or a human-in-the-loop. Extensive evaluations in high-fidelity simulators, including R2R-CE, RxR-CE, and VLNVerse, demonstrate that Talk2Escape exhibits consistent improvements across diverse base agents. Empirically, Talk2Escape achieves a 66.0% Success Rate on R2R-CE, outperforming the current supervised and zero-shot state-of-the-art methods. We further validate its sim-to-real transfer on a Unitree Go2 quadruped, proving that proactive dialogue drastically improves navigation robustness in physical environments.
Figures & tables
Fig. 1: Paradigm shift from single-turn to failure-aware dialogue-driven VLN: Left: Single-turn agents operate in an open-loop manner, making them architecturally brittle to physical uncertainties. Right: Our Talk2Escape framework introduces a closed-loop, proactive error-recovery mechanism, which actively solicits corrective feedback from a human guide, effectively rescuing the agent from severe real-world execution failures.
Fig. 2: Overview of the Talk2Escape Framework: ❶ Base navigation agents output spatial kinematics and action during execution. ❷ The Kinematic Monitor continuously evaluates these states. If cascading errors occur, it activates a Repeat ( Crep ) or Deviation ( Caway ) trigger, proactively pausing the robot and sending the error state copy to the Navigation Assistance block. ❸ The Vision-Language Translator grounds current RGB-D observations into a concise, active query. ❹ A Oracle Guide responds with a targeted corrective hint based on relative bearing and distance. ❺ This entire dialogue exchange is injected into the MLLM prompt, empowering the agent to re-plan its actions and successfully escape the unrecoverable state.
#
Methods
R2R-CE
RxR-CE
NE ↓
OSR ↑
SR ↑
SPL ↑
nDTW ↑
NE ↓
SR ↑
SPL ↑
nDTW ↑
Supervised Learning:
1
CMA [ 32 ]
6.30
49.0
38.0
33.0
–
10.4
24.1
19.1
37.4
2
VLN-BERT [ 32 ]
5.74
53.0
44.0
39.0
–
8.98
27.1
22.7
46.7
3
GridMM [ 26 ]
5.11
61.0
49.0
41.0
–
–
–
–
–
4
ETPNav [ 7 ]
4.71
65.0
57.0
49.0
–
5.64
54.8
44.9
61.9
TABLE I: Comparison in continuous environments on R2R-CE and RxR-CE Val-Unseen splits. The best supervised results are highlighted in bold , while the best zero-shot results are underlined . We report metrics on sampled subsets for R2R-CE (100 episodes, following Open-Nav [ 4 ] ) and RxR-CE (260 episodes), following GTA [ 31 ] .
Prompting Strategy
NE ( ↓ )
SR ( ↑ )
OSR ( ↑ )
SPL ( ↑ )
nDTW ( ↑ )
NavGPT [ 2 ]
6.10
19.30
61.50
8.13
52.60
NavGPT [ 2 ] + CoT
5.25
28.70
50.50
9.91
44.10
NavGPT [ 2 ] + T2E
2.92
84.60
88.30
35.20
60.50
MapGPT [ 3 ]
5.62
25.53
55.32
7.57
43.38
MapGPT [ 3 ] + CoT
4.51
42.19
72.40
11.32
45.20
MapGPT [ 3 ] + T2E
4.14
67.02
74.47
30.46
53.62
TABLE II: Zero-Shot Performance on VLNVerse. Evaluation on the fine-grained instruction task, comparing standard open-loop agents against their passive Chain-of-Thought (CoT) variants and our proposed interactive T2E module.
Method
SR ↑ (%)
NE ↓ (m)
Supervised Learning:
VLN-BERT [ 32 ]
16.0
5.36
RDP [ 16 ]
20.0
5.45
Zero-Shot Learning:
SmartWay [ 28 ]
32.0
4.85
GTA
40.0
3.66
TABLE III: Real-world navigation performance. We compare Talk2Escape against representative supervised (VLN-BERT, RDP) and zero-shot (SmartWay) baselines. ↑ indicates higher is better, ↓ indicates lower is better.
Figure 6
Fig. 5: Real-World Execution of the Talk2Escape Paradigm. (Top) The agent misinterprets the instruction and navigates toward an incorrect door (red cross), prompting a human guide to provide a grounded corrective hint via dialogue. (Bottom) Empowered by this feedback, the agent successfully corrects its route, reaches the intended target (green checkmark), and receives final confirmation.
Vision-and-Language Navigation (VLN) is a cornerstone of embodied intelligence. However, current agents often suffer from significant performance degradation when transitioning from simulation to real-world deployment, primarily due to perceptual instability (e.g., lighting variations and motion blur) and under-specified instructions. While existing methods attempt to bridge this gap by scaling up model size and training data, we argue that the bottleneck lies in the lack of robust spatial grounding and cross-domain priors. In this paper, we propose StereoNav, a robust Vision-Language-Action framework designed to enhance real-world navigation consistency. To address the inherent gap between synthetic training and physical execution, we introduce Target-Location Priors as a persistent bridge. These priors provide stable visual guidance that remains invariant across domains, effectively grounding the agent even when instructions are vague. Furthermore, to mitigate visual disturbances like motion blur and illumination shifts, StereoNav leverages stereo vision to construct a unified representation of semantics and geometry, enabling precise action prediction through enhanced depth awareness. Extensive experiments on R2R-CE and RxR-CE demonstrate that StereoNav achieves state-of-the-art egocentric RGB performance, with SR and SPL scores of 81.1% and 68.3%, and 67.5% and 52.0%, respectively, while using significantly fewer parameters and less training data than prior scaling-based approaches. More importantly, real-world robotic deployments confirm that StereoNav substantially improves navigation reliability in complex, unstructured environments. Project page: https://yunheng-wang.github.io/stereonav-public.github.io.
Vision-Language Navigation (VLN) requires embodied agents to generate actions based on instructions and observations. General-purpose multimodal agents offer a promising basis for this task, but selecting plausible local actions does not ensure that execution remains consistent with the intended route, particularly in long-horizon tasks. Moreover, the accumulated interaction history increases the input required for subsequent decisions, resulting in a significant inference overhead. To this end, we introduce \method, an Agentic VLN framework that includes a Goal Agent that sets adaptive goals for local actions, a Verify Agent that dynamically verifies whether a goal has been completed, a Memory Agent for multimodal context compression, and a Visuomotor Agent to execute adaptive goals. Specifically, the Goal Agent formulates adaptive goals based on the instruction, current observation, and execution history. Then the Visuomotor Agent executes navigation actions to achieve each goal, while the Verify Agent uses a goal-specific verification question to dynamically assess whether the observed outcomes satisfy the intended completion condition. Verified goal completion then marks a boundary for the Memory Agent to compress the corresponding multimodal interaction history while preserving information needed for subsequent navigation. We evaluate navigation on R2R-CE and RxR-CE, examine framework variants across three model backbones, and study context evolution during execution. For Real-World evaluation, \method achieves 83.3% success and 1.51,m navigation error across eight challenging routes evaluated three times each.
Haoxiang Shi, Zaijing Li, Muhe Ding +3
Harbin Institute of Technology (Shenzhen) · Pengcheng Laboratory
Vision-and-Language Navigation (VLN) requires agents to navigate photo-realistic environments following natural language instructions. Current methods predominantly rely on imitation learning, which suffers from limited generalization and poor robustness to execution perturbations. We present NavGRPO, a reinforcement learning framework that learns goal-directed navigation policies through Group Relative Policy Optimization. By exploring diverse trajectories and optimizing via within-group performance comparisons, our method enables agents to distinguish effective strategies beyond expert paths without requiring additional value networks. Built on ScaleVLN, NavGRPO achieves superior robustness on R2R and REVERIE benchmarks with +3.0% and +1.71% SPL improvements in unseen environments. Under extreme early-stage perturbations, we demonstrate +14.89% SPL gain over the baseline, confirming that goal-directed RL training builds substantially more robust navigation policies. Code and models will be released.
Jiangyang Li, Cong Wan, SongLin Dong +4
Xi’an Jiaotong University · Faculty of Computility Microelectronics, Shenzhen University of Advanced Technology