cs.CVFeb 17, 2026

Learning to Retrieve Navigable Candidates for Efficient Vision-and-Language Navigation

Authors: Shutian Gu, Chengkai Huang, Ruoyu Wang, Lina Yao

Organizations: University of New South Wales, Sydney, Australia · Macquarie University, Sydney, Australia · Data61, CSIRO, Sydney, Australia

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Vision-and-Language Navigation (VLN) requires an agent to follow natural-language instructions and navigate through previously unseen environments. Recent approaches increasingly employ large language models (LLMs) as high-level navigators due to their flexibility and reasoning capability. However, prompt-based LLM navigation often suffers from inefficient decision-making, as the model must repeatedly interpret instructions from scratch and reason over noisy and verbose navigable candidates at each step. In this paper, we propose a retrieval-augmented framework to improve the efficiency and stability of LLM-based VLN without modifying or fine-tuning the underlying language model. Our approach introduces retrieval at two complementary levels. At the episode level, an instruction-level embedding retriever selects semantically similar successful navigation trajectories as in-context exemplars, providing task-specific priors for instruction grounding. At the step level, an imitation-learned candidate retriever prunes irrelevant navigable directions before LLM inference, reducing action ambiguity and prompt complexity. Both retrieval modules are lightweight, modular, and trained independently of the LLM. We evaluate our method on the Room-to-Room (R2R) benchmark. Experimental results demonstrate consistent improvements in Success Rate, Oracle Success Rate, and SPL on both seen and unseen environments. Ablation studies further show that instruction-level exemplar retrieval and candidate pruning contribute complementary benefits to global guidance and step-wise decision efficiency. These results indicate that retrieval-augmented decision support is an effective and scalable strategy for enhancing LLM-based vision-and-language navigation.

Figures & tables

Fig. 1: The agent follows a language-centric pipeline. At each step, a candidate retriever prunes directional observations before LLM reasoning, while an embedding-based instruction retriever retrieves exemplar trajectories at the episode level for in-context guidance. A history summarizer maintains a compact navigation context. The prompt aggregates the system principles, instruction, retrieved exemplars, pruned observations, and history, enabling the LLM to generate reasoning traces and navigation actions in a closed-loop manner.
MethodsVal SeenVal Unseen
TLNE ↓\downarrowSR ↑\uparrowOSR ↑\uparrowSPL ↑\uparrowTLNE ↓\downarrowSR ↑\uparrowOSR ↑\uparrowSPL ↑\uparrow
Seq2Seq [ 1 ]11.336.013953-8.397.812128-
SpeakerFollower [ 39 ]-3.366674--6.623645-
HAMT [ 40 ]11.152.5276-7211.462.2966-61
DUET [ 41 ]12.322.2879867313.943.31728160
BEVBert [ 25 ]13.562.1781887414.552.81758464
TABLE I: Comparison with the state of the art on R2R. Bold highlight the best and runner-up performance in each column, while blue underscore our method row. ↑\uparrow indicates better performance with higher values.
SettingMethodVal SeenVal UnseenTokens / Time
TLNE ↓\downarrowSR ↑\uparrowOSR ↑\uparrowSPL ↑\uparrowTLNE ↓\downarrowSR ↑\uparrowOSR ↑\uparrowSPL ↑\uparrow(per episode)
Ablation ModeBaseline15.298.8115.7729.8710.3015.668.2718.2233.2511.4024.8k / 17.9s
Baseline + Embedding Retriever16.689.2117.0433.9910.8416.838.3920.1438.9512.5435.0k / 9.76s
Baseline + Candidate Retriever15.188.7521.6536.1414.0315.478.2820.3137.8912.9722.1k / 16.1s
Full Mode + Thinking Mode ∗14.937.9124.0039.0015.9114.118.0226.4044.4018.0057.6k / 536.5s
Full Mode (Ours)16.208.9619.8839.8613.2916.358.3323.4144.7014.7632.1k / 10.1s
TABLE II: Ablation Studies. Thinking Mode ∗ is enabled by using a reasoning-capable model with its internal “thinking” mode turned on. Unless otherwise specified, all baseline results are based on NavGPT with Qwen3-8B [ 11 ] as the backbone model. ↑\uparrow indicates better performance with higher values.
Fig. 2: Here is a case study of a successful navigation step. At this step, the candidate retriever selects the top 5 directional observations for LLM reasoning. The direction highlighted in blue corresponds to the correct navigable direction chosen by the LLM, while the red directions indicate irrelevant candidates pruned by the candidate retriever. This pruning reduces decision noise and enables the LLM to focus on instruction-consistent actions.
Fig. 3: SR, OSR, and SPL of the baseline agent with different ablation modules on the R2R val-unseen split.
More in this topic

Vision-Language Navigation

Vision-Language Navigation studies how embodied agents use natural-language instructions and visual observations to plan and execute routes.

See all 290 papers

Explore similar work

CardsList
Mar 9, 2026

CMMR-VLN: Vision-and-Language Navigation via Continual Multimodal Memory Retrieval

Although large language models (LLMs) are introduced into vision-and-language navigation (VLN) to improve instruction comprehension and generalization, existing LLM- based VLN lacks the ability to selectively recall and use relevant priori experiences to help navigation tasks, limiting their performance in long-horizon and unfamiliar scenarios. In this work, we propose CMMR-VLN (Continual Multimodal Memory Retrieval based VLN), a VLN framework that endows LLM agents with structured memory and reflection capabilities. Specifically, the CMMR-VLN constructs a multimodal experi- ence memory indexed by panoramic visual images and salient landmarks to retrieve relevant experiences during navigation, introduces a retrieved-augmented generation pipeline to mimick how experienced human navigators leverage priori knowledge, and incorporates a reflection-based memory update strategy that selectively stores complete successful paths and the key initial mistake in failure cases. Comprehensive tests illustrate average success rate improvements of 52.9%, 20.9% and 20.9%, and 200%, 50% and 50% over the NavGPT, the MapGPT, and the DiscussNav in simulation and real tests, respectively eluci- dating the great potential of the CMMR-VLN as a backbone VLN framework.

Haozhou Li, Xiangyu Dong, Huiyan Jiang +2

The authors are with the Foshan Graduate School of Innovation at Northeastern University, Foshan, China · The authors are with the College of Software at Northeastern University, Shenyang, China · The authors are with the School of Aeronautic Science and Engineering at Beihang University, Beijing, China +1

Vision-Language NavigationMemory-Augmented Agents
AskPDF
Oct 9, 2026

Robust Vision-Language Navigation with Awareness of Instruction-Execution Progress

In Vision-Language Navigation (VLN) tasks involving multi-step instructions as guidance, an agent must associate its observations with the relevant instruction content at each decision point. Most graph-based methods commonly match candidate trajectories with the entire instruction, allowing already executed content to interfere with subsequent decisions. Although instruction-execution progress provides a direct cue for separating executed and pending content, existing progress estimation approaches often rely on explicit progress supervision, sophisticated progress-learning stages, or costly online reasoning with LLMs. In this paper, we propose a robust VLN framework with Awareness of Instruction-Execution Progress (VLN-AIEP), which couples progress estimation with navigation decision-making. VLN-AIEP associates the visited trajectory with the instruction to suppress embeddings corresponding to executed content, and then matches candidate locations with the pending content. This coupling makes correct navigation decisions depend on accurate progress estimation, allowing the progress estimator to learn directly from navigation decision labels without explicit progress annotations. To support progress estimation, we construct a landmark-anchored trajectory representation that encodes the semantic labels and viewpoint-relative geometry of instruction-referred landmarks. These landmarks serve as spatial references for associating the visited trajectory with executed instruction content. Extensive experiments on R2R, RxR, CVDN, and R2R-CE demonstrate that the proposed method achieves favorable performance across multiple metrics.

Jiagan Huang, Zikun Zhou, Hongpeng Wang +3

Harbin Institute of Technology, Shenzhen

Vision-Language Navigation
AskPDF
Sep 1, 2026

VerNav: Verifier-First Low-Latency Vision-and-Language Navigation

Vision-and-Language Navigation (VLN) requires an agent to navigate through unseen 3D environments according to natural-language instructions. Explicit reasoning can improve instruction understanding and semantic grounding, but autoregressive generation at every step accumulates large decision-stage latency over multi-step navigation. We propose VerNav, a verifier-first framework for low-latency LLM-based VLN. The verifier reduces decision-stage latency by replacing per-step autoregressive generation with batched action verification, while an entropy-based adaptive generator is invoked only for uncertain decisions to produce compact state evidence. To further improve navigation performance with the verifier, we introduce a two-stage alignment scheme: (i) VPO improves local action-preference alignment in static verifier training, and (ii) step-level reinforcement fine-tuning provides dense progress rewards over multi-step navigation rollouts during dynamic task execution. Experiments on the Room-to-Room (R2R) benchmark show that the verifier-only decision path of VerNav achieves competitive navigation performance among representative LLM-based VLN agents while reducing average decision-stage LLM latency per step by more than 10×10\times compared with autoregressive methods.

Zhixin Wang, Chengzheyi Yao, Leyuan Liu +2

University of Electronic Science and Technology of China

Efficient VLM InferenceVision-Language NavigationRL for LLM Agents
AskPDF