NavHarness: Adaptive Goals for Agentic Vision-Language Navigation

Haoxiang Shi; Zaijing Li; Muhe Ding; Xiang Deng; Yaowei Wang; Liqiang Nie

cs.CVSep 30, 2026

NavHarness: Adaptive Goals for Agentic Vision-Language Navigation

Authors: Haoxiang Shi, Zaijing Li, Muhe Ding, Xiang Deng, Yaowei Wang, Liqiang Nie

Organizations: Harbin Institute of Technology (Shenzhen) · Pengcheng Laboratory

Abstract

Vision-Language Navigation (VLN) requires embodied agents to generate actions based on instructions and observations. General-purpose multimodal agents offer a promising basis for this task, but selecting plausible local actions does not ensure that execution remains consistent with the intended route, particularly in long-horizon tasks. Moreover, the accumulated interaction history increases the input required for subsequent decisions, resulting in a significant inference overhead. To this end, we introduce \method, an Agentic VLN framework that includes a Goal Agent that sets adaptive goals for local actions, a Verify Agent that dynamically verifies whether a goal has been completed, a Memory Agent for multimodal context compression, and a Visuomotor Agent to execute adaptive goals. Specifically, the Goal Agent formulates adaptive goals based on the instruction, current observation, and execution history. Then the Visuomotor Agent executes navigation actions to achieve each goal, while the Verify Agent uses a goal-specific verification question to dynamically assess whether the observed outcomes satisfy the intended completion condition. Verified goal completion then marks a boundary for the Memory Agent to compress the corresponding multimodal interaction history while preserving information needed for subsequent navigation. We evaluate navigation on R2R-CE and RxR-CE, examine framework variants across three model backbones, and study context evolution during execution. For Real-World evaluation, \method achieves 83.3% success and 1.51,m navigation error across eight challenging routes evaluated three times each.

Figures & tables

Figure 1: NavHarness connects adaptive goal execution, verified progress, and multimodal memory. Adaptive local goals guide navigation, while goal-specific questions verify whether observed outcomes meet completion conditions. Verified completion marks boundaries for summarizing completed interactions while retaining information for ongoing navigation and recovery. NavHarness significantly outperforms trained policies and other agent frameworks in VLN-CE.

Figure 2: NavHarness framework. Adaptive goals guide execution; observation-based verification provides feedback for the next step and goal revision or advancement. Verified completion triggers multimodal memory compression. The two situations illustrate route following and recovery.

			R2R-CE				RxR-CE
Method / backbone	Views	Training	NE $\downarrow$	OSR $\uparrow$	SR $\uparrow$	SPL $\uparrow$	NE $\downarrow$	SR $\uparrow$	SPL $\uparrow$	nDTW $\uparrow$
Trained navigation models
NaVid ( Zhang et al., 2024 )	M	Trained	5.47	49.1	37.4	35.9	8.41	23.8	21.2	—
NaVILA ( Cheng et al., 2025 )	M	Trained	5.22	62.5	54.0	49.0	6.77	49.3	44.0	58.8
StreamVLN ( Wei et al., 2026 )	M	Trained	4.90	63.6	56.4	50.2	5.65	54.4	45.4	63.7
JanusVLN ( Zeng et al., 2026 )	M	Trained	4.78	65.2	60.5	56.8	6.06	56.2	47.5	62.1

Table 1: Navigation performance on R2R-CE and RxR-CE. Bold : best reported value per metric; underline : best baseline within each group unless bold. Shaded rows denote NavHarness. M = monocular, P = panorama, D = depth; 3-cam = three fixed RGB cameras. NE is in meters; other metrics are percentages. Results retain their original protocols: SmartWay and AgenticNav use Open-Nav's 100 episodes; Dashes denote unreported or pending results.

R2R-CE
			Navigation Quality		Token Consumption			Decision Context		Compression
Model	Harness	Compress.	SR $\uparrow$	SPL $\uparrow$	Cache	Input $\downarrow$	Total $\downarrow$	Image/Dec. $\downarrow$	Text/Dec. (k) $\downarrow$	Total ratio (%) $\downarrow$
GPT-5.6-Sol	Codex CLI	Native	62.6	44.5	1351.40	1398.72	1410.31	12.43	68.99	—
	NavHarness	Off	67.0	47.9	1234.50	1334.64	1345.23	13.10	80.57	—
	NavHarness	Goal	66.0	50.3	770.45	851.85	861.82	11.80	32.31	64.06%
GPT-6-Astra	Codex CLI	Native	74.0	62.5	656.30	687.77	689.81	9.42	73.72	—

Table 2: Navigation quality and context consumption. Cache/Input/Total:

10^{3}

tokens per episode. Cache is included in Input. Image/Dec.: mean input images per external decision. Text/Dec.: mean input text length (

10^{3}

characters) per external decision. Total ratio: Goal/Off (%).

Figure 3: Input context during navigation. Vertical markers indicate compression events; percentages report selected local input reductions, not episode-level savings.

Figure 4: Framework ablation. R2R-CE SR (%), rounded. Plain NavHarness is Off.

Method	SR (%) $\uparrow$	NE (m) $\downarrow$
NaVILA	20.83	3.91
AwareVLN	20.83	3.89
StreamVLN	25.00	3.78
JanusVLN	50.00	2.34
NavHarness	83.30	1.51

Table 3: Real-world navigation. Baselines use the supplied aggregates; NavHarness uses eight routes with three trials each.

Figure 5: Indoor scene transitions. Following ordered landmarks, clearing the doorway before turning, and approaching the yellow chair with a white ball.

Panel of Figure 6 — Figure 6: Adaptive outdoor navigation. Obstacle-responsive goal adaptation toward the sculpture (top) and landmark-conditioned progress toward the trees (bottom).

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 7 — Figure 7: Successive landmark turns. The route links a doorway, a yellow chair, a white table, and a black table through successive landmark-conditioned turns. Panel (h) shows a clear view during the final approach; the run subsequently calls STOP.

Explore similar work

ReflectVLN: Training Vision-Language Navigation Agents with Reflective Reasoning
Jul 14, 2026Jiahang Wang, Yirong Yang, Yanqing Zhu +4Vision-Language Navigation Navigation
Amap, Alibaba Group, China · Beihang University
NavJev: Efficient Vision-Language Navigation via Action-Centric Visual Compression and Discriminative Action-Semantic Memory
Sep 28, 2026Kai Sheng, Liuyi Wang, Jinlong Li +3Vision-Language Navigation Diffusion-Based Vision-Language-Actions
College of Electronic and Information Engineering, Tongji University, Shanghai, China
SEDualVLN: A Spatially-Enhanced Dual-System for Vision-Language Navigation
May 17, 2026Jingzhi Huang, Junkai Huang, Wenxuan Song +4Vision-Language Navigation Multimodal Large Language Models
Hong Kong Polytechnic University · Institute of Automation, Chinese Academy of Sciences · Hong Kong University of Science and Technology (Guangzhou)

Jul 14, 2026, cs.CVJ/K move · Enter open · S save

ReflectVLN: Training Vision-Language Navigation Agents with Reflective Reasoning

Jiahang Wang, Yirong Yang, Yanqing Zhu, Minghua Luo+3

Amap, Alibaba Group, China · Beihang University

Vision-Language Navigation Navigation Chain-of-Thought Reasoning

Open paper AskPDF arXiv

Existing vision-language navigation methods often couple a VLM with waypoint decoders to produce multi-step action plans, but they typically lack an explicit closed-loop mechanism for tracking semantic progress, diagnosing execution failures, and recovering from error accumulation in long-horizon navigation. To address this gap, we propose ReflectVLN, an agentic VLN framework that organizes decision-making through bidirectionally interactive intention and execution agents. The intention agent performs subtask decomposition and reflection, generating executable subtask descriptions as corrective plans. Conditioned on these descriptions, the execution agent grounds them into short-horizon actions under current observations while monitoring sub-goal progress and detecting off-track behavior. Crucially, ReflectVLN enables closed-loop bidirectional communication: the execution agent emits progress and deviation signals to trigger reflection and subtask updates on demand, and the intention agent returns structured guidance that reconditions subsequent actions for recovery. To encourage temporally coherent decisions with interpretable intermediate rationales, we introduce Action Chain-of-Thought (Action-CoT), a path-conditioned dual-query training scheme for action generation. Experiments on standard VLN benchmarks show that ReflectVLN improves success rates and path efficiency under a constrained data budget, with favorable training cost and fewer high-level intention calls at inference time, while providing interpretable intermediate decisions for analysis and collaboration. Code is available at: https://github.com/AIprogrammer/ReflectVLN

Abstract

Explore similar work

ReflectVLN: Training Vision-Language Navigation Agents with Reflective Reasoning

NavJev: Efficient Vision-Language Navigation via Action-Centric Visual Compression and Discriminative Action-Semantic Memory

SEDualVLN: A Spatially-Enhanced Dual-System for Vision-Language Navigation