HaltNav: Reactive Visual Halting over Lightweight Topological Priors for Robust Vision-Language Navigation

Zihui Yu; Pingcong Li; Bichi Zhang; Sören Schwertfeger

cs.ROMar 13, 2026

HaltNav: Reactive Visual Halting over Lightweight Topological Priors for Robust Vision-Language Navigation

Authors: Zihui Yu, Pingcong Li, Bichi Zhang, Sören Schwertfeger

Organizations: Key Laboratory of Intelligent Perception and Human-Machine Collaboration - ShanghaiTech University, Ministry of Education, China.

Abstract

Vision-and-Language Navigation (VLN) is shifting from rigid, step-by-step instruction following toward open-vocabulary, goal-oriented autonomy. Achieving this transition without exhaustive routing prompts requires agents to leverage structural priors. While prior work often assumes computationally heavy 2D/3D metric maps, we instead exploit a lightweight, text-based osmAG (OpenStreetMap Area Graph), a floorplan-level topological representation that is easy to obtain and maintain. However, global planning over a prior map alone is brittle in real-world deployments, where local connectivity can change (e.g., closed doors or crowded passages), leading to execution-time failures. To address this gap, we propose a hierarchical navigation framework HaltNav that couples the robust global planning of osmAG with the local exploration and instruction-grounding capability of VLN. Our approach features an MLLM-based brain module, which is capable of high-level task grounding and obstruction awareness. Conditioned on osmAG, the brain converts the global route into a sequence of localized execution snippets, providing the VLN executor with prior-grounded, goal-centric sub-instructions. Meanwhile, it detects local anomalies via a mechanism we term Reactive Visual Halting (RVH), which interrupts the local control loop, updates osmAG by invalidating the corresponding topology, and triggers replanning to orchestrate a viable detour. To train this halting capability efficiently, we introduce a data synthesis pipeline that leverages generative models to inject realistic obstacles into otherwise navigable scenes, substantially enriching hard negative samples. Extensive experiments demonstrate that our hierarchical framework outperforms several baseline methods without tedious language instructions, and significantly improves robustness for long-horizon vision-language navigation under environmental changes.

Figures & tables

Fig. 1 : Overview of the proposed hierarchical navigation framework. To handle abstract user commands, our system decouples navigation into global semantic planning and local physical execution. (Top Left) The GGTD (Graph-Grounded Task Dispatcher) in the MLLM brain reasons the text-based static prior (osmAG) to formulate an initial route. (Bottom Left) The overarching plan is broken down into atomic instructions for the low-level VLN agent to execute. (Bottom Right) Upon encountering unmapped dynamic anomalies, the RVH (Reactive Visual Halting) mechanism interrupts the execution loop. (Top Right) The brain subsequently updates the internal map to prune the blocked passage and autonomously replans a collision-free detour.

Fig. 2 : An example of the osmAG map of the second floor of scene 00862 in HM3D dataset [ 26 ] . The map only contains the room info and the passage (e.g., doorway), shown in red dots, connecting the rooms.

Fig. 3 : Qualitative examples of Failure-Injected Counterfactual Synthesis. To bridge the sim-to-real gap and generate infinite diverse anomalies, we leverage generative visual inpainting models to modify RGB observations. The top row shows original Seed images denoting traversable paths. The bottom row displays the resulting Augmented images, where semantic obstacles, such as clutter, crowds, or physical hazards—are realistically synthesized. These serve as high-fidelity negative samples for training the reactive halting capability.

Method	Level	SR (%)			SPL			OS (%)			NE (m)
Method	Level	B	O	Drop	B	O	Drop	B	O	Drop	B	O	$\Delta$
Navid [ 27 ]	L0	73.13	6.25	66.88	0.68	0.05	0.63	83.13	6.25	76.88	2.82	9.64	6.82
	L1	58.13	6.25	51.88	0.52	0.02	0.50	61.25	6.25	55.00	4.24	7.87	3.62
	L2	49.38	0.00	49.38	0.43	0.00	0.43	56.25	0.00	56.25	5.16	8.45	3.29
OmniNav [ 28 ]	L0	90.63	12.50	78.13	0.82	0.06	0.76	96.88	18.75	78.13	1.30	9.54	8.24
	L1	80.63	18.75	61.88	0.76	0.13	0.63	87.50	18.75	68.75	2.35	9.84	7.49

TABLE I : Simulation results under standard (B) and obstacle-injected (O) settings across instruction levels (L0–L2). For SR/SPL/OS, Drop

=

-

O (lower is better). For NE,

\Delta

-

B (lower is better). Bold marks the best SR and OS under B and O per level.

Fig. 4 : Examples of our proposed methods in HM3D [ 26 ] dataset. The left example illustrates the reactive halting and re-planning ability of our pipeline, given the unexpected obstacle observed. The right one shows an ordinary task without injecting any obstacles.

Method	Level	SR (%)			SPL			OS (%)			NE (m)
Method	Level	B	O	Drop	B	O	Drop	B	O	Drop	B	O	$\Delta$
StreamVLN [ 29 ]	L0	33.33	0.00	33.33	0.33	0.00	0.33	40.00	0.00	40.00	15.20	18.80	3.60
	L2	13.33	0.00	13.33	0.13	0.00	0.13	13.33	0.00	13.33	12.05	19.58	7.53
InternVLA-N1 [ 31 ]	L0	40.00	0.00	40.00	0.39	0.00	0.39	53.33	0.00	53.33	6.86	19.94	13.08
	L2	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	27.14	25.76	$-$ 1.38
HaltNav (Ours)	L0	73.33	56.66	16.67	0.72	0.56	0.16	86.66	66.66	20.00	5.31	9.25	3.94

TABLE II : Real-world results on a Fetch robot under standard (B) and obstacle-injected (O) settings. L0 (detailed) and L2 (goal-only) are reported to contrast instruction granularity extremes. For SR/SPL/OS, Drop

=

-

O (lower is better). For NE,

\Delta

-

B (lower is better). Bold marks the best SR and OS under B and O per level.

Fig. 5 : An example of our real world test on Fetch robots. Given the instruction and osmAG, the robot sketches a global path starting in the hallway. However, one of the doors at the end of the hallway on the path is unexpectedly closed. With reactive halting module, the robot identified the situation and replanned via the other doorway on the farther side of the hallway.

Vision-Language Navigation

Vision-Language Navigation studies how embodied agents use natural-language instructions and visual observations to plan and execute routes.

See all 310 papers

May 21, 2026

AwareVLN: Reasoning with Self-awareness for Vision-Language Navigation

Vision-and-Language Navigation (VLN) requires an agent to ground language instructions to its own movement within a visual environment. While state-of-the-art methods leverage the reasoning capabilities of Vision-Language Models (VLMs) for end-to-end action prediction, they often lack an explicit and explainable understanding of the relationships between the agent, the instruction, and the scene. Conversely, explicitly building a scene map for heuristic planning is intuitively appealing but relies on additional 3D sensors and hinders large-scale vision-language pre-training. To bridge this gap, we propose AwareVLN, a novel framework that equips the navigation model with a self-aware reasoning mechanism, enabling it to understand the agent's state and task progress in a fully end-to-end and data-driven manner. Our approach features two key innovations: (1) a structural reasoning module that fosters spatial and task-oriented self-awareness, and (2) an automatic data engine with progress division for effective training. Extensive experiments on various datasets in Habitat simulator show our AwareVLN significantly outperforms previous state-of-the-art vision-language navigation methods. Project page: https://gwxuan.github.io/AwareVLN/.

Wenxuan Guo, Xiuwei Xu, Yichen Liu +7

Tsinghua University, China

Vision-Language Navigation VLM Reasoning Embodied AI Spatial Reasoning Reasoning for VLA Models

Ask PDF

Jul 31, 2026

HAM-VLN: Harnessing Hierarchical Agentic Memory for Zero-Shot Vision-and-Language Navigation

Vision-and-language navigation (VLN) enables robots to follow instructions in previously unseen environments. Recently, a training-free paradigm has emerged: the robot queries a multimodal LLM to understand its observations and plan the next action. However, long-horizon navigation based on either image streams or dense map inevitably introduces a growing memory and reasoning bottleneck. We present HAM-VLN, a decision-coupled, agent-authored memory that equips the robot with a persistent, depth-grounded world graph. In the same model call used to select the next action, HAM-VLN also records semantic and reflective information---including room type, objects, navigation progress, and failure notes. Recent waypoints remain verbatim within a bounded window, while older history re-enters the context only through retrieval scored by relevance, recency, and salience, together with one-hop topological expansion. This design requires no additional LLM calls beyond the per-waypoint decision. Compared to previous methods, HAM-VLN not only improves various navigation metrics but also reduces the context length by more than 65%. Specifically, HAM-VLN achieves 61.0% Success Rate (SR) on VLN-CE R2R, 52.7% SR on VLN-CE RxR, and 79.7% SR on HM3D-v2 ObjectNav without any training.

An Liu, Bingxi Liu, Hongyu Ding +6

Institute of Automation, Chinese Academy of Sciences, Beijing, China · Southern University of Science and Technology, Shenzhen, China · Pengcheng Laboratory, Shenzhen, China +3

Memory-Augmented VLMs Vision-Language Navigation Long-Horizon LLM Agents Memory-Augmented Agents

Ask PDF

Dec 2, 2025

SeeNav-Agent: Enhancing Vision-Language Navigation with Visual Prompt and Step-Level Policy Optimization

Existing Vision-Language Navigation (VLN) agents based on Large Vision-Language Models (LVLMs) often suffer from errors of perception, reasoning, and planning, which significantly hinder their navigation performance. To address these limitations, a novel VLN agent framework, named SeeNav-Agent, is proposed. First, to reduce hallucinations of the visual module of the VLN agent, a dual-view Visual Prompt (VP) technique is introduced, which can also improve the agent's understanding of current spatial states. Subsequently, a novel step-level Reinforcement Fine-Tuning (RFT) method, Step Reward Group Policy Optimization (SRGPO), is designed for post-training. In SRGPO, we first define verifiable process rewards for navigation, and then perform efficient step-level advantage estimation by randomly grouping navigation steps. SRGPO provides dense reward signals for the RFT of the VLN agent and enhances its planning capability. Experiments on both the EmbodiedBench Navigation (Embench-Nav) and VLN-CE validate the effectiveness of the proposed method. By introducing the zero-shot VP module, the GPT-4.1 achieves a navigation success rate of 86.7%, surpassing the current best LVLM by approximately 20 percentage points (pp) on Embench-Nav and achieves an 11.1 pp improvement on VLN-CE compared to the base model. With VP+SRGPO, the Qwen2.5-VL-3B model reaches a navigation success rate of 72.3%, outperforming the best existing LVLM model by 5.6 pp on Embench-Nav and achieves 8.9 pp improvement on VLN-CE compared to the baseline method. Moreover, compared to GRPO and GiGPO, the proposed SRGPO demonstrates significant improvements in training stability, convergence efficiency, and generalization capability.

Zhengcheng Wang, Zichuan Lin, Yijun Yang +2

Tencent AI Lab

Vision-Language Navigation

Ask PDF

Abstract

Vision-Language Navigation

Explore similar work

AwareVLN: Reasoning with Self-awareness for Vision-Language Navigation

HAM-VLN: Harnessing Hierarchical Agentic Memory for Zero-Shot Vision-and-Language Navigation

SeeNav-Agent: Enhancing Vision-Language Navigation with Visual Prompt and Step-Level Policy Optimization