LERR-Nav: Learning to Coordinate Exploration, Recovery, and Reminiscing for Object Navigation
Authors: Jingzhi Huang, Junkai Huang, Yuhang Peng, Keyu Mu, Hailong Huang, Haoyang Yang, Renbo Luo, Lujia Wang, Yi Wang
Organizations: Hong Kong Polytechnic University · Institute of Automation, Chinese Academy of Sciences · DeepRobotics · Hong Kong University of Science and Technology (Guangzhou) · Guangzhou University
Object Navigation in unseen environments remains a fundamental challenge in embodied AI, as it requires agents to generalize visual and spatial reasoning to novel layouts. While recent advances in large language model (LLM) and vision-language model (VLM) have enhanced semantic mapping, spatial memory, and waypoint selecting for navigation, successful navigation requires more than selecting the next promising waypoint. A robot must also recognize stalled progress, escape unproductive behaviors, and revisit overlooked target cues. Addressing these challenges requires both specialized navigation strategies and a high-level policy that coordinates their execution. Motivated by the analysis of failed navigation episodes, we propose LERR-Nav, an object navigation framework that coordinates exploration, recovery, and reminiscing through a learned high-level policy. The framework comprises four complementary navigation strategies: value-function-guided exploration, LLM-guided exploration, recovery from stalled or repetitive behavior, and memory-guided reminiscing. To support history-aware coordination, we maintain the execution memory that preserves textual summaries of all strategy executions alongside a sliding window of map sequences. To learn the coordination policy, we initialize a high-level vision-language agent from Qwen2.5-VL-7B-Instruct and train it through a two-stage curriculum. The first stage uses map-centric supervised fine-tuning (SFT) to establish spatial grounding. The second stage performs SFT on variable-length navigation segments to learn which navigation strategy to invoke next based on current maps and the hybrid execution memory. Experiments on HM3D and MP3D demonstrate state-of-the-art navigation performance, while ablation studies validate the contributions of individual components.
Figures & tables
Figure 1: LERR-Nav pipeline: Conditioned on the target instruction and execution memory, a learned high-level policy invokes one of four complementary strategies: value-function-guided exploration, LLM-guided exploration, recovery, or reminiscing. The selected strategy interacts with the environment until termination, after which its textual execution summary and RGB–map records are used to update the execution memory. The updated memory then informs the next high-level strategy invocation, forming a closed-loop navigation process.
Figure 2: Pipeline of the high-level coordination policy. At the k -th decision, execution memory offers textual summaries of all completed strategy executions ( e1,…,ek−1 ), and historical RGB observations and map snapshots to the agent.
Method
Venue
HM3D
MP3D
SR ↑
SPL ↑
SR ↑
SPL ↑
ZSON Majumdar et al. (2022)
NeurIPS’22
25.5
12.6
15.3
4.8
ESC Zhou et al. (2023)
ICML’23
39.2
22.3
28.7
14.2
L3MVN Yu et al. (2023)
IROS’23
50.4
23.1
34.9
14.5
PSL Sun et al. (2024)
ECCV’24
42.4
19.2
18.9
6.9
VLFM Yokoyama et al. (2024)
ICRA’24
52.5
30.4
36.4
17.5
Table 1: Comparison of LERR-Nav and other methods on the HM3D and MP3D datasets.
Method
Intra-Floor Scenarios
Inter-Floor Scenarios
All Episodes
SR ↑
SPL ↑
SR ↑
SPL ↑
SR ↑
SPL ↑
VLFM Yokoyama et al. (2024)
64.6
37.3
0.4
0.1
52.8
30.5
MFNP Zhang et al. (2025a)
68.4
30.5
13.4
9.8
58.3
26.7
ASCENT Gong et al. (2026)
72.6
37.7
33.3
14.9
65.4
33.5
LERR-Nav
75.8
38.1
62.5
27.3
72.3
35.6
Table 2: Performance comparison between intra and inter-floor navigation scenarios.
High-Level Policy
SR ↑
SPL ↑
GPT-5
52.2
23.2
LERR-Nav (Base)
43.3
19.6
LERR-Nav (Stage-I SFT)
58.1
27.3
LERR-Nav (Stage-I+II SFT)
72.3
35.6
Table 3: Impact of the high-level coordination policy on HM3D.
Variant
VF-E
LLM-E
Rec.
Rem.
SR ↑
SPL ↑
w/o VF-E
–
✓
✓
✓
63.5
32.6
w/o LLM-E
✓
–
✓
✓
51.8
30.2
w/o Rec.
✓
✓
–
✓
65.2
31.9
w/o Rem.
✓
✓
✓
–
66.3
32.5
Full
✓
✓
✓
✓
72.3
35.6
Table 4: Ablation of the navigation strategies on HM3D.
Figure 3: Real-world deployment of LERR-Nav on a Unitree Go2 quadruped robot.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: The uncertainty computation process. (a) Initial frontier point at the exploration boundary. (b) FOV projection defining the observable area S1 and uniformly sampled boundary points. (c) Fusion process with the Gaussian kernel function and the assigned value scores for each sampled boundary point. (d) Final Estimated Uncertainty, restricted to the region within S1 .
Exploration Strategy
HM3D
MP3D
SR ↑
SPL ↑
SR ↑
SPL ↑
(a) Semantic ( α=1,β=0 )
67.2
29.3
41.1
14.8
(b) Uncertainty ( α=0,β=1 )
66.8
31.1
41.8
15.5
(c) Semantic + Uncertainty ( α=0.5,β=0.5 )
68.5
33.4
43.3
16.3
(d) Semantic + Uncertainty ( α=0.5,β=0.5 )
69.8
34.2
45.4
17.1
(e) Semantic + Uncertainty (Dynamic α,β )
69.3
33.8
45.9
17.3
Appendix
Table 5: Ablation of value-function-guided exploration on HM3D and MP3D, comparing semantic and uncertainty components and static versus dynamic weighting.
Table 10
Figure 5: Explanation of the recovery strategy workflow: (a) and (d) illustrate abnormal behaviors caused by frontier points distance. (b) → (c) demonstrate how to handle overly distant frontier points using a segmented method. (e) → (f) show how LLM progressively approaches the target frontier point via fine-grained action adjustments.
Figure 6: Explanation of how the reminiscing strategy facilitates progress in task: (a) visualizes the representation of the KeyPoint Map. (b) illustrates how the LLM chose to keypoints and locate the staircase entrance. (c) shows the robot descending the stairs after locating the stairway entrance and then continuing to find the target.
Figure 7: Case Study1: (a)–(f)/(g)–(l) illustrate the complete process of LERR-Nav/ASCENT Gong et al. (2026) searching for the sofa.
Figure 8: Case Study2: The visualization starts after each method has finished exploring the single-floor. (a)–(f)/(g)–(l) show the process of our LERR-Nav/ASCENT Gong et al. (2026) searching for the potted plant.
Recent advancements in Large Language Models (LLMs) and Vision-Language Models (VLMs) have made them powerful tools in embodied navigation, enabling agents to leverage commonsense and spatial reasoning for efficient exploration in unfamiliar environments. Existing LLM-based approaches convert global memory, such as semantic or topological maps, into language descriptions to guide navigation. While this improves efficiency and reduces redundant exploration, the loss of geometric information in language-based representations hinders spatial reasoning, especially in intricate environments. To address this, VLM-based approaches directly process ego-centric visual inputs to select optimal directions for exploration. However, relying solely on a first-person perspective makes navigation a partially observed decision-making problem, leading to suboptimal decisions in complex environments. In this paper, we present a novel vision-language model (VLM)-based navigation framework that addresses these challenges by adaptively retrieving task-relevant cues from a global memory module and integrating them with the agent's egocentric observations. By dynamically aligning global contextual information with local perception, our approach enhances spatial reasoning and decision-making in long-horizon tasks. The proposed method surpasses previous state-of-the-art approaches by a significant margin on both the HSSD and HM3D benchmarks and demonstrates strong performance on a real robot.
Object navigation requires a robot to search for an unobserved target in an unknown environment by deciding where to explore next under partial observability. Effective search resembles human-like exploration: selectively probing visually promising frontiers while relying on spatial memory to avoid redundant revisits. We propose IntentNav, a spatial-visual imitation framework that learns human-like ObjectNav policies from human demonstrations. To infer high-level search intent from low-level human actions, we introduce Frontier-based Human-Intent Labeling, which looks ahead in human demonstrations and labels the frontier that best explains the demonstrator's future search direction. We construct a spatial-visual candidate space, where BEV memory tracks explored regions, unexplored frontiers, and trajectory history, while egocentric visual memory provides semantic cues for each candidate. A VLM policy is trained to select among these grounded candidates, using Intent-Aligned Objective to encourage consistent and human-like exploration. IntentNav achieves state-of-the-art performance on the MP3D, HM3D-v1 and HM3D-v2 ObjectNav benchmarks. The proposed candidate-level navigation interface transfers zero-shot to wheeled, quadruped, and humanoid robots without further VLM fine-tuning. Project page.
Yuxin Cai, Zongtai Li, Maonan Wang +9
Nanyang Technological University · Carnegie Mellon University · The Chinese University of Hong Kong +1
Constructing structured spatial memory is essential for enabling long-horizon reasoning in complex embodied navigation tasks. Current memory construction predominantly relies on a decoupled, two-stage paradigm: agents first aggregate environmental data through exploration, followed by the offline reconstruction of spatial memory. However, this post-hoc and geometry-centric approach precludes agents from leveraging high-level semantic intelligence, often causing them to overlook navigationally critical landmarks (e.g., doorways and staircases) that serve as fundamental semantic anchors in human cognitive maps. To bridge this gap, we propose ABot-Explorer, a novel active exploration framework that unifies memory construction and exploration into an online, RGB-only process. At its core, ABot-Explorer leverages Large Vision-Language Models (VLMs) to distill Semantic Navigational Affordances (SNA), which act as cognitive-aligned anchors to guide the agent's movement. By dynamically integrating these SNAs into a hierarchical SG-Memo, ABot-Explorer mirrors human-like exploratory logic by prioritizing structural transit nodes to facilitate efficient coverage. To support this framework, we contribute a large-scale dataset extending InteriorGS with SNA and SG-Memo annotations. Experimental results demonstrate that ABot-Explorer significantly outperforms current state-of-the-art methods in both exploration efficiency and environment coverage, while the resulting SG-Memo is shown to effectively support diverse downstream tasks.