RetroNav: Scene-Grounded Retrospection and Retry for Self-Evolving Lifelong Navigation

Diantao Tu; Yangdong Liu; Niannian Dai; Mengqi Rong; Shuhan Shen

cs.ROOct 26, 2026

RetroNav: Scene-Grounded Retrospection and Retry for Self-Evolving Lifelong Navigation

Authors: Diantao Tu, Yangdong Liu, Niannian Dai, Mengqi Rong, Shuhan Shen

Organizations: Institute of Automation, Chinese Academy of Sciences · School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences

Abstract

Existing robot navigation methods primarily focus on isolated tasks, overlooking the need to accumulate knowledge and experience across repeated navigation episodes. Although recent lifelong navigation methods have explored cross-task knowledge accumulation, they mainly focus on scene knowledge while neglecting historical navigation experience and its connection to scene evidence, limiting the ability to reflect on past mistakes and improve future decisions. To address this limitation, this paper presents RetroNav, a training-free self-evolving navigation framework that integrates navigation, memory accumulation, and failure reflection. Specifically, we develop a unified memory system that connects geometric and semantic scene knowledge with historical navigation experience, distinguishing objective observations from task-dependent interpretations. After each navigation attempt, an independent evaluator determines whether a retry is necessary. If so, a Retrospection Agent revisits historical decisions using accumulated scene knowledge and navigation records, generating an evidence-grounded retry brief to guide the next attempt. RetroNav achieves state-of-the-art performance across five navigation benchmarks. Our method will be open-sourced at https://github.com/3dv-casia/RetroNav .

Figures & tables

Figure 1: Overview of the proposed RetroNav framework. The navigator iteratively selects navigation directions and waypoints, while a unified memory system accumulates scene knowledge and historical navigation experience. After each navigation attempt, an independent Evaluator determines whether a retry is necessary. When triggered, a Retrospection Agent revisits historical decisions using accumulated scene knowledge and visual evidence, generating an evidence-grounded Retry Brief to guide the next attempt.

Figure 2: Structure and organization of the unified memory. Left: Cumulative scene knowledge acquired across navigation tasks, including regions, places and trajectories. Top right: Representative scene knowledge at Place 6 of Region 1. Bottom right: Navigation experiences at the same place across distinct tasks, recording different goals, judgments, and actions.

Training-based
	R2R-CE			RxR-CE
Method	SR	SPL	OSR	SR	SPL	OSR
NavMorph (HNR) [ 40 ]	64.0	53.0	68.0	58.0	49.0	–
Uni-NaVid [ 44 ]	47.0	42.7	53.3	48.7	40.9	55.5
StreamVLN-Prune-Mem [ 37 ]	57.4	51.1	65.5	53.9	45.1	–
DualVLN [ 36 ]	64.3	58.5	70.7	61.4	51.8	–

Table 1: Evaluation results on R2R-CE and RxR-CE val-unseen. – denotes unreported.

\dagger

indicates a method retested with an updated model. The best results are shown in bold , and the second best is underlined .

Training-based
Method	SR	SPL	OSR
Uni-NaVid [ 44 ]	39.5	19.8	–
AstraNav-Memory [ 14 ]	62.5	34.9	–
MTU3D [ 47 ]	40.8	12.1	–
Training-free
SpaceVLN (Qwen3.5-Plus/Flash) [ 8 ]	51.6	33.1	–

Table 2: Evaluation results on HM3D-OVON val-unseen. – denotes unreported. The best results are shown in bold , and the second best is underlined .

(a) Training-based
Method	SR	SPL	t-nDTW
CMA [ 19 ]	19.0 $\pm$ 1.0	18.0 $\pm$ 1.0	38.0 $\pm$ 2.0
TourCMA [ 18 ]	18.0 $\pm$ 0.0	17.0 $\pm$ 1.0	36.0 $\pm$ 1.0
PoolCMA [ 18 ]	16.0 $\pm$ 1.0	15.0 $\pm$ 0.0	36.0 $\pm$ 2.0
PoolEndCMA [ 18 ]	18.0 $\pm$ 1.0	16.0 $\pm$ 1.0	38.0 $\pm$ 2.0
OVER-NAV (iterative) [ 45 ]	35.0 $\pm$ 1.0	33.0 $\pm$ 1.0	50.0 $\pm$ 2.0

Table 3: Evaluation results on IR2R-CE val-unseen.

\dagger

indicates a method retested with an updated model. The best results are shown in bold , and the second best is underlined .

(a) Training-based
Method	SR	SPL	e-SR
SenseAct-NN Skill Chain [ 16 ]	29.5	11.3	–
SenseAct-NN Monolithic [ 16 ]	12.3	6.8	–
AstraNav-Memory [ 14 ]	62.7	56.9	–
MTU3D [ 47 ]	47.2	27.7	–
MemoryExplorer (LMEE) [ 34 ]	46.4	28.0	–

Table 4: Evaluation results on GOAT-Bench val-unseen. – denotes unreported. The best results are shown in bold , and the second best is underlined .

Figure 3: Qualitative trace of memory-guided navigation. Given “ Find and navigate to a rug ”, RetroNav retrieves room context and visual cues from Unified Memory $\mathcal{M}$ , steering the agent out of the kitchen (Steps 0–2) to locate and stop at the rug (Steps 3–6).

Row	Naive	$\mathcal{M}$	Eval.	Brief	SR	SPL	t-nDTW	SR	SPL	e-SR
	Component				IR2R-CE			GOAT-Bench
(1)	–	–	–	–	65.6	57.0	68.6	67.0	41.8	14.3
(2)	✓	–	–	–	76.7	62.6	71.6	81.0	47.8	28.6
(3)	–	✓	–	–	69.4	59.7	74.2	71.5	49.6	21.4
(4)	–	✓	✓	–	74.5	63.8	75.0	75.5	51.8	26.0
(5)	–	✓	✓	✓	84.1	71.9	75.2	82.3	57.6	35.3

Table 5: Component-wise ablation study on IR2R-CE and GOAT-Bench. We evaluate the contributions of Naive Retry (Naive), Unified Memory (

\mathcal{M}

), Evaluator (Eval.), and Retry Brief (Brief).

Figure 4: Failure retrospection and retry on R2R-CE. After the initial attempt branches into a dead end at Step 12 and stops prematurely (top), the Retry Brief isolates the branch confusion (middle), guiding the retry to bypass the error and reach the bedroom door (bottom).

Figure 5: Real-world navigation with RetroNav on a CASBOT 02 humanoid robot. Given the instruction “Find the vending machine,” the robot navigates through a lobby and reaches the target autonomously. (a) Bird’s-eye view of the recorded trajectory. (b,c) Third-person and onboard views at four numbered stages. Orange boxes highlight the target vending machine.

Row	Route	Review	Images	Rec.
(1)	–	–	–	12.1
(2)	✓	✓	–	21.2
(3)	✓	–	✓	18.2
(4)	✓	✓	✓	25.8

Table 6: Ablation of the Retry Brief on 33 fixed first-attempt failures from R2R-CE, without scene memory. Each variant receives one retry from the same first-attempt traces. Rec. denotes retry recovery rate (%). Checkmarks indicate enabled components.

Method	Succ. Recall	Fail. Recall	Accuracy
Self-Stopping	100.0	0.0	55.6
Evaluator (Ours)	76.0	65.0	71.1
Oracle	100.0	100.0	100.0

Table 7: Failure-detection quality on R2R-CE. We compare navigator self-stopping, our Evaluator, and the Oracle upper bound.

Benchmark	Evaluation size	Cross-task memory	Max. steps per attempt
R2R-CE	100 episodes	Disabled	30
RxR-CE	100 episodes	Disabled	50
HM3D-OVON	150 episodes	Disabled	40
IR2R-CE	299 episodes	Enabled	30
GOAT-Bench	307 subtasks	Enabled	40

Table 8: Evaluation subsets, cross-task memory, and navigation step budgets across the five benchmarks.

Track / Record	Stored fields and evidence references
Observation Anchor (Immutable Evidence)
Observation	Scene-global ID, source attempt ID, local step, 6-DoF pose, camera intrinsics/extrinsics, and filesystem image references.
Scene Knowledge ( $\mathcal{S}$ )
Place	Unique ID, parent region ID, member observation IDs, visual description, representative viewpoint image references.
Region	Unique ID, semantic room category label, natural-language architectural summary, and member place IDs.
Object	Instance ID, semantic category, 3D metric bounding box, fused point cloud, and multi-view RGB/mask evidence.

Table 9: Unified memory schemas and stored record fields.

Panel of Figure 6 — Figure 6: Progressive scene-memory accumulation. Gray lines show navigation trajectories, red dots mark places, and colored polygons indicate regions.

Panel of Figure 7 — Figure 7: Navigation performance over repeated tours in IR2R-CE. Solid lines show the mean across three independent runs; dashed lines show individual runs.

Benchmark	Model	First				Final				Cumulative
		SR	SPL	Steps	PL (m)	SR	SPL	Steps	PL (m)	SPL	Steps	PL (m)
R2R-CE	GPT-6-Astra	77.3	71.5	12.1	9.7	84.0	75.4	16.2	13.5	66.3	17.6	14.3
	GPT-5.6-Sol	66.3	52.2	14.3	11.4	76.3	66.4	18.3	15.0	52.0	21.2	17.1
	Qwen3.8-27B	50.7	39.6	14.5	11.7	58.0	49.5	18.7	15.5	38.3	21.6	17.6
RxR-CE	GPT-6-Astra	75.3	69.8	15.9	13.6	84.0	70.9	16.3	14.2	66.9	21.2	18.2
	GPT-5.6-Sol	62.0	55.5	18.0	15.2	76.7	61.0	17.8	15.1	54.2	26.1	22.0

Table 10: Retry performance and navigation cost per episode (subtask for GOAT-Bench). First and Final report the corresponding attempt; Cumulative includes all attempts. SR and SPL are percentages. PL denotes path length in meters.

Benchmark	Model	Calls	Tokens (K)		Cumulative model-call time (s)
			Input	Output	Navigation	Retry brief	Evaluator	Object annot.	Memory builder	Total
R2R-CE	GPT-5.6-Sol	65.3	663.0	12.2	302.0	47.7	49.9	/	/	399.6
	GPT-6-Astra	44.8	445.0	5.8	218.0	19.5	41.1	/	/	278.6
RxR-CE	GPT-5.6-Sol	53.0	629.5	9.2	366.5	25.0	60.9	/	/	452.4
	GPT-6-Astra	39.3	382.4	3.6	247.8	5.8	57.2	/	/	310.8
HM3D-OVON	GPT-5.6-Sol	29.2	199.1	5.3	162.0	23.1	27.6	/	/	212.7

Table 11: Total inference cost per episode (subtask for GOAT-Bench) across all attempts. / means the component was not used.

Scheme	SR $\uparrow$	SPL $\uparrow$	OSR $\uparrow$	Steps $\downarrow$	Tokens (K) $\downarrow$	Time (s) $\downarrow$
Two-stage	56.0	48.77	72.0	17.65	265.41	156.77
Joint	47.0	39.65	61.0	18.29	390.92	125.93

Table 12: Two-stage versus joint view and waypoint selection with Qwen on R2R-CE. Costs are averaged per episode. Tokens include input and output; time is cumulative model-call time. – denotes an unreported value.

Robot Navigation

Robot navigation studies how robots perceive, localize, plan, and move safely toward goals in static or changing environments.

See all 549 papers

AllDayNav: Lifelong Navigation via Real-World Reinforcement Learning
Jun 9, 2026Hang Yin, Yinan Liang, Jiazhao Zhang +4RL for Robotics Continual Robot Learning
Tsinghua University, Beijing 100084, China · Galbot Robotics, Beijing, China · Peking University, Beijing 100871, China +1
OmniNav: Robust Long-Horizon Target Navigation in Dynamic Environments
Sep 8, 2026Yujie Tang, Meiling Wang, Jinhao Jiang +4Robot Navigation Object-Goal Navigation
School of Automation, Beijing Institute of Technology, Beijing, China.
LERR-Nav: Learning to Coordinate Exploration, Recovery, and Reminiscing for Object Navigation
Oct 11, 2026Jingzhi Huang, Junkai Huang, Yuhang Peng +6Object-Goal Navigation Embodied Navigation
Hong Kong Polytechnic University · Institute of Automation, Chinese Academy of Sciences · DeepRobotics +2

Jun 9, 2026J/K move · Enter open · S save

AllDayNav: Lifelong Navigation via Real-World Reinforcement Learning

Hang Yin, Yinan Liang, Jiazhao Zhang, Jiahang Liu+3

Tsinghua University, Beijing 100084, China · Galbot Robotics, Beijing, China · Peking University, Beijing 100871, China+1

RL for Robotics Continual Robot Learning Embodied Navigation Robot Navigation Lifelong Learning Agents Visual Robot Navigation

Open paper AskPDF arXiv

Lifelong embodied navigation in dynamic environments requires robots to form persistent scene understanding from fragmentary observations, which remains difficult for existing methods that rely on explicit maps or scene graphs and struggle to generalize beyond structured settings. We propose AllDayNav, a lifelong self-learning navigation framework that implicitly encodes scene dynamics into the billion-scale parameters of a large model via reinforcement learning, powered by a self-evolving multimodal memory that maintains and updates visual keyframes, semantic descriptions, and temporal context while autonomously generating open-vocabulary instructions, image goals, and structured rewards. Experiments in both synthetic and real-world environments across cross-room, cross-episode, and cross-task scenarios show that AllDayNav achieves success rates approaching

100\%

and consistently surpasses strong map-based, VLM, and RL baselines in path efficiency and robustness, demonstrating implicit, memory-driven reinforcement learning as a scalable alternative to explicit mapping for reliable lifelong navigation.

Abstract

Robot Navigation

Explore similar work

AllDayNav: Lifelong Navigation via Real-World Reinforcement Learning

OmniNav: Robust Long-Horizon Target Navigation in Dynamic Environments

LERR-Nav: Learning to Coordinate Exploration, Recovery, and Reminiscing for Object Navigation