Ask to Evolve: A Unified Harness for Lifelong Embodied Navigation with Hybrid Action Space and Persistent Memory

Kai Sheng; Liuyi Wang; Jinlong Li; Haojie Dai; Chengju Liu; Qijun Chen

cs.ROOct 29, 2026

Ask to Evolve: A Unified Harness for Lifelong Embodied Navigation with Hybrid Action Space and Persistent Memory

Authors: Kai Sheng, Liuyi Wang, Jinlong Li, Haojie Dai, Chengju Liu, Qijun Chen

Organizations: School of Electronic and Information Engineering, Tongji University, Shanghai, China

Abstract

Giving embodied agents the freedom to act autonomously, initiate interactions, and understand their surroundings through experience is a step toward bringing them to life in the physical world. Realizing this vision requires action, interaction, and memory to work together, so that each encounter can inform current decisions and contribute experience for future tasks. We present EvoLife, a unified navigation harness that enables a multimodal embodied agent to choose how to move, when to ask, and what experience to carry forward. Its hybrid action space supports waypoint navigation, local discrete adjustments, pixel-based navigation, and object-directed navigation. Active interaction allows the agent to seek clarification or assistance when it encounters ambiguity or difficulty, while persistent spatial memory binds observations and dialogue to places, retaining useful experience to guide subsequent visits. We evaluate eight benchmarks across four task families: vision-language navigation, object navigation, interactive instance navigation, and continuous multi-task navigation. With an average of only 1.87 active interactions per episode, our method achieves a first-visit success rate (SR) of 100% across four 100-episode sets spanning vision-language navigation and object navigation. Complete-sequence SR reaches 98% on 50 LHPR-VLN sequences and 94% on 50 GOAT-Bench sequences. In the R2R revisit setting, success weighted by path length (SPL) increases from 82.23% to 93.31% with retained experience. Multi-day physical navigation further demonstrates exploration, proactive help-seeking, and revisiting remembered places. Together, these results advance the vision of bringing embodied agents to life through autonomous action, proactive interaction, and the accumulation and reuse of experience.

Figures & tables

Figure 1: EvoLife takes a step toward bringing embodied agents to life through autonomous action, proactive interaction, and the accumulation of experience. The left panel presents four task families: vision-language navigation [ 3 , 27 , 26 ] , object navigation [ 42 , 65 , 69 ] , interactive instance navigation [ 21 , 48 ] , and continuous multi-task navigation [ 47 , 24 ] . The center shows the unified harness, connecting a hybrid action space, active interaction, and persistent spatial memory. The right panel reports success rate (SR) across eight simulation benchmarks and illustrates multi-day physical navigation, demonstrating the agent’s ability to respond to environmental changes, proactively seek assistance, and revisit remembered places.

Figure 2: Overview of the unified EvoLife framework. A multimodal agent integrates hybrid actions, active interaction, and persistent memory within a unified decision loop. The agent adaptively selects navigation actions based on current observations and execution feedback, actively seeks external guidance to resolve navigation uncertainties, and retains spatially grounded experience for future tasks. $C_{i}$ denotes a geometric candidate waypoint. $P_{i}$ denotes a remembered place associated with its recorded pose and supporting evidence. $D_{i}$ denotes the $i$ -th tool-level decision, encompassing perception, navigation, interaction, and memory operations. $Q_{i}$ and $A_{i}$ denote the agent-generated question and the corresponding guide answer in the $i$ -th interaction, respectively. The illustrated example demonstrates how hybrid actions, interaction, and memory jointly support lifelong embodied navigation.

Tool	Agent-selected input	Returned evidence and function
Perception
observe observe_panorama_candidates	Observation request	Return current four-view or panoramic observations and geometric waypoint candidates.
observe_map	Memory Map	Display observed occupancy maps, executed routes, candidate waypoints, and remembered places.
process_image	Source view; processing pipeline and parameters	Return processed images and their processing records.
localize_object verify_object	View and box/point prompts; image version or object ID	Estimate object geometry from RGB-D observations and return visual overlays and depth-consistency evidence.
Hybrid Action Space

Table 1: Principal EvoLife tools organized into four functional groups. The agent autonomously selects tools and their parameters.

Figure 3: Online guide response generation. The navigation agent actively generates a question, while an independent multimodal guide generates a natural-language response based on the navigation instruction, current observations, target information, and reference route. The figure illustrates the current and target views, reference route, and spatial relationships between the agent and the destination. Only the generated natural-language response is returned to the navigation agent, which independently makes subsequent decisions. This process simulates natural human-to-human interaction, providing navigation guidance through directions, landmarks, and spatial relationships.

Benchmark	Task input	Episodes	Subtasks	Success criterion	Metrics
R2R-CE	Route instruction	100	–	STOP within $3\,\mathrm{m}$ of the destination	SR, OSR, SPL, NE
RxR-CE	English route instruction	100	–	STOP within $3\,\mathrm{m}$ of the destination	SR, SPL, NE, nDTW
HM3D-OVON	Open-vocabulary object category; Val-Unseen	100	–	STOP with $d_{g}<0.25\,\mathrm{m}$ to an accepted goal viewpoint	SR, SPL, NE
HM3D v2	Object category; validation split	100	–	STOP with $d_{g}<0.1\,\mathrm{m}$ to an accepted goal viewpoint	SR, SPL, NE
VL-LN	Category-only underspecified language goal; IIGN	100	–	STOP with $d_{g}<0.25\,\mathrm{m}$ to an accepted target viewpoint	SR, SPL, NQ
CoIN	Category-only language goal; hidden target instance; Unseen categories	100	–	STOP with $d_{g}<0.25\,\mathrm{m}$ to an accepted target-instance viewpoint	SR, SPL, NQ

Table 2: Evaluation protocols and sample sizes across eight navigation benchmarks. An episode denotes one complete task or continuous sequence.

Method

VL-LN

CoIN

LHPR-VLN

GOAT-Bench

\uparrow

SPL

\uparrow

\downarrow

\uparrow

SPL

\uparrow

\downarrow

\uparrow

ISR

\uparrow

CSR

\uparrow

CGT

\uparrow

s-SR

\uparrow

SPL

\uparrow

e-SR

\uparrow

VLFM [ 68 ]

8.8

5.6

–

0.2

0.1

–

MTU3D [ 85 ]

12.8

5.4

–

8.7

3.3

–

AIUTA [ 48 ]

4.0

2.2

5.42

6.7

2.3

1.13

–

ProCompNav [ 28 ]

–

17.0

6.2

2.30

–

InternVLA-N1 [ 23 , 84 ]

25.4

16.2

1.98

17.2

8.2

2.61

–

Table 4: Comparison on VL-LN, CoIN, LHPR-VLN, and GOAT-Bench.

Dataset	Model	Interaction	Memory	OSR $\uparrow$	SR $\uparrow$	SPL $\uparrow$	NE $\downarrow$	Steps $\downarrow$	ND $\downarrow$	NQ $\downarrow$	Time $\downarrow$	Cost $\downarrow$
Dataset	Model	Interaction	Memory	(%)	(%)	(%)	(m)	Steps $\downarrow$	task / sub.	task / sub.	(s)	($/ep.)
R2R	GPT-6 Astra	$\times$	$\times$	89	86	63.7	2.86	94.06	18.71	0.00	170.17	1.22
R2R	GPT-6 Astra	$\checkmark$	$\times$	100	100	82.2	0.95	70.12	15.57	1.66	164.08	1.06
R2R	GPT-6 Astra	$\times$	$\checkmark$	100	100	93.3	0.95	54.43	12.31	0.00	107.41	1.11
R2R	DeepSeek V4.1-Flash	$\times$	$\times$	68	55	39.5	5.23	136.64	43.02	0.00	361.21	0.03
R2R	DeepSeek V4.1-Flash	$\checkmark$	$\times$	99	98	70.8	1.21	118.44	44.72	3.21	459.12	0.03

Table 5: Navigation performance and efficiency across different benchmarks and interaction-memory settings. In the Interaction column,

\checkmark

and

\times

indicate whether active interaction is enabled or disabled, respectively. In the Memory column,

\times

denotes a first-visit run in an unseen environment, while

\checkmark

denotes a revisit run that reuses memory retained from the first visit. Memory accumulated during the current run remains available in both settings. ND denotes the mean number of completed tool-level decisions; NQ denotes the mean number of active interaction exchanges. For LHPR-VLN and GOAT-Bench, ND and NQ are reported as complete task / subtask, where one complete task is a continuous sequence. Subtask averages are computed by dividing the total counts by the total number of subtasks. Steps, time, and cost are averaged per episode, or per complete sequence for the continuous benchmarks.

Model	OSR $\uparrow$ (%)	SR $\uparrow$ (%)	SPL $\uparrow$ (%)	NE $\downarrow$ (m)	Steps $\downarrow$	ND $\downarrow$	NQ $\downarrow$	Time $\downarrow$ (s)	Cost $\downarrow$ ($/ep.)
GPT-6 Astra (ultra)	100	100	82.2	0.95	70.12	15.57	1.66	164.08	1.06
Qwen3.8-Max (xhigh)	93	91	70.2	1.71	104.90	72.98	1.59	906.20	0.46
DeepSeek V4.1-Flash (max)	99	98	70.8	1.21	118.44	44.72	3.21	459.12	0.03

Table 6: Backbone comparison of EvoLife on R2R-CE. ND denotes the mean number of completed tool-level decisions; NQ denotes the mean number of active interaction exchanges.

Benchmark	OSR $\uparrow$ (%)	SR $\uparrow$ (%)	SPL $\uparrow$ (%)	NE $\downarrow$ (m)	TL (m)
R2R-CE	91.0	90.0	74.0	1.44	11.21
RxR-CE	97.0	95.0	71.6	1.06	17.51
HM3D-OVON	72.0	69.0	51.1	1.57	7.44
HM3D v2	92.0	89.0	57.5	0.31	8.57

Table 7: Navigation performance beyond arrival confirmation.

Figure 4: Qualitative example of active interaction and persistent memory in R2R-CE. The agent actively asks whether a nearby wooden door is the intended destination or whether it should continue through a narrow archway. Based on the guide’s response, the agent adjusts its navigation direction, proceeds through the archway, and records its entrance as place P2. Q2 and A2 denote the agent’s second interaction question and the guide’s corresponding answer, respectively, while P2 denotes the stored place record.

Figure 5: Failure case analysis on GOAT-Bench. Left: the agent successfully completes the picture-finding subtask. Right: the agent subsequently identifies the bookcase with diamond-shaped compartments through active interaction, but stops 0.73 m from the reference target, exceeding the required 0.25 m success threshold. Orange dots on the maps indicate the agent’s position. $P_{i}$ denotes a saved place in persistent memory. $C_{i}$ labels candidate waypoint $i$ .

Figure 6: Three consecutive real-world robot navigation tasks across two days. In Task 1, the robot navigates from room 625 to the elevator on the opposite side of the floor, recording key locations along the route. In Task 2, the robot returns to room 625 using persistent memory and actively requests human assistance when encountering a closed door. In Task 3, the robot identifies and restores the matching saved map on the following morning and navigates to the glass office visited the previous day. The three trajectories are visualized in a shared coordinate frame, demonstrating active interaction, adaptation to environmental changes, and cross-day memory reuse.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1: Hybrid navigation on R2R-CE. The instruction requires passing through a bathroom to reach a gym. The observed map expands as the agent moves from the initial hallway through a bright workroom and the tiled bathroom. step and navigate_candidate requests provide complementary ways to expose and traverse the connecting doorways. The final view contains exercise equipment, and an explicit stop submits the completed task. Orange dots on the maps indicate the agent’s position after executing the corresponding decision. $P_{i}$ denotes a saved place in persistent memory. In the panoramas, $C_{i}$ labels candidate waypoint $i$ , and green markers indicate the selected waypoint or pixel. $D_{i}$ denotes the $i$ -th tool-level decision.

Figure S2: Pixel navigation and landmark recording on RxR-CE. The agent follows the passage between a couch and kitchen island, identifies the instructed right branch, and enters the room containing the sink. Pixel selections express local floor destinations, while a remember_place request records the eye-chart junction and its distinguishing geometry. The final observation shows the sink directly ahead before the task is submitted. Orange dots on the maps indicate the agent’s position after executing the corresponding decision. $P_{i}$ denotes a saved place in persistent memory. In the panoramas, green markers indicate the selected waypoint or pixel. $D_{i}$ denotes the $i$ -th tool-level decision.

Figure S3: Object grounding on HM3D v2. Starting from a living area, the agent approaches the hallway and bathroom entrance through visible free floor. Once the potted plant beside the vanity is visible, localize_object selects its image region with a bounding box. The subsequent approach ends beside the bathroom sink, where the agent submits an explicit stopping decision. Orange dots on the maps indicate the agent’s position after executing the corresponding decision. $P_{i}$ denotes a saved place in persistent memory. In the panoramas, $C_{i}$ labels candidate waypoint $i$ , and green markers indicate the selected waypoint or pixel. $D_{i}$ denotes the $i$ -th tool-level decision.

Figure S4: Active guidance during open-vocabulary object navigation. After entering the dining area, the agent asks where to look for a decorative plant. The guide identifies a tall white pot beneath artwork beside the blue fireplace wall and describes an aisle around the dining table. A subsequent candidate request follows that aisle, and the final view shows the agent beside the indicated plant. Orange dots on the maps indicate the agent’s position after executing the corresponding decision. $P_{i}$ denotes a saved place in persistent memory. In the panoramas, $C_{i}$ labels candidate waypoint $i$ , and green markers indicate the selected waypoint or pixel. $D_{i}$ denotes the $i$ -th tool-level decision.

Figure S5: Instance disambiguation on VL-LN. The category-level request to find a table is clarified through a description of a pale stone rectangular top on a central pedestal beneath an arched recess. The agent follows the narrow aisle, selects a point on the observed table for localization, and uses local movement to adjust its position. Its final summary refers to the same structural and spatial attributes. Orange dots on the maps indicate the agent’s position after executing the corresponding decision. $P_{i}$ denotes a saved place in persistent memory. In the panoramas, $C_{i}$ labels candidate waypoint $i$ , and green markers indicate the selected waypoint or pixel. $D_{i}$ denotes the $i$ -th tool-level decision.

Figure S6: Iterative clarification on CoIN. The agent first distinguishes the requested dresser from one beside a turquoise bed. A later question reveals that the furniture in a small pink-bed bedroom is also incorrect. Further guidance identifies a blue-runner passage through the kitchen. The final question verifies the long white dresser beneath a wide mirror in the larger bedroom. Orange dots on the maps indicate the agent’s position after executing the corresponding decision. $P_{i}$ denotes a saved place in persistent memory. In the panoramas, $C_{i}$ labels candidate waypoint $i$ , and green markers indicate the selected waypoint or pixel. $D_{i}$ denotes the $i$ -th tool-level decision.

Figure S7: Continuous two-stage navigation on LHPR-VLN. The agent first reaches the dining-room tray and submits that stage with stop . It then calls navigate_place with the saved start location to return toward the initial stair landing. A pixel request selects the bathroom opening, and the recorded execution continues with candidate movements before the final stop at the wash cabinet. Both destinations are visited in one continuous sequence. Orange dots on the maps indicate the agent’s position after executing the corresponding decision. $P_{i}$ denotes a saved place in persistent memory. In the panoramas, green markers indicate the selected pixel. $D_{i}$ denotes the $i$ -th tool-level decision.

Figure S8: Selected stages of a seven-subtask GOAT-Bench sequence. The goals combine language descriptions, reference images, and an object category. The execution panels show the initial mirror search, completion of the image-specified plant subtask, and completion of the final chest subtask. The observed map retains exploration across these stages. Only subtasks 1, 2, and 7 are shown in the execution panels. Orange dots on the maps indicate the agent’s position after executing the corresponding decision. $P_{i}$ denotes a saved place in persistent memory. $D_{i}$ denotes the $i$ -th tool-level decision.

Embodied Navigation

Embodied navigation studies how robots perceive, map, and move through environments to reach goals and follow instructions.

See all 139 papers

Lifelong Embodied Navigation Learning
Mar 6, 2026Xudong Wang, Jiahua Dong, Baichen Liu +3Embodied Navigation Continual Learning for LLM Agents
State Key Laboratory of Robotics and Intelligent Systems, Shenyang Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Mohamed bin Zayed University of Artificial Intelligence
NavHarness: Towards Lifelong Embodied Navigation
Sep 28, 2026Xunyi Zhao, Jian Zhou, Sihao Lin +6Embodied Navigation Memory-Augmented Agents
Adelaide University · Responsible AI Research Centre, AIML · CSIRO Data61
EvoNav-Bench: Benchmarking Lifelong Navigation in Evolving Environments
Sep 8, 2026Xilin Wang, Guoxi Zhang, Hongming Xu +3Embodied Navigation Robot Navigation
State Key Laboratory of General Artificial Intelligence, BIGAI · 2Tsinghua University

Mar 6, 2026J/K move · Enter open · S save

Lifelong Embodied Navigation Learning

Xudong Wang, Jiahua Dong, Baichen Liu, Qi Lyu+2

State Key Laboratory of Robotics and Intelligent Systems, Shenyang Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Mohamed bin Zayed University of Artificial Intelligence

Embodied Navigation Continual Learning for LLM Agents Vision-Language Navigation Low-Rank Adaptation

Open paper AskPDF arXiv

Embodied navigation agents powered by large language models have shown strong performance on individual tasks but struggle to continually acquire new navigation skills, which suffer from catastrophic forgetting. We formalize this challenge as lifelong embodied navigation learning (LENL), where an agent is required to adapt to a sequence of navigation tasks spanning multiple scenes and diverse user instruction styles, while retaining previously learned knowledge. To tackle this problem, we propose Uni-Walker, a lifelong embodied navigation framework that decouples navigation knowledge into task-shared and task-specific components with Decoder Extension LoRA (DE-LoRA). To learn the shared knowledge, we design a knowledge inheritance strategy and an experts co-activation strategy to facilitate shared knowledge transfer and refinement across multiple navigation tasks. To learn the specific knowledge, we propose an expert subspace orthogonality constraint together and a navigation-specific chain-of-thought reasoning mechanism to capture specific knowledge and enhance instruction-style understanding. Extensive experiments demonstrate the superiority of Uni-Walker for building universal navigation agents with lifelong learning.

Abstract

Embodied Navigation

Explore similar work

Lifelong Embodied Navigation Learning

NavHarness: Towards Lifelong Embodied Navigation

EvoNav-Bench: Benchmarking Lifelong Navigation in Evolving Environments