cs.ROOct 24, 2026
SaveReSight: Task-Driven Self-Improving Memory for Embodied Agents
Organizations: Tsinghua University · AGIBOT · Nanjing University · Nanyang Technological University
Abstract
Robots can use historical observations to answer spatiotemporal queries in familiar environments, but the presence of relevant evidence does not imply that existing tools support the operations a task requires. We introduce ReSight, a task-driven self-improving memory system for embodied agents that expands the capabilities for processing and accessing historical observations as task demands arise. ReSight retains keyframes and their spatiotemporal associations, separating observation evidence from extensible derived fields so that stored observations can be reprocessed for new needs. An execution agent and a coding agent share this memory through different interfaces: the former calls registered tools within a bounded online budget, while the latter uses a programmable workspace and a larger exploration budget to inspect evidence, write programs, and acquire models as needed. To realize capability expansion in ReSight, we develop Explore-Distill-Mount (EDM), which identifies capability gaps from execution and exploration records without task reference answers and converts exploration-derived solutions into reusable memory builders, query tools, and skills, extending memory representations and access capabilities as needed. Experiments on three public benchmarks demonstrate effective base memory access and capability expansion. With the execution model and online budget held fixed within each benchmark, EDM reduces mean position error on OC-NaVQA from 33.10 m to 24.72 m and improves high-level success on FindingDory from 38.3% to 63.8%. With the tools and skill frozen, success still improves by 28.3 percentage points on FindingDory episodes excluded from exploration and distillation. Experiments on real robot patrol memories further demonstrate reuse of these capabilities for subsequent queries and new memories, with fewer online tool calls on average.
Figures & tables
Figure 1: ReSight retains revisitable visual evidence and expands its processing capabilities through EDM. Evaluation across scan-based indoor simulation (GOAT-core), interactive household simulation (FindingDory), real campus recordings (OC-NaVQA), and real-robot office patrols demonstrates effective memory retrieval and task-driven capability expansion.
Figure 2: ReSight architecture. The two agents access the same evidence memory through different interfaces. EDM converts exploration-derived solutions into tools, skills, and memory builders for reuse by the execution agent.
Figure 3: EDM on robot-memory booth counting. After text-based retrieval fails, the coding agent develops photo retrieval and verifies two booths. Executor replay during distillation reveals that revisits confuse counting, prompting pose-and-time merging; the mounted tool returns both booths and supporting frames in one call.
| Method | Image | Object | Language | Average |
|---|---|---|---|---|
| DynaMem ( Liu et al., 2025 ) | 16.1 | 7.5 | 5.1 | 9.6 |
| VLMaps ( Huang et al., 2023 ) | 43.3 | 69.7 | 61.0 | 58.8 |
| LagMemo ( Zhou et al., 2025 ) | 56.4 | 88.4 | 66.8 | 70.8 |
| OpenFunGraph ( Zhang et al., 2025a ) | 53.4 | 67.1 | 69.0 | 63.1 |
| HOV-SG ( Werby et al., 2024 ) | – | 70.8 | 82.3 | 76.5 |
| MoM ( Zhou et al., 2026 ) | 80.1 | 79.5 | 82.3 | 80.6 |
Table 1: GOAT-core SR@5 (%); no EDM. The first six rows are from MoM Table I ( Zhou et al., 2026 ) ; HOV-SG averages two supported query types. Retrieval-only variants use no query-time VLM reasoning. Qwen/Sol denote annotation models (Appendix C.2 ); bold marks group bests.
Figure 4: Examples of EDM capability expansion: (a) object localization, (b) interaction-object tracking, and (c) OCR sign retrieval. Columns show execution before mounting, the generated programs, and execution after mounting. Call counts are per illustrated run; full trajectories appear in Appendix E .
| Accuracy (%) | Position | Temporal | |||
| Method | Binary | Text | Mean (m) | 10 m (%) | Error (min) |
| ReMEmbR-2B ( Anwar et al., 2024 ) | 43.2 | – | 53.47 | – | 2.287 |
| ReMEmbR-8B | 46.3 | – | 55.89 | – | 4.106 |
| ConceptGraphs ( Gu et al., 2023 ) | 29.9 | – | 111.29 | – | – |
| DAAAM-Mini ( Gorlo et al., 2026a ) | 71.1 | – | 41.75 | – | 1.792 |
| DAAAM-Sol | 80.6 | 57.6 | 36.34 | 52.1 | 2.225 |
Table 2: OC-NaVQA results on 210 questions. The first four rows are from DAAAM ( Gorlo et al., 2026a ) : Mini is GPT-5-mini; 2B/8B are NVILA-Lite models. RAVEN-Sol and ReSight-Sol † return camera poses. Bold compares Sol configurations. Orange denotes the Opus-5 exploration reference.
| Method | Spatial | Temporal | Multi-goal | Overall | Tokens/q (k) |
| Random | 13.4 | 6.9 | 0.5 | 9.2 | – |
| Embedder Only | 31.0 | 8.1 | 3.4 | 18.8 | – |
| VLM Only-Gemini | 36.0 | 28.1 | 2.2 | 28.2 | – |
| ReMEmbR-Gemini ( Anwar et al., 2024 ) | 36.7 | 17.1 | 4.5 | 25.0 | – |
| RAVEN-Gemini ( Hu et al., 2026 ) | 45.1 | 24.8 | 6.7 | 32.2 | – |
| RAVEN-Sol | 51.9 | 34.8 | 5.6 | 38.9 | 378 |
Table 3: FindingDory high-level success (%) on episodes 91–100 (596 queries). The first five rows are from RAVEN ( Hu et al., 2026 ) ; Gemini is Gemini-2.5-Flash. Bold compares Sol configurations; orange denotes the Opus-5 exploration reference. Tokens/q reports mean input-plus-output tokens in thousands. Annotation-model results appear in Appendix C.5 .
| Method | Spatial | Temporal | Multi-goal | Overall | Tokens/q (k) |
|---|---|---|---|---|---|
| ReSight-Sol (w/o EDM) | 48.3 | 36.2 | 4.5 | 37.5 | 128.8 |
| ReSight-Sol (EDM w/o Explore) | 51.0 | 48.6 | 25.8 | 46.4 | 26.9 |
| ReSight-Sol (w/ EDM) | 57.4 | 83.3 | 52.8 | 65.8 | 39.4 |
Table 4: Frozen-capability reuse on FindingDory episodes 81–90 (597 queries). EDM w/o Explore distills from the execution traces (Appendix C.3 ). High-level success (%); Tokens/q reports mean input-plus-output tokens in thousands.
| Success | Tool calls | Tokens/run (k) | ||||
| Task | w/o EDM | w/ EDM | w/o EDM | w/ EDM | w/o EDM | w/ EDM |
| Warmth Corner (8F) | 2/5 | 5/5 | 8.4 | 1.0 | 60.0 | 10.4 |
| Farthest point (8F) | 1/5 | 5/5 | 7.2 | 1.0 | 38.8 | 10.0 |
| Booth count (8F) | 0/5 | 5/5 | 7.6 | 1.0 | 50.2 | 12.9 |
| Room 805 (8F) | 5/5 | 5/5 | 2.0 | 1.0 | 12.5 | 11.2 |
| Patrol length (1F) | 0/5 | 5/5 | 3.4 | 1.8 | 14.5 | 15.1 |
Table 5: Real-robot memory queries, grouped into exploration and new tasks. Success is out of five runs; tool calls and input-plus-output tokens are per-run means. The last row aggregates 30 runs; bold marks pairwise bests. Queries and inputs: Appendix C.4 .
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
| Tool | Arguments | Function |
|---|---|---|
| semantic_search | query , [ top_k ] | RRF semantic search |
| recall_around_time | clock_time , [ limit ] | Time-neighbor recall |
| recall_near_position | position , [ limit ] | Pose-neighbor recall |
| inspect_frames | frame_ids , [ focus ] | Show original images |
| submit_answer | type , reasoning , source_frame_ids , type-specific answer field | Submit answer |
Table 6: Base execution interface for OC-NaVQA and FindingDory. Retrieval tools return text summaries; position recall requires camera poses.
| Host / benchmark | Semantic / temporal return | Position return | Rounds |
|---|---|---|---|
| ReSight / OC | 12 text / 8 default, 12 max | 8 default, 12 max text | 6 |
| ReSight / FD | 12 text / 8 default, 12 max | None | 15 |
| RAVEN / OC | 5 each, with images | 5 with images | 3 |
| RAVEN / FD | 30 each, with images | None | 15 |
Table 7: Base retrieval limits per call and decision-round caps. EDM keeps each host’s limits unchanged. Added tools can return event summaries, frame ids, or rendered panels.
| Query type | w/o EDM | EDM w/o Explore | w/ EDM | |
|---|---|---|---|---|
| Interaction Order | 160 | 31.9 | 46.9 | 82.5 |
| Conditional Interaction | 68 | 42.6 | 44.1 | 51.5 |
| Interaction | 60 | 68.3 | 68.3 | 73.3 |
| Unordered Revisitation | 59 | 6.8 | 18.6 | 30.5 |
| Room Visitation | 50 | 74.0 | 72.0 | 86.0 |
| Spatial Relationship | 50 | 4.0 | 8.0 | 12.0 |
Table 8: Success (%) on FindingDory episodes 81–90 by the benchmark’s low-level query category; is the number of queries.
| Accuracy (%) | Position | Temporal | ||||
|---|---|---|---|---|---|---|
| Annotator / configuration | Binary | Text | Mean (m) | 10 m (%) | Error (min) | Tokens/q (k) |
| GPT-5.6-sol (w/o EDM) | 86.6 | 63.6 | 33.10 | 19.7 | 0.587 | 18.8 |
| GPT-5.6-sol (w/ EDM) | 82.1 | 63.6 | 24.72 | 36.6 | 0.475 | 20.4 |
| Qwen3.8-27B (w/o EDM) | 85.1 | 69.7 | 31.54 | 16.9 | 0.427 | 19.1 |
| Qwen3.8-27B (w/ EDM) | 85.1 | 69.7 | 28.29 | 35.2 | 0.463 | 21.3 |
Table 9: OC-NaVQA annotator replacement on 210 questions. All rows use the GPT-5.6-sol executor. Tokens/q reports mean online input-plus-output tokens in thousands. Green denotes configurations with EDM; bold marks column bests.
| Annotator / configuration | Spatial | Temporal | Multi-goal | Overall | Tokens/q (k) |
|---|---|---|---|---|---|
| GPT-5.6-sol (w/o EDM) | 45.5 | 41.0 | 7.9 | 38.3 | 130 |
| GPT-5.6-sol (w/ EDM) | 54.2 | 82.9 | 50.6 | 63.8 | 43 |
| Qwen3.8-27B (w/o EDM) | 48.8 | 37.6 | 7.9 | 38.8 | 144 |
| Qwen3.8-27B (w/ EDM) | 54.2 | 87.1 | 50.6 | 65.3 | 42 |
Table 10: FindingDory annotator replacement on episodes 91–100 (596 queries). High-level success (%); all rows use the GPT-5.6-sol executor. Tokens/q reports mean online input-plus-output tokens in thousands. Green denotes configurations with EDM; bold marks column bests.
Figure 5: Mean online tokens per query (thousands). Gray denotes the base host and green its EDM-mounted counterpart. The two benchmarks use separate horizontal scales.
Figure 6: Mean online model requests per query. ReSight includes decision and finalization requests; RAVEN includes agent decisions and answer generation; DAAAM uses step-level request logs.
Figure 7: OC-NaVQA: red semi-truck. Both executions cite frame 67; EDM uses several views to estimate the truck’s position rather than returning the camera pose.
Figure 8: OC-NaVQA: green Lime scooters. The base execution misses the target. EDM’s presence scan narrows the candidate frames for image verification.
Figure 9: FindingDory: interaction time. The base execution cites a frame before placement; EDM identifies the interaction and cites evidence after placement.
Figure 10: FindingDory: the object with the longest rearrangement. The base execution estimates duration from retrieved frames; EDM ranks indexed interactions by duration and checks the selected object’s evidence. The repeated word “to” is retained from the original benchmark question.
Figure 11: Robot patrol, held-in: Warmth Corner sign. The base execution selects an unrelated recognition wall; text lookup returns a frame with the legible sign.
Figure 12: Robot patrol, held-in: farthest point. EDM computes the farthest frame from all 648 stored poses. The plotted trajectory is reconstructed from stored poses.
Figure 13: Robot patrol, held-in: booth count. Under the same short prompt and reference photo, the base execution answers zero, while EDM identifies both booths (frames 442 and 152).
Figure 14: Robot patrol, held-out: room 805 sign. Both executions answer correctly; the mounted text index reduces the count from three model requests to two.
Figure 15: Robot patrol, held-out: total distance walked. The base execution estimates 92.7 m from route fragments; EDM sums consecutive positions across 501 frames to obtain 111.0 m. The plotted trajectory is reconstructed from stored poses.
Figure 16: Robot patrol, held-out: independent phone-photo localization. Both executions answer correctly. In this paired run, EDM uses one tool call and two model requests, versus two tool calls and three model requests without EDM.
Figure 17: Visiting and inspecting two phone booths on floor 8. After the memory query identifies two booths, the user asks: “Help me plan a route to visit these two phone booths right now and confirm in person whether anyone is there.” The robot visits booth B and then booth A, checks their current occupancy, and reports that both are empty.
Figure 18: Navigating to a photo-matched location on floor 1. After the memory query locates the reference photograph, the user asks: “Go there and wait for me.” The robot navigates to the retrieved viewpoint, checks a fresh camera image, and waits at the destination.