Robots can use historical observations to answer spatiotemporal queries in familiar environments, but the presence of relevant evidence does not imply that existing tools support the operations a task requires. We introduce ReSight, a task-driven self-improving memory system for embodied agents that expands the capabilities for processing and accessing historical observations as task demands arise. ReSight retains keyframes and their spatiotemporal associations, separating observation evidence from extensible derived fields so that stored observations can be reprocessed for new needs. An execution agent and a coding agent share this memory through different interfaces: the former calls registered tools within a bounded online budget, while the latter uses a programmable workspace and a larger exploration budget to inspect evidence, write programs, and acquire models as needed. To realize capability expansion in ReSight, we develop Explore-Distill-Mount (EDM), which identifies capability gaps from execution and exploration records without task reference answers and converts exploration-derived solutions into reusable memory builders, query tools, and skills, extending memory representations and access capabilities as needed. Experiments on three public benchmarks demonstrate effective base memory access and capability expansion. With the execution model and online budget held fixed within each benchmark, EDM reduces mean position error on OC-NaVQA from 33.10 m to 24.72 m and improves high-level success on FindingDory from 38.3% to 63.8%. With the tools and skill frozen, success still improves by 28.3 percentage points on FindingDory episodes excluded from exploration and distillation. Experiments on real robot patrol memories further demonstrate reuse of these capabilities for subsequent queries and new memories, with fewer online tool calls on average.
Figures & tables
Figure 1: ReSight retains revisitable visual evidence and expands its processing capabilities through EDM. Evaluation across scan-based indoor simulation (GOAT-core), interactive household simulation (FindingDory), real campus recordings (OC-NaVQA), and real-robot office patrols demonstrates effective memory retrieval and task-driven capability expansion.
Figure 2: ReSight architecture. The two agents access the same evidence memory through different interfaces. EDM converts exploration-derived solutions into tools, skills, and memory builders for reuse by the execution agent.
Figure 3: EDM on robot-memory booth counting. After text-based retrieval fails, the coding agent develops photo retrieval and verifies two booths. Executor replay during distillation reveals that revisits confuse counting, prompting pose-and-time merging; the mounted tool returns both booths and supporting frames in one call.
Method
Image ↑
Object ↑
Language ↑
Average ↑
DynaMem ( Liu et al., 2025 )
16.1
7.5
5.1
9.6
VLMaps ( Huang et al., 2023 )
43.3
69.7
61.0
58.8
LagMemo ( Zhou et al., 2025 )
56.4
88.4
66.8
70.8
OpenFunGraph ( Zhang et al., 2025a )
53.4
67.1
69.0
63.1
HOV-SG ( Werby et al., 2024 )
–
70.8
82.3
76.5
MoM ( Zhou et al., 2026 )
80.1
79.5
82.3
80.6
Table 1: GOAT-core SR@5 (%); no EDM. The first six rows are from MoM Table I ( Zhou et al., 2026 ) ; HOV-SG averages two supported query types. Retrieval-only variants use no query-time VLM reasoning. Qwen/Sol denote annotation models (Appendix C.2 ); bold marks group bests.
Figure 4: Examples of EDM capability expansion: (a) object localization, (b) interaction-object tracking, and (c) OCR sign retrieval. Columns show execution before mounting, the generated programs, and execution after mounting. Call counts are per illustrated run; full trajectories appear in Appendix E .
Accuracy (%)
Position
Temporal
Method
Binary ↑
Text ↑
Mean (m) ↓
≤ 10 m (%) ↑
Error (min) ↓
ReMEmbR-2B ( Anwar et al., 2024 )
43.2
–
53.47
–
2.287
ReMEmbR-8B
46.3
–
55.89
–
4.106
ConceptGraphs ( Gu et al., 2023 )
29.9
–
111.29
–
–
DAAAM-Mini ( Gorlo et al., 2026a )
71.1
–
41.75
–
1.792
DAAAM-Sol
80.6
57.6
36.34
52.1
2.225
Table 2: OC-NaVQA results on 210 questions. The first four rows are from DAAAM ( Gorlo et al., 2026a ) : Mini is GPT-5-mini; 2B/8B are NVILA-Lite models. RAVEN-Sol and ReSight-Sol † return camera poses. Bold compares Sol configurations. Orange denotes the Opus-5 exploration reference.
Method
Spatial ↑
Temporal ↑
Multi-goal ↑
Overall ↑
Tokens/q (k) ↓
Random
13.4
6.9
0.5
9.2
–
Embedder Only
31.0
8.1
3.4
18.8
–
VLM Only-Gemini
36.0
28.1
2.2
28.2
–
ReMEmbR-Gemini ( Anwar et al., 2024 )
36.7
17.1
4.5
25.0
–
RAVEN-Gemini ( Hu et al., 2026 )
45.1
24.8
6.7
32.2
–
RAVEN-Sol
51.9
34.8
5.6
38.9
378
Table 3: FindingDory high-level success (%) on episodes 91–100 (596 queries). The first five rows are from RAVEN ( Hu et al., 2026 ) ; Gemini is Gemini-2.5-Flash. Bold compares Sol configurations; orange denotes the Opus-5 exploration reference. Tokens/q reports mean input-plus-output tokens in thousands. Annotation-model results appear in Appendix C.5 .
Method
Spatial ↑
Temporal ↑
Multi-goal ↑
Overall ↑
Tokens/q (k) ↓
ReSight-Sol (w/o EDM)
48.3
36.2
4.5
37.5
128.8
ReSight-Sol (EDM w/o Explore)
51.0
48.6
25.8
46.4
26.9
ReSight-Sol (w/ EDM)
57.4
83.3
52.8
65.8
39.4
Table 4: Frozen-capability reuse on FindingDory episodes 81–90 (597 queries). EDM w/o Explore distills from the execution traces (Appendix C.3 ). High-level success (%); Tokens/q reports mean input-plus-output tokens in thousands.
Success ↑
Tool calls ↓
Tokens/run (k) ↓
Task
w/o EDM
w/ EDM
w/o EDM
w/ EDM
w/o EDM
w/ EDM
Warmth Corner (8F)
2/5
5/5
8.4
1.0
60.0
10.4
Farthest point (8F)
1/5
5/5
7.2
1.0
38.8
10.0
Booth count (8F)
0/5
5/5
7.6
1.0
50.2
12.9
Room 805 (8F)
5/5
5/5
2.0
1.0
12.5
11.2
Patrol length (1F)
0/5
5/5
3.4
1.8
14.5
15.1
Table 5: Real-robot memory queries, grouped into exploration and new tasks. Success is out of five runs; tool calls and input-plus-output tokens are per-run means. The last row aggregates 30 runs; bold marks pairwise bests. Queries and inputs: Appendix C.4 .
Appendix figures & tables19 assets
Supplementary material from the paper’s appendix.
Appendix
Tool
Arguments
Function
semantic_search
query , [ top_k ]
RRF semantic search
recall_around_time
clock_time , [ limit ]
Time-neighbor recall
recall_near_position
position [x,y,z] , [ limit ]
Pose-neighbor recall
inspect_frames
frame_ids , [ focus ]
Show original images
submit_answer
type , reasoning , source_frame_ids , type-specific answer field
Submit answer
Appendix
Table 6: Base execution interface for OC-NaVQA and FindingDory. Retrieval tools return text summaries; position recall requires camera poses.
Host / benchmark
Semantic / temporal return
Position return
Rounds
ReSight / OC
12 text / 8 default, 12 max
8 default, 12 max text
6
ReSight / FD
12 text / 8 default, 12 max
None
15
RAVEN / OC
5 each, with images
5 with images
3
RAVEN / FD
30 each, with images
None
15
Appendix
Table 7: Base retrieval limits per call and decision-round caps. EDM keeps each host’s limits unchanged. Added tools can return event summaries, frame ids, or rendered panels.
Query type
n
w/o EDM
EDM w/o Explore
w/ EDM
Interaction Order
160
31.9
46.9
82.5
Conditional Interaction
68
42.6
44.1
51.5
Interaction
60
68.3
68.3
73.3
Unordered Revisitation
59
6.8
18.6
30.5
Room Visitation
50
74.0
72.0
86.0
Spatial Relationship
50
4.0
8.0
12.0
Appendix
Table 8: Success (%) on FindingDory episodes 81–90 by the benchmark’s low-level query category; n is the number of queries.
Accuracy (%)
Position
Temporal
Annotator / configuration
Binary ↑
Text ↑
Mean (m) ↓
≤ 10 m (%) ↑
Error (min) ↓
Tokens/q (k) ↓
GPT-5.6-sol (w/o EDM)
86.6
63.6
33.10
19.7
0.587
18.8
GPT-5.6-sol (w/ EDM)
82.1
63.6
24.72
36.6
0.475
20.4
Qwen3.8-27B (w/o EDM)
85.1
69.7
31.54
16.9
0.427
19.1
Qwen3.8-27B (w/ EDM)
85.1
69.7
28.29
35.2
0.463
21.3
Appendix
Table 9: OC-NaVQA annotator replacement on 210 questions. All rows use the GPT-5.6-sol executor. Tokens/q reports mean online input-plus-output tokens in thousands. Green denotes configurations with EDM; bold marks column bests.
Annotator / configuration
Spatial ↑
Temporal ↑
Multi-goal ↑
Overall ↑
Tokens/q (k) ↓
GPT-5.6-sol (w/o EDM)
45.5
41.0
7.9
38.3
130
GPT-5.6-sol (w/ EDM)
54.2
82.9
50.6
63.8
43
Qwen3.8-27B (w/o EDM)
48.8
37.6
7.9
38.8
144
Qwen3.8-27B (w/ EDM)
54.2
87.1
50.6
65.3
42
Appendix
Table 10: FindingDory annotator replacement on episodes 91–100 (596 queries). High-level success (%); all rows use the GPT-5.6-sol executor. Tokens/q reports mean online input-plus-output tokens in thousands. Green denotes configurations with EDM; bold marks column bests.
Figure 5: Mean online tokens per query (thousands). Gray denotes the base host and green its EDM-mounted counterpart. The two benchmarks use separate horizontal scales.
Figure 6: Mean online model requests per query. ReSight includes decision and finalization requests; RAVEN includes agent decisions and answer generation; DAAAM uses step-level request logs.
Figure 7: OC-NaVQA: red semi-truck. Both executions cite frame 67; EDM uses several views to estimate the truck’s position rather than returning the camera pose.
Figure 8: OC-NaVQA: green Lime scooters. The base execution misses the target. EDM’s presence scan narrows the candidate frames for image verification.
Figure 9: FindingDory: interaction time. The base execution cites a frame before placement; EDM identifies the interaction and cites evidence after placement.
Figure 10: FindingDory: the object with the longest rearrangement. The base execution estimates duration from retrieved frames; EDM ranks indexed interactions by duration and checks the selected object’s evidence. The repeated word “to” is retained from the original benchmark question.
Figure 11: Robot patrol, held-in: Warmth Corner sign. The base execution selects an unrelated recognition wall; text lookup returns a frame with the legible sign.
Figure 12: Robot patrol, held-in: farthest point. EDM computes the farthest frame from all 648 stored poses. The plotted trajectory is reconstructed from stored poses.
Figure 13: Robot patrol, held-in: booth count. Under the same short prompt and reference photo, the base execution answers zero, while EDM identifies both booths (frames 442 and 152).
Figure 14: Robot patrol, held-out: room 805 sign. Both executions answer correctly; the mounted text index reduces the count from three model requests to two.
Figure 15: Robot patrol, held-out: total distance walked. The base execution estimates 92.7 m from route fragments; EDM sums consecutive positions across 501 frames to obtain 111.0 m. The plotted trajectory is reconstructed from stored poses.
Figure 16: Robot patrol, held-out: independent phone-photo localization. Both executions answer correctly. In this paired run, EDM uses one tool call and two model requests, versus two tool calls and three model requests without EDM.
Figure 17: Visiting and inspecting two phone booths on floor 8. After the memory query identifies two booths, the user asks: “Help me plan a route to visit these two phone booths right now and confirm in person whether anyone is there.” The robot visits booth B and then booth A, checks their current occupancy, and reports that both are empty.
Figure 18: Navigating to a photo-matched location on floor 1. After the memory query locates the reference photograph, the user asks: “Go there and wait for me.” The robot navigates to the retrieved viewpoint, checks a fresh camera image, and waits at the destination.
Long-horizon manipulation requires robots to retain task-relevant history while revising conclusions no longer supported by physical evidence. Retaining observations alone does not specify which completed conditions should be withdrawn after a disturbance or which obligations require repair. We introduce REMM, Revisable Embodied Memory for Manipulation, a supervisory memory layer for a frozen G0.5 vision-language-action executor. REMM separates an append-only evidence history from revisable task state and explicitly links derived relations and completed conditions to their supporting facts. When accepted evidence changes a supporting fact, it withdraws dependent conclusions and schedules targeted recovery at verified skill boundaries without updating the executor. Across 51 physical trials, including operator-assisted runs, REMM improves mean recorded task progress over the strongest baseline from 57% to 91% in intervention recovery and from 43% to 70% in sequence recall. Complementary video diagnostics show improved grounding of retained history into the next instruction, including comparisons with π0.5-compatible high-level planner adaptations. These results support revisable memory as a practical interface for continued manipulation without policy retraining. Project: https://revanthsenthil.github.io/remm/.
Memory allows large language model (LLM) agents to use past experience, often stored as textual records, when responding to requests. For embodied agents that control robots, visual and spatial observations introduce a broader choice of memory modalities, affecting both the information available for subsequent tasks and the model inference costs of constructing and using memory. We introduce MeMoREval, a benchmark for studying this problem through a controlled experience-request protocol. Agents first experience a sequence of events without knowing the subsequent tasks, and subsequently they use memory to respond to information and action requests without receiving new observations. The benchmark comprises 53 tasks across four simulated indoor environments and measures task success alongside the inference costs of memory construction and use. Our reference evaluations reveal that storing frames outperforms storing generated descriptions at lower write-time inference cost for the primary model, while storing frames with scene graphs achieves the highest aggregate task success for every tested model. Extensions to accumulated experience, selective frame retention, and visual input to memory writers demonstrate how the benchmark accommodates further memory-design investigations. Two physical-robot demonstrations establish the feasibility of applying its protocol beyond simulation.
Junxiang Wang, Yuchen Liu, Yi Wu +6
Dexmate · Carnegie Mellon University · Purdue University
Spatial memory enables robots to reuse past observations for localization, navigation, and question answering. Designing such memory involves jointly deciding what to retain and how to retrieve it for diverse semantic, metric, and temporal needs. We investigate whether coding agents can automate this design process through iterative implementation and evaluation. Under a fixed development budget, a Designer Agent writes and revises code for memory construction and retrieval using feedback from a fixed Query Agent. The resulting implementations are frozen before building memories from held-out observation histories, without access to downstream questions, and evaluated across five complementary task tracks using the same Query Agent. Across three independent design searches, Codex / GPT-6 Astra averages 44.23 on the equally weighted five-track score, versus 29.32 for ReMEmbR, the strongest evaluated baseline supporting all five tracks, a 50.9% relative improvement. Qualitative analysis relates performance differences to evidence retention and retrieval, while identifying grounding and evidence-use failures. These results demonstrate the potential of coding agents to automate spatial memory design, while revealing task-dependent trade-offs in performance and computational cost. Project page: https://ba-fa-3344.github.io/.
Qianwei Wang, Ce Zhang, Zhanpeng Luo +3
Qianwei Wang is with Princeton University · Ce Zhang is with Carnegie Mellon University · Zhanpeng Luo is with Johns Hopkins University +2