Robots that work alongside people should be able to use what they have seen when new requests arise. The same experience may later support identifying an object or returning it to its original location, although these uses were unknown during observation. ReuseMem learns to use one visual memory for multiple later manipulation goals. It records visual features independently of future goals, retrieves complementary evidence for each request, and combines it with the current scene through a lightweight action interface. Retrieval is supervised by how well candidate evidence supports expert-action prediction across different current states, linking memory selection to its use in manipulation. Repeated visual features are stored once, and memory computations that remain unchanged during action generation are reused. Experiments on four simulation tasks and two physical scenarios demonstrate reuse of a common recording across different requests. With matched action adaptation, ReuseMem improves joint goal completion on the benchmark tasks by 24.5 percentage points over uniform memory. These findings show that one memory can support multiple later manipulation goals as the scene changes.
Figures & tables
Figure 1: Overview of ReuseMem. The robot records case contents and origins before later requests are known. The same memory supplies different historical evidence for each goal, while current observations guide the next manipulation.
Figure 2: ReuseMem architecture. The Writer avoids duplicate appearance storage while retaining a separate temporal entry for each observation. A later request retrieves complementary evidence from this common bank, and current-scene fusion connects it to the action policy.
Figure 3: Retrieving complementary evidence. The Reader searches candidate anchors, selects reference observations, and gathers temporally related support. Coverage frames preserve context across the recording. Deduplication and filling produce 24 historical slots for the action policy.
Figure 4: From memory to action. Retrieved history and recent execution observations are fused with current context, then combined with current-frame tokens. The action expert generates fresh action queries using this memory input. Memory keys and values are prepared once per sampling call.
Figure 5: Learning retrieval and action use. Expert-action prediction across current states compares candidate evidence selections to supervise the Reader. The updated Reader then supplies the history used to train fusion and low-rank modulation, while the base policy stays frozen.
1
ψˉ←sg(ψ) ; Ge+←∅ .
2
for each g∈Ge do
3
Zg←ProposeUniqueϕ(Be0,g;C) ; Cg←∣Zg∣ .
4
Compute Ug,c and pg by Eq. ( 4 ), with fixed expert/noise operands.
5
if Cg>1 and (maxcUg,c−mincUg,c)/T>δ , add g to Ge+ .
6
end for
7
if Ge+=∅ : ϕ←OptStepϕ(ϕ,∇ϕLϕ) by Eq. ( 4 ).
Algorithm 1 Learning retrieval and action use
Figure 6: Simulation tasks and later goals. Each row shows a common observation history, the current scene, and the manipulation specified by each later goal. The same past determines two destinations or targets through event timing, order, concealed identity or transient highlighting.
Method
Goal (%)
All-goal (%)
Sequence (%)
No-Memory ( π0.5 )
25.3 (101/400)
5.5 (11/200)
7.5 (15/200)
MME-VLA ( π0.5 )
32.5 (130/400)
14.5 (29/200)
11.5 (23/200)
MemER (RoboMME)
39.3 (157/400)
20.0 (40/200)
20.5 (41/200)
Uniform-Memory ( π0.5 )
54.0 (216/400)
36.0 (72/200)
35.0 (70/200)
ReuseMem
74.5 (298/400)
60.5 (121/200)
58.0 (116/200)
Table 1: Success on four simulation tasks. Each task contributes 50 common recordings with two later goals. Goal pools 400 independent requests; All-goal and Sequence each use 200 recordings. All-goal combines reset branches, whereas Sequence requires both requests without reset.
Figure 7: All-goal success by task. Each point is the proportion of 50 common recordings for which both independent goals succeed. Learned evidence selection improves reuse across event-relative, temporal, hidden-identity and transient-cue tasks.
Figure 8: Learning retrieval and current-scene action use. Bars compare independent All-goal success and no-reset Sequence success over 200 common recordings. Controls share the prepared policy, expert data and action-training budget.
Figure 9: Real-world manipulation sequences. Each row shows a common observation history, the scene before requests, and three successive goals. (a) Case contents determine the transfer targets, and the recorded origin guides return. (b) The same toy origin supports relocation, localized brushing and restoration from the changing scene.
Method
G1
G2
G3
Goal
All-goal
Sequence
Case preparation and return
No-Memory
50
50
20
40.00
5
20
Uniform-Memory
75
75
60
70.00
45
50
ReuseMem
85
85
85
85.00
70
65
Toy relocation, cleaning and restoration
No-Memory
50
45
45
46.67
10
10
Table 2: Physical manipulation success (%). Each scenario uses 20 common recordings. Goal averages 60 independent requests; All-goal and Sequence require all three independent or sequential requests, respectively.
Figure 10: Success, response and retained memory. Left: new-request response versus All-goal success. Right: bank storage for matched policy configurations at 2,048 observations. Sharing reduces stored visual content while retaining all temporal entries.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Value
Optimizer
AdamW
Reader/index learning rate
10−4
Action-interface learning rate
5×10−5
Batch size
4 experience groups
Examples per group
2 goals × 2 current states
Training budget
Up to 20,000 steps
Appendix
Table 3: Optimization settings. Checkpoints are selected using validation performance; test recordings are reserved for final evaluation.
Component
Setting
Appearance pool
Immutable 16×1024 BF16 appearance prototypes
Memory entry
Original ordinal, prototype reference, validity, metadata, and 256-dimensional key
Causal access index
Width 256; two blocks; four heads; 64-row local window
Reference/support search
Eight searched anchors; four references; three supports per reference
Historical packet
Eight coverage rows; stable deduplication/fill to 24 slots
Training group
Associated goals; two current-scene queries per goal
Appendix
Table 4: Model configuration. Frame budgets, representation widths, and adaptation ranks have different roles.
1
qg←Poolϕ(g),P←() .
2
for u=1,…,8 do
3
iu←Selectμ(fϕref(⋅;qg,P),I∖P);P←P∥iu .
4
end for
5
R←Selectμ(4)(fϕemit(⋅;qg),P) .
6
for u=1,…,4 , with ru=Ru do
7
Au←Selectμ(3)(ℓ⋅∣ru,g,I) if ru=⊥ ; otherwise Au←(⊥,⊥,⊥) .
Appendix
Algorithm 2 Reference and support retrieval
1
Initialize an empty bank B .
2
for each observation (oi,ηi) in He do
3
Compute frozen Ai ; match or append a prototype by Eq. ( 8 ).
4
Append its occurrence metadata and reference pi ; update ki by Eq. ( 1 ).
5
end for ; commit Be0←B and initialize Je←() .
6
for each subsequent policy call t do
7
Acquire (gt,ot,st) and update Je with newly available strict-past observations; discard unissued commands on a request change.
Appendix
Algorithm 3 Memory recording and policy execution
Method
G1
G2
All-goal
Sequence
VideoPlaceButton
No-Memory
13
14
3
4
MME-VLA
29
27
17
13
MemER
15
15
4
5
Uniform-Memory
34
33
26
25
ReuseMem
44
45
40
38
Appendix
Table 5: Per-task simulation counts. Every cell reports successes out of 50 common recordings. G1 and G2 use independent branches; All-goal requires both, and Sequence uses a separate two-request execution without reset.
Configuration
Goal (%)
All-goal (%)
Sequence (%)
Without retrieval learning
54.0 (216/400)
36.5 (73/200)
34.0 (68/200)
Single-state supervision
69.0 (276/400)
53.0 (106/200)
49.0 (98/200)
Without current fusion
70.5 (282/400)
55.5 (111/200)
47.0 (94/200)
ReuseMem
74.5 (298/400)
60.5 (121/200)
58.0 (116/200)
Without feature sharing
74.8 (299/400)
61.0 (122/200)
58.0 (116/200)
Without within-call KV cache
74.5 (298/400)
60.5 (121/200)
58.0 (116/200)
Appendix
Table 6: Complete ablation results. Goal pools 400 requests; All-goal and Sequence each use 200 recordings. Sufficient historical evidence supplies the annotated past evidence as a diagnostic control.
Method
Evidence coverage (%)
Goal success (%)
Uniform-Memory
80.5 (322/400)
54.0 (216/400)
Without retrieval learning
75.0 (300/400)
54.0 (216/400)
Single-state supervision
87.5 (350/400)
69.0 (276/400)
ReuseMem
90.0 (360/400)
74.5 (298/400)
Appendix
Table 7: Evidence selection and action success. Coverage requires the selected 24 historical observations to cover every necessary annotated fact group; equivalent views within a group are interchangeable. Each metric uses 400 goal queries.
Figure 11: Dependence on historical evidence. Removing relevant observations or supplying mismatched history reduces goal success more than removing unrelated observations. All conditions retain the original task truth and current inputs.
Stage
Time (ms)
ReuseMem
Observation, I/O and dispatch
90
Current vision and language
37
Reader
11
Reconstruction, fusion and memory K/V
10
Action sampling
27
Appendix
Table 8: New-request response components. Components sum to each method's complete instruction-to-first-command latency.
Figure 12: Response composition and within-call caching. Left: instruction-to-first-command response split into input/dispatch, memory processing and action-policy computation. Memory processing includes the high-level VLM in MemER. Right: memory-K/V caching compares continuing-goal calls and reduces mean response by 9 ms.
Long-horizon manipulation requires robots to retain task-relevant history while revising conclusions no longer supported by physical evidence. Retaining observations alone does not specify which completed conditions should be withdrawn after a disturbance or which obligations require repair. We introduce REMM, Revisable Embodied Memory for Manipulation, a supervisory memory layer for a frozen G0.5 vision-language-action executor. REMM separates an append-only evidence history from revisable task state and explicitly links derived relations and completed conditions to their supporting facts. When accepted evidence changes a supporting fact, it withdraws dependent conclusions and schedules targeted recovery at verified skill boundaries without updating the executor. Across 51 physical trials, including operator-assisted runs, REMM improves mean recorded task progress over the strongest baseline from 57% to 91% in intervention recovery and from 43% to 70% in sequence recall. Complementary video diagnostics show improved grounding of retained history into the next instruction, including comparisons with π0.5-compatible high-level planner adaptations. These results support revisable memory as a practical interface for continued manipulation without policy retraining. Project: https://revanthsenthil.github.io/remm/.
Mobile manipulators deployed across many rooms and visits should improve with experience: after discovering that a cabinet is locked or finding an object in a drawer, the robot should reuse that knowledge rather than start each task from scratch. Yet today's robots often treat each task as new: compact scene representations omit interaction-derived knowledge, raw video histories are difficult to query, and VLM planners reason at inference time without persistently updating what the robot knows. We present MessyMem, a persistent memory system that enables mobile manipulators to learn from experience and reuse that knowledge across future tasks. It maintains a spatially grounded 3D scene graph of objects and locations, augments it with properties and outcomes learned through interaction, and links visual observations for fine-grained recall. We evaluate MessyMem in simulation and on a real mobile manipulator. In a continuous 25-task simulation spanning over 3 hours, MessyMem achieves 80.0% task progress, outperforming the strongest ablation by 14.8 percentage points and the strongest external baseline by 28.9 points, while retrieving task-relevant evidence from thousands of stored keyframes and over an hour into the past.
Anuva Banwasi, William Muckelroy, Priya Sundaresan +3
Memory-dependent robotic manipulation requires policies to use information that is no longer available in the current observation. Retaining history alone is insufficient: memory must preserve information that supports future actions. One challenge is whether a memory-free foundation model can learn to retain and use historical information from action demonstrations alone, without external memory support. We introduce T2Mem, a framework that develops this capability within a pretrained vision-language-action policy, without external reasoning models or memory-specific annotations. T2Mem uses test-time training to encode observation history into compact fast weights through online self-supervised updates, avoiding repeated processing of the full history. An observation-grounded interface extracts vision-language information for memory formation and supplies retrieved context to the action expert. Action supervision shapes what the memory learns to retain and use, while alternating memory-policy learning gives each component a fixed counterpart during optimization. Across 16 RoboMME tasks, T2Mem improves average success from 17.93% to 56.83% over the memory-free base policy and outperforms the recurrent-memory methods reported in the benchmark, while controlled profiling indicates at least 3x inference speedup over explicit methods. Project website: https://yzliu84.github.io/T2MEM-project/
Yize Liu, Huang Huang, Yining Hong +4
Stanford University · NVIDIA · University of Michigan, Ann Arbor