Robots that work alongside people should be able to use what they have seen when new requests arise. The same experience may later support identifying an object or returning it to its original location, although these uses were unknown during observation. ReuseMem learns to use one visual memory for multiple later manipulation goals. It records visual features independently of future goals, retrieves complementary evidence for each request, and combines it with the current scene through a lightweight action interface. Retrieval is supervised by how well candidate evidence supports expert-action prediction across different current states, linking memory selection to its use in manipulation. Repeated visual features are stored once, and memory computations that remain unchanged during action generation are reused. Experiments on four simulation tasks and two physical scenarios demonstrate reuse of a common recording across different requests. With matched action adaptation, ReuseMem improves joint goal completion on the benchmark tasks by 24.5 percentage points over uniform memory. These findings show that one memory can support multiple later manipulation goals as the scene changes.
Figures & tables
Figure 1: Overview of ReuseMem. The robot records case contents and origins before later requests are known. The same memory supplies different historical evidence for each goal, while current observations guide the next manipulation.
Figure 2: ReuseMem architecture. The Writer avoids duplicate appearance storage while retaining a separate temporal entry for each observation. A later request retrieves complementary evidence from this common bank, and current-scene fusion connects it to the action policy.
Figure 3: Retrieving complementary evidence. The Reader searches candidate anchors, selects reference observations, and gathers temporally related support. Coverage frames preserve context across the recording. Deduplication and filling produce 24 historical slots for the action policy.
Figure 4: From memory to action. Retrieved history and recent execution observations are fused with current context, then combined with current-frame tokens. The action expert generates fresh action queries using this memory input. Memory keys and values are prepared once per sampling call.
Figure 5: Learning retrieval and action use. Expert-action prediction across current states compares candidate evidence selections to supervise the Reader. The updated Reader then supplies the history used to train fusion and low-rank modulation, while the base policy stays frozen.
1
ψˉ←sg(ψ) ; Ge+←∅ .
2
for each g∈Ge do
3
Zg←ProposeUniqueϕ(Be0,g;C) ; Cg←∣Zg∣ .
4
Compute Ug,c and pg by Eq. ( 4 ), with fixed expert/noise operands.
5
if Cg>1 and (maxcUg,c−mincUg,c)/T>δ , add g to Ge+ .
6
end for
7
if Ge+=∅ : ϕ←OptStepϕ(ϕ,∇ϕLϕ) by Eq. ( 4 ).
Algorithm 1 Learning retrieval and action use
Figure 6: Simulation tasks and later goals. Each row shows a common observation history, the current scene, and the manipulation specified by each later goal. The same past determines two destinations or targets through event timing, order, concealed identity or transient highlighting.
Method
Goal (%)
All-goal (%)
Sequence (%)
No-Memory ( π0.5 )
25.3 (101/400)
5.5 (11/200)
7.5 (15/200)
MME-VLA ( π0.5 )
32.5 (130/400)
14.5 (29/200)
11.5 (23/200)
MemER (RoboMME)
39.3 (157/400)
20.0 (40/200)
20.5 (41/200)
Uniform-Memory ( π0.5 )
54.0 (216/400)
36.0 (72/200)
35.0 (70/200)
ReuseMem
74.5 (298/400)
60.5 (121/200)
58.0 (116/200)
Table 1: Success on four simulation tasks. Each task contributes 50 common recordings with two later goals. Goal pools 400 independent requests; All-goal and Sequence each use 200 recordings. All-goal combines reset branches, whereas Sequence requires both requests without reset.
Figure 7: All-goal success by task. Each point is the proportion of 50 common recordings for which both independent goals succeed. Learned evidence selection improves reuse across event-relative, temporal, hidden-identity and transient-cue tasks.
Figure 8: Learning retrieval and current-scene action use. Bars compare independent All-goal success and no-reset Sequence success over 200 common recordings. Controls share the prepared policy, expert data and action-training budget.
Figure 9: Real-world manipulation sequences. Each row shows a common observation history, the scene before requests, and three successive goals. (a) Case contents determine the transfer targets, and the recorded origin guides return. (b) The same toy origin supports relocation, localized brushing and restoration from the changing scene.
Method
G1
G2
G3
Goal
All-goal
Sequence
Case preparation and return
No-Memory
50
50
20
40.00
5
20
Uniform-Memory
75
75
60
70.00
45
50
ReuseMem
85
85
85
85.00
70
65
Toy relocation, cleaning and restoration
No-Memory
50
45
45
46.67
10
10
Table 2: Physical manipulation success (%). Each scenario uses 20 common recordings. Goal averages 60 independent requests; All-goal and Sequence require all three independent or sequential requests, respectively.
Figure 10: Success, response and retained memory. Left: new-request response versus All-goal success. Right: bank storage for matched policy configurations at 2,048 observations. Sharing reduces stored visual content while retaining all temporal entries.
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Value
Optimizer
AdamW
Reader/index learning rate
10−4
Action-interface learning rate
5×10−5
Batch size
4 experience groups
Examples per group
2 goals × 2 current states
Training budget
Up to 20,000 steps
Appendix
Table 3: Optimization settings. Checkpoints are selected using validation performance; test recordings are reserved for final evaluation.
Component
Setting
Appearance pool
Immutable 16×1024 BF16 appearance prototypes
Memory entry
Original ordinal, prototype reference, validity, metadata, and 256-dimensional key
Causal access index
Width 256; two blocks; four heads; 64-row local window
Reference/support search
Eight searched anchors; four references; three supports per reference
Historical packet
Eight coverage rows; stable deduplication/fill to 24 slots
Training group
Associated goals; two current-scene queries per goal
Appendix
Table 4: Model configuration. Frame budgets, representation widths, and adaptation ranks have different roles.
1
qg←Poolϕ(g),P←() .
2
for u=1,…,8 do
3
iu←Selectμ(fϕref(⋅;qg,P),I∖P);P←P∥iu .
4
end for
5
R←Selectμ(4)(fϕemit(⋅;qg),P) .
6
for u=1,…,4 , with ru=Ru do
7
Au←Selectμ(3)(ℓ⋅∣ru,g,I) if ru=⊥ ; otherwise Au←(⊥,⊥,⊥) .
Appendix
Algorithm 2 Reference and support retrieval
1
Initialize an empty bank B .
2
for each observation (oi,ηi) in He do
3
Compute frozen Ai ; match or append a prototype by Eq. ( 8 ).
4
Append its occurrence metadata and reference pi ; update ki by Eq. ( 1 ).
5
end for ; commit Be0←B and initialize Je←() .
6
for each subsequent policy call t do
7
Acquire (gt,ot,st) and update Je with newly available strict-past observations; discard unissued commands on a request change.
Appendix
Algorithm 3 Memory recording and policy execution
Method
G1
G2
All-goal
Sequence
VideoPlaceButton
No-Memory
13
14
3
4
MME-VLA
29
27
17
13
MemER
15
15
4
5
Uniform-Memory
34
33
26
25
ReuseMem
44
45
40
38
Appendix
Table 5: Per-task simulation counts. Every cell reports successes out of 50 common recordings. G1 and G2 use independent branches; All-goal requires both, and Sequence uses a separate two-request execution without reset.
Configuration
Goal (%)
All-goal (%)
Sequence (%)
Without retrieval learning
54.0 (216/400)
36.5 (73/200)
34.0 (68/200)
Single-state supervision
69.0 (276/400)
53.0 (106/200)
49.0 (98/200)
Without current fusion
70.5 (282/400)
55.5 (111/200)
47.0 (94/200)
ReuseMem
74.5 (298/400)
60.5 (121/200)
58.0 (116/200)
Without feature sharing
74.8 (299/400)
61.0 (122/200)
58.0 (116/200)
Without within-call KV cache
74.5 (298/400)
60.5 (121/200)
58.0 (116/200)
Appendix
Table 6: Complete ablation results. Goal pools 400 requests; All-goal and Sequence each use 200 recordings. Sufficient historical evidence supplies the annotated past evidence as a diagnostic control.
Method
Evidence coverage (%)
Goal success (%)
Uniform-Memory
80.5 (322/400)
54.0 (216/400)
Without retrieval learning
75.0 (300/400)
54.0 (216/400)
Single-state supervision
87.5 (350/400)
69.0 (276/400)
ReuseMem
90.0 (360/400)
74.5 (298/400)
Appendix
Table 7: Evidence selection and action success. Coverage requires the selected 24 historical observations to cover every necessary annotated fact group; equivalent views within a group are interchangeable. Each metric uses 400 goal queries.
Figure 11: Dependence on historical evidence. Removing relevant observations or supplying mismatched history reduces goal success more than removing unrelated observations. All conditions retain the original task truth and current inputs.
Stage
Time (ms)
ReuseMem
Observation, I/O and dispatch
90
Current vision and language
37
Reader
11
Reconstruction, fusion and memory K/V
10
Action sampling
27
Appendix
Table 8: New-request response components. Components sum to each method's complete instruction-to-first-command latency.
Figure 12: Response composition and within-call caching. Left: instruction-to-first-command response split into input/dispatch, memory processing and action-policy computation. Memory processing includes the high-level VLM in MemER. Right: memory-K/V caching compares continuing-goal calls and reduces mean response by 9 ms.