Memory Without Curation: Learning Multi-Stream Latent Memory for Robot Policies

Yilin Wu; Rutav Shah; Shuaiyi Huang; Sumedh A. Sontakke; Homanga Bharadhwaj; Andrea Bajcsy; Max Simchowitz; Guanya Shi

cs.ROOct 9, 2026

Memory Without Curation: Learning Multi-Stream Latent Memory for Robot Policies

Authors: Yilin Wu, Rutav Shah, Shuaiyi Huang, Sumedh A. Sontakke, Homanga Bharadhwaj, Andrea Bajcsy, Max Simchowitz, Guanya Shi

Organizations: Amazon FAR · Carnegie Mellon University · The University of Texas at Austin · Johns Hopkins University

Abstract

It is widely appreciated that robotic policies require long-term memory to perform complex tasks. However, directly conditioning policies on all past observations leaves policy learning susceptible to learning spurious shortcuts, and introduces large latency at policy deployment. As such, today's memory-augmented policies curate what to remember, either by retaining only recent frames, selected keyframes, or textual summaries, or by learning latent representations of the visual stream alone. In this work, we propose an alternative approach IMPRINT, a policy-agnostic latent memory framework that augments vision-language-action (VLA) models with multi-stream memory without explicitly constraining what the policy should remember. IMPRINT compresses different streams into a latent memory that is updated incrementally at inference, enabling low latency. To avoid shortcut learning, IMPRINT adopts a two-stage training procedure, first supervising memory to reconstruct multiple streams (observations, actions, proprioception) of past information, as well as estimate current task progress, followed by end-to-end finetuning, jointly with the policy, on action prediction. Across both simulation and real-world tasks demanding memory, IMPRINT improves average success over the base VLA by at least 26%, outperforming popular alternatives, and maintaining lowest inference latency among memory-augmented VLAs. Through careful ablations, we demonstrate the utility of incorporating diverse information streams into robotic memory, validating the advantages of foregoing history curation. Project website: https://yilin-wu98.github.io/imprint_website/.

Figures & tables

Figure 2: Real World Tasks & Result Highlight

Figure 3: Average Task Success on Rmbench

Figure 4: Average Task Performance For Memory and Non-Memory Tasks in Robocasa

Figure 5: Real World Task Performance Across 3 Tasks

Figure 6: Runtime vs Success on RMBench

Figure 7: Ablations on Memory Capacity. We evaluate Imprint trained with different memory tokens against task performance (left) and supervision loss evaluated on the validation set (right).

Figure 8: Ablations of Memory Learning. We ablate different input streams (right) and varied supervision objectives (middle) and training recipes (left) of Imprint on RMBench.

Figure 9: Analysis of Memory Bottleneck. We compare E2E, frozen memory encoder, and our adapted memory with t-SNE embeddings, linear probes, and attention visualization on Cover Block .

Figure 10: Imprint Improves Different VLA Backbones with Memory.

Appendix figures & tables57 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Value
Memory width $d$	1024
Memory tokens $B$	256 (ablated over $\{1,4,16,64,256,1024\}$ )
Visual stream projections	Linear
Action / proprioception projections	2-layer MLP, GELU, hidden width 512
Language conditioning	FiLM on queries, 2048-d instruction embedding
Perceiver blocks	2, pre-LayerNorm

Appendix

Table 1: Full-attention memory encoder hyperparameters.

Figure 11: Architecture of Memory Adapter for VLAs.

Setting	Value
Memory / action-expert width	1024
Query heads	4
Key-value heads	1 (grouped-query attention)
Head dimension	256
Positional encoding	Separate rotary positions for memory and action tokens
FiLM projection initialization	$\mathcal{N}(0,0.002^{2})$ , zero bias

Appendix

Table 2: Memory adapter hyperparameters for

\pi_{0}

and

\pi_{0.5}

Setting	Stage 1: memory pretraining	Stage 2: joint adaptation
Updates	160,000	100,000
Global batch size	64	64
GPUs	4	4
Optimizer	AdamW	AdamW
Peak learning rate	$1.0\times 10^{-4}$	$2.5\times 10^{-5}$
Final learning rate	$1.0\times 10^{-5}$	$2.5\times 10^{-6}$

Appendix

Table 3: Optimization settings for IMPRINT on RMBench.

Method	Category	Backbone	RMBench result
non-memory VLA	None	$\pi_{0}$ , $\pi_{0.5}$ , MolmoAct2	Trained by us
Uniform sampling	Heuristic	$\pi_{0.5}$	Trained by us
EventVLA (visual anchors only) †	Heuristic	QwenOFT (Qwen3-VL-4B)	Yang et al. [2026a]
MemER	Hierarchical	VLM planner + VLA policy	Yang et al. [2026a]
Mem-0	Hierarchical	Qwen3-VL-8B planner + DiT policy	Chen et al. [2026b]
HAMLET	Latent	GR00T N1.6	Trained by us

Appendix

Table 4: Compared policies, their backbones, and the source of their RMBench results.

	Heuristic		Hierarchical		Latent
	Uniform sampling	EventVLA †	MemER	Mem-0	HAMLET	MemoryVLA	IMPRINT
What memory stores
Visual history	✓	✓	✓	✓	✓	✓	✓
Action history	✗	✗	✗	✗	✗	✗	✓
Proprio. history	✗	✗	✗	✗	✗	✗	✓
Memory form	Frames	Frames	Frames, text	Frames, text	Latent	Latent	Latent

Appendix

Table 5: Comparison of memory designs. ✓ : yes;

\boldsymbol{\sim}

: partially; ✗ : no. † On RMBench, EventVLA is evaluated only in its visual-anchor configuration, which keeps the initial frame and a few recent frames and omits its keyframe evidence memory [ Yang et al., 2026a ] . The reported result therefore reflects this heuristic variant rather than the full EventVLA, and the column describes this evaluated variant.

Study	Variant	Stage 1 init.	Memory updated	Change from default	Reported in
Encoding	Memory capacity	yes	yes	$B\in\{1,4,16,64,256,1024\}$ (full-attention encoder)	Fig. 7
	Full attention	yes	yes	Perceiver attention instead of recurrence	Fig. 6 , Sec. 5.3
Recipe	End-to-end	no	yes	Random encoder, action loss only	Fig. 8
	Frozen memory	yes	no	Encoder fixed during Stage 2	Fig. 8
	IMPRINT	yes	yes	None (default)	Fig. 8
Objectives	Without progress	yes	yes	$\lambda_{\rho}=0$	Fig. 8

Appendix

Table 6: Configurations used in the ablation study. Stage 1 init. : whether the memory encoder starts from Stage 1 pretraining. Memory updated : whether the encoder is optimized during Stage 2. Unlisted settings match the default IMPRINT configuration.

Task	Type	Task and memory requirement	Limit
Observe and pickup	M(1)	Pick the object matching a target shown before occlusion.	250
Rearrange blocks	M(1)	Move the center block to the empty mat, press, and move the initially mat-placed block to the center.	700
Put back block	M(1)	Move the block to the center, press, and return it to its original mat.	500
Swap blocks	M(1)	Use the spare tray to exchange the blocks’ initial tray assignments, then press once.	1000
Swap T	M(1)	Exchange the initial positions and orientations of two T-shaped blocks.	600
Battery	M( $n$ )	Use gauge feedback from tested orientation pairs to place both batteries correctly.	1000

Appendix

Table 7: RMBench tasks and episode limits, measured in simulator steps.

Task	Instruction and memory requirement	Horizon	L/S	Variation across configurations
Memory
Retrieve fruit	Place the {fruit} in the sink; recall which side of the sink it was observed on.	1600	1/1	9 fruit categories; target at the far left or right of the sink.
Retrieve oil	Pick up the olive-oil bottle; recall which of two bottles is the target after both sides are shown.	1600	1/1	Four initial/target-side combinations (LL, LR, RL, RR).
Wash and return	Wash the fruit and return it to its original container; recall the source container.	1600	1/1	Left or right source container; fruit, receptacle, and placements vary.
Wash and return to same location	Wash the fruit and return it to its exact initial location; recall that location.	1600	1/1	Initial fruit location varies.
Sweeten hot chocolate	Place {1, 2, 3} sugar cubes in the saucepan; track how many have been placed.	2100	15/17	Target counts 1/2/3 in 20/20/10 configurations; 3–5 cubes per scene, including 0–3 extras.

Appendix

Table 8: Tasks in the customized RoboCasa benchmark. Horizon : maximum number of policy-controlled simulator steps. L/S : number of distinct layouts and styles among the 50 evaluation configurations. Instructions are abbreviated.

Figure 12: Real World Tasks with Three Camera Views

Figure 13: Real World Fold Cloth Task Configuration

Task	Rollouts	Variation
Dust Table	20	2 object sets $\times$ duster left/right (5 rollouts each)
Scan Bottle	20	bottles on left/right (10 rollouts each)
Fold Cloth	32	16 towel color/style contexts (2 rollouts each)

Appendix

Table 9: Real-world rollout protocol.

Figure 14: OOD Scenarios for Each Task in the Real World

	M(1)					M(n)
Method	Observe / pick	Rearrange blocks	Put back block	Swap blocks	Swap T	Battery	Block ranking	Cover blocks	Press button	Average
$\pi_{0.5}$ (no memory)	8	63	20	16	8	30	66	15	3	25.4
Uniform sampling	5	79	100	62	30	28	82	13	3	44.7
EventVLA †	21	96	95	96	87	35	81	97	3	67.8
MemER †	7	17	0	14	7	27	0	6	0	8.7
Mem-0 †	4	89	90	67	14	28	18	68	0	42.0

Appendix

Table 10: Per-task success rates on the nine RMBench memory tasks, grouped into single-event, M(1), and multi-event, M(n), tasks. The final column is the task-balanced average reported in Fig. 3 . The best result for each task and for the average is shown in bold. † Results reported by prior work (App. A.2 ).

Task	Type	$\pi_{0.5}$	Heuristic action	Heuristic VLM	IMPRINT full	IMPRINT recurrent
Retrieve fruit	Memory	42	88	60	84	90
Retrieve oil	Memory	48	98	50	76	80
Wash and return	Memory	46	58	46	86	96
Wash and return to same location	Memory	46	36	44	62	58
Rinse cutting board	Memory	4	8	0	0	4
Sweeten hot chocolate	Memory	46	36	34	58	64

Appendix

Table 11: Per-task success rates on the customized RoboCasa benchmark (Fig. 4 ). Heuristic action supplies uniformly sampled historical frames to the action expert, whereas heuristic VLM supplies them as additional VLM-context images. The best result for each task and each average is shown in bold.

Method	Dust Table	Scan Bottle	Fold Cloth	Average
$\pi_{0}$	6/20 (30%)	5/20 (25%)	7/32 (22%)	25.6%
HAMLET	5/20 (25%)	5/20 (25%)	6/32 (19%)	22.9%
MemER	8/20 (40%)	7/20 (35%)	12/32 (38%)	37.5%
Uniform sampling	10/20 (50%)	10/20 (50%)	15/32 (47%)	49.0%
IMPRINT	16/20 (80%)	15/20 (75%)	23/32 (72%)	75.6%
IMPRINT (OOD)	9/20 (45%)	14/20 (70%)	10/32 (31%)	48.8%

Appendix

Table 12: Real-world results (Fig. 5 ). Each cell reports successful rollouts / total rollouts, followed by the success rate in parentheses. The average is the mean of the per-task success rates. The best in-distribution result for each task and the average is shown in bold. OOD denotes the held-out object variations.

$B$	Total	Vision	Action	State	Progress
	$\times 10^{-2}$	$\times 10^{-2}$	$\times 10^{-4}$	$\times 10^{-4}$	$\times 10^{-4}$
1	$3.649\pm 0.295$	3.599	2.37	2.43	5.13
4	$3.308\pm 0.273$	3.278	1.31	1.35	3.31
16	$3.406\pm 0.277$	3.368	1.50	1.55	4.41
64	$3.389\pm 0.282$	3.359	1.50	1.52	2.94
256	$3.338\pm 0.280$	3.311	1.27	1.31	2.81

Appendix

Table 13: Stage-1 losses across memory capacities. Total loss is mean

\pm

SEM across tasks; component losses are unweighted. The second header row indicates scaling.

Task	Type	$B{=}1$	$B{=}4$	$B{=}16$	$B{=}64$	$B{=}256$	$B{=}1024$
RMBench
Observe / pick	Memory	7	8	8	8	9	10
Rearrange blocks	Memory	100	100	100	100	100	99
Put back block	Memory	100	100	100	100	100	100
Swap blocks	Memory	97	97	99	99	100	93
Swap T	Memory	99	99	99	99	100	99

Appendix

Table 14: Per-task success as the capacity of the full-attention memory encoder is varied (Fig. 7 ). All other architecture, training, and evaluation settings are fixed.

B=1

and

B=1024

are evaluated on RMBench only (–). The best result in each row is shown in bold; rows where all settings tie are left unmarked.

	Training recipe			Stage 1 objective
Task	E2E	IMPRINT frozen	IMPRINT	IMPRINT	Without progress	Visual only	Action + state
Observe / pick	12	7	9	9	6	6	5
Rearrange blocks	90	99	100	100	91	78	100
Put back block	100	100	100	100	100	100	100
Swap blocks	90	99	100	100	90	89	89
Swap T	45	99	100	100	97	99	98

Appendix

Table 15: Per-task success for the training-recipe and Stage 1 objective ablations on RMBench (Fig. 8 ). The best result in each row is shown in bold separately within each ablation; rows where all variants tie are left unmarked.

	$\pi_{0.5}$		$\pi_{0}$		MolmoAct2
Task	No memory	IMPRINT	No memory	IMPRINT	No memory	IMPRINT
Observe / pick	8	9	8	6	4	4
Rearrange blocks	63	100	13	99	52	100
Put back block	20	100	49	100	0	100
Swap blocks	16	100	4	100	11	65
Swap T	8	100	0	98	15	100

Appendix

Table 16: Per-task success with and without IMPRINT across three VLA backbones on RMBench (Fig. 10 ). For each backbone, the better result of the two variants is shown in bold; ties are left unmarked.

Memory training	Timestep accuracy	Balanced accuracy	Episode accuracy	Task success
E2E	19.94	17.29	42.00	15.00
Frozen	83.31	83.68	90.00	35.00
IMPRINT	91.53	85.95	98.00	100.00

Appendix

Table 17: Cover Blocks color-order probes and closed-loop task success (%). Probes are evaluated on held-out episode folds drawn from the memory model’s training demonstrations.

Figure 15: Inference Time Analysis For Different Bottleneck Size and Different Methods

	Full attention		Recurrent
$B$	1 frame	1,451 frames	1 frame	1,451 frames
4	109.0	144.1	114.1	124.4
16	107.4	143.4	115.3	125.4
64	109.7	143.7	116.2	128.3
256	111.9	148.2	116.8	127.1

Appendix

Table 18: Policy-call latency in milliseconds at the shortest and longest history. All variants of our method predict 50 actions.

Historical inputs	Obs./pick	Rearrange	Put back	Swap blocks	Swap T	Battery	Ranking	Cover	Press	Avg.
Vision only	9	93	100	81	45	30	80	100	7	60.6
Vision + Action	6	100	100	100	99	38	72	83	18	68.4
Vision + State	9	94	100	98	99	28	46	70	4	60.9
Vision + Action + State (Imprint)	9	100	100	100	100	41	83	100	21	72.7

Appendix

Table 20: Historical input-stream ablation on RMBench with the full-attention encoder at

B=256

. Each variant removes the listed streams from both the memory input and the Stage-1 reconstruction targets. The best result in each column is shown in bold; columns where all variants tie are left unmarked.

Figure 16: RMBench: Battery Try. Representative rollouts of Imprint , HAMLET, and Uniform Sampling.

Figure 17: RMBench: Battery Try. Representative rollouts of Imprint and the Frozen and End-to-End training variants.

Figure 18: RMBench: Block Ranking Try. Representative rollouts of Imprint , HAMLET, and Uniform Sampling.

Figure 19: RMBench: Block Ranking Try. Representative rollouts of Imprint and the Frozen and End-to-End training variants.

Figure 20: RMBench: Swap Blocks. Representative rollouts of Imprint , HAMLET, and Uniform Sampling.

Figure 21: RMBench: Swap Blocks. Representative rollouts of Imprint and the Frozen and End-to-End training variants.

Figure 22: RMBench: Swap T. Representative rollouts of Imprint , HAMLET, and Uniform Sampling.

Figure 23: RMBench: Swap T. Representative rollouts of Imprint and the Frozen and End-to-End training variants.

Figure 24: RMBench: Put Back Block. Representative rollouts of Imprint , HAMLET, and Uniform Sampling.

Figure 25: RMBench: Put Back Block. Representative rollouts of Imprint and the Frozen and End-to-End training variants.

Figure 26: RMBench: Cover Blocks. Representative rollouts of Imprint , HAMLET, and Uniform Sampling.

Figure 27: RMBench: Cover Blocks. Representative rollouts of Imprint and the Frozen and End-to-End training variants.

Figure 29: RMBench: Press Button. Representative rollouts of Imprint and the Frozen and End-to-End training variants.

Figure 30: RMBench: Observe and Pick Up. Representative rollouts of Imprint , HAMLET, and Uniform Sampling.

Figure 31: RMBench: Observe and Pick Up. Representative rollouts of Imprint and the Frozen and End-to-End training variants.

Figure 32: RMBench: Rearrange Blocks. Representative rollouts of Imprint , HAMLET, and Uniform Sampling.

Figure 33: RMBench: Rearrange Blocks. Representative rollouts of Imprint and the Frozen and End-to-End training variants.

Figure 34: RoboCasa: Frying Pan Adjustment. Representative rollouts of Imprint , $\pi_{0.5}$ , and Uniform Sampling with VLM-context or action-expert conditioning.

Figure 35: RoboCasa: Sweeten Hot Chocolate. Representative rollouts of Imprint , $\pi_{0.5}$ , and Uniform Sampling with VLM-context or action-expert conditioning.

Figure 36: RoboCasa: Pan Transfer. Representative rollouts of Imprint , $\pi_{0.5}$ , and Uniform Sampling with VLM-context or action-expert conditioning.

Figure 37: RoboCasa: Wash and Return to the Original Plate. Representative rollouts of Imprint , $\pi_{0.5}$ , and Uniform Sampling with VLM-context or action-expert conditioning.

Figure 38: RoboCasa: Wash and Return to the Same Location. Representative rollouts of Imprint , $\pi_{0.5}$ , and Uniform Sampling with VLM-context or action-expert conditioning.

Figure 39: RoboCasa: Rinse Cutting Board. Representative rollouts of Imprint , $\pi_{0.5}$ , and Uniform Sampling with VLM-context or action-expert conditioning.

Figure 40: RoboCasa: Scrub Cutting Board. Representative rollouts of Imprint , $\pi_{0.5}$ , and Uniform Sampling with VLM-context or action-expert conditioning.

Figure 41: RoboCasa: Organize Condiments. Representative rollouts of Imprint , $\pi_{0.5}$ , and Uniform Sampling with VLM-context or action-expert conditioning.

Figure 42: RoboCasa: Pick and Place to Sink. Representative rollouts of Imprint , $\pi_{0.5}$ , and Uniform Sampling with VLM-context or action-expert conditioning.

Figure 43: RoboCasa: Retrieve Fruit. Representative rollouts of Imprint , $\pi_{0.5}$ , and Uniform Sampling with VLM-context or action-expert conditioning.

Figure 44: RoboCasa: Retrieve Oil. Representative rollouts of Imprint , $\pi_{0.5}$ , and Uniform Sampling with VLM-context or action-expert conditioning.

Figure 45: RoboCasa: Place Equal Ice Cubes. Representative rollouts of Imprint , $\pi_{0.5}$ , and Uniform Sampling with VLM-context or action-expert conditioning.

Figure 46: Real World: Dust Table. Representative rollouts of Imprint (in-distribution and OOD objects), HAMLET, MemER, Uniform Sampling, and $\pi_{0}$ .

Figure 47: Real World: Scan Bottle. Representative rollouts of Imprint (in-distribution and OOD objects), HAMLET, MemER, Uniform Sampling, and $\pi_{0}$ .

Figure 48: Real World: Fold Cloth. The top row shows the in-context demonstration provided to Imprint’s memory, followed by representative rollouts of Imprint (in-distribution and OOD towel), HAMLET, MemER, Uniform Sampling, and $\pi_{0}$ .

Memory-Augmented VLMs

Memory-augmented vision-language models use persistent or retrieved context to improve reasoning across long inputs and interactions.

See all 160 papers

Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation
Jul 8, 2026Hongyu Qu, Jianzhe Gao, Xiaobin Hu +6Memory-Augmented VLMs Robotic Manipulation
Nanjing University of Science and Technology · Zhejiang University · National University of Singapore
NativeMEM: Native Memory Compression for Long-Horizon Robotic Manipulation
Jul 7, 2026Ziye Wang, Modi Shi, Chaojun Ni +5Memory-Augmented VLMs Long-Horizon Robotic Manipulation
The University of Hong Kong · Beihang University · Peking University +2
ActMem-VLA: Action-History Memory with Dual-Expert Denoising for Long-Horizon Vision-Language-Action Policies
Sep 29, 2026Yaxin Zhao, Dianye Huang, Chenwei Wang +2Memory-Augmented VLMs Long-Horizon Robotic Manipulation
Harbin Institute of Technology, Harbin, China. · Medical Intelligence and Robotic Cognition (MIRoC) Lab, Department of Mechanical Engineering, The University of Hong Kong (HKU), Hong Kong SAR, China. · Department of Computing, The Hong Kong Polytechnic University, Hong Kong SAR, China

Jul 8, 2026J/K move · Enter open · S save

Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation

Hongyu Qu, Jianzhe Gao, Xiaobin Hu, Shaohuan Yang+5

Nanjing University of Science and Technology · Zhejiang University · National University of Singapore

Memory-Augmented VLMs Robotic Manipulation Vision-Language-Action Models Memory-Augmented Agents Reasoning for VLA Models Temporal Modeling for VLA Models

Open paper AskPDF arXiv

Mainstream Vision-Language-Action (VLA) models predict actions primarily from the current observation under a Markovian assumption, thus struggling with long-horizon, temporally dependent tasks. Existing memory-augmented VLAs either expand the observation window or retrieve history from the memory bank as auxiliary policy-side context. However, they leave memory outside the native latent embedding space of VLA reasoning, preventing historical experience from being fluidly interleaved with multimodal reasoning and action formation. To this end, we introduce LaMem-VLA, a latent-memory-native framework that reconstructs historical experience into latent memory tokens and directly interweaves them with VLA reasoning. At its core, LaMem-VLA introduces four coordinated components: (i) a curator that organizes historical experience into two complementary short-term and long-term memory vaults; (ii) a seeker that queries both vaults using the multimodal cognition to retrieve context-relevant evidence; (iii) a condenser that reconstructs the retrieved evidence into compact short-term and long-term latent memory tokens; and (iv) a weaver that injects these memory tokens with the current observation and instruction into one continuous embedding sequence. By representing, retrieving, and consuming historical experience entirely in the same continuous latent space, LaMem-VLA enables memory to directly participate in VLA reasoning and guide action generation under a bounded context. Extensive experiments on SimplerEnv and LIBERO demonstrate the superiority of our LaMem-VLA.

Abstract

Memory-Augmented VLMs

Explore similar work

Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation

NativeMEM: Native Memory Compression for Long-Horizon Robotic Manipulation

ActMem-VLA: Action-History Memory with Dual-Expert Denoising for Long-Horizon Vision-Language-Action Policies