UniMem: Unifying Multimodal Memory and Control for Vision-Language-Action Models

Lars Osterberg; Maggie Wang; Mac Schwager

cs.ROAug 24, 2026

UniMem: Unifying Multimodal Memory and Control for Vision-Language-Action Models

Authors: Lars Osterberg, Maggie Wang, Mac Schwager

Organizations: Stanford University, Stanford, CA, USA

Abstract

While Vision-Language-Action (VLA) models have leveraged internet-scale pretraining and task-focused finetuning to achieve strong performance on long-horizon tasks, they often struggle with non-Markovian tasks that require memory. Existing approaches to memory typically involve additional Vision-Language-Models (VLMs) for long-term memory management, introducing a memory bottleneck and a fractured training pipeline. Conditioning on multiple historical frames can provide the VLA with access to more descriptive features of past scenes, but can degrade performance if frames are chosen at arbitrary, fixed intervals. To address these limitations, we present UniMem, a framework that unifies high-level, multimodal memory and low-level control under one backbone. UniMem employs an event classifier for memory updates, a keyframe encoder for dense spatial memory, and a keyframe caching technique to minimize overhead during policy rollouts. We evaluate UniMem across five simulation and four hardware tasks targeting sequential and spatial memory, demonstrating that our unified, single-model system outperforms fixed-interval image sampling baselines (93.4% vs. 68.2%) in simulation and hierarchical baselines (80.0% vs. 43.5%) in hardware, while offering faster inference and a simple training pipeline for easy adoption. Project website: https://losterberg3.github.io/unimem-vla/

Figures & tables

Fig. 1 : Overview of UniMem . An event classifier ( $f_{\phi}$ ) detects sparse sub-task transition events from the backbone latent space, autoregressively updating textual memory and a cache of precomputed keyframe hidden states. This routes to our keyframe encoder and a tokenizer, providing event-driven, multimodal memory directly to the backbone and action expert while unifying memory and control in one self-sustaining, low-latency VLA.

Fig. 2 : Overview of UniMem ’s automated data labeling procedure. We prompt an agent to generate a script for the given task that labels demonstrations with $e_{t}$ and $\mathcal{M}_{t}$ . Multiple frames in the window $W_{i}$ are labeled with the corresponding event, and $\mathcal{M}_{t}$ is only updated once this window has passed. At training time, a keyframe from each past $W_{i}$ along with $I_{t}$ is used to build $\mathcal{H}_{t}$ .

Fig. 3 : Overview of Tasks. (Top) Manipulation tasks in robosuite . (Bottom) Real-world tasks on the xArm6 setup.

Task	$\pi_{0.5}+\text{V.E.}$	N.M.	T.O.	K.O.	Ours
UpDown	84	52	96	92	100
UpDown3Times †	16	22	93	23	96
OccludedTap	96	60	88	100	96
UpDownSpatial ‡	49	6	30	52	79
PlateRecall	96	8	20	96	96
Avg. Performance	68.2	29.6	65.4	72.6	93.4

TABLE I : Simulation Results & Ablations. Success rates (%) (

N=25

). (

\dagger

) reports mean subtask success due to long-horizon complexity. (

\ddagger

) reports spatial accuracy.

Task	MemER	N.M.	T.O.	K.O.	Ours
HammerMeasure	87	13	53	53	87
BeanScoop	67	0	27	20	93
TableClean	13	0	0	47	80
TapScoopPour	7	7	7	27	60
Avg. Performance	43.5	5.0	21.8	36.8	80.0
N.M. = No Memory, T.O. = Text Only, K.O. = Keyframe Only

TABLE II : Hardware Results & Ablations. Success rates (%) (

N=15

Fig. 4 : Cumulative Success Rates. Hardware tasks of BeanScoop , TableClean , and TapScoopPour .

Panel of Figure 7 — Fig. 5 : Benchmarking inference latency on an RTX 4090. We compare inference speed when varying images per camera stream for 2 streams (left) and a simulated bimanual setup with 4 streams (right).

Memory-Augmented VLMs

Memory-augmented vision-language models use persistent or retrieved context to improve reasoning across long inputs and interactions.

See all 144 papers

Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation
Jul 8, 2026Hongyu Qu, Jianzhe Gao, Xiaobin Hu +6Memory-Augmented VLMs Robotic Manipulation
Nanjing University of Science and Technology · Zhejiang University · National University of Singapore
ReMem-VLA: Empowering Vision-Language-Action Model with Memory via Dual-Level Recurrent Queries
Mar 13, 2026Hang Li, Fengyi Shen, Dong Chen +6Memory-Augmented VLMs Vision-Language-Action Models
Technical University of Munich, Munich, Germany · Huawei Heisenberg Research Center, Munich, Germany · University of Freiburg, Freiburg, Germany +2
ActMem-VLA: Action-History Memory with Dual-Expert Denoising for Long-Horizon Vision-Language-Action Policies
Sep 29, 2026Yaxin Zhao, Dianye Huang, Chenwei Wang +2Memory-Augmented VLMs Long-Horizon Robotic Manipulation
Harbin Institute of Technology, Harbin, China. · Medical Intelligence and Robotic Cognition (MIRoC) Lab, Department of Mechanical Engineering, The University of Hong Kong (HKU), Hong Kong SAR, China. · Department of Computing, The Hong Kong Polytechnic University, Hong Kong SAR, China

Jul 8, 2026J/K move · Enter open · S save

Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation

Hongyu Qu, Jianzhe Gao, Xiaobin Hu, Shaohuan Yang+5

Nanjing University of Science and Technology · Zhejiang University · National University of Singapore

Memory-Augmented VLMs Robotic Manipulation Vision-Language-Action Models Memory-Augmented Agents Reasoning for VLA Models

Open paper AskPDF arXiv

Mainstream Vision-Language-Action (VLA) models predict actions primarily from the current observation under a Markovian assumption, thus struggling with long-horizon, temporally dependent tasks. Existing memory-augmented VLAs either expand the observation window or retrieve history from the memory bank as auxiliary policy-side context. However, they leave memory outside the native latent embedding space of VLA reasoning, preventing historical experience from being fluidly interleaved with multimodal reasoning and action formation. To this end, we introduce LaMem-VLA, a latent-memory-native framework that reconstructs historical experience into latent memory tokens and directly interweaves them with VLA reasoning. At its core, LaMem-VLA introduces four coordinated components: (i) a curator that organizes historical experience into two complementary short-term and long-term memory vaults; (ii) a seeker that queries both vaults using the multimodal cognition to retrieve context-relevant evidence; (iii) a condenser that reconstructs the retrieved evidence into compact short-term and long-term latent memory tokens; and (iv) a weaver that injects these memory tokens with the current observation and instruction into one continuous embedding sequence. By representing, retrieving, and consuming historical experience entirely in the same continuous latent space, LaMem-VLA enables memory to directly participate in VLA reasoning and guide action generation under a bounded context. Extensive experiments on SimplerEnv and LIBERO demonstrate the superiority of our LaMem-VLA.

Abstract

Memory-Augmented VLMs

Explore similar work

Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation

ReMem-VLA: Empowering Vision-Language-Action Model with Memory via Dual-Level Recurrent Queries

ActMem-VLA: Action-History Memory with Dual-Expert Denoising for Long-Horizon Vision-Language-Action Policies