Variational Cognitive Memory Abstraction for Long-Horizon Vision-Language-Action Models

Xinyu Liu; Daiying Tian; Chenwei Tang; Ruyang Han; Chengyang Zhang; Qingyu Zeng; Wei Ju; Liangli Zhen

cs.AIOct 25, 2026

Variational Cognitive Memory Abstraction for Long-Horizon Vision-Language-Action Models

Authors: Xinyu Liu, Daiying Tian, Chenwei Tang, Ruyang Han, Chengyang Zhang, Qingyu Zeng, Wei Ju, Liangli Zhen,

Organizations: College of Computer Science, Sichuan University, and Engineering Research Center of Machine Learning and Industry Intelligence, Ministry of Education, Chengdu 610065, PR China · Institute of High Performance Computing, Agency for Science, Technology and Research (A*STAR), Singapore 138632, Singapore

Abstract

Vision-Language-Action (VLA) models have shown remarkable success in robotic manipulation, yet their ability to execute long-horizon tasks remains limited by insufficient long- term memory modeling. Existing methods either rely on implicit temporal representations within policy networks, where historical information can be gradually diluted, or maintain explicit memory buffers that suffer from redundancy, noise accumulation, and inefficient retrieval. In this paper, we propose CogMemory, a novel variational cognitive memory framework that enables VLA systems to selectively preserve and reuse task-relevant historical information. Our approach encodes multimodal trajectories into compact latent memory representations through a selective state- space memory encoder, multi-slot temporal abstraction, and variational information compression. During action generation, dynamic cross-attention retrieval with gated fusion adaptively incorporates relevant memories into policy prediction. With merely 3.45M additional parameters, CogMemory stays lightweight and seamlessly fits both early- and intermediate-fusion VLA backbones. Extensive experiments on long-horizon robotic manipulation benchmarks demonstrate consistent improvements across diverse VLA backbones.

Figures & tables

Fig. 1 : Early-fusion architectures jointly model all modalities but lack explicit long-range history retention, while intermediate-fusion approaches rely on implicit temporal aggregation within policy networks, leaving them vulnerable to noise and decay of early signals. Our proposed CogMemory abstracts long-horizon history into compact latent slots via selective state-space modeling (SSM) and variational information compression (VIC), and enables the policy to dynamically query task-relevant memory via cross-attention and gated fusion for action generation. This lightweight module can be seamlessly integrated into both early-fusion and intermediate-fusion VLA models.

Fig. 2 : Overview of CogMemory . Given visual observations, robot states, and language instructions, the VLA backbone first produces multimodal representations. CogMemory then constructs compact cognitive memory through three stages: (1) selective state-space modeling (SSM) for long-range temporal encoding, (2) multi-slot temporal pooling for memory abstraction, and (3) variational information compression (VIC) for learning informative latent memory distributions. During action generation, the action decoder queries task-relevant cognitive memory through cross-attention, followed by dynamic gated fusion to adaptively integrate queried memory into action prediction. The framework is compatible with both early-fusion (e.g., $\pi_{0}$ -style architectures) and intermediate-fusion VLA models (e.g., FLOWER-style architectures).

Method	Spatial $\uparrow$	Object $\uparrow$	Goal $\uparrow$	Long-10 $\uparrow$	Long-90 $\uparrow$	Avg. $\uparrow$
UniACT [ 30 ]	77.0	87.0	77.0	70.0	73.0	76.8
SpatialVLA [ 31 ]	88.2	89.9	78.6	55.5	46.2	71.7
TraceVLA [ 23 ]	84.6	85.2	75.1	54.1	–	74.8
CoT-VLA [ 32 ]	87.5	91.6	87.6	69.0	–	83.9
$\pi_{0}$ -FAST [ 33 ]	96.4	96.8	88.6	60.2	83.1	85.0
CronusVLA [ 34 ]	90.1	94.7	91.3	68.7	–	86.2

TABLE I: Comparison of the task success rates (%) on LIBERO using the Franka Panda robot across five evaluation suites. For methods that do not report Long-90 results, we follow the original reporting protocol and compute Avg. over the available suites.

Setting	Method	Backbone	LH-1 $\uparrow$	LH-2 $\uparrow$	LH-3 $\uparrow$	LH-4 $\uparrow$	LH-5 $\uparrow$	Avg. Len. $\uparrow$
D $\to$ D	$\pi_{0.5}$ [ 37 ]	PaliGemma	86.8	68.8	53.3	42.5	32.4	2.84
	CogMemory (Ours)	PaliGemma	89.7	75.5	63.5	53.6	44.9	3.27
	MDT [ 40 ]	Diffusion	93.7	84.5	74.1	64.4	55.6	3.72
	RoboUniView [ 41 ]	Cont	96.2	88.8	77.6	66.6	56.3	3.85
	FLOWER [ 11 ]	Florence-2-L	97.4	92.4	86.9	81.3	74.9	4.35
	ThinkProprio [ 42 ]	Florence-2-L	96.9	89.8	83.6	80.5	72.7	4.23

TABLE II: Results on CALVIN benchmark with the Franka Emika robot. We report success rates (%) for different task-chain lengths and the average completed chain length (Avg. Len.) under two evaluation settings: D

\to

D and ABC

\to

Variant	Method	L-1	L-2	L-3	L-4	L-5	Avg. Len.
Comp.	Ours	89.7	75.5	63.5	53.6	44.9	3.27
	w/o Memory	85.7	68.6	53.2	42.4	33.0	2.83
	w/o VIC	87.3	67.7	50.2	39.4	30.3	2.75
	w/o Gate	89.6	70.9	53.6	42.6	33.6	2.90
# Slots	Slots=4	86.9	70.1	57.1	47.2	38.2	2.99
	Slots=8	89.7	75.5	63.5	53.6	44.9	3.27

TABLE III : Ablation studies on CALVIN D

\to

D using the PaliGemma-based backbone. We ablate (top) key components of CogMemory and (bottom) the number of memory slots. Success rates (%)

\uparrow

are reported at each chain length, along with the average completed chain length (Avg. Len.)

\uparrow

Fig. 3 : Qualitative comparison between the baseline policy and CogMemory on representative long-horizon manipulation tasks. Each row shows a sequential task requiring multiple operations, including stacking, turning, placing, picking, opening, and closing. Red and green markers indicate failure and successful completion, respectively.

Module	Number of Parameters
SSM	432.14 K
Slot Pooling	2.06 K
VIC	131.58 K
Memory Fusion & Gate	2.10 M
Projection	787.71 K
TOTAL	3.45 M

TABLE IV : Parameter counts of different components in CogMemory .

Hyperparameters	Values
GPU type	NVIDIA L20X
GPU num	8
fsdp devices	8
Optimizer	Adamw
- b1	0.9
- b2	0.95

TABLE V : Hyperparameters for training CogMemory .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Property	Value
Robot Platform	Franka Panda
Simulation Engine	MuJoCo
Observation FPS	10
Top Camera Resolution	$256\times 256$
Wrist Camera Resolution	$256\times 256$
Robot State Dimension	8

Appendix

TABLE VI: Statistics of the LIBERO benchmark used in our experiments.

Property	Value
Robot Platform	Franka Emika
Simulation Engine	PyBullet
Atomic Manipulation Skills	34
Language Instructions	389 annotations
Evaluation Protocol	1000 instruction chains
Instructions per Chain	5 tasks

Appendix

TABLE VII: Statistics of the CALVIN benchmark used in our experiments.

Task	Baseline	CogMemory
Close bottom drawer	0:24	0:09
Close microwave	0:30	0:17
Close top drawer	0:20	0:06
Open bottom drawer	0:15	0:14
Open top drawer	0:35	0:08
Soup $\rightarrow$ basket	0:22	0:11

Appendix

TABLE VIII: Execution time comparison between the baseline policy and CogMemory on tasks successfully completed by both methods. Task names are abbreviated for readability.

Fig. 4 : Qualitative results of CogMemory on Libero tasks. Representative examples include tasks from Spatial, Object, Goal, Long-10, and Long-90 suites.

Fig. 5 : Qualitative results of CogMemory on Calvin tasks. Representative examples include tasks from Calvin validation set.

Memory-Augmented VLMs

Memory-augmented vision-language models use persistent or retrieved context to improve reasoning across long inputs and interactions.

See all 160 papers

Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation
Jul 8, 2026Hongyu Qu, Jianzhe Gao, Xiaobin Hu +6Memory-Augmented VLMs Robotic Manipulation
Nanjing University of Science and Technology · Zhejiang University · National University of Singapore
MemoryVLA++: Temporal Modeling via Memory and Imagination in Vision-Language-Action Models
Jun 8, 2026Hao Shi, Weiye Li, Bin Xie +6Long-Horizon Robotic Manipulation World Models for Robotics
Department of Automation, BNRist, Tsinghua University, Beijing, China · Department of Computer Science, The University of Hong Kong, Hong Kong, China · Dexmal, Beijing, China +1
Explicit Language Memory for Long-Horizon Planning in Vision-Language-Action Models
Aug 5, 2026Houze Xu, Jizhong Li, Ziyi YeMemory-Augmented VLMs Vision-Language-Action Models
Fudan University

Jul 8, 2026J/K move · Enter open · S save

Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation

Hongyu Qu, Jianzhe Gao, Xiaobin Hu, Shaohuan Yang+5

Nanjing University of Science and Technology · Zhejiang University · National University of Singapore

Memory-Augmented VLMs Robotic Manipulation Vision-Language-Action Models Memory-Augmented Agents Reasoning for VLA Models Temporal Modeling for VLA Models

Open paper AskPDF arXiv

Mainstream Vision-Language-Action (VLA) models predict actions primarily from the current observation under a Markovian assumption, thus struggling with long-horizon, temporally dependent tasks. Existing memory-augmented VLAs either expand the observation window or retrieve history from the memory bank as auxiliary policy-side context. However, they leave memory outside the native latent embedding space of VLA reasoning, preventing historical experience from being fluidly interleaved with multimodal reasoning and action formation. To this end, we introduce LaMem-VLA, a latent-memory-native framework that reconstructs historical experience into latent memory tokens and directly interweaves them with VLA reasoning. At its core, LaMem-VLA introduces four coordinated components: (i) a curator that organizes historical experience into two complementary short-term and long-term memory vaults; (ii) a seeker that queries both vaults using the multimodal cognition to retrieve context-relevant evidence; (iii) a condenser that reconstructs the retrieved evidence into compact short-term and long-term latent memory tokens; and (iv) a weaver that injects these memory tokens with the current observation and instruction into one continuous embedding sequence. By representing, retrieving, and consuming historical experience entirely in the same continuous latent space, LaMem-VLA enables memory to directly participate in VLA reasoning and guide action generation under a bounded context. Extensive experiments on SimplerEnv and LIBERO demonstrate the superiority of our LaMem-VLA.

Abstract

Memory-Augmented VLMs

Explore similar work

Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation

MemoryVLA++: Temporal Modeling via Memory and Imagination in Vision-Language-Action Models

Explicit Language Memory for Long-Horizon Planning in Vision-Language-Action Models