HiMEM: Hippocampus-Inspired Memory Consolidation, Pruning, and Composition for Vision-Language-Action Models in 3D Scenes
Authors: Boran Zhao, Yanbin Hu, Jun Ye, Zhenxian Hu, Yuxin Zhai, Jin Cui, Pengju Ren
Organizations: School of Software Engineering, Xi’an Jiaotong University, Xi’an, China. · State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Xi’an, China. · School of Artificial Intelligence, Xi’an Jiaotong University, Xi’an, China.
Memory plays a crucial role in long-horizon task execution and cross-task generalization for Vision-Language-Action (VLA) models operating in complex 3D scenes. However, existing VLA memory mechanisms suffer from memory noise, particularly in complex 3D scenes, as stored episodic memories contain extensive redundant details about scene layouts, object configurations, and robot-specific states, which obscure generalizable manipulation principles and hinder cross-task transfer. To address this challenge, we first introduce a hippocampus-inspired memory consolidation mechanism that synthesizes extensive episodic experiences into a compact set of essential semantic memories, enabling VLA models to derive generalizable principles from practical experience. Second, we propose an episodic memory pruning mechanism that removes redundant episodic memories whose contents are captured by consolidated semantic memories, thereby increasing effective memory capacity and improving long-horizon task execution. Finally, to handle complex task scenarios, we develop a flexible memory composition mechanism that jointly leverages multiple episodic and semantic memories to enhance cross-scenario generalization. Experiments on LIBERO, LIBERO-Plus demonstrate the effectiveness of HiMEM across different VLA backbones, improving the average success rate of pi_0 from 91.88% to 98.20% on LIBERO.
Figures & tables
Fig. 1: Illustration of memory noise and its effects on 3D manipulation. (a) Stored manipulation experiences for the same target object and subgoal contain varying non-target object details, introducing memory redundancy. (b) Using ECHO [ 6 ] for manipulation execution, we group memories by target and subgoal and randomly remove 30% of redundant entries differing only in non-target object details while preserving task-relevant experiences. On LIBERO-Long, in-distribution success rates decrease slightly from 93.50% to 93.25%, whereas out-of-distribution success rates improve from 92.78% to 94.67%. These results suggest that removing redundant memory details can improve out-of-distribution generalization while largely preserving in-distribution performance.
Method
Memory Mechanisms
Benefits
ECHO [ 6 ]
Hierarchical episodic memory
Enhancing retrieval efficiency through structured memory
Reduced memory noise and redundancy, improved storage efficiency, and enhanced cross-task generalization
TABLE I: Method comparison of this work and our previous conference version.
Fig. 2: Overview of the episodic and semantic memory framework in HiMEM. The hippocampus-inspired module organizes manipulation experiences into a shared hierarchical hyperbolic memory tree. The neocortex-inspired module consolidates experiences into compact semantic memories capturing scene and object conditions, reusable manipulation strategies, and success probabilities. Semantic consolidation guides the pruning of redundant episodic memories, while episodic retrieval and semantic guidance jointly support manipulation in complex 3D scenes.
Fig. 3: Overview of HiMEM for long-horizon spatial manipulation. During offline memory formation, verified execution episodes are organized into a shared hierarchical hyperbolic memory, while a VLM consolidates cross-episode experiences into compact meta-semantic rules that capture scene conditions, manipulation strategies, and outcome-based success probabilities. During online execution, the VLA backbone provides hidden states and scene context for hierarchical episodic retrieval and semantic rule matching. The matched rules guide the weighting of successful episodic memories and suppress risky operations using failure evidence. The resulting semantic-conditioned memory is fused with short-term representations in Lorentz space, decoded into a residual vector, and injected into the VLA action-generation pathway to support long-horizon execution and cross-task generalization.
Method
LIBERO
Avg.
LIBERO- Plus
Spatial
Object
Goal
Long
Reported baselines
Octo [ 14 ]
78.9
85.7
84.6
51.1
75.08
–
OpenVLA [ 2 ]
84.7
88.4
79.2
53.7
76.50
15.6
CronusVLA [ 29 ]
97.3
99.6
96.9
94.0
97.00
–
MAP-VLA [ 9 ]
96.3
98.4
95.4
83.4
93.38
–
TABLE II: Success rates (%) on LIBERO, LIBERO-Plus. LIBERO Avg. averages the four standard suites. Shaded rows denote matched +HiMEM comparisons. ZS denotes zero-shot evaluation. A dash (–) indicates that the method was not evaluated on the corresponding benchmark.
Task
Backbone
ECHO
HiMEM
Place
75.0
90.0
90.0
Stack
45.0
55.0
70.0
Insert
55.0
65.0
75.0
Basic-task average
58.3
70.0
78.3
G1
40.0
50.0
65.0
G2
45.0
55.0
60.0
TABLE III: Physical manipulation success rates (%). Averages are reported separately for basic and composite tasks.
Fig. 4: Hierarchical memory retrieval and semantic-guided cross-suite compositional experience reuse. (A) Representative query states at three key execution steps (S1, S3, and S4) of the target task. (B) Coarse-to-fine traversal of the hierarchical manipulation memory, illustrating activated retrieval paths and pruned irrelevant branches. (C) Historical experiences retrieved from the Spatial and Object suites provide reusable manipulation priors for grasping, in-drawer placement, and drawer closure, enabling compositional experience transfer across subgoals.
Fig. 5: Representative real-robot rollouts for five held-out compositional tasks (G1–G5). Each row shows four temporally ordered frames from one execution.
Model Variant
Memory / Retrieval Configuration
Success (%)
Vanilla π0
–
80.70
π0 + Short-term Buffer
Euclidean within-episode buffer
88.81
π0 + Flat Memory
Flat Euclidean k NN retrieval
83.25
π0 + Hyperbolic Memory
Hyperbolic distance-based k NN
91.11
π0 + Cone Tree
Entailment-aware cone-tree retrieval
92.04
π0 + HiMEM w/o Virtual Synthesis
Cone tree + consolidation
92.69
TABLE IV: Ablations on LIBERO-Long. Geometric controls assess memory organization and retrieval; three semantic controls assess outcome evidence, cross-episode consolidation, and online conditioning.
Setting
Backbone
ECHO
HiMEM
Held-out memory
Same-target memory
Held-out memory
Same-target memory
LIBERO cross-suite
80.70
89.31
93.48
92.78
95.30
Real robot (G1–G5)
37.0
47.0
–
61.0
-
TABLE V: Three additional ablations assess virtual synthesis, failure evidence, and cross-episode consolidation.
Source pool
Method
Segments
Success (%)
Tree (ms)
kNN (ms)
25%
ECHO
692
91.67
4.9
7.8
HiMEM
420
92.25
4.4
4.8
50%
ECHO
1384
92.25
5.5
15.3
HiMEM
834
92.67
5.1
9.0
75%
ECHO
2077
93.25
5.8
22.1
HiMEM
1250
94.75
5.2
13.9
TABLE VI: Memory-scale efficiency on LIBERO-Long. ECHO and HiMEM use matched source-pool fractions; segment counts reflect each method’s resulting episodic memory. Tree and kNN report episodic retrieval latency (ms); other costs are measured separately.
Target pruned (%)
Actual pruned (%)
Retained vectors
Success (%)
0
0
2769
95.3
20
13.1
2406
95.2
40
25.2
2071
94.9
60
42.8
1584
94.0
80
56.6
1202
92.9
TABLE VII: Effect of episodic memory pruning on LIBERO-Long with π0 + HiMEM. Pruning ratios use the pre-pruning episodic-vector count as the denominator.
Memory capacity is a critical factor determining the performance of Vision-Language-Action (VLA) models in long-horizon manipulation tasks. Existing memory-augmented architectures primarily rely on linear or flat storage, lacking structural priors for manipulation categories and hierarchical organization. This deficiency hinders efficient experience retrieval and limits generalization to unseen long-horizon task compositions. Inspired by the hierarchical organization of human experience, we propose ECHO (Experience Consolidation and Hierarchical Organization), a novel memory framework operating within a Continuous Hierarchical Space. By employing a hyperbolic autoencoder, ECHO maps VLA hidden states into this space. Leveraging hyperbolic metrics and entailment constraint mechanisms, experience vectors are organized into a semantic memory tree that supports efficient top-down retrieval. In parallel, a background consolidation mechanism continuously refines the memory tree through geometric interpolation and structural splitting, supporting virtual memory synthesis in the continuous space. We integrate ECHO into the π0 foundation model. Evaluations on LIBERO and preliminary real-world experiments demonstrate the effectiveness of our approach, notably achieving a 12.8% absolute improvement in execution success rate over the π0 baseline on LIBERO-Long, while improving compositional generalization on cross-suite unseen long-horizon tasks.
Yanbin Hu, Jin Cui, Jiayi Lu +6
School of Software, Xi’an Jiaotong University · State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University · School of Artificial Intelligence, Xi’an Jiaotong University
Mainstream Vision-Language-Action (VLA) models predict actions primarily from the current observation under a Markovian assumption, thus struggling with long-horizon, temporally dependent tasks. Existing memory-augmented VLAs either expand the observation window or retrieve history from the memory bank as auxiliary policy-side context. However, they leave memory outside the native latent embedding space of VLA reasoning, preventing historical experience from being fluidly interleaved with multimodal reasoning and action formation. To this end, we introduce LaMem-VLA, a latent-memory-native framework that reconstructs historical experience into latent memory tokens and directly interweaves them with VLA reasoning. At its core, LaMem-VLA introduces four coordinated components: (i) a curator that organizes historical experience into two complementary short-term and long-term memory vaults; (ii) a seeker that queries both vaults using the multimodal cognition to retrieve context-relevant evidence; (iii) a condenser that reconstructs the retrieved evidence into compact short-term and long-term latent memory tokens; and (iv) a weaver that injects these memory tokens with the current observation and instruction into one continuous embedding sequence. By representing, retrieving, and consuming historical experience entirely in the same continuous latent space, LaMem-VLA enables memory to directly participate in VLA reasoning and guide action generation under a bounded context. Extensive experiments on SimplerEnv and LIBERO demonstrate the superiority of our LaMem-VLA.
Hongyu Qu, Jianzhe Gao, Xiaobin Hu +6
Nanjing University of Science and Technology · Zhejiang University · National University of Singapore
Recent advances in Vision-Language-Action (VLA) models have enabled fine manipulation capabilities, yet performing tasks that require working memory remains a key challenge. Most existing VLA architectures operate under a Markovian assumption, mapping only the current observation to an action, which leads to failure in non-Markovian environments. Inspired by the human 'inner monologue', we propose Hierarchical Explicit Language Memory (HeLM), a hierarchical VLA framework that maintains task-relevant history in explicit language through factorized event detection and event-triggered memory updates. We evaluate HeLM across 10 non-Markovian real-world tasks with 56 variants, including five newly formulated inter-episode tasks that require persistent context across task boundaries. HeLM improves intra-episode success by 12.8 percentage points over the strongest baseline and achieves 78% subtask success on compositional generalization tasks, while using up to 4.5x less memory and running 2.5x faster than the keyframe baseline at long horizons.