Remember Smarter: Visual History Compressor and Hyperbolic Experience Space for Robotic Memory

Dai Zhou; Jiexi Yan; Tong Li; Yuxuan Wang; Cheng Deng

cs.ROAug 15, 2026

Remember Smarter: Visual History Compressor and Hyperbolic Experience Space for Robotic Memory

Authors: Dai Zhou, Jiexi Yan, Tong Li, Yuxuan Wang, Cheng Deng

Organizations: Xidian University

Abstract

Long-horizon robot policies require compact access to recent observations and reusable experience without expanding the vision-language-action (VLA) context. We introduce Remember Smarter (RS), a plug-and-play module with complementary visual-history and hyperbolic experience-memory branches. Its visual branch compresses multi-view patch histories using bidirectional spatial Mamba and causal temporal Mamba, then exposes the resulting memory to action-facing hidden states through residual cross-attention while leaving the VLM visual-token stream unchanged. Its experience branch stores successful final-layer VLM states in a Poincare VAE space, organizes them hierarchically, and asynchronously converts retrieved experience into geodesic prompt tokens without blocking action inference. When adapted to pi0, RS increases total success on LIBERO-Plus from 53.6% to 70.6% and achieves substantial performance gains in real-robot experiments designed to evaluate memory retention and experience utilization.

Figures & tables

Figure 1: High-level concept of Remember Smarter. The fine-grained branch compresses multi-view visual patch history, while the coarse-grained branch retrieves structured hyperbolic experience.

Figure 2: Mamba-compressed visual-history pipeline. Historical multi-view patch features are extracted by the shared visual encoder and mapped into a unified feature space by a lightweight adapter. Each frame is encoded through bidirectional spatial scans and learned-query pooling, after which causal temporal Mamba layers recurrently aggregate the per-view history. The resulting visual-history representations provide the keys and values for residual cross-attention, while the final hidden representations used for action prediction provide the queries. The enhanced hidden representations are then passed to the action head.

Figure 3: P-VAE experience space and HypHC hierarchy for the first 300 successful

\pi_{0}

inference records from https://huggingface.co/datasets/physical-intelligence/libero . Each colored point is a final-layer VLM hidden state encoded for visualization; color denotes its source task T00–T09. The black star in (b) marks the virtual root of the decoded HypHC hierarchy.

Method	Camera	Robot	Language	Light	Background	Noise	Layout	Total
MemoryVLA++	36.4	68.9	88.7	93.8	90.6	63.5	83.8	73.1
MemoryVLA	42.7	44.9	84.4	92.8	95.0	62.1	84.7	70.2
OpenVLA-OFT	56.4	31.9	79.5	88.7	93.3	75.8	74.2	69.6
$\pi_{0}$ -Fast	65.1	21.6	61.0	73.2	73.2	74.4	68.8	61.6
$\pi_{0}$	13.8	6.0	58.8	85.0	81.4	79.0	68.9	53.6
$\pi_{0}$ +RS	50.8	45.8	65.3	92.1	94.0	81.2	78.0	70.6

Table 1: Robustness on the full LIBERO-Plus benchmark. Entries are strict success rates (%), and bold indicates the highest score in each reported column. Total is computed over all 10,030 tasks rather than averaging the seven dimension-level scores. Published baselines retain the settings reported in their original work; the MemoryVLA variants are evaluated in the zero-shot setting [ 22 ] . The final row is the absolute change of

\pi_{0}

+RS relative to

\pi_{0}

Method	Press-2	Press-4	Press-6	Order	Occl.	Chen.-S	Chen.-H
$\pi_{0}$	64	20	0	20	72	64	20
$\pi_{0}$ +RS	92	70	60	40	88	60	80

Table 2: Real-robot strict success rate (%). Press-

k

denotes

k

repeated presses; Order denotes ordered button pressing; Occl. denotes occluded placement; Chen.-S/H denote the simple/hard Chenchen tasks.

Figure 4: Representative real-robot rollouts on the compositional Chen.-H task. The green sequence shows a successful $\pi_{0}$ +RS rollout. The red sequences show $\pi_{0}$ failures: (a) failure to lift Chenchen, (b) failure to press the required button, and (c) failure to place the ball in the cup. These examples are consistent with the difficulty of recognizing a failed subtask and recovering without access to successful experience.

Figure 5: Component ablation on the four standard LIBERO suites. VMC is the visual Mamba compressor without the Poincare experience space.

\pi_{0}

is a single run (standard deviation unavailable);

\pi_{0}

+VMC and

\pi_{0}

+RS use five runs, and error bars denote standard deviation.

Figure 6: Effect of Mamba history length and TBPTT truncation. (a) Long success-rate change relative to a no-Mamba baseline, measured every 5k training steps. (b) Training loss for a fixed 64-frame history under different TBPTT truncation lengths.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Representative successful keyframe sequences for the real-robot task families: (a) Chen.-S name grounding, (b) Press-2 counting, (c) Order sequence following, (d) occluded placement, and (e) Chen.-H compositional long-horizon execution.

Item	Setting
Visual history	3 views, 1 FPS, 64 frames; TBPTT window 16
VMC	$d=256$ , state 16, convolution 4, expansion 2, head width 64
Precision / parallelism	bf16 activations; fp32 parameters and gradients; 4-device FSDP
Optimizer	Grouped AdamW; weight decay $10^{-4}$ ; gradient clip 1; EMA 0.999
Batch and checkpoints	Global batch 32; checkpoint every 2,500 steps
Stage A	5k steps; RS adapter, VMC, and history reader trainable

Appendix

Table 3: Default

\pi_{0}

+RS training configuration.

Memory-Augmented VLMs

Memory-augmented vision-language models use persistent or retrieved context to improve reasoning across long inputs and interactions.

See all 159 papers

ActMem-VLA: Action-History Memory with Dual-Expert Denoising for Long-Horizon Vision-Language-Action Policies
Sep 29, 2026Yaxin Zhao, Dianye Huang, Chenwei Wang +2Memory-Augmented VLMs Long-Horizon Robotic Manipulation
Harbin Institute of Technology, Harbin, China. · Medical Intelligence and Robotic Cognition (MIRoC) Lab, Department of Mechanical Engineering, The University of Hong Kong (HKU), Hong Kong SAR, China. · Department of Computing, The Hong Kong Polytechnic University, Hong Kong SAR, China
NativeMEM: Native Memory Compression for Long-Horizon Robotic Manipulation
Jul 7, 2026Ziye Wang, Modi Shi, Chaojun Ni +5Memory-Augmented VLMs Long-Horizon Robotic Manipulation
The University of Hong Kong · Beihang University · Peking University +2
Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation
Jul 8, 2026Hongyu Qu, Jianzhe Gao, Xiaobin Hu +6Memory-Augmented VLMs Robotic Manipulation
Nanjing University of Science and Technology · Zhejiang University · National University of Singapore

Sep 29, 2026J/K move · Enter open · S save

ActMem-VLA: Action-History Memory with Dual-Expert Denoising for Long-Horizon Vision-Language-Action Policies

Yaxin Zhao, Dianye Huang, Chenwei Wang, Chenguang Yang+1

Harbin Institute of Technology, Harbin, China. · Medical Intelligence and Robotic Cognition (MIRoC) Lab, Department of Mechanical Engineering, The University of Hong Kong (HKU), Hong Kong SAR, China. · Department of Computing, The Hong Kong Polytechnic University, Hong Kong SAR, China

Memory-Augmented VLMs Long-Horizon Robotic Manipulation Visuomotor Policy Learning Vision-Language-Action Models

Open paper AskPDF arXiv

Vision-language-action (VLA) models have driven rapid progress in robotic manipulation, demonstrating strong fine-grained control and promising performance on long-horizon tasks. However, many existing VLAs lack explicit access to interaction history, making them vulnerable to perceptual aliasing: similar current observations and robot states at different task stages may induce action ambiguity and lower success rate. Existing methods incorporate temporal or progress cues through feature conditioning, action-prior modification, or sampling guidance. However, methods that jointly fine-tune memory modules and the base VLA incur additional policy-training costs, motivating the separation of trainable history-conditioned steering from frozen base-policy refinement. We propose ActMem-VLA, a dual-expert handover architecture that augments a frozen, fine-tuned VLA with a memory plugin comprising a Mamba-based memory module and a lightweight PreAction Expert (PAE). Specifically, Mamba encodes executed-action history into memory that conditions PAE alongside current context. With these inputs, PAE steers task progression during early, high-noise denoising, then passes the partially denoised action to the frozen Action Expert (AE) to refine action details during the remaining low-noise steps. The fine-tuned base VLA remains frozen throughout training, while only the Mamba module and PAE are jointly optimized. On LIBERO-Mem, ActMem-VLA achieves 80.8% average success across all ten tasks, compared with 65.2% for

π_{0.5}

and 49.5% for MemoryVLA, while introducing only 3.45% additional parameters. Across four real-world tasks, it improves the average success rate over

π_{0.5}

by 28.8%.

Abstract

Memory-Augmented VLMs

Explore similar work

ActMem-VLA: Action-History Memory with Dual-Expert Denoising for Long-Horizon Vision-Language-Action Policies

NativeMEM: Native Memory Compression for Long-Horizon Robotic Manipulation

Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation