Dual-Stream Memory for Vision-Language-Action Policies

Yang Wang; Ping Liu; Ruiyu Wang; Xuhua Chen; Lingfeng Zhang; Tao Zhang

cs.ROOct 9, 2026

Dual-Stream Memory for Vision-Language-Action Policies

Authors: Yang Wang, Ping Liu, Ruiyu Wang, Xuhua Chen, Lingfeng Zhang, Tao Zhang

Organizations: Magic-Lab Team, Magiclab Robotics Inc. · University of California, Los Angeles, USA

Abstract

Memory-dependent manipulation requires a policy to track execution state and recover visual evidence that is no longer observable. We propose dual-stream memory for vision-language-action policies, assigning these requirements to complementary representations. A task-agnostic Gated DeltaNet compresses history into one temporal token, supervised by progress prediction and action-conditioned feature reconstruction. A task-conditioned retriever retains frames above a fixed relevance threshold; linear multi-scale fusion encodes each frame into sixteen spatial tokens. We then freeze the temporal compressor and jointly optimize retrieval and policy learning. Across sixteen RoboMME tasks, the complete model achieves 49.22% mean success versus 44.51% for the published FrameSamp+Modul reference, improving fourteen tasks and all four suite means. On four real-robot tasks, mean success reaches 62.5% versus 37.5%, with the largest gains on repeated actions requiring execution-stage discrimination. These results support combining execution context with selective visual access, while failures under cup swaps expose persistent identity tracking as an unresolved requirement.

Figures & tables

Fig. 1: Dual-stream memory combines recurrent execution context with retrieved spatial evidence. Bars show mean task success rate (TSR) across sixteen RoboMME tasks and four real-world tasks evaluated on the Piper robot (Sections IV-D and IV-E ). Images illustrate a RoboMME episode and a human ShellGame demonstration. Orange boxes mark the visible target and its final covering cup; memory symbols are schematic.

Fig. 2: Dual-stream memory. (a) GDN compresses visual history into one temporal token; threshold retrieval and multi-scale fusion produce $16m_{t}$ spatial tokens from $m_{t}$ retained frames. Dashed arrows denote action-expert queries to the memory readouts. Their gated sum modulates the action expert. (b) Stage 2 combines action and retrieval losses, with GDN frozen. (c) Stage 1 pretrains GDN with compression objectives; its auxiliary heads are then removed. Snowflakes mark frozen modules in their respective stages. Camera images show a RoboMME episode; score displays are schematic.

Component / suite	Task	Base	Addition	$\Delta$ (pp)
GDN / Counting	BinFill	39.56	43.33	+3.77
	StopCube	42.00	57.33	+15.33
	PickXtimes	87.33	94.00	+6.67
	SwingXtimes	92.00	94.00	+2.00
	Suite mean	65.22	$\mathbf{72.17\pm 4.25}$	+6.94
Retrieval / Permanence	ButtonUnmask	25.11	30.67	+5.56

TABLE I: Component comparisons. TSR (%) across seeds 0, 7, and 42; suite rows report mean

\pm

standard deviation. Base: published FrameSamp+Modul. Bold marks improvements over Base.

\Delta

: percentage-point difference before rounding. Per-seed scores are in the supplement.

Fig. 3: Current-progress predictions for the first eight tasks in the source ordering, with two episodes per task. Blue and orange identify episodes; solid and dashed curves denote predictions and reference labels. Episode identifiers appear below task names. The horizontal axis is normalized execution time. The readout probes execution-stage information in recurrent memory; the selected traces illustrate task-dependent errors.

Fig. 4: Sixteen-slot visual-history comparisons for (a) InsertPeg, (b) PatternLock, and (c) SwingXtimes. Blue and orange borders mark reference (GT) and prediction-associated (Pred) images. On each Pred row, a thick green box marks a scene match and a thick red box marks a mismatch. Columns preserve the supplied slot order. These selected examples illustrate action-conditioned visual-history reconstruction, rather than a dataset-level retrieval metric.

Fig. 5: Temporal retrieval on four training episodes: (a) PickHighlight, (b) VideoUnmask, (c) ButtonUnmask, and (d) StopCube. Blue curves show normalized scores; shading and gold lines mark positive ranges and keyframe centers. Red markers and thumbnails show peaks above the dashed visualization threshold (0.04). AP denotes average precision; R@ $|$ GT $|$ denotes recall with as many retrieved frames as positive frames. Inference instead thresholds LSE logits at 0.6 (Section III-C ).

Suite	Task	$\pi_{0.5}$ (no memory)	NativeMEM	FrameSamp+Modul	Ours	$\Delta$ (pp)
Counting	BinFill	30.00	21.33	39.56	42.00	+2.44
	StopCube	6.67	4.00	42.00	55.33	+13.33
	PickXtimes	42.89	24.00	87.33	93.33	+6.00
	SwingXtimes	35.56	40.67	92.00	92.67	+0.67
	Counting mean	28.78	22.50	65.22	70.83	+5.61
Permanence	ButtonUnmask	22.22	14.00	25.11	28.00	+2.89

TABLE II: RoboMME evaluation. TSR (%) averaged across seeds 0, 7, and 42; suite and overall means weight tasks equally. Bold marks improvements over published FrameSamp+Modul.

\Delta

: percentage-point difference from reported means.

Fig. 6: Real-robot task examples from recorded demonstrations: (a) repeated picking, (b) repeated swinging, (c) revisiting demonstrated objects, and (d) selecting a cup after swaps. Overhead views show robot execution; these frames illustrate the task settings, not policy-evaluation outcomes.

Method	Pick $\times 3$	Swing $\times 2$	RePick	Shell	Avg.
$\pi_{0.5}$ base	20.0	6.7	3.3	30.0	15.0
NativeMEM	20.0	30.0	40.0	36.7	31.7
FrameSamp+Modul	26.7	36.7	33.3	53.3	37.5
Ours	80.0	93.3	36.7	40.0	62.5

TABLE III: Real-robot evaluation. TSR (%) over 30 rollouts per task. Bold marks gains over FrameSamp+Modul; Avg. weights tasks equally.

Explore similar work

ActMem-VLA: Action-History Memory with Dual-Expert Denoising for Long-Horizon Vision-Language-Action Policies
Sep 29, 2026Yaxin Zhao, Dianye Huang, Chenwei Wang +2Memory-Augmented VLMs Long-Horizon Robotic Manipulation
Harbin Institute of Technology, Harbin, China. · Medical Intelligence and Robotic Cognition (MIRoC) Lab, Department of Mechanical Engineering, The University of Hong Kong (HKU), Hong Kong SAR, China. · Department of Computing, The Hong Kong Polytechnic University, Hong Kong SAR, China
ReMem-VLA: Empowering Vision-Language-Action Model with Memory via Dual-Level Recurrent Queries
Mar 13, 2026Hang Li, Fengyi Shen, Dong Chen +6Memory-Augmented VLMs Vision-Language-Action Models
Technical University of Munich, Munich, Germany · Huawei Heisenberg Research Center, Munich, Germany · University of Freiburg, Freiburg, Germany +2
Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation
Jul 8, 2026Hongyu Qu, Jianzhe Gao, Xiaobin Hu +6Memory-Augmented VLMs Robotic Manipulation
Nanjing University of Science and Technology · Zhejiang University · National University of Singapore

Sep 29, 2026J/K move · Enter open · S save

ActMem-VLA: Action-History Memory with Dual-Expert Denoising for Long-Horizon Vision-Language-Action Policies

Yaxin Zhao, Dianye Huang, Chenwei Wang, Chenguang Yang+1

Harbin Institute of Technology, Harbin, China. · Medical Intelligence and Robotic Cognition (MIRoC) Lab, Department of Mechanical Engineering, The University of Hong Kong (HKU), Hong Kong SAR, China. · Department of Computing, The Hong Kong Polytechnic University, Hong Kong SAR, China

Memory-Augmented VLMs Long-Horizon Robotic Manipulation Visuomotor Policy Learning Vision-Language-Action Models

Open paper AskPDF arXiv

Vision-language-action (VLA) models have driven rapid progress in robotic manipulation, demonstrating strong fine-grained control and promising performance on long-horizon tasks. However, many existing VLAs lack explicit access to interaction history, making them vulnerable to perceptual aliasing: similar current observations and robot states at different task stages may induce action ambiguity and lower success rate. Existing methods incorporate temporal or progress cues through feature conditioning, action-prior modification, or sampling guidance. However, methods that jointly fine-tune memory modules and the base VLA incur additional policy-training costs, motivating the separation of trainable history-conditioned steering from frozen base-policy refinement. We propose ActMem-VLA, a dual-expert handover architecture that augments a frozen, fine-tuned VLA with a memory plugin comprising a Mamba-based memory module and a lightweight PreAction Expert (PAE). Specifically, Mamba encodes executed-action history into memory that conditions PAE alongside current context. With these inputs, PAE steers task progression during early, high-noise denoising, then passes the partially denoised action to the frozen Action Expert (AE) to refine action details during the remaining low-noise steps. The fine-tuned base VLA remains frozen throughout training, while only the Mamba module and PAE are jointly optimized. On LIBERO-Mem, ActMem-VLA achieves 80.8% average success across all ten tasks, compared with 65.2% for

π_{0.5}

and 49.5% for MemoryVLA, while introducing only 3.45% additional parameters. Across four real-world tasks, it improves the average success rate over

π_{0.5}

by 28.8%.

Abstract

Explore similar work

ActMem-VLA: Action-History Memory with Dual-Expert Denoising for Long-Horizon Vision-Language-Action Policies

ReMem-VLA: Empowering Vision-Language-Action Model with Memory via Dual-Level Recurrent Queries

Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation