SimpleMemVLA: A Simple but Effective Native-Video Memory for Vision-Language-Action Models

Cheng Yin; Wang Xu; Junpeng Yang; Sikyuen Tam; Hanyu Liu; Yuan Yao; Xiangrui Zeng; Junbo Cui

cs.CVSep 2, 2026

SimpleMemVLA: A Simple but Effective Native-Video Memory for Vision-Language-Action Models

Authors: Cheng Yin, Wang Xu, Junpeng Yang, Sikyuen Tam, Hanyu Liu, Yuan Yao, Xiangrui Zeng, Junbo Cui,

Organizations: Research Center for Advanced Electronics Manufacturing, MSE†, HUST‡ · Zhongguancun Academy, Beijing, China · Department of Computer Science and Technology, Tsinghua University · Modelbest ·

Abstract

Long-horizon manipulation is partially observable: the information needed to choose the next action may appear only in observations from minutes earlier. Existing memory mechanisms for VLAs, such as retrieval banks, learned compressors and recurrent states, must decide what to keep from the past before knowing what a future decision will require. They were motivated by the assumption that minute-scale history is too large to process directly, which no longer holds for modern VLM backbones. We propose SimpleMemVLA, a VLA without a dedicated memory module that uses the backbone's native video context directly as memory. It keeps the sampled history intact in the timestamped video format the backbone was pretrained to process, routes the evidence it finds to a standard flow-matching action head through the hidden states of a generated sub-task, and prefills the history shared by consecutive decisions during action execution, keeping latency close to that of a single-frame VLA. SimpleMemVLA achieves state-of-the-art results on four memory benchmarks without loss on general-purpose control, and with the same backbone and training setup it outperforms retrieval, compression and recurrent-state methods by a wide margin. History interventions show that the policy reads specific evidence from its past and follows edited histories without parameter updates, a visual form of in-context learning. On a physical dual-arm robot, it completes two tasks whose decisive evidence disappears before the robot acts. Code available at https://github.com/OpenBMB/SimpleMemVLA

Figures & tables

Figure 1: VLA memory mechanisms versus SimpleMemVLA. Left: Three prior design families insert dedicated memory machinery between the observation stream and policy, while SimpleMemVLA uses the timestamped stream directly as native context. Right: A 60 s history uses only 5.6k tokens of the backbone’s 262k-token context window, which can hold roughly 45 minutes.

Figure 2: SimpleMemVLA architecture and streaming inference. (a) The architecture uses only standard VLA components, with self-attention over plaintext-timestamped history serving as memory. (b) Consecutive decisions differ by only one temporal patch, enabling shared-prefix prefill during action execution and reducing latency from 1.02 s to 0.68 s with identical outputs (Section 4.6 ).

Figure 3: SimpleMemVLA leads all memory suites and matches the best results on the general-purpose suites. All methods from the per-suite tables are shown in score order. SimpleMemVLA uses one model per suite, while most RMBench baselines are task-specific specialists. Gray oracle and human references are excluded from ranking.

	Single-memory $M(1)$						Multi-memory $M(n)$
Method	Obs&PU	Rearr.	PutBack	SwapB	SwapT	Avg	Battery	Rank.	Cover	Press	Place-Mat †	Avg	Overall
One specialist model per task
DP ( Chi et al., 2023 )	1	0	0	11	20	6.4	10	10	0	0	–	5.0	5.8
ACT ( Zhao et al., 2023 )	1	29	0	2	2	6.8	19	0	0	0	–	4.8	5.9
X-VLA ( Zheng et al., 2026 )	9	13	18	16	3	11.8	26	1	2	0	–	7.3	9.8
Mem-0 ( Chen et al., 2026 )	4	89	90	67	14	52.8	28	18	68	0	–	28.5	42.0

Table 1: RMBench per-task success rates (%), averaged over the nine tasks with published baselines. Most baselines train one specialist model per task while SimpleMemVLA is a single multi-task model evaluated at

n=100

seeds per task in the streaming deployment of Section 4.6 . † No prior VLA reports Place-Mat so it is excluded from all averages. Best per column in bold .

Memory	Method	Integr. / VLM	Counting	Permanence	Reference	Imitation	AVG
Human	Human Performance	-	88.5	91	93	89.5	90.5
Symbolic (Oracle)	SimpleSG ( Dai et al., 2026 )	GT VLM	82.6	21.6	32.3	61.9	49.6
Symbolic (Oracle)	GroundSG ( Dai et al., 2026 )	GT VLM	83.9	93.3	95.2	64.0	84.1
Symbolic	SimpleSG	Gemini	39.0	13.5	22.5	18.0	23.3
Symbolic	SimpleSG	QwenVL	44.6	19.6	25.2	26.6	29.0
Symbolic	GroundSG	Gemini	12.8	15.8	11.3	6.5	11.6

Table 2: RoboMME category-level success rates (%). AVG is over all sixteen tasks. Gray rows are reference-only and excluded from ranking. Bold and underline mark ranks 1 and 2. The SimpleMemVLA variant rows re-create one mechanism family each on the otherwise unchanged SimpleMemVLA stack. Per-task results are in Table 10 .

Method	ShellGame	Intercept	RC-3	RC-5	RC-9	Avg
Vision-language-action models
CronusVLA ( Li et al., 2026 )	32	5	31	13	9	18.0
SpatialVLA ( Qu et al., 2025 )	23	27	27	17	11	21.0
OpenVLA-OFT ( Kim et al., 2025 )	47	14	59	16	6	28.4
$\pi_{0}$ ( Black et al., 2025 )	33	42	35	22	15	29.4
Octo ( Team et al., 2024 )	46	39	45	17	11	31.6

Table 3: MIKASA-Robo five-task success rates (%). GMP is a per-task non-VLA reference. Best VLA per column in bold .

	Transferring		Occlusion		Counting		Sequence		Average
Method	TSR	CSR	TSR	CSR	TSR	CSR	TSR	CSR	TSR	CSR
$\pi_{0.5}$ ( Intelligence et al., 2025 )	20.0	42.8	12.7	17.2	14.3	50.9	60.0	71.6	21.5	38.7
HiF-VLA ( Lin et al., 2025b )	17.5	38.9	12.7	27.1	8.6	45.9	42.5	70.2	16.9	39.8
MemoryVLA ( Shi et al., 2026a )	15.0	37.2	7.3	13.1	14.3	55.1	37.5	65.2	15.0	35.3
MemER ( Sridhar et al., 2026 )	20.0	36.1	16.4	33.2	27.1	65.1	65.0	79.1	27.3	49.1
FrameSamp+Modul ( Dai et al., 2026 )	63.8	72.1	39.1	56.5	31.4	57.8	73.8	86.9	46.2	63.9

Table 4: RoboMemArena category-level TSR and CSR (%) over all 26 tasks under the official protocol. MemER is the benchmark authors’ reimplementation, FrameSamp+Modul the only external leaderboard entry, the gray oracle row excluded from ranking. Best per column in bold .

General-purpose policies and VLAs
Method	Spatial	Object	Goal	Long	Avg
Diffusion Policy ( Chi et al., 2023 )	78.3	92.5	68.3	50.5	72.4
TraceVLA ( Zheng et al., 2025 )	84.6	85.2	75.1	54.1	74.8
Octo ( Team et al., 2024 )	78.9	85.7	84.6	51.1	75.1
OpenVLA ( Kim et al., 2024 )	84.7	88.4	79.2	53.7	76.5
SpatialVLA ( Qu et al., 2025 )	88.2	89.9	78.6	55.5	78.1

Table 5: LIBERO success rates (%) on the four standard suites, 500 trials per suite. Best per column in bold .

General-purpose policies and VLAs
Method	Camera	Robot	Lang.	Light	Backg.	Noise	Layout	Total
OpenVLA ( Kim et al., 2024 )	0.8	3.5	23.0	8.1	34.8	15.2	28.5	15.6
WorldVLA ( Cen et al., 2025 )	0.1	27.9	41.6	43.7	17.1	10.9	38.0	25.0
Diffusion Policy ( Chi et al., 2023 )	1.6	32.3	77.0	25.0	19.8	20.3	42.2	31.7
NORA ( Hung et al., 2025 )	2.2	37.0	65.1	45.7	58.6	12.8	62.1	39.0
UniVLA ( Bu et al., 2025 )	1.8	46.2	69.6	69.0	81.0	21.2	31.9	42.9

Table 6: LIBERO-Plus zero-shot robustness transfer to the 10,030 perturbed tasks, all policies trained on standard LIBERO only. Best per column in bold .

Figure 5: Task-level effects of restricted memory interfaces on RoboMME. The three controlled variants differ from SimpleMemVLA only in how history enters the model. (a) Success rates across 16 tasks grouped by benchmark dimension. (b) The same results normalized to native-context performance. Retrieval, token compression and recurrent state retain at most 79%, 96% and 58%, respectively, with the token-compression peak confined to the count task SwingXtimes. Results are from Table 10 , with

n=50

episodes per task.

Figure 6: History interventions redirect the output across all suites. Blocks denote benchmarks and rows show histories from oldest to most recent, processed through the unchanged deployment pipeline. Orange borders mark evidence frames, gray fills evidence ablations and red borders donor-episode insertions. Removing evidence changes the output, donor evidence redirects it toward the donor content and matched control edits leave it unchanged.

Figure 7: Streaming inference reduces decision latency. Measured on one H100 (bf16, batch size 1). (a) With 60 s histories, streaming reduces decision latency from 1.02 s to 0.68 s, close to the same-model single-frame baseline of approximately 0.65 s. (b) On artificially constructed 45 min inputs (245k tokens), decision-path latency falls from 32.1 s to 1.18 s.

Figure 8: Native-context memory relies on retained temporal evidence and contextual hidden states. (a) Cover-blocks fails once the covering event leaves the window, whereas press-button degrades gradually as completed presses are removed. (b) Both tasks depend on frame order, while timestamps matter primarily for counting. (c) Stale sub-task hidden states sharply reduce success, whereas equally stale position IDs leave it unchanged. (d) Behavior follows the source of the hidden states rather than the token embeddings, identifying contextual hidden states as the memory-to-action interface.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

	RMBench	RoboMME	MIKASA-Robo	RoboMemArena	LIBERO
Embodiment	Aloha-AgileX, bimanual	Panda, single	Panda, single	Franka, single	Franka, single
Action dimension $d_{a}$	14	8	8	7	7
History camera $\mathcal{C}_{\mathrm{hist}}$	head	front	overhead	front	agentview
Current cameras $\mathcal{C}_{\mathrm{cur}}$	both wrists	wrist	wrist	wrist	wrist
Window $T_{w}$ (s)	60	60	3	126	30
Video rate $f_{v}$ (fps)	2	2	20	1	2

Appendix

Table 8: Per-benchmark instantiation of the SimpleMemVLA configuration tuple and the resulting training cost. Everything else about the method is identical across suites. Training time is wall-clock hours on 128 H100 GPUs.

Cover Blocks	RGB	GRB	GBR	RBG	BRG	BGR	Overall
Successes/trials	8/10	6/10	7/10	6/10	5/10	3/10	35/60

Appendix

Table 9: Real-world results by initial configuration. Each entry reports successes over ten autonomous trials. Cover Blocks labels list colors from far to near. Put Back Block labels follow the task coordinate convention.

Figure 9: Additional real-world autonomous rollouts. (a) Cover Blocks under three initial color arrangements, labeled from far to near. (b) Put Back Block from four initial positions, labeled using the task coordinate convention rather than camera-image directions. Each row shows chronological frames from one successful rollout.

Figure 10: First on all sixteen RoboMME tasks among the 21 deployable methods. Gray dots denote the 20 other non-Human, non-Oracle methods evaluated by the benchmark. Boxes show the interquartile range of the 21-method ranked pool, with median and full range. Orange diamonds mark SimpleMemVLA and labels report its competition rank out of 21. Human performance and the two symbolic Oracle variants, which replace VLM outputs with ground-truth outputs, are shown only as references and are excluded from all ranks and distribution statistics.

Figure 11: Task-wise RoboMME success rates (%, higher is better) for representative methods. Panels group the 16 tasks into Counting, Permanence, Reference and Imitation. Overall is recomputed from all 16 task scores. Human and GroundSG (Oracle, using ground-truth VLM outputs) are shown only as references and are excluded from the ranked comparison. For each remaining baseline family, the row with the highest overall AVG is shown. Values above orange hatched bars report SimpleMemVLA scores.

Figure 12: Complete task-wise RoboMME success rates (%, higher is better) for all 24 rows. Tasks follow the source-table order and methods retain the same order in every panel. Colors encode memory families. The orange hatched bar denotes SimpleMemVLA.

Memory	Method	Integr. / VLM	Counting				Permanence				Reference				Imitation				AVG
			BinFill	PickX	SwingX	StopC	V-Umsk	B-Umsk	V-UmskS	B-UmskS	PickHL	V-Repk	V-PlcB	V-PlcO	MoveC	InsPeg	PatLock	RouteS
Human	Human Performance	-	96	100	80	78	90	92	92	90	92	92	98	90	90	98	84	86	90.5
Symbolic (Oracle)	SimpleSG ( Dai et al., 2026 )	GT VLM	85.78	99.78	100	44.67	33.11	22	15.56	15.56	44	27.78	31.33	26	87.33	10	95.33	55.11	49.58
Symbolic (Oracle)	GroundSG ( Dai et al., 2026 )	GT VLM	85.78	100	100	49.67	98.78	95	99.22	80.22	83.33	97.33	100	100	87.78	15.56	97	55.56	84.08
Symbolic	SimpleSG	Gemini	46	63	45	2	29	9	14	2	20	15	26	29	61	4	7	0	23.25
Symbolic	SimpleSG	QwenVL	77.56	95.33	5.11	0.44	34.22	19.33	15.33	9.56	17.11	25.33	33.33	25.11	82	3.78	12.67	7.78	29

Appendix

Table 10: Complete task-level RoboMME success rates (%). Human and GT-VLM Oracle rows (gray) are reference-only and excluded from ranking. Bold and underline mark ranks 1 and 2 among the remaining 21 methods. Task abbreviations follow Table 2 ’s four dimensions in source order. The three SimpleMemVLA variant rows are the controlled within-stack rebuilds of Section 4.4 and are excluded from ranking.

By source suite
	Spatial	Object	Goal	Long
Tasks	2,402	2,518	2,591	2,519
Success (%)	81.8	83.9	75.0	73.1
By difficulty level
	Level 1	Level 2	Level 3	Level 4	Level 5
Tasks	1,644	2,202	2,094	1,886	2,083

Appendix

Table 11: LIBERO-Plus breakdowns. Top: by source LIBERO suite. Bottom: by the benchmark’s five difficulty levels, which stratify tasks by the accuracy of four reference models ( Fei et al., 2025 ) . The 121 tasks that carry no difficulty label in the benchmark metadata are omitted from the bottom panel.

Corruption type	Overall	Low	Mid	High
Motion blur	87.8	100	95	76
Glass blur	85.7	93	89	79
Gaussian blur	69.2	95	78	47
Zoom blur	62.5	96	77	44
Fog	60.7	93	80	39

Appendix

Table 12: The Sensor Noise column of Table 6 , split by corruption type and severity tier. Types are encoded in the benchmark’s task naming. Success rates in %.

Figure 13: The SWA variant: one context construction, trained and deployed. (a) Two consecutive streaming decisions: each appends one unit (orange) to the persistent stream, the softmax window slides by one unit, and the unit leaving it (red) has its cache rows freed; the linear-attention state (green) integrates from

t=0

and is never reset. (b) Training forwards each episode once as a trunk and supervises

B

branches at random anchors, each attending to its last

\leq M

units and resuming the linear state snapshotted at its fork — exactly the deployed construction. Details in the accompanying text.

Figure 14: Figure 7 , extended with the SWA streaming variant. Measurements use one H100, bf16 and batch size 1, with the setup of Figure 7 . (a) The SWA decision takes 0.92 s, within the 0.96 s execution interval. (b) The SWA variant uses a fixed

L=5{,}888

-token window to bound its resident cache.

Method	Obs&PU	Rearr.	PutBack	SwapB	SwapT	Avg	Battery	Rank.	Cover	Press	Place-Mat †	Avg	Overall
	Single-memory $M(1)$						Multi-memory $M(n)$
SimpleMemVLA (Table 1 )	65	100	100	100	93	91.6	90	100	98	100	100	97.0	94.0
$+$ SWA streaming (evicted cache)	56	100	100	100	93	89.8	71	100	99	100	99	92.5	91.0

Appendix

Table 13: Closed-loop success of the SWA streaming deployment , on the RMBench protocol of Table 1 (100 held-out seeds per task).

Vision-Language-Action Models

Vision-language-action models turn visual observations and natural-language instructions into actions for robots and other embodied agents.

See all 1,154 papers

Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation
Jul 8, 2026Hongyu Qu, Jianzhe Gao, Xiaobin Hu +6Memory-Augmented VLMs Robotic Manipulation
Nanjing University of Science and Technology · Zhejiang University · National University of Singapore
ReMem-VLA: Empowering Vision-Language-Action Model with Memory via Dual-Level Recurrent Queries
Mar 13, 2026Hang Li, Fengyi Shen, Dong Chen +6Memory-Augmented VLMs Vision-Language-Action Models
Technical University of Munich, Munich, Germany · Huawei Heisenberg Research Center, Munich, Germany · University of Freiburg, Freiburg, Germany +2
NativeMEM: Native Memory Compression for Long-Horizon Robotic Manipulation
Jul 7, 2026Ziye Wang, Modi Shi, Chaojun Ni +5Memory-Augmented VLMs Long-Horizon Robotic Manipulation
The University of Hong Kong · Beihang University · Peking University +2

Jul 8, 2026J/K move · Enter open · S save

Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation

Hongyu Qu, Jianzhe Gao, Xiaobin Hu, Shaohuan Yang+5

Nanjing University of Science and Technology · Zhejiang University · National University of Singapore

Memory-Augmented VLMs Robotic Manipulation Vision-Language-Action Models Memory-Augmented Agents Reasoning for VLA Models

Open paper AskPDF arXiv

Mainstream Vision-Language-Action (VLA) models predict actions primarily from the current observation under a Markovian assumption, thus struggling with long-horizon, temporally dependent tasks. Existing memory-augmented VLAs either expand the observation window or retrieve history from the memory bank as auxiliary policy-side context. However, they leave memory outside the native latent embedding space of VLA reasoning, preventing historical experience from being fluidly interleaved with multimodal reasoning and action formation. To this end, we introduce LaMem-VLA, a latent-memory-native framework that reconstructs historical experience into latent memory tokens and directly interweaves them with VLA reasoning. At its core, LaMem-VLA introduces four coordinated components: (i) a curator that organizes historical experience into two complementary short-term and long-term memory vaults; (ii) a seeker that queries both vaults using the multimodal cognition to retrieve context-relevant evidence; (iii) a condenser that reconstructs the retrieved evidence into compact short-term and long-term latent memory tokens; and (iv) a weaver that injects these memory tokens with the current observation and instruction into one continuous embedding sequence. By representing, retrieving, and consuming historical experience entirely in the same continuous latent space, LaMem-VLA enables memory to directly participate in VLA reasoning and guide action generation under a bounded context. Extensive experiments on SimplerEnv and LIBERO demonstrate the superiority of our LaMem-VLA.

Abstract

Vision-Language-Action Models

Explore similar work

Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation

ReMem-VLA: Empowering Vision-Language-Action Model with Memory via Dual-Level Recurrent Queries

NativeMEM: Native Memory Compression for Long-Horizon Robotic Manipulation