SimpleMemVLA: A Simple but Effective Native-Video Memory for Vision-Language-Action Models
Authors: Cheng Yin, Wang Xu, Junpeng Yang, Sikyuen Tam, Hanyu Liu, Yuan Yao, Xiangrui Zeng, Junbo Cui, Yequan Wang, Zhouping Yin, Yankai Lin
Organizations: Research Center for Advanced Electronics Manufacturing, MSE†, HUST‡ · Zhongguancun Academy, Beijing, China · Department of Computer Science and Technology, Tsinghua University · Modelbest · Peking University · College of AI, Tsinghua University · Beijing Academy of Artificial Intelligence · Gaoling School of Artificial Intelligence, Renmin University of China
Long-horizon manipulation is partially observable: the information needed to choose the next action may appear only in observations from minutes earlier. Existing memory mechanisms for VLAs, such as retrieval banks, learned compressors and recurrent states, must decide what to keep from the past before knowing what a future decision will require. They were motivated by the assumption that minute-scale history is too large to process directly, which no longer holds for modern VLM backbones. We propose SimpleMemVLA, a VLA without a dedicated memory module that uses the backbone's native video context directly as memory. It keeps the sampled history intact in the timestamped video format the backbone was pretrained to process, routes the evidence it finds to a standard flow-matching action head through the hidden states of a generated sub-task, and prefills the history shared by consecutive decisions during action execution, keeping latency close to that of a single-frame VLA. SimpleMemVLA achieves state-of-the-art results on four memory benchmarks without loss on general-purpose control, and with the same backbone and training setup it outperforms retrieval, compression and recurrent-state methods by a wide margin. History interventions show that the policy reads specific evidence from its past and follows edited histories without parameter updates, a visual form of in-context learning. On a physical dual-arm robot, it completes two tasks whose decisive evidence disappears before the robot acts. Code available at https://github.com/OpenBMB/SimpleMemVLA
Figures & tables
Figure 1: VLA memory mechanisms versus SimpleMemVLA. Left: Three prior design families insert dedicated memory machinery between the observation stream and policy, while SimpleMemVLA uses the timestamped stream directly as native context. Right: A 60 s history uses only 5.6k tokens of the backbone’s 262k-token context window, which can hold roughly 45 minutes.
Figure 2: SimpleMemVLA architecture and streaming inference. (a) The architecture uses only standard VLA components, with self-attention over plaintext-timestamped history serving as memory. (b) Consecutive decisions differ by only one temporal patch, enabling shared-prefix prefill during action execution and reducing latency from 1.02 s to 0.68 s with identical outputs (Section 4.6 ).
Figure 3: SimpleMemVLA leads all memory suites and matches the best results on the general-purpose suites. All methods from the per-suite tables are shown in score order. SimpleMemVLA uses one model per suite, while most RMBench baselines are task-specific specialists. Gray oracle and human references are excluded from ranking.
Single-memory M(1)
Multi-memory M(n)
Method
Obs&PU
Rearr.
PutBack
SwapB
SwapT
Avg
Battery
Rank.
Cover
Press
Place-Mat †
Avg
Overall
One specialist model per task
DP ( Chi et al., 2023 )
1
0
0
11
20
6.4
10
10
0
0
–
5.0
5.8
ACT ( Zhao et al., 2023 )
1
29
0
2
2
6.8
19
0
0
0
–
4.8
5.9
X-VLA ( Zheng et al., 2026 )
9
13
18
16
3
11.8
26
1
2
0
–
7.3
9.8
Mem-0 ( Chen et al., 2026 )
4
89
90
67
14
52.8
28
18
68
0
–
28.5
42.0
Table 1: RMBench per-task success rates (%), averaged over the nine tasks with published baselines. Most baselines train one specialist model per task while SimpleMemVLA is a single multi-task model evaluated at n=100 seeds per task in the streaming deployment of Section 4.6 . † No prior VLA reports Place-Mat so it is excluded from all averages. Best per column in bold .
Memory
Method
Integr. / VLM
Counting
Permanence
Reference
Imitation
AVG
Human
Human Performance
-
88.5
91
93
89.5
90.5
Symbolic (Oracle)
SimpleSG ( Dai et al., 2026 )
GT VLM
82.6
21.6
32.3
61.9
49.6
Symbolic (Oracle)
GroundSG ( Dai et al., 2026 )
GT VLM
83.9
93.3
95.2
64.0
84.1
Symbolic
SimpleSG
Gemini
39.0
13.5
22.5
18.0
23.3
Symbolic
SimpleSG
QwenVL
44.6
19.6
25.2
26.6
29.0
Symbolic
GroundSG
Gemini
12.8
15.8
11.3
6.5
11.6
Table 2: RoboMME category-level success rates (%). AVG is over all sixteen tasks. Gray rows are reference-only and excluded from ranking. Bold and underline mark ranks 1 and 2. The SimpleMemVLA variant rows re-create one mechanism family each on the otherwise unchanged SimpleMemVLA stack. Per-task results are in Table 10 .
Method
ShellGame
Intercept
RC-3
RC-5
RC-9
Avg
Vision-language-action models
CronusVLA ( Li et al., 2026 )
32
5
31
13
9
18.0
SpatialVLA ( Qu et al., 2025 )
23
27
27
17
11
21.0
OpenVLA-OFT ( Kim et al., 2025 )
47
14
59
16
6
28.4
π0 ( Black et al., 2025 )
33
42
35
22
15
29.4
Octo ( Team et al., 2024 )
46
39
45
17
11
31.6
Table 3: MIKASA-Robo five-task success rates (%). GMP is a per-task non-VLA reference. Best VLA per column in bold .
Transferring
Occlusion
Counting
Sequence
Average
Method
TSR
CSR
TSR
CSR
TSR
CSR
TSR
CSR
TSR
CSR
π0.5 ( Intelligence et al., 2025 )
20.0
42.8
12.7
17.2
14.3
50.9
60.0
71.6
21.5
38.7
HiF-VLA ( Lin et al., 2025b )
17.5
38.9
12.7
27.1
8.6
45.9
42.5
70.2
16.9
39.8
MemoryVLA ( Shi et al., 2026a )
15.0
37.2
7.3
13.1
14.3
55.1
37.5
65.2
15.0
35.3
MemER ( Sridhar et al., 2026 )
20.0
36.1
16.4
33.2
27.1
65.1
65.0
79.1
27.3
49.1
FrameSamp+Modul ( Dai et al., 2026 )
63.8
72.1
39.1
56.5
31.4
57.8
73.8
86.9
46.2
63.9
Table 4: RoboMemArena category-level TSR and CSR (%) over all 26 tasks under the official protocol. MemER is the benchmark authors’ reimplementation, FrameSamp+Modul the only external leaderboard entry, the gray oracle row excluded from ranking. Best per column in bold .
Method
Spatial
Object
Goal
Long
Avg
General-purpose policies and VLAs
Diffusion Policy ( Chi et al., 2023 )
78.3
92.5
68.3
50.5
72.4
TraceVLA ( Zheng et al., 2025 )
84.6
85.2
75.1
54.1
74.8
Octo ( Team et al., 2024 )
78.9
85.7
84.6
51.1
75.1
OpenVLA ( Kim et al., 2024 )
84.7
88.4
79.2
53.7
76.5
SpatialVLA ( Qu et al., 2025 )
88.2
89.9
78.6
55.5
78.1
Table 5: LIBERO success rates (%) on the four standard suites, 500 trials per suite. Best per column in bold .
Method
Camera
Robot
Lang.
Light
Backg.
Noise
Layout
Total
General-purpose policies and VLAs
OpenVLA ( Kim et al., 2024 )
0.8
3.5
23.0
8.1
34.8
15.2
28.5
15.6
WorldVLA ( Cen et al., 2025 )
0.1
27.9
41.6
43.7
17.1
10.9
38.0
25.0
Diffusion Policy ( Chi et al., 2023 )
1.6
32.3
77.0
25.0
19.8
20.3
42.2
31.7
NORA ( Hung et al., 2025 )
2.2
37.0
65.1
45.7
58.6
12.8
62.1
39.0
UniVLA ( Bu et al., 2025 )
1.8
46.2
69.6
69.0
81.0
21.2
31.9
42.9
Table 6: LIBERO-Plus zero-shot robustness transfer to the 10,030 perturbed tasks, all policies trained on standard LIBERO only. Best per column in bold .
Figure 10
Figure 5: Task-level effects of restricted memory interfaces on RoboMME. The three controlled variants differ from SimpleMemVLA only in how history enters the model. (a) Success rates across 16 tasks grouped by benchmark dimension. (b) The same results normalized to native-context performance. Retrieval, token compression and recurrent state retain at most 79%, 96% and 58%, respectively, with the token-compression peak confined to the count task SwingXtimes. Results are from Table 10 , with n=50 episodes per task.
Figure 6: History interventions redirect the output across all suites. Blocks denote benchmarks and rows show histories from oldest to most recent, processed through the unchanged deployment pipeline. Orange borders mark evidence frames, gray fills evidence ablations and red borders donor-episode insertions. Removing evidence changes the output, donor evidence redirects it toward the donor content and matched control edits leave it unchanged.
Figure 7: Streaming inference reduces decision latency. Measured on one H100 (bf16, batch size 1). (a) With 60 s histories, streaming reduces decision latency from 1.02 s to 0.68 s, close to the same-model single-frame baseline of approximately 0.65 s. (b) On artificially constructed 45 min inputs (245k tokens), decision-path latency falls from 32.1 s to 1.18 s.
Figure 8: Native-context memory relies on retained temporal evidence and contextual hidden states. (a) Cover-blocks fails once the covering event leaves the window, whereas press-button degrades gradually as completed presses are removed. (b) Both tasks depend on frame order, while timestamps matter primarily for counting. (c) Stale sub-task hidden states sharply reduce success, whereas equally stale position IDs leave it unchanged. (d) Behavior follows the source of the hidden states rather than the token embeddings, identifying contextual hidden states as the memory-to-action interface.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
RMBench
RoboMME
MIKASA-Robo
RoboMemArena
LIBERO
Embodiment
Aloha-AgileX, bimanual
Panda, single
Panda, single
Franka, single
Franka, single
Action dimension da
14
8
8
7
7
History camera Chist
head
front
overhead
front
agentview
Current cameras Ccur
both wrists
wrist
wrist
wrist
wrist
Window Tw (s)
60
60
3
126
30
Video rate fv (fps)
2
2
20
1
2
Appendix
Table 8: Per-benchmark instantiation of the SimpleMemVLA configuration tuple and the resulting training cost. Everything else about the method is identical across suites. Training time is wall-clock hours on 128 H100 GPUs.
Cover Blocks
RGB
GRB
GBR
RBG
BRG
BGR
Overall
Successes/trials
8/10
6/10
7/10
6/10
5/10
3/10
35/60
Appendix
Table 9: Real-world results by initial configuration. Each entry reports successes over ten autonomous trials. Cover Blocks labels list colors from far to near. Put Back Block labels follow the task coordinate convention.
Figure 9: Additional real-world autonomous rollouts. (a) Cover Blocks under three initial color arrangements, labeled from far to near. (b) Put Back Block from four initial positions, labeled using the task coordinate convention rather than camera-image directions. Each row shows chronological frames from one successful rollout.
Figure 10: First on all sixteen RoboMME tasks among the 21 deployable methods. Gray dots denote the 20 other non-Human, non-Oracle methods evaluated by the benchmark. Boxes show the interquartile range of the 21-method ranked pool, with median and full range. Orange diamonds mark SimpleMemVLA and labels report its competition rank out of 21. Human performance and the two symbolic Oracle variants, which replace VLM outputs with ground-truth outputs, are shown only as references and are excluded from all ranks and distribution statistics.
Figure 11: Task-wise RoboMME success rates (%, higher is better) for representative methods. Panels group the 16 tasks into Counting, Permanence, Reference and Imitation. Overall is recomputed from all 16 task scores. Human and GroundSG (Oracle, using ground-truth VLM outputs) are shown only as references and are excluded from the ranked comparison. For each remaining baseline family, the row with the highest overall AVG is shown. Values above orange hatched bars report SimpleMemVLA scores.
Figure 12: Complete task-wise RoboMME success rates (%, higher is better) for all 24 rows. Tasks follow the source-table order and methods retain the same order in every panel. Colors encode memory families. The orange hatched bar denotes SimpleMemVLA.
Memory
Method
Integr. / VLM
Counting
Permanence
Reference
Imitation
AVG
BinFill
PickX
SwingX
StopC
V-Umsk
B-Umsk
V-UmskS
B-UmskS
PickHL
V-Repk
V-PlcB
V-PlcO
MoveC
InsPeg
PatLock
RouteS
Human
Human Performance
-
96
100
80
78
90
92
92
90
92
92
98
90
90
98
84
86
90.5
Symbolic (Oracle)
SimpleSG ( Dai et al., 2026 )
GT VLM
85.78
99.78
100
44.67
33.11
22
15.56
15.56
44
27.78
31.33
26
87.33
10
95.33
55.11
49.58
Symbolic (Oracle)
GroundSG ( Dai et al., 2026 )
GT VLM
85.78
100
100
49.67
98.78
95
99.22
80.22
83.33
97.33
100
100
87.78
15.56
97
55.56
84.08
Symbolic
SimpleSG
Gemini
46
63
45
2
29
9
14
2
20
15
26
29
61
4
7
0
23.25
Symbolic
SimpleSG
QwenVL
77.56
95.33
5.11
0.44
34.22
19.33
15.33
9.56
17.11
25.33
33.33
25.11
82
3.78
12.67
7.78
29
Appendix
Table 10: Complete task-level RoboMME success rates (%). Human and GT-VLM Oracle rows (gray) are reference-only and excluded from ranking. Bold and underline mark ranks 1 and 2 among the remaining 21 methods. Task abbreviations follow Table 2 ’s four dimensions in source order. The three SimpleMemVLA variant rows are the controlled within-stack rebuilds of Section 4.4 and are excluded from ranking.
By source suite
Spatial
Object
Goal
Long
Tasks
2,402
2,518
2,591
2,519
Success (%)
81.8
83.9
75.0
73.1
By difficulty level
Level 1
Level 2
Level 3
Level 4
Level 5
Tasks
1,644
2,202
2,094
1,886
2,083
Appendix
Table 11: LIBERO-Plus breakdowns. Top: by source LIBERO suite. Bottom: by the benchmark’s five difficulty levels, which stratify tasks by the accuracy of four reference models ( Fei et al., 2025 ) . The 121 tasks that carry no difficulty label in the benchmark metadata are omitted from the bottom panel.
Corruption type
Overall
Low
Mid
High
Motion blur
87.8
100
95
76
Glass blur
85.7
93
89
79
Gaussian blur
69.2
95
78
47
Zoom blur
62.5
96
77
44
Fog
60.7
93
80
39
Appendix
Table 12: The Sensor Noise column of Table 6 , split by corruption type and severity tier. Types are encoded in the benchmark’s task naming. Success rates in %.
Figure 13: The SWA variant: one context construction, trained and deployed. (a) Two consecutive streaming decisions: each appends one unit (orange) to the persistent stream, the softmax window slides by one unit, and the unit leaving it (red) has its cache rows freed; the linear-attention state (green) integrates from t=0 and is never reset. (b) Training forwards each episode once as a trunk and supervises B branches at random anchors, each attending to its last ≤M units and resuming the linear state snapshotted at its fork — exactly the deployed construction. Details in the accompanying text.
Figure 14: Figure 7 , extended with the SWA streaming variant. Measurements use one H100, bf16 and batch size 1, with the setup of Figure 7 . (a) The SWA decision takes 0.92 s, within the 0.96 s execution interval. (b) The SWA variant uses a fixed L=5,888 -token window to bound its resident cache.
Single-memory M(1)
Multi-memory M(n)
Method
Obs&PU
Rearr.
PutBack
SwapB
SwapT
Avg
Battery
Rank.
Cover
Press
Place-Mat †
Avg
Overall
SimpleMemVLA (Table 1 )
65
100
100
100
93
91.6
90
100
98
100
100
97.0
94.0
+ SWA streaming (evicted cache)
56
100
100
100
93
89.8
71
100
99
100
99
92.5
91.0
Appendix
Table 13: Closed-loop success of the SWA streaming deployment , on the RMBench protocol of Table 1 (100 held-out seeds per task).
Mainstream Vision-Language-Action (VLA) models predict actions primarily from the current observation under a Markovian assumption, thus struggling with long-horizon, temporally dependent tasks. Existing memory-augmented VLAs either expand the observation window or retrieve history from the memory bank as auxiliary policy-side context. However, they leave memory outside the native latent embedding space of VLA reasoning, preventing historical experience from being fluidly interleaved with multimodal reasoning and action formation. To this end, we introduce LaMem-VLA, a latent-memory-native framework that reconstructs historical experience into latent memory tokens and directly interweaves them with VLA reasoning. At its core, LaMem-VLA introduces four coordinated components: (i) a curator that organizes historical experience into two complementary short-term and long-term memory vaults; (ii) a seeker that queries both vaults using the multimodal cognition to retrieve context-relevant evidence; (iii) a condenser that reconstructs the retrieved evidence into compact short-term and long-term latent memory tokens; and (iv) a weaver that injects these memory tokens with the current observation and instruction into one continuous embedding sequence. By representing, retrieving, and consuming historical experience entirely in the same continuous latent space, LaMem-VLA enables memory to directly participate in VLA reasoning and guide action generation under a bounded context. Extensive experiments on SimplerEnv and LIBERO demonstrate the superiority of our LaMem-VLA.
Hongyu Qu, Jianzhe Gao, Xiaobin Hu +6
Nanjing University of Science and Technology · Zhejiang University · National University of Singapore
Vision-language-action (VLA) models for closed-loop robot control are typically cast under the Markov assumption, making them prone to errors on tasks requiring historical context. To incorporate memory, existing VLAs either retrieve from a memory bank, which can be misled by distractors, or extend the frame window, whose fixed horizon still limits long-term retention. In this paper, we introduce ReMem-VLA, a Recurrent Memory VLA model equipped with two sets of learnable queries: frame-level recurrent memory queries for propagating information across consecutive frames to support short-term memory, and chunk-level recurrent memory queries for carrying context across temporal chunks for long-term memory. These queries are trained end-to-end to aggregate and maintain relevant context over time, implicitly guiding the model's decisions without additional training or inference cost. Furthermore, to enhance visual memory, we introduce Past Observation Prediction as an auxiliary training objective. Through extensive memory-centric simulation and real-world robot experiments, we demonstrate that ReMem-VLA exhibits strong memory capabilities across multiple dimensions, including spatial, sequential, episodic, temporal, and visual memory. ReMem-VLA significantly outperforms memory-free VLA baselines π0.5 and OpenVLA-OFT and surpasses MemoryVLA on memory-dependent tasks by a large margin.
Hang Li, Fengyi Shen, Dong Chen +6
Technical University of Munich, Munich, Germany · Huawei Heisenberg Research Center, Munich, Germany · University of Freiburg, Freiburg, Germany +2
How can pretrained Vision-Language-Action (VLA) models retain long-horizon visual histories with high-frequency updates without sacrificing efficiency? Existing approaches rely on external memory management, which restrains either the memory horizon or the reactiveness of pretrained policies. To this end, we present NativeMEM, a VLA policy that features long-term and real-time updated memory. At its core is an efficient memory encoding scheme, Native Memory Compression, which repurposes the VLA's own vision encoder to compress each historical frame from each camera view into a single token. Appended to the input sequence, these memory tokens enable the pretrained VLA to attend over long-term history with negligible latency overhead, requiring neither an external planner nor a freshly initialized memory module. To align the memory tokens with the pretrained policy, we first develop a generic memory tokenizer under the supervision of a frozen VLA on memory-demanding data, and then unfreeze the VLA for task-specific fine-tuning. NativeMEM consistently outperforms prior methods, boosting success rates from 32.4% to 84.0% in simulation and up to 98.7% on real robots, while maintaining low inference latency and GPU memory usage. Notably, NativeMEM exhibits high data efficiency by achieving competitive results with prior arts using only 20% of the training data.
Ziye Wang, Modi Shi, Chaojun Ni +5
The University of Hong Kong · Beihang University · Peking University +2