ActMem: From Latent Memory Reasoning to Grounded Robotic Action
Organizations: Beihang University · Beta Infinity
Abstract
Vision-Language-Action (VLA) models have shown promising results in real-world long-horizon manipulation, where memory plays a crucial role. However, existing memory-augmented VLAs either retrieve past observations without reasoning about what they imply, or reason over the full history in a shared context where it crowds out current perception. Cognitive science treats them as distinct systems: memory reconstructs past experience to decide what to do, while perception guides how to act. We therefore propose ActMem, an in-context memory framework that decouples memory reasoning from perceptual grounding within a single vision-language backbone. Built on a vision-language model with native video understanding, ActMem infers the current subtask from a timestamped interaction history and keeps the hidden states formed during this inference as a Latent Memory Chain-of-Thought (Memory CoT), which conditions a flow-matching action expert. Perception Attention further lets the expert attend directly to the current view. To keep inference fast under long histories, Efficient Memory CoT Inference reuses computation across consecutive decisions without changing the output. ActMem achieves 71.1% task success and 80.8% completion success on RoboMemArena, surpassing the strongest baseline by 7.5 and 8.7 percentage points, and 73.8% average success on MIKASA-Robo, the best among all compared methods. In the real world, it reaches 68.8% average success on exogenous, endogenous, and traceless memory tasks, 10.0 percentage points above the strongest memory-augmented VLA. Code and checkpoints will be released at https://github.com/LigphiDonk/ActMem and https://huggingface.co/donk111/ActMem.
Figures & tables
| Method | Transferring | Occlusion | Counting | Sequence | Average | |||||
|---|---|---|---|---|---|---|---|---|---|---|
| TSR | CSR | TSR | CSR | TSR | CSR | TSR | CSR | TSR | CSR | |
| Reference | ||||||||||
| Qwen3-VL-8B (frozen) [ 19 ] | 15.0 | 34.6 | 0.0 | 6.8 | 9.3 | 44.6 | 7.5 | 39.2 | 6.0 | 26.2 |
| GPT-5.4 [ 19 ] | 13.8 | 32.9 | 1.8 | 9.2 | 12.9 | 50.7 | 15.0 | 47.3 | 8.7 | 30.5 |
| Reactive / short-horizon VLAs | ||||||||||
| [ 13 ] | 20.0 | 42.8 | 12.7 | 17.2 | 14.3 | 50.9 | 60.0 | 71.6 | 21.5 | 38.7 |
| Method | ShellGame | Intercept | RC-3 | RC-5 | RC-9 | Avg |
| Reference | ||||||
| GMP [ 11 ] | 98 | 83 | 80 | 61 | 17 | 67.8 |
| Vision-language-action models | ||||||
| CronusVLA [ 21 ] | 32 | 5 | 31 | 13 | 9 | 18.0 |
| SpatialVLA [ 25 ] | 23 | 27 | 27 | 17 | 11 | 21.0 |
| OpenVLA-OFT [ 17 ] | 47 | 14 | 59 | 16 | 6 | 28.4 |
| Configuration | TSR (%) | CSR (%) |
|---|---|---|
| w/o History | 17.7 | 34.6 |
| 10 s History | 31.2 | 47.7 |
| Text-only Subtask | 40.4 | 56.9 |
| Full-token Conditioning | 38.8 | 54.6 |
| w/o Perception Attention | 47.7 | 64.2 |
| ActMem (Full) | 52.3 | 68.5 |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Hyperparameter | Value |
|---|---|
| Vision-language backbone | Qwen3-VL-4B |
| Action expert | Flow-matching DiT, 16 layers |
| Perception Attention layers | 3, 7, 11, 15 |
| Condition for the action expert | Subtask-span hidden states + token embeddings |
| History (RoboMemArena) | 1 fps, 126 s |
| History (MIKASA-Robo) | 20 fps, 3 s |
| History (frames) | 0–30 | 30–60 | 60–90 | 90–126 |
|---|---|---|---|---|
| Decisions | 51 | 60 | 60 | 74 |
| SimpleMemVLA | 0.70 / 1.05 | 0.85 / 1.29 | 1.02 / 1.67 | 1.25 / 1.96 |
| ActMem, standard | 0.87 / 1.45 | 0.98 / 1.68 | 1.09 / 1.94 | 1.42 / 2.27 |
| ActMem, Efficient Memory CoT | 0.33 / 0.52 | 0.37 / 0.56 | 0.52 / 0.75 | 0.50 / 0.74 |
| Model | Latency (ms) |
|---|---|
| [ 13 ] | 78.6 / 88.8 |
| MEM- (short memory) [ 32 ] | 93.6 / 123.1 |
| MEM- (long memory) [ 32 ] | 78.3 / 122.2 |
| MemoryVLA [ 27 ] (reimplemented on ) | 94.8 / 103.3 |
Memory-Augmented VLMs
Memory-augmented vision-language models use persistent or retrieved context to improve reasoning across long inputs and interactions.
See all 178 papers