Memory-Augmented VLMs

VLM: Vision-Language Model

Memory-augmented vision-language models study how systems can retain and reuse visual, textual, and action-related information beyond the current image or context window. Approaches include retrieval from episodic or structured memory, recurrent and latent memory tokens, compressed visual histories, and external world models for tasks such as long-video question answering, navigation, and robot manipulation. Work is evaluated on long-context video and spatial-reasoning benchmarks as well as memory-dependent robotic tasks, with designs ranging from training-free retrieval to end-to-end learned memory. A central challenge is preserving fine-grained, trustworthy evidence while controlling memory and inference costs; compressed or stale memories can omit useful details or mislead later decisions.

Latest papers

160 papers49 papers in the last four weeks, up 96% on the four weeks before. 0.3% of all new papers.

Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation
Jul 8, 2026Hongyu Qu, Jianzhe Gao, Xiaobin Hu +6Memory-Augmented VLMs Robotic Manipulation
Nanjing University of Science and Technology · Zhejiang University · National University of Singapore
Do You Remember? Toward Memory-Centric Multimodal AI
Jul 7, 2026Xuguang Yu, Weigang Zheng, Minyue YuMemory-Augmented VLMs Multimodal Memory
NativeMEM: Native Memory Compression for Long-Horizon Robotic Manipulation
Jul 7, 2026Ziye Wang, Modi Shi, Chaojun Ni +5Memory-Augmented VLMs Long-Horizon Robotic Manipulation
The University of Hong Kong · Beihang University · Peking University +2
Latent Visual Cache for Video Reasoning
Jul 1, 2026Yongheng Zhang, Zhipeng Xu, Hao Wu +4Memory-Augmented VLMs Vision-Language Models
1Tencent Youtu Lab · 2Tsinghua University · University of Illinois at Chicago
M2Note: Continual Evolution of Vision Language Models via Mistake Notebook Learning
Jul 1, 2026Haiwen Li, Jing Tang, Rui Chen +2Memory-Augmented VLMs VLM Reasoning
AMAP, Alibaba Group
ComMem: Complementary Memory Systems for Test-Time Adaptation of Vision-Language Models
Jun 27, 2026Guanglong Sun, Shuang Cui, Bo Lei +6Memory-Augmented VLMs Vision-Language Models
School of Life Sciences, IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing, China · Institute of Software Chinese Academy of Sciences, Beijing, China · 3Beijing Academy of Artificial Intelligence, Beijing, China +2
$μ$ VLA: On Recurrent Memory for Partially Observable Manipulation in VLA Models
Jun 10, 2026Egor Cherepanov, Nikita Kachaev, Daniil Zelezetsky +6Memory-Augmented VLMs Recurrent Transformers
CogAI Lab, Moscow, Russia · MIRAI, Moscow, Russia
Key-Gram: Extensible World Knowledge for Embodied Manipulation
May 18, 2026Jingjing Fan, Siyuan Li, Botao Ren +1Memory-Augmented VLMs Language-Conditioned Robot Manipulation
Department of Computer Science and Technology Tsinghua University · Department of Automation Tsinghua University
AtlasVA: Self-Evolving Visual Skill Memory for Teacher-Free VLM Agents
May 18, 2026Pan Wang, Yihao Hu, Xiujin Liu +3Memory-Augmented VLMs Reward Shaping
Ant Group · University of Science and Technology of China · Westlake University +2
MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Models
May 14, 2026Xiyu Ren, Zhaowei Wang, Yiming Du +11VLM Evaluation Memory-Augmented VLMs
CSE Deparment, HKUST · CUHK · OmniMemory (Shenzhen) Intelligent Technology Co., Ltd. +1
ECHO: Continuous Hierarchical Memory for Vision-Language-Action Models
May 9, 2026Yanbin Hu, Jin Cui, Jiayi Lu +6Memory-Augmented VLMs Long-Horizon Robotic Manipulation
School of Software, Xi’an Jiaotong University · State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University · School of Artificial Intelligence, Xi’an Jiaotong University
Persistent Visual Memory: Sustaining Perception for Deep Generation in LVLMs
May 1, 2026Siyuan Huang, Xiaoye Qu, Yafu Li +6Memory-Augmented VLMs VLM Robustness
Shanghai AI Laboratory · Shanghai Jiao Tong University · The Chinese University of Hong Kong +3
POINTS-Seeker: An Open Recipe for Multimodal Search Agents with Visual Memory Management
Apr 15, 2026Yikun Liu, Yuan Liu, Haicheng Wang +6Multimodal Search Agents Multimodal Agents
School of Artificial Intelligence, Shanghai Jiao Tong University · CMIC, Shanghai Jiao Tong University · WeChat AI, Tencent
ReflectCAP: Detailed Image Captioning with Reflective Memory
Apr 14, 2026Kyungmin Min, Minbeom Kim, Kang-il Lee +2Image Captioning VLM Hallucination Mitigation
IPAI, Seoul National University · Dept. of ECE, Seoul National University · Adobe Research
POINTS-Long: Adaptive Dual-Mode Visual Reasoning in MLLMs
Apr 13, 2026Haicheng Wang, Yuan Liu, Yikun Liu +9Multimodal Large Language Models Long-Context VLMs
SAI, Shanghai Jiao Tong University, China · WeChat AI, Tencent, China
Belief-Aware VLM Model for Human-like Reasoning
Apr 5, 2026Anshul Nayak, Shahil Shaik, Yue WangVLM Reasoning Memory-Augmented VLMs
Mechanical Engineering Department, Clemson University.
Stop Wandering: Efficient Vision-Language Navigation via Metacognitive Reasoning
Apr 2, 2026Xueying Li, Feng Lyu, Hao Wu +3Vision-Language Navigation Memory-Augmented VLMs
Central South University, Changsha, China · Nanjing University, Nanjing, China · State Grid Hubei Electric Power Research Institute, Wuhan, China +1
Synergizing Discriminative Exemplars and Self-Refined Experience for MLLM-based In-Context Learning in Medical Diagnosis
Mar 29, 2026Wenkai Zhao, Zipei Wang, Mengjie Fang +3Multimodal ICL Medical VLMs
CAS Key Laboratory of Molecular Imaging, Institute of Automation, Chinese Academy of Sciences · School of Software, North University of China
Memory-Augmented Vision-Language Agents for Persistent and Semantically Consistent Object Captioning
Mar 25, 2026Tommaso Galliena, Stefano Rosa, Tommaso Apicella +3Memory-Augmented VLMs VLMs for Robotics
Italian Institute of Technology, Genoa, Italy · University of Genoa, Genoa, Italy
RoboHarness: A Memory-Augmented Policy Harness for Vision-Language-Action Model Robustness via In-Context Adaptation
Mar 25, 2026Zhuoran Li, Zhiyang Li, Kaijun Zhou +1Robustness of VLA Models Test-Time Adaptation of VLA Models
School of Computer Science, Shanghai Jiao Tong University, Shanghai, China. · Shanghai Syslong Information Technology Co., Ltd.
See, Remember, Explore: A Benchmark and Baselines for Streaming Spatial Reasoning
Mar 25, 2026Yuxi Wei, Wei Huang, Qirui Chen +2Spatial Reasoning Benchmarks Visual Spatial Reasoning
The University of Hong Kong · Shanghai Jiao Tong University · Yinwang Intelligent Technology Co. Ltd.
ME-IQA: Memory-Enhanced Image Quality Assessment via Re-Ranking
Mar 21, 2026Kanglong Fan, Tianhe Wu, Wen Wen +6Memory-Augmented VLMs Vision-Language Models
City University of Hong Kong, Hong Kong SAR, China · ByteDance Inc., China
SARE: Sample-wise Adaptive Reasoning for Training-free Fine-grained Visual Recognition
Mar 18, 2026Jingxiao Yang, DaLin He, Miao Pan +7Fine-Grained Visual Perception Adaptive Inference
Zhejiang University · Netease Yidun AI Lab
LightZeroNav: Zero-Shot Vision Language Navigation in Continuous Environments Based on Lightweight VLMs
Mar 16, 2026Kun Luo, Xiangyu Dong, Xiaoguang Ma +2Vision-Language Navigation Vision-Language Models
Foshan Graduate School of Innovation, Northeastern University, Foshan 528311, China · Faculty of Robot Science and Engineering, Northeastern University, Shenyang 110819, China · QingniaoAI, China +1
HiMemVLN: Enhancing Reliability of Open-Source Zero-Shot Vision-and-Language Navigation with Hierarchical Memory System
Mar 16, 2026Kailin Lyu, Kangyi Wu, Pengna Li +9Vision-Language Navigation Memory-Augmented VLMs
Institute of Automation, Chinese Academy of Sciences, and the Zhongguancun Academy, Beijing, China · Zhongguancun Academy, Beijing, China · Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University, Xi’an, China +7
ReMem-VLA: Empowering Vision-Language-Action Model with Memory via Dual-Level Recurrent Queries
Mar 13, 2026Hang Li, Fengyi Shen, Dong Chen +6Memory-Augmented VLMs Vision-Language-Action Models
Technical University of Munich, Munich, Germany · Huawei Heisenberg Research Center, Munich, Germany · University of Freiburg, Freiburg, Germany +2
AnchorVLA4D: an Anchor-Based Spatial-Temporal Vision-Language-Action Model for Robotic Manipulation
Mar 13, 2026Juan Zhu, Zhanying Shao, Xiaoqi Li +4Vision-Language-Action Models Language-Conditioned Robot Manipulation
PrimeBot · School of Computer Science, Peking University
OnFly: Onboard Zero-Shot Aerial Vision-Language Navigation toward Safety and Efficiency
Mar 11, 2026Guiyong Zheng, Yueting Ban, Mingjie Zhang +2Vision-Language Navigation Aerial VLN
School of Artificial Intelligence, Sun Yat-Sen University, Zhuhai, China. · Southern University of Science and Technology, Shenzhen, China · The Hong Kong University of Science and Technology, Guangzhou, China.
Hybrid Self-evolving Structured Memory for GUI Agents
Mar 11, 2026Sibo Zhu, Wenyi Wu, Kun Zhou +2GUI Agents Agent Memory Management
University of California, San Diego
PathMem: Toward Cognition-Aligned Memory Transformation for Pathology MLLMs
Mar 10, 2026Jinyue Li, Yuci Liang, Qiankun Li +7Memory-Augmented VLMs Medical VLMs
University of Science and Technology of China · Shenzhen University · Nanyang Technological University +2

Jul 8, 2026J/K move · Enter open · S save

Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulation

Hongyu Qu, Jianzhe Gao, Xiaobin Hu, Shaohuan Yang+5

Nanjing University of Science and Technology · Zhejiang University · National University of Singapore

Memory-Augmented VLMs Robotic Manipulation Vision-Language-Action Models Memory-Augmented Agents Reasoning for VLA Models Temporal Modeling for VLA Models

Open paper AskPDF arXiv

Mainstream Vision-Language-Action (VLA) models predict actions primarily from the current observation under a Markovian assumption, thus struggling with long-horizon, temporally dependent tasks. Existing memory-augmented VLAs either expand the observation window or retrieve history from the memory bank as auxiliary policy-side context. However, they leave memory outside the native latent embedding space of VLA reasoning, preventing historical experience from being fluidly interleaved with multimodal reasoning and action formation. To this end, we introduce LaMem-VLA, a latent-memory-native framework that reconstructs historical experience into latent memory tokens and directly interweaves them with VLA reasoning. At its core, LaMem-VLA introduces four coordinated components: (i) a curator that organizes historical experience into two complementary short-term and long-term memory vaults; (ii) a seeker that queries both vaults using the multimodal cognition to retrieve context-relevant evidence; (iii) a condenser that reconstructs the retrieved evidence into compact short-term and long-term latent memory tokens; and (iv) a weaver that injects these memory tokens with the current observation and instruction into one continuous embedding sequence. By representing, retrieving, and consuming historical experience entirely in the same continuous latent space, LaMem-VLA enables memory to directly participate in VLA reasoning and guide action generation under a bounded context. Extensive experiments on SimplerEnv and LIBERO demonstrate the superiority of our LaMem-VLA.