Continual Learning for VLMs

VLM: Vision-Language Model

Follow this topic Ask about this topic

Momentum

7 papers in the last four weeks, against 1 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 51

All topics

Frozen Models, Evolving Expertise: Model-Agnostic Learning from Deployment Experience for Multimodal Medical AI
Oct 6, 2026Yexiao He, Yucheng Tang, Pengfei Guo +6Medical LLMs Continual Learning for VLMs
University of Maryland, College Park · NVIDIA
ReCAP: Retrieval-Guided Capability Reuse for Multimodal Continual Instruction Tuning
Sep 29, 2026Tao Hu, Zhinuo Zhou, Xialiang Tong +2Continual Learning Continual Learning for VLMs
School of Artificial Intelligence, Nanjing University · State Key Laboratory for Novel Software Technology, Nanjing University · Noah’s Ark Lab, Huawei Technologies
Visual Branch is What You Need for CLIP-based Class-Incremental Learning
Sep 29, 2026Tao Hu, Zhen-Hao Xie, Jingcai Guo +2Class-Incremental Learning Modality Gap in VLMs
School of Artificial Intelligence, Nanjing University · State Key Laboratory for Novel Software Technology, Nanjing University · Hong Kong Polytechnic University
Dual-Mode Low-Rank Learner with Bridge-Prototype Ensemble for Vision-Language Class-Incremental Learning
Sep 29, 2026Chiyuan He, Zihuan Qiu, Fanman Meng +5Prototype-Based Learning Low-Rank Adaptation
University of Electronic Science and Technology of China, Chengdu, China · Qiyuan Lab, Beijing, China
Brain-Inspired Hierarchical Modularity for General Continual Learning
Sep 21, 2026Hongwei Yan, Kanglei Zhou, Qi Cheng +7Continual Learning for LLMs Continual Learning for VLA Models
School of Life Sciences, Tsinghua University, Beijing, China. · IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing, China. · Tsinghua-Peking Center for Life Sciences, Beijing, China. +2
ReDraft, Don't Just Distill: Reference-Driven Revision for Continual VLLM Post-Training
Sep 15, 2026Zhihao Zhang, Mingqi Wu, Qiaole Dong +15Fine-Tuning Multimodal Large Language Models
Fudan University · Shanghai Artificial Intelligence Laboratory · Tencent
AnchorGUI: Asymmetric Memory for Dual-Scale Learning in GUI Navigation
Sep 14, 2026Shengjie Jin, Zelong Sun, Hengbo Xu +2Memory-Augmented VLMs Continual Learning for VLMs
Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China
Drive by Hindsight and Foresight: Tool-Grounded Synergistic Reasoning over Hierarchical Memory for Autonomous Driving
Sep 8, 2026Baojie Chen, Zijun Jia, Jing ZhongMemory-Augmented VLMs Tool Use in VLMs
Beihang University · Tsinghua University
Towards Continual Test-Time Adaptation of Vision-Language Models in Open-Vocabulary Semantic Segmentation
Aug 30, 2026Chandler Timm C. Doloriel, Yunbei Zhang, Sarthak Kumar Maharana +5Test-Time Adaptation of VLMs Open-Vocabulary Semantic Segmentation
Faculty of Science and Technology (REALTEK), Norwegian University of Life Sciences (NMBU) · Tulane University · University of Texas at Dallas
DARAD: Dual Adapters and Ranking-Aware Distillation for Continual Remote Sensing Image-Text Retrieval
Aug 6, 2026Xi Chen, Xu Chen, Xiangyang Jia +3VLM Adaptation Image-Text Retrieval
School of Computer Science, Wuhan University, Wuhan 430072, China · Beijing Institute for General Artificial Intelligence (BIGAI)
Taming the Implicit: Dual-Channel Risk-Aware Reinforcement Fine-Tuning for Continual Multimodal Post-Training
Aug 4, 2026Yibei Liu, Jiajun Chen, Qianle Zhang +4Multimodal Large Language Models Risk-Sensitive RL
University of Electronic Science and Technology of China · Zhejiang University · South China University of Technology +1
SAFE-Merge: Data-Free Continual Model Merging with General Knowledge Preservation
Aug 2, 2026Zihuan Qiu, Zhiyang Liao, Chiyuan He +5Language Model Merging Model Merging
University of Electronic Science and Technology of China, Chengdu, China · Dalian University of Technology, Dalian, China
Progressive Multimodal Alignment for Continual Instruction Tuning
Jul 29, 2026Duzhen Zhang, Yahan Yu, Qiaoyi Su +2Cross-Modal Alignment Multimodal Large Language Models
Mohamed bin Zayed University of Artificial Intelligence · Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences · Kyoto University +2
Continual Video-MLLM Adaptation over Evolving Domains
Jul 21, 2026Rui Cheng, Meixing Shi, Yuxiang Cai +3Video QA Domain-Incremental Learning
School of Software Technology, Zhejiang University, Ningbo, China · Zhejiang Key Laboratory of Digital-Intelligence Service Technology, China · Hong Kong Polytechnic University, Hong Kong SAR +1
AlphaWiSE: Adaptive Weight Interpolation for Continual Multimodal Representation Learning
Jul 16, 2026Sarthak Jain, Qiran Hu, Zhen Zhu +1Cross-Modal Representation Learning Cross-Modal Alignment
University of Illinois Urbana-Champaign · Google DeepMind
An Empirical Analysis of Continual Learning for Heterogeneous Medical Visual Question Answering
Jul 13, 2026Mai A. Shaaban, Tausifa Jan Saleem, Alaa Mohamed +3Medical VQA Continual Learning for VLMs
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE · Department of Mathematics and Computer Science, Faculty of Science, Alexandria University, Alexandria, Egypt
RL Forgets! Towards Continual Policy Optimization
Jul 5, 2026Mao-Lin Luo, Zhe-Xu Wang, Zi-Hao Zhou +4RL Post-Training Continual Learning for VLMs
School of Computer Science and Engineering, Southeast University, Nanjing 210096, China · Key Laboratory of Computer Network and Information Integration (Southeast University), Ministry of Education, China · Zhongguancun Academy +1
CL-Anomaly: Layer-Adaptive Mixture-of-Experts with Multimodal Large Language Model for Continual Learning in Anomaly Detection
Jul 3, 2026Wen Dong, Zhao Wang, Shuangqing Zhang +5Mixture of Experts Multimodal Large Language Models
Nanjing University, Nanjing, China · China Mobile Zijin Innovation Institute, Nanjing, China · Nanjing University of Science and Technology, Nanjing, China
Hidden Forgetting in Continual Multimodal Learning: When Accuracy Survives but Grounding Fails
Jul 2, 2026Qianyu Chen, Canran Xiao, Runxuan TangMultimodal Grounding Multimodal Foundation Models
1Nanyang Technological University · 2Shenzhen Campus of Sun Yat-sen University
InduceKV: Fixed-Footprint Continual Adaptation of Multimodal LLMs via Inducing KV Memories
Jul 2, 2026Qianyu Chen, Ziteng Feng, Canran Xiao +1Continual Learning for LLMs Memory-Efficient Fine-Tuning
1Nanyang Technological University · University of Science and Technology of China · 3Shenzhen Campus of Sun Yat-sen University
M2Note: Continual Evolution of Vision Language Models via Mistake Notebook Learning
Jul 1, 2026Haiwen Li, Jing Tang, Rui Chen +2Memory-Augmented VLMs VLM Reasoning
AMAP, Alibaba Group
Wake up for Touch! Mask-isolated Tactile Alignment Learning in MLLMs
Jul 1, 2026Yoonhyung Park, Minji Kim, Sungwon Moon +1Visuo-Tactile Representation Learning Multimodal Large Language Models
Division of Artificial Intelligence & Software, Ewha Womans University
Rosetta: Composable Native Multimodal Pretraining
Jul 1, 2026Xiangyue Liu, Zijian Zhang, Miles Yang +3Unified Multimodal Models Multimodal Pretraining
HKUST · Tencent Hunyuan
Few-Shot Domain Incremental Learning via Continual Vision-Language Consolidation
Jun 29, 2026Naeem Paeedeh, Mahardhika Pratama, Wolfgang Mayer +3VLM Adaptation Domain-Incremental Learning
School of Computer Science and IT, Adelaide University, Australia. · Faculty of Engineering and IT, University of Technology Sydney (UTS), Australia · School of Artificial Intelligence and Computer Science, Nantong University, China +1
ComMem: Complementary Memory Systems for Test-Time Adaptation of Vision-Language Models
Jun 27, 2026Guanglong Sun, Shuang Cui, Bo Lei +6Memory-Augmented VLMs Vision-Language Models
School of Life Sciences, IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing, China · Institute of Software Chinese Academy of Sciences, Beijing, China · 3Beijing Academy of Artificial Intelligence, Beijing, China +2
ReasonCLIP-58M: Visually Grounded Commonsense Reasoning Supervision for CLIP
Jun 25, 2026Sicheng Zhang, Muzammal Naseer, Binzhu Xie +5Compositional Reasoning Multimodal Reasoning
Khalifa University · University of Western Australia · The Chinese University of Hong Kong +2
CADRE: Stable, Parameter Efficient Adaptation of Medical Vision Language Models with Bounded Forgetting and Prior Drift
Jun 22, 2026Amrita Singh, Rishabh JhaVLM Adaptation Elastic Weight Consolidation
Mindriser’s Consortium, Kathmandu, Nepal · University of Victoria, BC, Canada
Attention-Spectrum Regularization for Replay-Free Continual Multimodal LLMs
Jun 22, 2026Chuangxin Zhao, Canran Xiao, Siyuan Ma +5Visual Question Answering Cross-Modal Attention
The Hong Kong University of Science and Technology (Guangzhou) · Sun Yat-sen University · Nanyang Technological University +2
Black-Box Continual Learning for Vision-Language Models
Jun 22, 2026Yuting Li, Weihang Fang, Haoyuan Gao +4Incremental Learning Continual Learning for VLMs
School of Computer Science, Shanghai Jiao Tong University, Shanghai, China · School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai, China · State Key Laboratory of General Artificial Intelligence, Beijing Institute for General Artificial Intelligence (BIGAI), Beijing, China +2
Is Our Benchmark Enough? An Analysis of Continual Learning for MLLMs
Jun 18, 2026Van-Tuan Tran, Shruthi Gowda, Merim Dzaferagic +1Continual Learning LLM Routing
School of Computer Science and Statistics, Trinity College Dublin, Ireland · Department of Mathematics and Computer Science, Eindhoven University of Technology, The Netherlands

Oct 6, 2026, cs.AIJ/K move · Enter open · S save

Frozen Models, Evolving Expertise: Model-Agnostic Learning from Deployment Experience for Multimodal Medical AI

Yexiao He, Yucheng Tang, Pengfei Guo, Yufan He+5

University of Maryland, College Park · NVIDIA

Medical LLMs Continual Learning for VLMs Online Continual Learning Memory-Augmented VLMs Medical VLMs

Open paper AskPDF arXiv

Large language models (LLMs) and vision-language models (VLMs) are usually frozen after deployment, so they do not learn from the cases they solve. This is especially concerning in medicine, where new clinical evidence, updated guidelines, and new therapies can change established practice. Fine-tuning can update the model, but it requires access to model weights and additional training. Parameter-free methods avoid training, but they may overfit a fixed validation set, lack reliable domain knowledge, or lose visual details by saving experience only as text. To address these limitations, we present a model-agnostic framework that allows frozen LLMs and VLMs to learn from deployment experience through three forms of external expertise: a Skill that guides reasoning and tool use, a Knowledge Memory that stores reliable facts supported by earlier cases or trusted external evidence, and a Multimodal Knowledge Base that keeps visual examples and guides the model to relate each retrieved case to the current image. Instead of relying on a fixed validation set, a validation strategy keeps an update only if it helps on new cases without degrading performance on earlier ones. Across six benchmarks covering clinical diagnosis, clinical workflows, medical reasoning, and medical and non-medical visual reasoning, and with four open-weight and closed-source base models, our framework improves performance during online deployment by up to 34.2% over the base model on medical tasks, generalizes to unseen cases, transfers to other models without further optimization, and works in non-medical domains.

Momentum

Related topics

Latest papers 51