VLM Adaptation

VLM: Vision-Language Model

Follow this topic Ask about this topic

Momentum

34 papers in the last four weeks, up 209% on the four weeks before. 0.3% of all new papers.

Jul 13Week of Sep 28

Latest papers 395

All topics

Unlocking Patch-Level Features for CLIP-Based Class-Incremental Learning
May 13, 2026Hao Sun, Zi-Jun Ding, Da-Wei ZhouClass-Incremental Learning VLM Adaptation
School of Artificial Intelligence, Nanjing University · State Key Laboratory for Novel Software Technology, Nanjing University
Test-Time Hinting for Black-Box Vision-Language Models
May 13, 2026Kaihua Hou, Abhijith Varma Mudunuri, Jiaxing Qiu +3Visual Question Answering VLM Adaptation
UC Berkeley · UCSF · University of Virginia +1
A $_3$ B $_2$ : Adaptive Asymmetric Adapter for Alleviating Branch Bias in Vision-Language Image Classification with Few-Shot Learning
May 13, 2026Yiyun Zhou, Zhonghua Jiang, Wenkang Han +4Vision-Language Models VLM Adaptation
1Zhejiang University · 2Swansea University
Anomaly-Aware Vision-Language Adapters for Zero-Shot Anomaly Detection
May 12, 2026Muhammad Aqeel, Maham Nazir, Uzair Khan +2VLM Adaptation Medical Image Anomaly Detection
Dept. of Engineering for Innovation Medicine, University of Verona, Italy · School of Computer Science and Engineering, Beihang University, China · Dept. of Engineering for Innovation +1
Cluster-Aware Neural Collapse Prompt Tuning for Long-Tailed Generalization of Vision-Language Models
May 12, 2026Boyang Guo, Liang Li, Lin Peng +3Long-Tail Learning VLM Adaptation
Hangzhou Dianzi University, Hangzhou, China · Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China · The Hong Kong Polytechnic University, Hong Kong, China +2
Reviving In-domain Fine-tuning Methods for Source-Free Cross-domain Few-shot Learning
May 12, 2026Yaze Zhao, Yicong Liu, Yixiong Zou +2Source-Free Domain Adaptation Few-Shot Domain Adaptation
CapVector: Learning Transferable Capability Vectors in Parametric Space for Vision-Language-Action Models
May 11, 2026Wenxuan Song, Han Zhao, Fuhao Li +7Task Vectors VLM Adaptation
1HKUST (GZ) · 2Zhejiang University · 3Westlake University +2
Learning More from Less: Exploiting Counterfactuals for Data-Efficient Chart Understanding
May 11, 2026Jianzhu Bao, Haozhen Zhang, Kuicai Dong +5Data Visualization VLM Adaptation
1Nanyang Technological University · 2Aumovio Singapore Pte. Ltd.
UniSteer: Unified Noise Steering for Efficient Human-Guided VLA Adaptation
May 11, 2026Junjie Lu, Xinyao Qin, Yuhua Jiang +6RL for Robotics Robot Policy Adaptation
University of Technology Sydney · Work conducted at Microsoft Research. · Tsinghua University +1
Thermal-Det: Language-Guided Cross-Modal Distillation for Open-Vocabulary Thermal Object Detection
May 11, 2026Yasiru Ranasinghe, Elim Schenck, Florence Yellin +3Open-Vocabulary Object Detection Cross-Modal Knowledge Distillation
Johns Hopkins University · Kitware · DEVCOM Army Research Laboratory
Hystar: Hypernetwork-driven Style-adaptive Retrieval via Dynamic SVD Modulation
May 11, 2026Yujia Cai, Boxuan Li, Chenghao Xu +1Multimodal IR VLM Adaptation
School of Computer Science and Technology, Xidian University, Xi’an, Shaanxi, China · School of Electronic Engineering, Xidian University, Xi’an, Shaanxi, China
Fashion Florence: Fine-Tuning Florence-2 for Structured Fashion Attribute Extraction
May 11, 2026Anushree BerliaVLM Adaptation Multimodal Foundation Models
CrossVL: Complexity-Aware Feature Routing and Paired Curriculum for Cross-View Vision-Language Detection
May 10, 2026Zhipeng Liu, Chunbo LuoMulti-View Consistency VLM Adaptation
Department of Computer Science, University of Exeter
When Adaptation Fails: A Gradient-Based Diagnosis of Collapsed Gating in Vision-Language Prompt Learning
May 10, 2026Yunxuan Fang, Ziwei Zhang, Xinhe WangPrompt Learning VLM Adaptation
Beihang University · Beijing, China
DAPE: Dynamic Non-uniform Alignment and Progressive Detail Enhancement Techniques for Improving the Performance of Efficient Visual Language Models
May 9, 2026Mengyuan Tian, Qiyan Zhao, Yanan Wang +1VLM Adaptation Efficient VLM Inference
The Wang Yanan Institute for Studies in Economics, Xiamen University, Xiamen 361005, China · Fujian Key Laboratory of Pattern Recognition and Image Understanding, School of Computer and Information Engineering, Xiamen University of Technology, Xiamen 361024, China
FragileFlow: Spectral Control of Correct-but-Fragile Predictions for Foundation Model Robustness
May 9, 2026Zhuoyun Li, Boxuan Wang, Jinwei Hu +2VLM Robustness Spectral Regularization
School of Computer Science and Informatics, University of Liverpool, UK
Semantic Alignment in Hyperbolic Space for Open-Vocabulary Semantic Segmentation
May 9, 2026Hoang M. Truong, Hai Nguyen-Truong, Dang HuynhVLM Adaptation Hierarchical Representation Learning
Fulbright University Vietnam
Gate-and-Merge: Zero-shot Compositional Personalization of Vision Language Models
May 9, 2026Guodong Ding, Angela YaoVision-Language Models VLM Adaptation
National University of Singapore
IPAD-CLIP: Teaching CLIP to Detect Image Local Perceptual Artifacts
May 9, 2026Juan Wang, Xinyu Sun, Ke Zhang +4VLM Adaptation
Institute of Automation Chinese Academy of Sciences Beijing, China · School of Computer Science Beijing Jiaotong University Beijing, China · Department of Electronic Engineering Tsinghua University Beijing, China +2
Decoupling Endpoint and Semantic Transition Learning for Zero-Shot Composed Image Retrieval
May 8, 2026Mingyu Liu, Sihan Huang, Yijia Fan +4Cross-Modal Learning VLM Adaptation
1Sun Yat-sen University · 2Guangdong Province Key Laboratory of Information Security Technology, China · 3Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China
jina-embeddings-v5-omni: Geometry-preserving Embeddings via Locked Aligned Towers
May 8, 2026Florian Hönicke, Michael Günther, Andreas Koukounas +4Cross-Modal Learning Cross-Modal Representation Learning
Jina by Elastic
One Adapter, Every Resolution: Gated Low-Rank Adaptation for Remote Sensing VLMs
May 8, 2026Song Zhang, Yanlong Chen, Yining Chen +2Remote Sensing Image Understanding Remote Sensing VQA
Nanjing University · ETH Zurich · Nanjing University of Posts and Telecommunications
Hierarchical Dual-Subspace Decoupling for Continual Learning in Vision-Language Models
May 8, 2026Mengxin Qin, Xiang Zhang, Kun Wei +2Class-Incremental Learning Vision-Language Models
School of Electronic Engineering, Xidian University Xi’an 710071, China
DIMoE-Adapters: Dynamic Expert Evolution for Continual Learning in Vision-Language Models
May 8, 2026Mengxin Qin, Xiang Zhang, Xi Wang +3Vision-Language Models Fine-Tuning
School of Electronic Engineering, Xidian University Xi’an 710071, China
Escaping the Diversity Trap in Robotic Manipulation via Anchor-Centric Adaptation
May 8, 2026Yanzhe Chen, Kevin Yuchen Ma, Qi Lv +4Robot Policy Adaptation VLM Adaptation
National University of Singapore · Institude for Information Research, A*STAR · Harbin Institute of Technology (Shenzhen).
AT-VLA: Adaptive Tactile Injection for Enhanced Feedback Reaction in Vision-Language-Action Models
May 8, 2026Xiaoqi Li, Muhe Cai, Jiadong Xu +5Contact-Rich Robotic Manipulation VLM Adaptation
School of Computer Science, Peking University · PrimeBot · PKU Lab
StrLoRA: Towards Streaming Continual Visual Instruction Tuning for MLLMs
May 8, 2026Chang Che, Ziqi Wang, Hui Ma +2VLM Adaptation Low-Rank Adaptation
Hefei University of Technology · Tsinghua University
Fine-tuning a vision-language model for fracture-surface morphology recognition
May 8, 2026Quanliang Liu, Jungtaek Kim, Kangwook Lee +1VLM Adaptation Materials Science
Department of Materials Science & Engineering, University of Wisconsin–Madison, Madison, WI, USA, 53706 · Department of Electrical & Computer Engineering, University of Wisconsin–Madison, Madison, WI, USA, 53706 · KRAFTON, Gangnam-gu, Seoul, Republic of Korea, 06142 +1
GeoStack: A Framework for Quasi-Abelian Knowledge Composition in VLMs
May 7, 2026Pranav Mantini, Shishir K. ShahVision-Language Models VLM Adaptation
Department of Computer Science University of Houston · The University of Oklahoma School of Computer Science
Plug-and-play Class-aware Knowledge Injection for Prompt Learning with Visual-Language Model
May 7, 2026Junhui Yin, Nan Pu, Xinyu Zhang +4Prompt Learning VLM Adaptation
University of Science and Technology Beijing, China. · Hefei University of Technology, China. · University of Auckland, New Zealand. +3

May 13, 2026, cs.CVJ/K move · Enter open · S save

Unlocking Patch-Level Features for CLIP-Based Class-Incremental Learning

Hao Sun, Zi-Jun Ding, Da-Wei Zhou

School of Artificial Intelligence, Nanjing University · State Key Laboratory for Novel Software Technology, Nanjing University

Class-Incremental Learning VLM Adaptation Incremental Learning Vision-Language Alignment

Open paper AskPDF arXiv

Class-Incremental Learning (CIL) enables models to continuously integrate new knowledge while mitigating catastrophic forgetting. Driven by the remarkable generalization of CLIP, leveraging pre-trained vision-language models has become a dominant paradigm in CIL. However, current work primarily focuses on aligning global image embeddings (i.e., [CLS] token) with their corresponding text prompts (i.e., [EOS] token). Despite their good performance, we find that they discard the rich patch-level semantic information inherent in CLIP's encoders. For instance, when recognizing a rabbit, local patches may encode its distinctive cues, such as long ears and a fluffy tail, which can provide complementary evidence for recognition. Based on the above observation, we propose SPA (Semantic-guided Patch-level Alignment) for CLIP-based CIL, which aims to awaken long-neglected local representations within CLIP. Specifically, for each class, we first construct representative and diverse visual samples and feed them to GPT-5 as visual guidance to generate class-wise semantic descriptions. These descriptions are used to guide the selection of discriminative patch-level visual features. Building upon these selected patches, we further employ optimal transport to align selected patch tokens with semantic tokens from class-wise descriptions, yielding a structured cross-modal alignment that improves recognition. Furthermore, we introduce task-specific projectors for effective adaptation to downstream incremental tasks, and sample pseudo-features from stored class-wise Gaussian statistics to calibrate old-class representations, thereby mitigating catastrophic forgetting. Extensive experiments demonstrate that SPA achieves state-of-the-art performance.

Momentum

Related topics

Latest papers 395