Modality Gap in VLMs

VLM: Vision-Language Model

Follow this topic Ask about this topic

Momentum

9 papers in the last four weeks, against 2 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 24

All topics

Visual Branch is What You Need for CLIP-based Class-Incremental Learning
Sep 29, 2026Tao Hu, Zhen-Hao Xie, Jingcai Guo +2Class-Incremental Learning Modality Gap in VLMs
School of Artificial Intelligence, Nanjing University · State Key Laboratory for Novel Software Technology, Nanjing University · Hong Kong Polytechnic University
Seeing Is Not Addressing: Auditing Linguistic Access to Frozen Visual Geometry
Sep 29, 2026Woosang Jeon, Jiwon Yang, Soo Chung +1Vision-Language Models Vision-Language Grounding
Seoul National University
Dual-Mode Low-Rank Learner with Bridge-Prototype Ensemble for Vision-Language Class-Incremental Learning
Sep 29, 2026Chiyuan He, Zihuan Qiu, Fanman Meng +5Prototype-Based Learning Low-Rank Adaptation
University of Electronic Science and Technology of China, Chengdu, China · Qiyuan Lab, Beijing, China
How Medical VLMs Underutilize Their Vision Encoders: A Dermatology Perspective
Sep 29, 2026Janet Wang, Yunbei Zhang, Xiao Wang +1Medical Diagnosis VLM Interpretability
Tulane University · Oak Ridge National Laboratory
Render Before Reading: Visual Rendering as a Prompt Injection Defense
Sep 28, 2026Jie Zhang, Andrei Baroian, Jan N. van Rijn +2Multimodal Large Language Models Modality Gap in VLMs
ETH Zurich · Leiden University
One Geometry, Different Outcomes: Readout-Dependent Effects of the Modality Gap in Vision-Language Models
Sep 28, 2026Aditya Sharma, Divya SaxenaCross-Modal Retrieval Modality Gap in VLMs
Indian Institute of Technology Jodhpur
Efficient Quantization-Aware Distillation with Cross-Modal Alignment for Edge Vision-Language Models
Sep 15, 2026Jinwoo Jeon, GyuYeop Do, Yubin Lim +4VLM Quantization Efficient VLM Inference
Korea University · Seoul National University
TwinICL: Diagnosing Multimodal In-Context Learning through Paired Counterfactuals
Sep 14, 2026Zihan Xue, Po-Yi Lu, Serhii Honcharenko +5Multimodal ICL Modality Gap in VLMs
University of California, Los Angeles · 2National Taiwan University · 3Texas A&M University +2
UOT-Gap: A Variational Principle for the Modality Gap in Vision-Language Models via Unbalanced Optimal Transport
Sep 9, 2026Zonglin Yang, Huilan Ma, Xudan Zheng +1Modality Gap in VLMs Vision-Language Retrieval
Guangdong Police College
The Visual Insensitivity Gap: Diagnosing When Vision-Language Models Fail to Use Visual Evidence
Sep 1, 2026Genpei ZhangVLM Evaluation Large Vision-Language Models
University of Wisconsin–Madison
Slow to See, Slow to Suppress: Understanding the Effects of Modality in Context-Memory Conflicts
Aug 31, 2026Athulith Paraselli, Etha Tianze Hua, Ellie PavlickVision-Language Models Knowledge Conflicts in Language Models
Brown University
Where did the ambiguity go? Examining how multimodal models interpret polysemous words
Aug 1, 2026Jasin Cekinmez, Addison J. Wu, Raja Marjieh +1Multimodal Generation Modality Gap in VLMs
Princeton University
MIRROR: Aligning Semantic Relations from Language to Image via Gromov--Wasserstein
Jun 28, 2026Hong-Han Wang, Yuntao Wang, Hu DingLarge Vision-Language Models VLM Reasoning
University of Science and Technology of China, Hefei, China
MLLMs Get It Right, Then Get It Wrong: Tracing and Correcting Late-Layer Textual Bias
Jun 16, 2026Xingming Li, Ao Cheng, Qiyao Sun +4VLM Robustness Vision-Language Models
National University of Defense Technology, changsha, China · 2Chinese University of Hong Kong, Shenzhen, China · 3Intelligent Game and Decision Lab, Beijing, China
Beyond Symmetric Alignment: Spectral Diagnostics of Modality Imbalance in Vision-Language Models in the Medical Domain
Jun 3, 2026Alessandro Gambetti, Qiwei Han, Cláudia Soares +1VLM Evaluation Modality Imbalance
NOVA School of Science and Technology · Lisbon, Portugal · Nova School of Business and Economics +2
Density-Aware Translation of Spurious Correlations in Zero-Shot VLMs
Jun 1, 2026Afsaneh Hasanebrahimi, Hanxun Huang, Christopher Leckie +1VLM Robustness Vision-Language Models
School of Computing and Information Systems, The University of Melbourne, Victoria, Australia.
LoMo: Local Modality Substitution for Deeper Vision-Language Fusion
May 28, 2026Feng Han, Zhixiong Zhang, Zheming Liang +2Cross-Modal Learning VLM Robustness
Fudan University · Shanghai Innovation Institute · Shanghai Jiao Tong University +1
Respecting Modality Gap in Post-hoc Out-of-distribution Detection with Pre-trained Vision-Language Models
May 26, 2026Yuanwei Hu, Bo Peng, Yadan Luo +3Vision-Language Models Test-Time Adaptation
University of Technology Sydney · The University of Queensland
Medical Context Distorts Decisions in Clinical Vision Language Models
May 17, 2026David Restrepo, Ira Ktena, Maria Vakalopoulou +2VLM Evaluation VLM Robustness
1MICS, CentraleSup´elec - Universit´e Paris-Saclay, France · 2Cancer Data Science Unit, IHU PRISM, National Institute in Precision Oncology, University Paris-Saclay, CentraleSupelec, Gustave Roussy, INSERM, Gif-sur-Yvette · 3Ellison Institute of Technology (EIT), Oxford, UK +1
When Language Overwrites Vision: Over-Alignment and Geometric Debiasing in Vision-Language Models
May 7, 2026Harshvardhan Saini, Samyak Jha, Yiming Tang +1Vision-Language Models Object Hallucination in VLMs
Indian Institute of Technology Dhanbad · National University of Singapore
Jailbreaking Vision-Language Models Through the Visual Modality
May 1, 2026Aharon Azulay, Jan Dubiński, Zhuoyun Li +2Adversarial Attacks on VLMs Vision-Language Models
Independent · NASK National Research Institute, Warsaw, Poland · Warsaw University of Technology, Warsaw, Poland +3
The Expense of Seeing: Attaining Trustworthy Multimodal Reasoning Within the Monolithic Paradigm
Apr 22, 2026Karan GoyalVLM Evaluation Multimodal Reasoning
IIIT Delhi, India
Do Vision-Language Models Truly Perform Vision Reasoning? A Rigorous Study of the Modality Gap
Apr 17, 2026Yige Xu, Yongjie Wang, Zizhuo Wu +3Vision-Language Models VLM Adaptation
College of Computing and Data Science, Nanyang Technological University, Singapore · Alibaba-NTU Global e-Sustainability CorpLab (ANGEL) · Tongyi Lab, Alibaba Group, China
BabyVision: Visual Reasoning Beyond Language
Jan 10, 2026Liang Chen, Weichu Xie, Yiyan Liang +27VLM Evaluation Visual Reasoning
1UniPat AI · 6Peking University · 3Alibaba Group +8

Sep 29, 2026, cs.CVJ/K move · Enter open · S save

Visual Branch is What You Need for CLIP-based Class-Incremental Learning

Tao Hu, Zhen-Hao Xie, Jingcai Guo, De-Chuan Zhan+1

School of Artificial Intelligence, Nanjing University · State Key Laboratory for Novel Software Technology, Nanjing University · Hong Kong Polytechnic University

Class-Incremental Learning Modality Gap in VLMs Continual Learning for VLMs

Open paper AskPDF arXiv

Class-Incremental Learning (CIL) requires models to recognize new classes over time without forgetting previously learned ones. With the rise of vision-language pre-training, CLIP has become a strong foundation for CIL. A common design in CLIP-based CIL is to construct textual classifier weights by encoding class-name templates with the CLIP text encoder, and then classify visual features by image-text cosine similarity. This design is appealing: since CLIP aligns images and text in a shared embedding space, textual weights appear to provide an off-the-shelf classifier for incremental classes. However, we show that this seemingly natural design is not always beneficial, as a modality gap can still separate the two modalities and make textual classifier weights deviate from visual class distributions. Empirically, under identical task-wise CIL training, initializing the cosine classifier with visual class centers yields lower loss and better incremental accuracy than using CLIP textual features. Motivated by these observations, we propose VIS, a visual-only method for CLIP-based CIL that removes the deployed textual branch and constructs the incremental classifier entirely in the visual space. To obtain stronger task-adaptive visual representations, VIS uses only base-session data to enhance CLIP's final visual representation with informative visual-layer features. Built on the enhanced visual representation, VIS employs a simple kernelized incremental least-squares SVM, whose classifier weights are solved in closed form from additive sufficient statistics. When new classes arrive, VIS accumulates their sufficient statistics and recomputes the classifier weights for all seen classes, enabling efficient incremental updates while preserving historical class knowledge. Extensive experiments show that VIS achieves state-of-the-art performance without a textual branch.

Momentum

Related topics

Latest papers 24