Modality Gap in VLMs

VLM: Vision-Language Model

The modality gap describes differences between how vision-language models represent or use images and text, even when the two convey the same information. Researchers study it through embedding geometry and paired-input evaluations, including image-text retrieval, visual reasoning, and tests that swap text for equivalent images. Approaches include contrastive and optimal-transport alignment, feature projection, modality-aware training, and diagnostics that measure whether outputs respond to relevant visual evidence. A central tension is that reducing representational separation can help some tasks but hurt others, while many models still favor language over images; reliable measures must connect the gap to task-specific performance.

Latest papers

59 papers12 papers in the last four weeks, up 100% on the four weeks before. 0.1% of all new papers.

Closing the Modality Gap via Simplex-Constrained Representations
Oct 23, 2026Shubham Gupta, Siva Reddy, Perouz Taslakian +2Modality Gap in VLMs Cross-Modal Alignment
Mila – Québec AI Institute · ServiceNow Research · Université Laval +1
Refining Open-Vocabulary Semantic Segmentation via Regional Semantics and Visual Prototypes
Oct 22, 2026Aijing Yu, Zhengbo Wang, Lijun Sheng +2Open-Vocabulary Semantic Segmentation Vision-Language Alignment
School of Cyber Security, University of Chinese Academy of Sciences · Institute of Information Engineering, Chinese Academy of Sciences · NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences +2
Persistent Representational Misalignment Induces Vision-Language Model Failures
Oct 18, 2026Yonatan Gideoni, Yoav Gelberg, Tim G. J. Rudner +1Vision-Language Alignment Modality Gap in VLMs
University of Oxford · University of Toronto
Vision-Language Models Integrate Conflicting Evidence Like Biased Bayesian Observers
Oct 14, 2026Josh Barua, Kyle Mahowald, William RudmanVision-Language Models Modality Gap in VLMs
The University of Texas at Austin
Omni-Modal LLMs Know More Than They Say
Oct 12, 2026Na Min An, Taekyung Kim, Jaemyung Yu +2Multimodal Large Language Models Cross-Modal Alignment
KAIST AI · NAVER AI Lab
Confidence-Gated Cloud-Edge Cascade Triage via Variational Risk Minimization for Medical Imaging
Oct 1, 2026Xinye Yang, Zhusi Zhong, Scott Collins +7Knowledge Distillation Chest X-Ray Classification
Department of Computer Science, Brown University, Providence, RI, USA · Brown University and Brown University Health, Providence, RI, USA · School of Computing and Information Sciences, Florida International University, Miami, FL, USA
Visual Branch is What You Need for CLIP-based Class-Incremental Learning
Sep 29, 2026Tao Hu, Zhen-Hao Xie, Jingcai Guo +2Class-Incremental Learning Modality Gap in VLMs
School of Artificial Intelligence, Nanjing University · State Key Laboratory for Novel Software Technology, Nanjing University · Hong Kong Polytechnic University
Seeing Is Not Addressing: Auditing Linguistic Access to Frozen Visual Geometry
Sep 29, 2026Woosang Jeon, Jiwon Yang, Soo Chung +1Vision-Language Models Vision-Language Grounding
Seoul National University
Dual-Mode Low-Rank Learner with Bridge-Prototype Ensemble for Vision-Language Class-Incremental Learning
Sep 29, 2026Chiyuan He, Zihuan Qiu, Fanman Meng +5Prototype-Based Learning Low-Rank Adaptation
University of Electronic Science and Technology of China, Chengdu, China · Qiyuan Lab, Beijing, China
How Medical VLMs Underutilize Their Vision Encoders: A Dermatology Perspective
Sep 29, 2026Janet Wang, Yunbei Zhang, Xiao Wang +1Medical Diagnosis VLM Interpretability
Tulane University · Oak Ridge National Laboratory
Render Before Reading: Visual Rendering as a Prompt Injection Defense
Sep 28, 2026Jie Zhang, Andrei Baroian, Jan N. van Rijn +2Multimodal Large Language Models Modality Gap in VLMs
ETH Zurich · Leiden University
One Geometry, Different Outcomes: Readout-Dependent Effects of the Modality Gap in Vision-Language Models
Sep 28, 2026Aditya Sharma, Divya SaxenaCross-Modal Retrieval Modality Gap in VLMs
Indian Institute of Technology Jodhpur
Efficient Quantization-Aware Distillation with Cross-Modal Alignment for Edge Vision-Language Models
Sep 15, 2026Jinwoo Jeon, GyuYeop Do, Yubin Lim +4VLM Quantization Efficient VLM Inference
Korea University · Seoul National University
TwinICL: Diagnosing Multimodal In-Context Learning through Paired Counterfactuals
Sep 14, 2026Zihan Xue, Po-Yi Lu, Serhii Honcharenko +5Multimodal ICL Modality Gap in VLMs
University of California, Los Angeles · 2National Taiwan University · 3Texas A&M University +2
UOT-Gap: A Variational Principle for the Modality Gap in Vision-Language Models via Unbalanced Optimal Transport
Sep 9, 2026Zonglin Yang, Huilan Ma, Xudan Zheng +1Modality Gap in VLMs Vision-Language Retrieval
Guangdong Police College
NOVA: Normal-Side Modeling for Training-Free Zero-Shot Video Anomaly Detection
Sep 6, 2026Wei-Chih Yin, Yun-Ching Kao, Cheng-Kuan Lin +1Training-Free Video Anomaly Detection Video Anomaly Detection
Department of Computer Science, National Yang Ming Chiao Tung University, Hsinchu, Taiwan
The Visual Insensitivity Gap: Diagnosing When Vision-Language Models Fail to Use Visual Evidence
Sep 1, 2026Genpei ZhangVLM Evaluation Large Vision-Language Models
University of Wisconsin–Madison
Slow to See, Slow to Suppress: Understanding the Effects of Modality in Context-Memory Conflicts
Aug 31, 2026Athulith Paraselli, Etha Tianze Hua, Ellie PavlickVision-Language Models Knowledge Conflicts in Language Models
Brown University
EgoArgus: Benchmarking VLMs as Situational Assistants for Modality-Grounded User Supports
Aug 26, 2026Yu-Chien Tang, Yu-Hsiang Liu, An-Zi YenVLM Evaluation Agent Benchmarks
Department of Computer Science, National Yang Ming Chiao Tung University, Taiwan
Where did the ambiguity go? Examining how multimodal models interpret polysemous words
Aug 1, 2026Jasin Cekinmez, Addison J. Wu, Raja Marjieh +1Multimodal Generation Modality Gap in VLMs
Princeton University
MIRROR: Aligning Semantic Relations from Language to Image via Gromov--Wasserstein
Jun 28, 2026Hong-Han Wang, Yuntao Wang, Hu DingLarge Vision-Language Models VLM Reasoning
University of Science and Technology of China, Hefei, China
MLLMs Get It Right, Then Get It Wrong: Tracing and Correcting Late-Layer Textual Bias
Jun 16, 2026Xingming Li, Ao Cheng, Qiyao Sun +4VLM Robustness Vision-Language Models
National University of Defense Technology, changsha, China · 2Chinese University of Hong Kong, Shenzhen, China · 3Intelligent Game and Decision Lab, Beijing, China
Beyond Symmetric Alignment: Spectral Diagnostics of Modality Imbalance in Vision-Language Models in the Medical Domain
Jun 3, 2026Alessandro Gambetti, Qiwei Han, Cláudia Soares +1VLM Evaluation Modality Imbalance
NOVA School of Science and Technology · Lisbon, Portugal · Nova School of Business and Economics +2
Density-Aware Translation of Spurious Correlations in Zero-Shot VLMs
Jun 1, 2026Afsaneh Hasanebrahimi, Hanxun Huang, Christopher Leckie +1VLM Robustness Vision-Language Models
School of Computing and Information Systems, The University of Melbourne, Victoria, Australia.
LoMo: Local Modality Substitution for Deeper Vision-Language Fusion
May 28, 2026Feng Han, Zhixiong Zhang, Zheming Liang +2Cross-Modal Learning VLM Robustness
Fudan University · Shanghai Innovation Institute · Shanghai Jiao Tong University +1
Respecting Modality Gap in Post-hoc Out-of-distribution Detection with Pre-trained Vision-Language Models
May 26, 2026Yuanwei Hu, Bo Peng, Yadan Luo +3Vision-Language Models Test-Time Adaptation
University of Technology Sydney · The University of Queensland
TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs
May 20, 2026Andong Hua, Colton Bishop, Igor Mordatch +5Multimodal Large Language Models Modality Gap in VLMs
University of California, Santa Barbara · Google DeepMind
Medical Context Distorts Decisions in Clinical Vision Language Models
May 17, 2026David Restrepo, Ira Ktena, Maria Vakalopoulou +2VLM Evaluation VLM Robustness
1MICS, CentraleSup´elec - Universit´e Paris-Saclay, France · 2Cancer Data Science Unit, IHU PRISM, National Institute in Precision Oncology, University Paris-Saclay, CentraleSupelec, Gustave Roussy, INSERM, Gif-sur-Yvette · 3Ellison Institute of Technology (EIT), Oxford, UK +1
When Language Overwrites Vision: Over-Alignment and Geometric Debiasing in Vision-Language Models
May 7, 2026Harshvardhan Saini, Samyak Jha, Yiming Tang +1VLM Hallucination Mitigation Vision-Language Models
Indian Institute of Technology Dhanbad · National University of Singapore
Jailbreaking Vision-Language Models Through the Visual Modality
May 1, 2026Aharon Azulay, Jan Dubiński, Zhuoyun Li +2Adversarial Attacks on VLMs Vision-Language Models
Independent · NASK National Research Institute, Warsaw, Poland · Warsaw University of Technology, Warsaw, Poland +3

Oct 23, 2026J/K move · Enter open · S save

Closing the Modality Gap via Simplex-Constrained Representations

Shubham Gupta, Siva Reddy, Perouz Taslakian, Valentina Zantedeschi+1

Mila – Québec AI Institute · ServiceNow Research · Université Laval+1

Modality Gap in VLMs Cross-Modal Alignment Representation Learning Multimodal Contrastive Learning Cross-Modal Retrieval

Open paper AskPDF arXiv

While multimodal contrastive models like CLIP and CLAP successfully learn shared representations, their embeddings typically exhibit a persistent modality gap. We propose addressing this separation through a purely geometric approach: changing the embedding domain from the unit sphere,

S^{d-1}

, to the probability simplex,

Δ^{d-1}

. Although both manifolds share the same intrinsic dimension, the simplex acts as a structural prior; its nonnegative, unit-mass constraints serve as a cross-modal calibration that forces both modalities to ground themselves in the same coordinate system. By attaching lightweight softmax adapters and using Total Variation similarity, we reduce the centroid

\ell_2

gap by 97--99% across five multimodal retrieval benchmarks. This confirms that simplicial geometry induces genuine semantic alignment, matching or exceeding the retrieval performance of standard unconstrained models.