Modality Gap in VLMs

VLM: Vision-Language Model

The modality gap describes differences between how vision-language models represent or use images and text, even when the two convey the same information. Researchers study it through embedding geometry and paired-input evaluations, including image-text retrieval, visual reasoning, and tests that swap text for equivalent images. Approaches include contrastive and optimal-transport alignment, feature projection, modality-aware training, and diagnostics that measure whether outputs respond to relevant visual evidence. A central tension is that reducing representational separation can help some tasks but hurt others, while many models still favor language over images; reliable measures must connect the gap to task-specific performance.

Latest papers

59 papers12 papers in the last four weeks, up 100% on the four weeks before. 0.1% of all new papers.

The Expense of Seeing: Attaining Trustworthy Multimodal Reasoning Within the Monolithic Paradigm
Apr 22, 2026Karan GoyalVLM Evaluation Multimodal Reasoning
IIIT Delhi, India
Do Vision-Language Models Truly Perform Vision Reasoning? A Rigorous Study of the Modality Gap
Apr 17, 2026Yige Xu, Yongjie Wang, Zizhuo Wu +3Vision-Language Models VLM Adaptation
College of Computing and Data Science, Nanyang Technological University, Singapore · Alibaba-NTU Global e-Sustainability CorpLab (ANGEL) · Tongyi Lab, Alibaba Group, China
The Cost of Language: Centroid Erasure Exposes and Exploits Modal Competition in Multimodal Language Models
Apr 15, 2026Akshay Paruchuri, Ishan Chatterjee, Henry Fuchs +2Modality Reliance in Multimodal Models Multimodal Large Language Models
Università della Svizzera italiana · Stanford University · University of Washington +1
Penny Wise, Pixel Foolish: Bypassing Price Constraints in Multimodal Agents via Visual Adversarial Perturbations
Apr 15, 2026Jiachen Qian, Zhaolu KangVLM Robustness Adversarial Attacks on VLMs
City University of Hong Kong · Peking University
Why Multimodal In-Context Learning Lags Behind? Unveiling the Inner Mechanisms and Bottlenecks
Apr 15, 2026Yu Wang, Sharon LiMultimodal ICL In-Context Learning
Department of Computer Sciences, University of Wisconsin-Madison
Visual Enhanced Depth Scaling for Multimodal Latent Reasoning
Apr 12, 2026Yudong Han, Yong Wang, Zaiquan Yang +3Latent Visual Reasoning Continuous Latent Reasoning
Beijing Institute of Technology · AMAP, Alibaba Group · City University of Hong Kong +2
DISSECT: Diagnosing Where Vision Ends and Language Priors Begin in Scientific VLMs
Apr 6, 2026Dikshant Kukreja, Kshitij Sah, Karan Goyal +2Reasoning for VLMs VLM Evaluation
IIIT Delhi, India
Bridging the Missing-Modality Gap: Improving Text-Only Calibration of Vision Language Models
Apr 3, 2026Mingyeong Kim, Jungwon Choi, Chaeyun Jang +1Modality Gap in VLMs VLM Inference
Kim Jaechul Graduate School of AI, KAIST
The Geometry of Compromise: Unlocking Generative Capabilities via Controllable Modality Alignment
Mar 31, 2026Hongyuan Liu, Qinli Yang, Wen Li +6Cross-Modal Alignment Modality Gap in VLMs
University of Electronic Science and Technology of China · University of Bristol
Is the Modality Gap a Bug or a Feature? A Robustness Perspective
Mar 30, 2026Rhea Chowers, Oshri Naparstek, Udi Barzelay +1Modality Gap in VLMs Multimodal Embedding
Hebrew University, Jerusalem, Israel · IBM Research, Haifa, Israel
RoboAlign: Learning Test-Time Reasoning for Language-Action Alignment in Vision-Language-Action Models
Mar 22, 2026Dongyoung Kim, Sumin Park, Woomin Song +6Vision-Language-Action Models Reasoning for VLA Models
KAIST · RLWRLD · Yonsei University +1
IsoCLIP: Decomposing CLIP Projectors for Efficient Intra-modal Alignment
Mar 20, 2026Simone Magistri, Dipam Goswami, Marco Mistretta +3Latent Space Alignment Modality Gap in VLMs
Media Integration and Communication Center (MICC), University of Florence, Italy · Department of Computer Science, Universitat Aut`onoma de Barcelona, Spain · Computer Vision Center, Barcelona, Spain +1
On the Cone Effect and Modality Gap in Medical Vision-Language Embeddings
Mar 18, 2026David Restrepo, Miguel L Martins, Chenwei Wu +5Modality Gap in VLMs Multimodal Learning
CentraleSupélec, Université Paris-Saclay, France · University of Porto, Portugal · University of Michigan, USA +3
Two Birds, One Projection: Harmonizing Safety and Utility in LVLMs via Inference-time Feature Projection
Mar 16, 2026Yewon Han, Yumin Seol, EunGyung Kong +2VLM Safety VLM Robustness
Graduate School of Data Science, Seoul National University · Mobilint
SCoCCA: Multi-modal Sparse Concept Decomposition via Canonical Correlation Analysis
Mar 14, 2026Ehud Gordon, Meir Yossef Levi, Guy GilboaCanonical Correlation Analysis Multimodal Model Interpretability
Viterbi Faculty of Electrical and Computer Engineering, Technion – Israel Institute of Technology, Haifa, Israel
Towards Cognitive Defect Analysis in Active Infrared Thermography with Vision-Text Cues
Mar 11, 2026Mohammed Salah, Eman Ouda, Giuseppe Dell'Avvocato +6Fault Detection Thermal Imaging
Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMs
Mar 10, 2026Kaiser Sun, Xiaochuang Yuan, Hongjun Liu +4Multimodal Large Language Models Modality Gap in VLMs
Johns Hopkins University · Amazon · New York University +1
K-MaT: Knowledge-Anchored Manifold Transport for Cross-Modal Prompt Learning in Medical Imaging
Mar 6, 2026Jiajun Zeng, Shadi AlbarqouniCross-Modal Learning Medical VLMs
University of Bonn, University Hospital Bonn, Clinic for Diagnostic and Interventional Radiology, 53127 Bonn, Germany
Aligning the True Semantics: Constrained Decoupling and Distribution Sampling for Cross-Modal Alignment
Mar 5, 2026Xiang Ma, Lexin Fang, Litian Xu +1Cross-Modal Alignment Vision-Language Alignment
Shandong University, Jinan, China · The University of Exeter, Exeter, United Kingdom
Cross-modal Identity Mapping: Minimizing Information Loss in Modality Conversion via Reinforcement Learning
Mar 2, 2026Haonan Jia, Shichao Dong, Xin Dong +6Image Captioning Modality Gap in VLMs
Taobao & Tmall Group of Alibaba · The University of Hong Kong
3D Modality-Aware Pre-training for Vision-Language Model in MRI Multi-organ Abnormality Detection
Feb 27, 2026Haowen Zhu, Ning Yin, Xiaogen ZhouMedical VLMs Vision-Language Pretraining
School of Computer Science and Engineering, Southeast University, China · Department of Medical Imaging, Suzhou Traditional Chinese Medicine Hospital, China · School of Electronic, Electrical Engineering and Physics, Fujian University of Technology
ViCLIP-OT: The First Foundation Vision-Language Model for Vietnamese Image-Text Retrieval with Optimal Transport
Feb 26, 2026Quoc-Khang Tran, Minh-Thien Nguyen, Nguyen-Khang PhamVision-Language Retrieval Cross-Modal Retrieval
Can Tho University, Can Tho, Vietnam
Closing the gap in multimodal medical representation alignment
Feb 23, 2026Eleonora Grassucci, Giordano Cicchetti, Danilo ComminielloModality Gap in VLMs Vision-Language Alignment
Dept. of Information Engineering, Electronics, and Telecommunications, Sapienza University of Rome, Italy
CaReFlow: Cyclic Adaptive Rectified Flow for Multimodal Fusion
Feb 22, 2026Sijie Mai, Shiqin HanMultimodal Fusion Cross-Modal Alignment
South China Normal University
Bridging Modality Disconnect in Self-Reflection via Closed-Loop Visually Grounded Verification
Feb 21, 2026Haoyu Zhang, Yuwei Wu, Pengxiang Li +6Self-Correction in VLMs Vision-Language Models
Beijing Institute of Technology · Shenzhen MSU-BIT University
MiSCHiEF: A Benchmark in Minimal-Pairs of Safety and Culture for Holistic Evaluation of Fine-Grained Image-Caption Alignment
Feb 21, 2026Sagarika Banerjee, Tangatar Madi, Advait Swaminathan +4Cultural Safety Evaluation of VLMs VLM Evaluation
Algoverse AI Research · PocketFM
Can Vision-Language Models See Squares? Text-Recognition Mediates Spatial Reasoning Across Three Model Families
Feb 17, 2026Yuval LeventalVision-Language Models VLM Reasoning
Rochester Institute of Technology
BabyVision: Visual Reasoning Beyond Language
Jan 10, 2026Liang Chen, Weichu Xie, Yiyan Liang +27VLM Evaluation Visual Reasoning
1UniPat AI · 6Peking University · 3Alibaba Group +8
Stabilizing the Modality Gap: Null-Space Projected Continual Learning for Vision-Language Models
Jul 26, 2025Tiantian Peng, Yuyang Liu, Shuo Yang +2Continual Learning for VLMs Modality Gap in VLMs
Peking University, Shenzhen Graduate School · Peng Cheng Laboratory

Apr 22, 2026J/K move · Enter open · S save

The Expense of Seeing: Attaining Trustworthy Multimodal Reasoning Within the Monolithic Paradigm

Karan Goyal

IIIT Delhi, India

VLM Evaluation Multimodal Reasoning Vision-Language Grounding Modality Gap in VLMs Multimodal Model Evaluation Multimodal Understanding

Open paper AskPDF arXiv

The rapid proliferation of Vision-Language Models (VLMs) is often framed as enabling unified multimodal knowledge discovery but rests on an under-examined assumption: that current VLMs faithfully synthesise multimodal data. We argue they often do not, and this gap reflects a trustworthiness problem in the dominant Vision Encoder-Projector-LLM paradigm. Rather than extracting grounded knowledge from visual inputs, state-of-the-art models frequently exhibit functional blindness, i.e., exploiting strong language priors to bypass severe visual representation bottlenecks. In this work, we challenge the conventional methodology of multimodal evaluation, which relies on data ablation or new dataset creation and therefore conflates dataset biases with architectural incapacity. We propose an information-theoretic departure: the Modality Translation Protocol, designed to quantify what we call the Expense of Seeing. By translating semantic payloads rather than ablating them, we formulate three novel metrics -- the Toll (ToS), Curse (CoS), and Fallacy (FoS) of Seeing -- culminating in the Semantic Sufficiency Criterion (SSC). Furthermore, we hypothesise a Divergence Law of Multimodal Scaling: as the underlying language engines scale to unprecedented reasoning capabilities, the penalty of the visual knowledge bottleneck may increase rather than diminish. We argue the community should move beyond "multimodal gain" as a primary evaluation target. By elevating the SSC from a passive diagnostic constraint to an active architectural blueprint, we provide a foundation for guiding the next generation of AI systems toward genuine multimodal reasoning.