Modality Gap in VLMs

VLM: Vision-Language Model

Overview

The modality gap describes differences between how vision-language models represent or use images and text, even when the two convey the same information. Researchers study it through embedding geometry and paired-input evaluations, including image-text retrieval, visual reasoning, and tests that swap text for equivalent images. Approaches include contrastive and optimal-transport alignment, feature projection, modality-aware training, and diagnostics that measure whether outputs respond to relevant visual evidence. A central tension is that reducing representational separation can help some tasks but hurt others, while many models still favor language over images; reliable measures must connect the gap to task-specific performance.

Latest papers

CardsList

63 papers6 papers in the last four weeks, down 40% on the four weeks before. 0.0% of all new papers.

  1. TMT: Adapting CLIP to Infrared Imagery via Text-Derived Modality Knowledge Transfer

    Nov 9, 2026Ruofan Wang, Puzuo Wang, Xin Chen +3VLM AdaptationCross-Modal Representation Transfer

  2. Seeing Isn't Enough: Unlocking Visual Reasoning by Bridging the Reasoning-Access Gap with TIGER

    Nov 7, 2026Gregory Kang Ruey Lau, Minh Huynh Nguyen, Bryan Kian Hsiang LowReasoning for VLMsModality Gap in VLMs

  3. Grounding in the Dark: Text-Only Training for Zero-Shot Video Temporal Localization

    Nov 4, 2026Divyanshu Mishra, Sabine Sternig, Rakshith Shetty +1Temporal Video GroundingModality Gap in VLMs

  4. Closing the Modality Gap via Simplex-Constrained Representations

    Oct 23, 2026Shubham Gupta, Siva Reddy, Perouz Taslakian +2Modality Gap in VLMsCross-Modal Alignment

  5. Refining Open-Vocabulary Semantic Segmentation via Regional Semantics and Visual Prototypes

    Oct 22, 2026Aijing Yu, Zhengbo Wang, Lijun Sheng +2Open-Vocabulary Semantic SegmentationVision-Language Alignment

  6. Persistent Representational Misalignment Induces Vision-Language Model Failures

    Oct 18, 2026Yonatan Gideoni, Yoav Gelberg, Tim G. J. Rudner +1Vision-Language AlignmentModality Gap in VLMs

  7. Vision-Language Models Integrate Conflicting Evidence Like Biased Bayesian Observers

    Oct 14, 2026Josh Barua, Kyle Mahowald, William RudmanVision-Language ModelsModality Gap in VLMs

  8. Omni-Modal LLMs Know More Than They Say

    Oct 12, 2026Na Min An, Taekyung Kim, Jaemyung Yu +2Multimodal Large Language ModelsCross-Modal Alignment

  9. Confidence-Gated Cloud-Edge Cascade Triage via Variational Risk Minimization for Medical Imaging

    Oct 1, 2026Xinye Yang, Zhusi Zhong, Scott Collins +7Knowledge DistillationChest X-Ray Classification

  10. Visual Branch is What You Need for CLIP-based Class-Incremental Learning

    Sep 29, 2026Tao Hu, Zhen-Hao Xie, Jingcai Guo +2Class-Incremental LearningModality Gap in VLMs

  11. Dual-Mode Low-Rank Learner with Bridge-Prototype Ensemble for Vision-Language Class-Incremental Learning

    Sep 29, 2026Chiyuan He, Zihuan Qiu, Fanman Meng +5Prototype-Based LearningLow-Rank Adaptation

  12. How Medical VLMs Underutilize Their Vision Encoders: A Dermatology Perspective

    Sep 29, 2026Janet Wang, Yunbei Zhang, Xiao Wang +1Medical DiagnosisVLM Interpretability

  13. Render Before Reading: Visual Rendering as a Prompt Injection Defense

    Sep 28, 2026Jie Zhang, Andrei Baroian, Jan N. van Rijn +2Multimodal Large Language ModelsModality Gap in VLMs

  14. Efficient Quantization-Aware Distillation with Cross-Modal Alignment for Edge Vision-Language Models

    Sep 15, 2026Jinwoo Jeon, GyuYeop Do, Yubin Lim +4VLM QuantizationEfficient VLM Inference

  15. TwinICL: Diagnosing Multimodal In-Context Learning through Paired Counterfactuals

    Sep 14, 2026Zihan Xue, Po-Yi Lu, Serhii Honcharenko +5Multimodal ICLModality Gap in VLMs

  16. NOVA: Normal-Side Modeling for Training-Free Zero-Shot Video Anomaly Detection

    Sep 6, 2026Wei-Chih Yin, Yun-Ching Kao, Cheng-Kuan Lin +1Training-Free Video Anomaly DetectionVideo Anomaly Detection

  17. EgoArgus: Benchmarking VLMs as Situational Assistants for Modality-Grounded User Supports

    Aug 26, 2026Yu-Chien Tang, Yu-Hsiang Liu, An-Zi YenVLM EvaluationAgent Benchmarks

  18. MIRROR: Aligning Semantic Relations from Language to Image via Gromov--Wasserstein

    Jun 28, 2026Hong-Han Wang, Yuntao Wang, Hu DingLarge Vision-Language ModelsVLM Reasoning

  19. MLLMs Get It Right, Then Get It Wrong: Tracing and Correcting Late-Layer Textual Bias

    Jun 16, 2026Xingming Li, Ao Cheng, Qiyao Sun +4VLM RobustnessVision-Language Models

  20. Beyond Symmetric Alignment: Spectral Diagnostics of Modality Imbalance in Vision-Language Models in the Medical Domain

    Jun 3, 2026Alessandro Gambetti, Qiwei Han, Cláudia Soares +1VLM EvaluationModality Imbalance

  21. Density-Aware Translation of Spurious Correlations in Zero-Shot VLMs

    Jun 1, 2026Afsaneh Hasanebrahimi, Hanxun Huang, Christopher Leckie +1VLM RobustnessVision-Language Models

  22. LoMo: Local Modality Substitution for Deeper Vision-Language Fusion

    May 28, 2026Feng Han, Zhixiong Zhang, Zheming Liang +2Cross-Modal LearningVLM Robustness

  23. Respecting Modality Gap in Post-hoc Out-of-distribution Detection with Pre-trained Vision-Language Models

    May 26, 2026Yuanwei Hu, Bo Peng, Yadan Luo +3Vision-Language ModelsTest-Time Adaptation