Modality Gap in VLMs

VLM: Vision-Language Model

Overview

The modality gap describes differences between how vision-language models represent or use images and text, even when the two convey the same information. Researchers study it through embedding geometry and paired-input evaluations, including image-text retrieval, visual reasoning, and tests that swap text for equivalent images. Approaches include contrastive and optimal-transport alignment, feature projection, modality-aware training, and diagnostics that measure whether outputs respond to relevant visual evidence. A central tension is that reducing representational separation can help some tasks but hurt others, while many models still favor language over images; reliable measures must connect the gap to task-specific performance.

Latest papers

CardsList

63 papers6 papers in the last four weeks, down 40% on the four weeks before. 0.0% of all new papers.

  1. BabyVision: Visual Reasoning Beyond Language

    Jan 10, 2026Liang Chen, Weichu Xie, Yiyan Liang +27VLM EvaluationVisual Reasoning

  2. Stabilizing the Modality Gap: Null-Space Projected Continual Learning for Vision-Language Models

    Jul 26, 2025Tiantian Peng, Yuyang Liu, Shuo Yang +2contrastive knowledge distillationContinual Learning for VLMs