cs.CVNov 17, 2026

Can Vision-Language Models Truly Understand Images? A Benchmark Beyond Commonsense Bias

Authors: Ling Lin, Heng Su, Xiangzeng Liu, Yang Bai, Congcong Zhu, Jingrun Chen

Abstract

Multimodal large language models (MLLMs) have achieved impressive performance on a wide range of vision-language tasks. However, it remains unclear whether their predictions are grounded in the visual content itself or are largely driven by statistical co-occurrence patterns and commonsense correlations learned from large-scale image-text corpora. Existing vision-language benchmarks predominantly consist of samples that conform to human commonsense, allowing models to achieve strong performance by exploiting dataset priors rather than genuinely understanding image content. To address this limitation, we present ContraSense, a systematically constructed benchmark for evaluating faithful vision-language understanding. ContraSense consists of carefully curated counter-intuitive image-text pairs that deliberately violate common expectations, substantially reducing the effectiveness of statistical biases and requiring models to rely on visual evidence for reasoning. We conduct comprehensive evaluations on a diverse set of representative open- and closed-source MLLMs. The results reveal a substantial performance gap between current MLLMs and humans: even the strongest closed-source model achieves only about 60% accuracy, compared with 96.04% for human participants. These findings demonstrate that current MLLMs still struggle with faithful visual understanding and highlight the need for more robust multimodal reasoning grounded in image content. The ContraSense benchmark is publicly available at https://huggingface.co/datasets/LingLin00/ContraSense.

Figures & tables

Explore similar work

CardsList
  1. Can MLLMs Reason Beyond Language? VisReason: A Comprehensive Benchmark for Vision-Centric Reasoning

    May 25, 2026Longteng Guo, Yifan Wang, Pengkang Huo +4Visually Grounded ReasoningVisual Reasoning

  2. HumanSightBench: A Benchmark for Human Visual Perception in MLLMs

    Nov 9, 2026Zixin Su, Yangguang Lin, Zegang Peng +10VLM EvaluationMLLM Evaluation

  3. Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models

    Jul 28, 2026Jiaang Li, Chengzu Li, Zhaochong An +4VLM EvaluationMultimodal Large Language Models