Multimodal large language models (MLLMs) have achieved impressive performance on a wide range of vision-language tasks. However, it remains unclear whether their predictions are grounded in the visual content itself or are largely driven by statistical co-occurrence patterns and commonsense correlations learned from large-scale image-text corpora. Existing vision-language benchmarks predominantly consist of samples that conform to human commonsense, allowing models to achieve strong performance by exploiting dataset priors rather than genuinely understanding image content. To address this limitation, we present ContraSense, a systematically constructed benchmark for evaluating faithful vision-language understanding. ContraSense consists of carefully curated counter-intuitive image-text pairs that deliberately violate common expectations, substantially reducing the effectiveness of statistical biases and requiring models to rely on visual evidence for reasoning. We conduct comprehensive evaluations on a diverse set of representative open- and closed-source MLLMs. The results reveal a substantial performance gap between current MLLMs and humans: even the strongest closed-source model achieves only about 60% accuracy, compared with 96.04% for human participants. These findings demonstrate that current MLLMs still struggle with faithful visual understanding and highlight the need for more robust multimodal reasoning grounded in image content. The ContraSense benchmark is publicly available at https://huggingface.co/datasets/LingLin00/ContraSense.
Figures & tables
Fig. 1: We propose a new benchmark called ContraSense, which is both creative and imaginative. The benchmark consists of 4,698 carefully crafted and manually calibrated high-resolution (1024 × 1024 pixels) text QA pairs. Each image contains at least one counterintuitive scenario designed to assess whether visual-language models can provide accurate responses based on the image content itself without relying on common-sense prior knowledge from the training data. This benchmark provides a unique and powerful tool for studying the robustness and real-world visual understanding capabilities of VLMs.
Fig. 2: Pipeline of counterintutive data collection.
No.
Type Name
Example
Description
Common Sense Detection Capability
1
Context Violation
”A small wooden boat inside a modern bathroom, photorealistic”
Objects appearing in unlikely backgrounds or scenes
Scene Common Knowledge
2
Attribute Violation
”A black banana”
Unreasonable attributes such as colour, material, orientation, etc.
Attribute recognition
3
Size/Proportion Anomalies
”A giant cat towering over city buildings”
Proportion Extreme Anomalies
Scale Comparison
4
Physics-Impossible
”A full glass of water floating in mid-air”
Violation of the laws of physics
Physics common sense
5
Function/State Perversity
”A burning ice cube on a metal tray”
State does not correspond to the function of the object
State Identification
TABLE I: ContraSense defines five counterintuitive categories, and each generated sample contains at least one of these categories, or a combination of multiple counterintuitive categories.
Fig. 3: Distribution of different types of common sense violations in ContraSense. Phys-Imposs: Physics-Impossible. Attr-Viol: Attribute Violation. Ctx-Viol: Context Violation. Sz-Prop-Anom: Size/Proportion Anomalies. Func-State-Perv: Function/State Perversity.
Model
Image Encoder
Language Model
Data Type
ContraSense Bench Performance
Contra-Acc( % ) ↑
ROUGE-1 ↑
ROUGE-2 ↑
ROUGE-L ↑
METEOR ↑
Times
Human Performance
-
-
ContraSense
96.04
-
-
-
-
-
Open-source Models
ContraSense
36.91
14.50
4.17
11.20
21.00
01:28:16
Normal
79.18
17.48
7.52
13.75
25.97
01:22:05
LLaVA-NeXT-8B
CLIP-L-14
Llama-3-8B-Instruct
Δ
-42.27
-2.98
-3.35
-2.55
-4.97
-
TABLE II: Model performance evaluation on the ContraSense Benchmark. We systematically evaluated the overall performance of eight state-of-the-art VLM models on the ContraSense dataset and normal data. The Δ row indicates the difference between the two datasets across each metric. The experimental results show that all models perform significantly lower on the ContraSense dataset compared to the normal data, indicating that these models still have limited capability in accurately reflecting image content.
Model
Image Encoder
Language Model
Data Type
ContraSense Bench Performance
Contra-Acc( % ) ↑
ROUGE-1 ↑
ROUGE-2 ↑
ROUGE-L ↑
METEOR ↑
Open-source Models
Normal
79.18
17.48
7.52
13.75
25.97
Context Violation
42.49
15.08
4.34
11.40
21.42
Attribute Violation
38.10
15.00
4.36
11.50
21.51
Size/Proportion Anomalies
34.83
14.35
4.23
11.22
20.89
TABLE III: Model performance was evaluated on the ContraSense dataset for different types of counterintuitive samples.
Fig. 4: We replaced the input images with noisy images and conducted blind tests on ANOMALY RECOGNITION QUESTIONS and COMMONSENSE REASONING QUESTION. The results showed that the model’s blind test performance on the normal data was significantly higher than its performance on the ContraSense dataset, averaging a high level of 95%, indicating that even in the absence of image content, the model can still obtain the correct answer based on statistical co-occurrence. The red dashed line represents the randomly selected probability baseline.
Fig. 5: Row-normalized confusion matrices for eight MLLMs on the ContraSense benchmark. Each row corresponds to the true answer slot (A–E), and each column denotes the predicted answer slot.
Model
A
B
C
D
E
Open-source
LLaVA-NeXT-8B
1.381
1.231
1.127
0.746
0.515
DeepSeek-VL-7B-Chat
1.478
1.207
1.000
0.713
0.603
Llama-3.2-11B-Vision-Instruct
1.523
0.919
1.092
0.781
0.685
Qwen2.5-VL-7B-Instruct
1.016
1.200
1.151
0.859
0.774
Closed-source
TABLE IV: Column-wise sums of row-normalized confusion matrices, reflecting models’ positional bias over options.
Models
Data Scale
Answer Type
Contra-intuitive
Human Annotation
HumanEval [ 34 ]
164
Open
✗
✗
MM-Vet [ 35 ]
218
Open
✗
✗
INTERLEAVEDBENCH [ 36 ]
815
Open
✗
✓
MMIE [ 37 ]
20K
Multi-Choice & Open
✗
✓
ContraSense (Ours)
4,698
Multi-Choice & Open
✓
✓
TABLE V: The comparison between ContraSense with other LVLM benchmarks.
Model
Image Encoder
Language Model
Data Type
ContraSense Bench Performance
Accuracy( % ) ↑
F1 Macro( % ) ↑
F1 Weighted( % ) ↑
Precision( % ) ↑
Recall( % ) ↑
MCC( % ) ↑
Open-source Models
Object Class
19.05
12.62
21.02
14.47
11.89
-7.72
Object Colour
30.00
25.16
32.17
27.47
28.72
11.01
Object Num
37.50
33.01
41.34
38.70
30.03
18.36
Counterintuitive
36.05
27.53
36.47
27.38
27.91
9.06
TABLE VI: Model performance was evaluated on the ContraSense dataset for different types of questions.
Fig. 6: The figure shows the performance trends of mainstream multimodal models at different resolutions.