VLM-SubtleBench: How Far Are VLMs from Human-Level Subtle Comparative Reasoning?

Minkyu Kim; Sangheon Lee; Dongmin Park

cs.CVMar 9, 2026

VLM-SubtleBench: How Far Are VLMs from Human-Level Subtle Comparative Reasoning?

Authors: Minkyu Kim, Sangheon Lee, Dongmin Park

Organizations: KRAFTON · KAIST

Abstract

The ability to distinguish subtle differences between visually similar images is essential for diverse domains such as industrial anomaly detection, medical imaging, and aerial surveillance. While comparative reasoning benchmarks for vision-language models (VLMs) have recently emerged, they primarily focus on images with large, salient differences and fail to capture the nuanced reasoning required for real-world applications. In this work, we introduce VLM-SubtleBench, a benchmark designed to evaluate VLMs on subtle comparative reasoning. Our benchmark covers ten difference types - Attribute, State, Emotion, Temporal, Spatial, Existence, Quantity, Quality, Viewpoint, and Action - and curate paired question-image sets reflecting these fine-grained variations. Unlike prior benchmarks restricted to natural image datasets, our benchmark spans diverse domains, including industrial, aerial, and medical imagery. Through extensive evaluation of both proprietary and open-source VLMs, we reveal systematic gaps between model and human performance across difference types and domains, and provide controlled analyses highlighting where VLMs' reasoning sharply deteriorates. Together, our benchmark and findings establish a foundation for advancing VLMs toward human-level comparative reasoning.

Figures & tables

Figure 1: Comparison of VLM-SubtleBench and MLLM-CompBench with GPT-4o.

Figure 2: Example tasks from the VLM-SubtleBench, covering ten difference categories (Attribute, State, Emotion, Temporal, Spatial, Existence, Quality, Quantity, Viewpoint, Action) and six domains (natural, game, medical, industry, aerial, synthetic). For each example, the correct answer is highlighted in bold green . Model responses from GPT-5-main, Claude-sonnet-4, and Gemini-2.5-pro are shown beneath each question in order. Some VQA instances are simplified due to space constraints; full versions and additional examples are provided in the appendix.

Benchmarks	Is Subtle?	# Domains	# Diff. Types	MCQ	Captioning
Birds-to-Words ( Forbes et al., 2019 )	✗	1	1	✗	✓
Spot-the-Diff ( 20 )	✓	1	1	✗	✓
MLLM-CompBench ( Kil et al., 2024 )	✗	1	8	✓	✗
VLM-SubtleBench (Ours)	✓	6	10	✓	✓

Table 1: Summary of Comparative Reasoning Benchmarks. ‘Is Subtle?’ indicates whether the benchmark contains image pairs whose DINOv3 similarity averages at least 0.8, meaning the differences are subtle. VLM-SubtleBench is the only benchmark that focuses on subtle comparison, spans diverse domains, and includes both multiple-choice questions and captioning tasks.

Figure 3: Data Curation Pipeline of VLM-SubtleBench.

Figure 4: Statistics of the test split of VLM-SubtleBench.

Open-source
Model	AT	ST	EM	TM	SP	EX	QN	QL	VP	AC	AVG
Random Guess	35.9	50.0	50.0	50.0	36.6	23.2	48.9	50.0	42.1	50.0	43.3
Human Eval	92.0	93.0	93.0	93.0	95.0	97.0	97.0	99.0	98.0	98.0	95.5
0 0 LLaVA-NeXT-7B	37.0	51.3	51.8	47.4	37.3	25.6	49.5	48.0	43.7	46.9	43.6
0 0 LLaVA-OneVision-7B	41.6	56.8	73.9	48.7	35.5	44.2	54.9	62.7	49.1	60.5	52.0
0 0 Qwen2.5-VL-7B	46.5	63.7	87.8	50.2	39.5	73.8	58.0	70.9	47.5	69.3	59.4

Table 2: Performance of open-source and proprietary vision-language models in VLM-SubtleBench. Human evaluation was conducted on a randomly selected 10% of the samples.

Prompting Strategies
Model	AT	ST	EM	TM	SP	EX	QN	QL	VP	AC	AVG
Random Guess	35.9	50.0	50.0	50.0	36.6	23.2	48.9	50.0	42.1	50.0	43.3
0 0 GPT-5-main	72.9	78.4	92.7	53.6	50.1	75.4	72.6	84.5	57.5	83.6	71.3
0 0 + Reasoning	76.5	79.1	91.2	56.1	51.6	80.2	75.8	86.1	57.0	85.4	73.1
0 0 + Two-Step Reasoning	70.8	79.1	93.4	56.9	47.4	81.3	66.4	83.5	58.0	83.6	71.0
0 0 + Grid	71.6	77.5	89.1	52.8	51.0	75.8	72.5	82.6	57.2	84.2	70.6

Table 4: Effect of prompting strategies and fine-tuning in VLM-SubtleBench.

Table 9

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Type	Domain	Source	#
Attribute	Natural	COCO	114
	Industry	MVTEC-AD	300
	Medical	MIMIC-Diff-VQA	330
	Synthetic	Synthetic	700
State	Natural	ChangeIt	465
State	Industry	MVTEC-AD	586

Appendix

Table 7: Number of test examples per difference type, domain, and source.

Figure 6: Example annotation interface for sources with ground-truth annotations (ChangeIt). Annotators select frame pairs from pre-sampled candidates based on the provided labels.

Figure 7: Example annotation interface for sources without ground-truth annotations (YT8M temporal). Annotators provide labels (e.g., whether the temporal order is irreversible) for each frame pair.

Figure 8: Standard prompt used in our benchmark.

Figure 9: Reasoning prompt used in our benchmark.

Figure 10: Two-step reasoning prompt used in our benchmark.

Figure 11: Grid prompt used in our benchmark.

Figure 12: Concatenating prompt used in our benchmark.

Figure 13: Highlighting prompt used in our benchmark.

Figure 14: Overlapping prompt used in our benchmark.

Figure 15: Subtracting prompt used in our benchmark.

Figure 16: Captioning prompt used in our benchmark.

Panel of Figure 17 — Figure 17: Example of overlap-image construction. Two aligned inputs are blended with equal weights (50% each) to produce a composite image that visually merges both inputs.

Panel of Figure 18 — Figure 18: Example of subtraction-image construction. Given two aligned images ( $I_{1}$ , $I_{2}$ ), the subtraction map $S$ highlights regions with maximal pixel-wise change.

Panel of Figure 19 — Figure 19: Example highlight images. Significant regions of change, extracted via percentile thresholding and morphological filtering, are emphasized by dimming the background and drawing green bounding-box boundaries.

Models	VLM-SubtleBench	MLLM-CompBench	MMAD	QAG-360K
Qwen-2.5-VL-7B	59.4	73.6	65.0	34.4
Qwen-2.5-VL-32B	62.2	74.6	67.6	35.5
Qwen-2.5-VL-72B	65.4	76.9	68.9	41.2
GPT-4o	61.6	75.7	67.7	35.2
o3	75.7	86.3	72.9	39.7
GPT-5-main	71.3	83.9	70.6	36.3

Appendix

Table 8: Model-wise accuracy on VLM-SubtleBench, MLLM-CompBench, MMAD, and QAG-360K.

Category	GPT-5-main	Two-stage Reasoning
Emotion	92.7	87.8
Existence	75.4	63.2
Quality	84.5	84.6

Appendix

Table 9: Effect of pure language-based comparison for emotion, existence, and quality categories.

Open-source
Model	Natural	Game	Industry	Aerial	Synthetic	Medical
Random Guess	49.2	50.0	50.0	26.9	29.3	50.0
0 0 LLaVA-NeXT-7B	48.6	47.9	53.8	35.4	30.0	41.5
0 0 LLaVA-OneVision-7B	59.3	56.0	54.7	62.2	32.4	50.9
0 0 Qwen2.5-VL-7B	65.2	61.8	66.3	62.8	43.8	50.3
0 0 Qwen2.5-VL-32B	66.1	63.4	64.4	64.6	53.0	54.5

Appendix

Table 10: Domain-wise accuracy (%) of open-source and proprietary VLMs.

Figure 20: Color-sensitivity analysis of GPT-4o under controlled synthetic settings.

Type	Accuracy	CSS	LLM-Judge
Real	60.6	0.51	26.3
Reconstructed	60.6	0.51	27.3

Appendix

Table 11: Comparison between real and nano-banana -reconstructed image pairs on VQA and captioning metrics.

Model	AT	ST	EM	TM	SP	EX	QN	QL	VP	AC
Random Guess	35.9	50.0	50.0	50.0	36.6	23.2	48.9	50.0	42.1	50.0
Qwen2.5-VL-72B	53.9	68.9	85.9	49.9	47.8	81.7	67.7	78.4	56.2	74.1
+ Reasoning	50.6	68.0	87.5	51.0	44.4	81.7	65.0	77.1	54.7	75.0
+ Grid	54.9	67.6	85.6	49.8	46.3	83.4	70.1	75.7	54.4	71.1
+ Concat	55.7	68.0	86.4	50.0	39.6	67.5	61.0	65.4	47.0	73.7
+ Overlap	46.3	63.2	85.3	49.7	48.8	78.7	63.1	72.5	49.7	71.6

Appendix

Table 12: Effect of different prompting strategies in VLM-SubtleBench.

Attribute
Dataset	Random	GPT-4o	GPT-5-main	GPT-5-thinking	GPT-o3	Claude-sonnet-4	Gemini-2.5-flash	Gemini-2.5-pro
MVTEC-AD	50.0	76.7	79.0	82.0	79.0	66.3	76.3	83.3
COCO	25.0	78.1	82.5	86.0	84.2	71.1	69.3	77.2
MIMIC-DIFF-VQA	50.0	62.4	78.8	82.4	68.5	54.8	62.4	68.8
Synthetic	25.0	31.3	66.0	84.4	81.1	35.0	28.3	33.4
State

Appendix

Table 13: Performance of proprietary models by data source

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 989 papers

CoRe: A Comprehensive Framework for Cross-Image Comparative Reasoning in Vision-Language Models
Jul 14, 2026Lin Peng, Cong Wan, Zeyu Guo +2VLM Evaluation Visual Reasoning
Xi’an Jiaotong University Xi’an, China
VDiff-Bench: A Challenging Benchmark for Fine-Grained Image Difference Identification
Sep 5, 2026Yixin Wan, Tianle Zheng, Kai-Wei ChangVLM Evaluation Vision-Language Models
Department of Computer Science University of California, Los Angeles
A Study of Commonsense Reasoning over Visual Object Properties
Aug 14, 2025Abhishek Kolari, Mohammadhossein Khojasteh, Yifan Jiang +2VLM Evaluation Visual Question Answering
Department of Computer Science, Vrije Universiteit, Amsterdam, The Netherlands · Information Sciences Institute, University of Southern California, Marina del Rey, CA, USA

Jul 14, 2026J/K move · Enter open · S save

CoRe: A Comprehensive Framework for Cross-Image Comparative Reasoning in Vision-Language Models

Lin Peng, Cong Wan, Zeyu Guo, SongLin Dong+1

Xi’an Jiaotong University Xi’an, China

VLM Evaluation Visual Reasoning Vision-Language Grounding VLM Reasoning

Open paper AskPDF arXiv

Cross-image comparative reasoning remains challenging for vision-language models (VLMs), especially when correct prediction requires fine-grained attribute grounding and globally consistent reasoning. We present CoRe, a unified framework for this problem. CoRe includes: (i) CoRe-20K, a large-scale triplet-based training set automatically constructed from structured visual metadata through a multi-expert collaborative pipeline, covering counting, depth, distance, and spatial relations; (ii) TriSR, a structured reward framework that jointly supervises attribute grounding, judgment alignment, and triplet consistency under GRPO optimization; and (iii) CoRe-Bench, the first benchmark dedicated to fine-grained cross-image comparative reasoning. Experiments show that CoRe substantially outperforms existing VLMs on CoRe-Bench while remaining competitive on standard multimodal benchmarks, achieving a 28.2-point gain in partial accuracy over the strongest baseline.

Abstract

VLM Evaluation

Explore similar work

CoRe: A Comprehensive Framework for Cross-Image Comparative Reasoning in Vision-Language Models

VDiff-Bench: A Challenging Benchmark for Fine-Grained Image Difference Identification

A Study of Commonsense Reasoning over Visual Object Properties