cs.CVSep 5, 2026

VDiff-Bench: A Challenging Benchmark for Fine-Grained Image Difference Identification

Authors: Yixin Wan, Tianle Zheng, Kai-Wei Chang

Organizations: Department of Computer Science University of California, Los Angeles

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Multimodal Large Language Models (MLLMs) perform strongly on general visual understanding tasks such as visual question answering, yet they often struggle with a basic comparative skill: identifying what has changed between two similar images. We introduce VDiff-Bench, a challenging multiple-choice benchmark for fine-grained Image Difference Identification. VDiff-Bench contains 1,756 four-way questions over image pairs and covers 10 change categories: position, motion, regional image color, overall image color, appearance/disappearance, noise/resolution, texture, substitution/size, OCR/text, and illumination. Each question corresponds to two image inputs with 4 choices: the true difference, two hard negative descriptions, and a "no difference" distractor. To make the task challenging, we specifically curate ground-truth-conditioned negatives that require models to distinguish the actual change from nearby semantic alternatives. Experiments with 11 state-of-the-art open- and closed-source MLLMs show that fine-grained visual comparison remains brittle: models exhibit uneven performance across sources and change categories, with persistent failures on subtle low-level changes like noises and textures. For instance, three 7-8B-scale open-source MLLMs score 52.5-70.6% on semantic changes but only 8.7-33.3% on low-level changes like noise and texture, falsely assuming no changes between two image inputs. Surprisingly, despite strong performance of other closed-source commercial models, Grok 4.3 demonstrate remarkable performance drop on identifying noise and texture differences between images, falling significantly behind large open-source models like Kimi K2.5 and K3. Overall, VDiff-Bench provides a targeted diagnostic for evaluating comparative visual understanding in MLLMs, exposing failures that are not captured by standard single-image vision-language tasks.

Figures & tables

Figure 1: Comparison with prior visual difference identification benchmarks. Existing benchmarks primarily focus on open-ended difference captioning and are often limited in visual domains, change categories, or evaluation reliability. VDiff-Bench expands to 10 fine-grained change categories and evaluates models through challenging multiple-choice questions for more deterministic and objective IDI performance quantization.
Figure 2: Overall and low-level accuracy on VDiff-Bench across 11 MLLMs. Bars report overall four-choice accuracy; purple diamonds report accuracy on whole-image color, noise/resolution, texture, and illumination changes. Models are sorted by overall accuracy, and horizontal lines mark the 25% uniform-guess and 33.3% informed-guess baselines. Similar overall scores can conceal substantial low-level deficits.
Figure 3: Each evaluation example in VDiff-Bench contains a ground truth difference, two false difference distractor options, and one no-difference distractor option. We show an example of higher-level direct visual change (i.e. changing an object clearly visible) on the left and an example of lower-level visual change (i.e. changing the image’s trait like texture) on the right.
Category# Q.Representative distinction
Position158left/right, nearer/farther, relocated object
Motion138action, pose, orientation, or state transition
Regional color170localized object or region color
Whole-image color147global RGB or color-cast shift
Appearance/disappearance201added, removed, or missing entity
Noise/resolution150Gaussian noise or resolution degradation
Table 1: Current VDiff-Bench taxonomy. Counts refer to questions; the present inventory contains 1,756 questions over 1,543 distinct image pairs.
BenchmarkPairsImage RegimeTarget-Change CoverageEvaluation Design
RealEditedRendered2D puzzleSemanticOCR / textGlobal photo.Noise / res.TextureHard alternativesExact scoring
Spot-the-Diff ( Jhamtani and Berg-Kirkpatrick, 2018 )13,192✓✗✗✗✓✗✗✗✗✗✗
CLEVR-Change ( Park et al., 2019 )79,606✗✗✓✗✓✗✗✗✓✗✗
OmniDiff ( Liu et al., 2025 )15,598✓✗✓✗✓✓✓✗✗✗✗
DiffCap-Bench ( Wei et al., 2026 )1,075✓✓✓✗✓✓✓✓✓✗✗
VDiff-Bench (ours)1,543✓✓✓✓✓✓✓✓✓✓✓
Table 2: Coverage and evaluation design of visual difference benchmarks. Check marks indicate dimensions explicitly included in benchmark construction or evaluated as target differences. “Global photo.” includes whole-image color and illumination changes, while “Noise/res.” includes noise or resolution degradation. Hard alternatives are plausible competing change statements, and exact, judge-free scoring denotes direct answer-key matching without reference-caption metrics or a learned evaluator.
ModelChange CategoryOverall
Pos.MotionReg. col.Whole col.App./dis.NoiseTextureSub./sizeOCRIllum.
Baselines
Random25.025.025.025.025.025.025.025.025.025.025.0
Informed guess33.333.333.333.333.333.333.333.333.333.333.3
Text-only style57.633.334.127.261.232.732.835.733.531.338.2
Closed Source Models
Table 3: Choice accuracy (%) on VDiff-Bench . Results are reported by change category and overall; invalid outputs count as incorrect. Random guessing over four options yields 25.0%, while informed guessing uniformly selects among the three descriptive options, exploiting that “no difference” is always incorrect, and yields 33.3%. The text-only baseline uses only capitalization and punctuation cues without viewing the images. Bold indicates the best MLLM performance in each column, including ties.
Figure 4: Semantic and low-level performance across 11 MLLMs. (a) Accuracy on semantic and low-level change groups; connectors expose within-model gaps, labels mark gaps of at least 25 percentage points, and the dashed line denotes the 33.3% informed-guess baseline. (b) Low-level predictions decomposed into correct, no-difference, candidate-alternative, and invalid outcomes; labels mark no-difference rates of at least 15%.
ModelPair macroChange GroupError Type
SemanticLow-levelOCRNo diff.AlternativeInvalid
Closed Source Models
GPT-5.486.283.585.186.05.210.40.0
Gemini 2.5 Flash67.467.357.586.012.118.63.4
Gemini 3.1 Pro91.488.590.092.81.68.30.6
Gemini 3.5 Flash90.488.189.789.10.57.82.9
Table 4: Grouped performance on VDiff-Bench (%). “Pair macro” indicates accuracy averaged over unique image pairs. Group columns report question-level accuracy for semantic, low-level, and OCR changes. Error columns give rates across all questions of selecting no difference, either hard negative, or an invalid response; lower is better.
Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Source subsetQuestionsPairsPrimary category/categories
Spot-the-Diff10855appearance, position
MotionEdit13767motion
OmniEdit color137137regional color
OmniEdit position120120position
Custom illumination4444illumination
Kids games10212mixed puzzle differences
Appendix
Table 5: Source composition of the current benchmark inventory. Source names are reported for provenance; main-paper coverage is summarized by change category.
ModelWhole colorNoiseTextureIllum.Overall
GPT-5.49.50.09.614.45.2
Gemini 2.5 Flash24.50.039.021.012.1
Gemini 3.1 Pro1.40.02.34.61.6
Gemini 3.5 Flash2.00.00.01.00.5
Grok 4.34.179.372.335.421.8
Doubao Seed 1.6 Vision0.00.04.06.21.3
Appendix
Table 6: Rate (%) at which each model selects the no-difference distractor. The first four columns are category-specific; “Overall” covers all 1,756 questions. High rates indicate a missed-change selection rather than confusion with another detailed description.
Figure 6: Representative no-difference selections. (a) Smoothing: Qwen, Gemini, LLaVA, and InternVL select no difference; GPT-5.4 selects the keyed answer. (b) Global RGB shift: Qwen, LLaVA, and InternVL select no difference. (c) Two people disappear: LLaVA and InternVL select no difference.
Figure 7: Qualitative VDiff-Bench results across all ten change categories (part 1 of 2). Each panel shows the complete image pair at a common height, all four answer choices with the ground truth colored, and correctness for six representative MLLMs. Green checks and red crosses indicate correct and incorrect selections.
Figure 8: Qualitative VDiff-Bench results across all ten change categories (part 2 of 2). Layout and symbols follow Figure 7 . Examples are selected to illustrate mixed model outcomes rather than category-level prevalence.
More in this topic

VLM Evaluation

Vision-language model evaluation measures multimodal systems’ perception, reasoning, grounding, robustness, and reliability across controlled and real-world tasks.

See all 989 papers

Explore similar work

CardsList
May 6, 2026

DiffCap-Bench: A Comprehensive, Challenging, Robust Benchmark for Image Difference Captioning

Image Difference Captioning (IDC) generates natural language descriptions that precisely identify differences between two images, serving as a key benchmark for fine-grained change perception, cross-modal reasoning, and image editing data construction. However, existing benchmarks lack diversity and compositional complexity, and standard lexical-overlap metrics (e.g., BLEU, METEOR) fail to capture semantic consistency or penalize hallucinations, which together prevent a comprehensive and robust evaluation of multimodal large language models (MLLMs) on IDC. To address these gaps, we introduce DiffCap-Bench, a comprehensive IDC benchmark covering ten distinct difference categories to ensure diversity and compositional complexity. Furthermore, we propose an LLM-as-a-Judge evaluation protocol grounded in human-validated Difference Lists, enabling a robust assessment of models' ability to both capture and describe visual changes. Through extensive evaluation of state-of-the-art MLLMs, we reveal significant performance gaps between proprietary and open-source models, highlight the critical importance of reasoning capability, and identify clear limitations in model scaling. Our framework also demonstrates strong alignment with human expert judgments and strong correlation with downstream image editing data construction quality. These findings establish DiffCap-Bench as both a reliable IDC evaluation framework and a practical predictor of downstream utility. The benchmark and code will be made publicly available to support further research.

Yuancheng Wei, Haojie Zhang, Linli Yao +7

South China University of Technology · Peking University · The University of Hong Kong +2

Image CaptioningMultimodal Model Evaluation
AskPDF
May 28, 2026

DiffSpot: Can VLMs Spot Fine-Grained Visual Differences in Web Interfaces?

Vision-language models (VLMs) have made strong progress on high-level image-text alignment, yet their ability to perceive subtle visual differences remains limited. We study this problem in rendered web interfaces, where localized visual changes are both a diagnostic test of fine-grained perception and a practical requirement for GUI agents and design tools. We introduce DiffSpot, a code-driven benchmark for open-ended spot-the-difference on web interfaces. DiffSpot constructs controlled image pairs by mutating a single CSS property of a target element in self-contained HTML, re-rendering the page, and recording the changed property, element, and mutation magnitude. A grounding gate retains only pairs whose rendered pixel difference is confined to the target element. The benchmark contains 4{,}400 pairs, including 3{,}900 has-diff pairs balanced across 13 CSS-property operators and three difficulty tiers, plus 500 no-diff pairs for hallucination control. Evaluating 13 frontier VLMs zero-shot, we find that even the best model identifies only 40.7%40.7\% of true changes, with Hard-tier Recall below 23%23\% for every model. DiffSpot further shows that difficulty is strongly property-dependent: across CSS operators, neither pixel magnitude nor CLIP distance reliably predicts Recall.

Linhao Zhang, Aiwei Liu, Yuan Liu +1

WeChat AI, Tencent Inc.

VLM EvaluationFine-Grained Visual Perception
AskPDF
Mar 9, 2026

VLM-SubtleBench: How Far Are VLMs from Human-Level Subtle Comparative Reasoning?

The ability to distinguish subtle differences between visually similar images is essential for diverse domains such as industrial anomaly detection, medical imaging, and aerial surveillance. While comparative reasoning benchmarks for vision-language models (VLMs) have recently emerged, they primarily focus on images with large, salient differences and fail to capture the nuanced reasoning required for real-world applications. In this work, we introduce VLM-SubtleBench, a benchmark designed to evaluate VLMs on subtle comparative reasoning. Our benchmark covers ten difference types - Attribute, State, Emotion, Temporal, Spatial, Existence, Quantity, Quality, Viewpoint, and Action - and curate paired question-image sets reflecting these fine-grained variations. Unlike prior benchmarks restricted to natural image datasets, our benchmark spans diverse domains, including industrial, aerial, and medical imagery. Through extensive evaluation of both proprietary and open-source VLMs, we reveal systematic gaps between model and human performance across difference types and domains, and provide controlled analyses highlighting where VLMs' reasoning sharply deteriorates. Together, our benchmark and findings establish a foundation for advancing VLMs toward human-level comparative reasoning.

Minkyu Kim, Sangheon Lee, Dongmin Park

KRAFTON · KAIST

VLM EvaluationVLM ReasoningFine-Grained Visual Perception
AskPDF