OddGridBench: Exposing the Lack of Fine-Grained Visual Discrepancy Sensitivity in Multimodal Large Language Models

Tengjin Weng; Wenhao Jiang; Jingyi Wang; Ming Li; Lin Ma; Zhong Ming

cs.CVMar 10, 2026

OddGridBench: Exposing the Lack of Fine-Grained Visual Discrepancy Sensitivity in Multimodal Large Language Models

Authors: Tengjin Weng, Wenhao Jiang, Jingyi Wang, Ming Li, Lin Ma, Zhong Ming

Organizations: College of Computer Science and Software Engineering, Shenzhen University · Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ) · Tsinghua Shenzhen International Graduate School · Meituan ·

Abstract

Multimodal large language models (MLLMs) have achieved remarkable performance across a wide range of vision language tasks. However, their ability in low-level visual perception, particularly in detecting fine-grained visual discrepancies, remains underexplored and lacks systematic analysis. In this work, we introduce OddGridBench, a controllable benchmark for evaluating the visual discrepancy sensitivity of MLLMs. OddGridBench comprises over 1,400 grid-based images, where a single element differs from all others by one or multiple visual attributes such as color, size, rotation, or position. Experiments reveal that all evaluated MLLMs, including open-source families such as Qwen3-VL and InternVL3.5, and proprietary systems like Gemini-2.5-Pro and GPT-5, perform far below human levels in visual discrepancy detection. We further propose OddGrid-GRPO, a reinforcement learning framework that integrates curriculum learning and distance-aware reward. By progressively controlling the difficulty of training samples and incorporating spatial proximity constraints into the reward design, OddGrid-GRPO significantly enhances the model's fine-grained visual discrimination ability. We hope OddGridBench and OddGrid-GRPO will lay the groundwork for advancing perceptual grounding and visual discrepancy sensitivity in multimodal intelligence. Code and dataset are available at https://wwwtttjjj.github.io/OddGridBench/.

Figures & tables

Figure 1: Illustration of human perceptual visual discrepancy sensitivity, showing the transition from imperceptible to perceptible visual differences in color, rotation, and size.

Figure 2: Overview of OddGridBench. OddGridBench encompasses four primary visual attributes, including color, size, rotation, and position, and supports both single-attribute and multi-attribute discrepancy compositions, providing a systematic framework for evaluating the perceptual discrepancy sensitivity of MLLMs.

Figure 3: Evaluation results of MLLMs on the OddGridBench. Human performance significantly surpasses all evaluated MLLMs across color, size, rotation, and position dimensions, as well as multi-type combinations.

Figure 4: Overview of the OddGridBench data generation pipeline, which constructs grid-based images from collected icons under precisely controlled perceptual conditions for evaluating visual discrepancy sensitivity.

Figure 5: Overview of OddGrid-GRPO framework. OddGrid-GRPO integrates curriculum-guided optimization with spatially guided reward shaping to enhance perceptual grounding and improve fine-grained visual discrimination in MLLMs.

Method	Color	Size	Rotation	Position	2-Type	3-Type	4-Type	Total
Random	2.00	2.00	2.00	2.00	1.50	4.00	3.50	2.43
Open-source MLLMs
Phi-3.5-vision	2.00	2.00	2.00	1.50	1.50	1.50	2.00	1.79
SAIL-VL2-2B	13.50	2.50	3.00	2.00	7.50	5.00	9.50	6.14
SAIL-VL2-8B	45.00	2.50	5.00	2.50	21.50	24.00	30.50	18.71
LLaVA-OneVision-1.5-4B	27.50	3.00	7.50	1.00	14.00	17.50	26.00	13.79

Table 1: Accuracies (%) of various MLLMs on the OddGridBench dataset. The benchmark evaluates fine-grained visual discrimination across four perceptual dimensions as well as their multi-type combinations. Dark gray and Light gray indicate the best and second-best results among all models, respectively.

Figure 6: Accuracy across varying perceptual discrepancy magnitudes in four visual dimensions. The x-axis represents the incremental difference in each attribute (

\Delta d=\sqrt{(\Delta x)^{2}+(\Delta y)^{2}}

). As the discrepancy increases from imperceptible to clearly perceptible levels, model performance gradually improves.

Method	Acc	TolAcc	LabeledAcc
InternVL3.5-38B	27.07	53.29 $(\uparrow+26.22)$	56.40 $(\uparrow+19.33)$
Qwen3-VL-4B	52.43	74.14 $(\uparrow+21.71)$	70.40 $(\uparrow+17.97)$
Qwen3-VL-32B	68.07	75.43 $(\uparrow+7.36)$	73.80 $(\uparrow+5.73)$
Gemini-2.5-Pro	49.29	61.21 $(\uparrow+11.92)$	58.40 $(\uparrow+9.11)$

Table 2: Comparison of strict accuracy (Acc), Tolerance Accuracy (TolAcc), and Labeled Grid Accuracy (LabeledAcc). Values in green indicate improvements relative to strict accuracy.

Method	Color	Size	Rotation	Position	2-Type	3-Type	4-Type	Total
Baseline	23.00	5.00	12.50	7.00	19.00	22.50	31.00	17.14
GRPO	88.50	44.00	67.50	41.50	78.50	83.00	93.00	70.86
GSPO	70.00	55.00	81.50	59.00	85.50	85.50	95.00	75.93
OddGrid-GRPO (w/o $r_{\text{d}}$ )	87.50	44.50	67.00	45.50	80.50	91.00	91.50	72.50
OddGrid-GRPO (w/o Cur-Guided)	87.50	60.50	69.00	64.00	84.00	88.50	95.50	78.43
OddGrid-GRPO	89.50	64.50	80.50	64.50	90.50	91.50	97.50	82.64

Table 3: Quantitative comparison of OddGrid-GRPO with existing reinforcement learning methods (upper block) and its ablated variants (lower block) across all perceptual attribute types on OddGridBench.

Figure 7: Training curves of GRPO, GSPO, and OddGrid-GRPO. OddGrid-GRPO converges faster and achieves higher final rewards, indicating more stable and efficient optimization.

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Phi-3.5-vision	microsoft/Phi-3.5-vision-instruct
SAIL-VL2-2B	BytedanceDouyinContent/SAIL-VL2-2B
SAIL-VL2-8B	BytedanceDouyinContent/SAIL-VL2-8B
LLaVA-OneVision-1.5-4B	lmms-lab/LLaVA-OneVision-1.5-4B-Instruct
LLaVA-OneVision-1.5-8B	lmms-lab/LLaVA-OneVision-1.5-8B-Instruct
LLaVA-v1.6-34B	liuhaotian/llava-v1.6-34b
InternVL3.5-38B	OpenGVLab/InternVL3_5-38B

Appendix

Table 4: List of MLLMs evaluated in this study, with model names shown exactly as they appear on Hugging Face or in official APIs.

Model	Grid-Small	Grid-Mid	Grid-Large
InternVL3.5-38B	27.74	26.94	25.42
Qwen3-VL-32B	70.49	66.97	64.41
Gemini-2.5-Pro	54.06	47.34	41.24
GPT-5	35.51	26.79	15.82

Appendix

Table 5: Performance across different grid sizes.

	Resolution
Model	50	100	150
Qwen3-VL-32B	69.00	70.00	73.00
Qwen3-VL-8B	56.00	60.00	59.00
Gemini-2.5-Pro	47.00	57.00	57.00
GPT-5	31.00	35.00	39.00

Appendix

Table 6: Performance across different image resolutions.

	Resolution
Model	50	100	150
Gemini-2.5-Pro	47.00	57.00	57.00
GPT-5	31.00	35.00	39.00

Appendix

Table 7: Ablation on image resolution. We evaluate the performance of different models under three input resolutions (50, 100, and 150). Higher resolutions generally provide richer visual details and lead to improved odd-item detection accuracy.

Model	[email protected]	[email protected]	[email protected]
Qwen3-VL-2B	0.074	0.053	0.036
Qwen3-VL-4B	0.089	0.062	0.041
Qwen3-VL-8B	0.102	0.078	0.055
Qwen3-VL-32B	0.133	0.087	0.053

Appendix

Table 8: Bounding-box localization accuracy of Qwen3-VL models at different IoU thresholds.

Figure 8: Comparison of MLLMs’ performance under different grid densities. Grid size is categorized by total cell count (

r\times c

): Small Grid (

\leq 44

cells), Medium Grid (45–64 cells), and Large Grid (65–81 cells).

Method	MVTec-AD	VisA	MNIST	SCC
Qwen3-VL-32B	67.00	66.00	39.00	87.00
Qwen3-VL-2B (Baseline)	20.00	9.00	6.00	17.00
GRPO	47.00	39.00	18.00	54.00
OddGrid-GRPO (Ours)	49.00	40.00	37.00	60.00

Appendix

Table 9: Cross-dataset generalization results on four external datasets, including two real-image anomaly detection benchmarks (MVTec-AD and VisA) and two handwritten/character datasets (MNIST and SCC).

Figure 9: Visualization of the four datasets used in the image sequence settings.

Method	MVTec-AD	VisA	MNIST	SCC
Qwen3-VL-32B	76.00	72.00	62.00	86.00
Qwen3-VL-2B (Baseline)	29.00	32.00	2.00	28.00
GRPO	42.00	38.00	11.00	51.00
OddGrid-GRPO (Ours)	46.00	40.00	16.00	64.00

Appendix

Table 10: Cross-format generalization results under a non-grid setting where the model receives 8–15 independent images and predicts the odd-one-out index.

Figure 10: Visualization of the four datasets used in the row and column settings.

Model	MVTec-AD		VisA
Model	EM	F1	EM	F1
Qwen3-VL-8B	37.18	48.63	29.71	43.30
Qwen3-VL-32B	47.08	58.46	40.00	53.48

Appendix

Table 11: Evaluation on real-world industrial anomaly detection datasets. We report Exact Match (EM) and F1 scores on MVTec-AD and VisA.

Odd Type	$\Delta$ Level 1	$\Delta$ Level 2	$\Delta$ Level 3	$\Delta$ Level 4	$\Delta$ Level 5
Color	53.66	79.07	92.31	97.96	96.55
Position	31.25	28.57	44.26	53.85	63.64
Rotation	38.10	31.48	67.65	70.59	69.44
Size	7.50	25.00	50.00	52.17	67.86

Appendix

Table 12: Accuracy across increasing discrepancy levels on different odd types (Qwen3-VL-32B) Higher

\Delta

levels correspond to larger visual differences.

Odd Type	$\Delta$ Level 1	$\Delta$ Level 2	$\Delta$ Level 3	$\Delta$ Level 4	$\Delta$ Level 5
Color	58.54	69.77	86.54	89.80	94.83
Position	6.25	4.29	9.84	3.85	9.09
Rotation	14.29	16.67	32.35	41.18	33.33
Size	2.50	5.00	10.87	13.04	17.86

Appendix

Table 13: Accuracy across increasing discrepancy levels on different odd types (Gemini-2.5-Pro). Higher

\Delta

levels correspond to larger visual differences.

Odd Type	$\Delta$ Level 1	$\Delta$ Level 2	$\Delta$ Level 3	$\Delta$ Level 4	$\Delta$ Level 5
Color	39.02	53.49	59.62	57.14	67.24
Position	0.00	2.86	8.20	7.69	9.09
Rotation	4.76	22.22	17.65	26.47	36.11
Size	0.00	12.50	4.35	15.22	17.86

Appendix

Table 14: Accuracy across increasing discrepancy levels on different odd types (GPT-5). Higher

\Delta

levels correspond to larger visual differences.

Odd Type	$\Delta$ Level 1	$\Delta$ Level 2	$\Delta$ Level 3	$\Delta$ Level 4	$\Delta$ Level 5
Color	14.63	44.19	51.92	51.02	60.34
Position	9.38	5.71	9.84	19.23	18.18
Rotation	0.00	11.11	14.71	32.35	27.78
Size	0.00	5.00	2.17	0.00	3.57

Appendix

Table 15: Accuracy across increasing discrepancy levels on different odd types (InternVL3.5-38B). Higher

\Delta

levels correspond to larger visual differences.

Figure 11: Example of grid images with row and column indices used in OddGridBench.

Figure 12: Example of model outputs for fine-grained perceptual discrimination on OddGridBench, showing predictions from Qwen3-VL-32B and Gemini-2.5 Pro (Example 1/6).

Figure 13: Example of model outputs for fine-grained perceptual discrimination on OddGridBench, showing predictions from Qwen3-VL-32B and Gemini-2.5 Pro (Example 2/6).

Figure 14: Example of model outputs for fine-grained perceptual discrimination on OddGridBench, showing predictions from Qwen3-VL-32B and Gemini-2.5 Pro (Example 3/6).

Figure 15: Example of model outputs for fine-grained perceptual discrimination on OddGridBench, showing predictions from Qwen3-VL-32B and Gemini-2.5 Pro (Example 4/6).

Figure 16: Example of model outputs for fine-grained perceptual discrimination on OddGridBench, showing predictions from Qwen3-VL-32B and Gemini-2.5 Pro (Example 5/6).

Figure 17: Example of model outputs for fine-grained perceptual discrimination on OddGridBench, showing predictions from Qwen3-VL-32B and Gemini-2.5 Pro (Example 6/6).

Figure 18: Example of model outputs for fine-grained perceptual discrimination on OddGridBench, showing predictions from Qwen3-VL-2B and oddGrid-GRPO trained model (Example 1/3).

Figure 19: Example of model outputs for fine-grained perceptual discrimination on OddGridBench, showing predictions from Qwen3-VL-2B and oddGrid-GRPO trained model (Example 2/3).

Figure 20: Example of model outputs for fine-grained perceptual discrimination on OddGridBench, showing predictions from Qwen3-VL-2B and oddGrid-GRPO trained model (Example 3/3).

Fine-Grained Visual Perception

Fine-grained visual perception studies how models detect, distinguish, and ground subtle visual details, attributes, parts, and changes.

See all 153 papers

Jun 25, 2026

DiCoBench: Benchmarking Multi-Image Fine-Grained Perception via Differential and Commonality Visual Cues

Recent advancements in Multimodal Large Language Models (MLLMs) have demonstrated impressive fine-grained perception capabilities. However, existing benchmarks predominantly rely on explicit textual cues or low-resolution inputs, failing to evaluate a model's ability to autonomously perceive implicit visual cues in high-resolution. To bridge this gap, we introduce DiCoBench, a comprehensive, multi-image high-resolution benchmark designed for cross-image fine-grained perception. DiCoBench consists of 765 meticulously curated samples categorized into two progressive tracks: Differential Visual Cues and Commonality Visual Cues, covering 8 distinct perception tasks. By formulating the benchmark as a multiple-choice question task and utilizing high-resolution imagery (approaching 2K), we eliminate evaluation metric bias and pose a substantial challenge to current state-of-the-art MLLMs. Our extensive evaluation of 18 diverse MLLMs reveals a striking performance gap compared to human accuracy (98.3%), with top-performing models struggling significantly with micro-scale detail capture. We believe DiCoBench will serve as a challenging testbed to drive future research in autonomous, high-resolution multi-image perception.

Geng Li, Yuxin Peng

Wangxuan Institute of Computer Technology, Peking University, Beijing, China

Visual Reasoning Multimodal Large Language Models Multimodal Model Evaluation Fine-Grained Visual Perception

Ask PDF

Jul 27, 2026

PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Models

We introduce PerceptionBench, a benchmark specifically designed to evaluate the atomic visual perception capabilities of Multimodal Large Language Models (MLLMs). Existing benchmarks often fail to isolate perception: holistic evaluations conflate perceptual errors with failures in reasoning or domain knowledge, while application-driven benchmarks only cover narrow, fragmented domains shaped by heuristic designs. To address these limitations, PerceptionBench adopts a bottom-up approach: by diagnosing the earliest failure points in the responses of frontier MLLMs across 42 existing benchmarks, we construct an error taxonomy whose perception branch defines ten atomic perceptual capabilities. Guided by this taxonomy, we construct 3,000 verified questions with short, unambiguous answers, each isolating a single capability, with difficulty stemming from perception rather than reasoning or knowledge. Benchmark results across sixteen frontier MLLMs reveal that atomic perception remains largely unsolved---no model reaches 60% accuracy, perception-related hallucination is the weakest capability on average, and similar overall scores conceal sharply divergent capability profiles. PerceptionBench thus provides a capability-level standard for measuring and diagnosing the visual perception boundaries of MLLMs.

Zichao Lin, Yifeng Xie, Bowen Qu +30

Moonshot AI

Object Hallucination in VLMs Multimodal Large Language Models Multimodal Model Evaluation Fine-Grained Visual Perception

Ask PDF

Sep 5, 2026

VDiff-Bench: A Challenging Benchmark for Fine-Grained Image Difference Identification

Multimodal Large Language Models (MLLMs) perform strongly on general visual understanding tasks such as visual question answering, yet they often struggle with a basic comparative skill: identifying what has changed between two similar images. We introduce VDiff-Bench, a challenging multiple-choice benchmark for fine-grained Image Difference Identification. VDiff-Bench contains 1,756 four-way questions over image pairs and covers 10 change categories: position, motion, regional image color, overall image color, appearance/disappearance, noise/resolution, texture, substitution/size, OCR/text, and illumination. Each question corresponds to two image inputs with 4 choices: the true difference, two hard negative descriptions, and a "no difference" distractor. To make the task challenging, we specifically curate ground-truth-conditioned negatives that require models to distinguish the actual change from nearby semantic alternatives. Experiments with 11 state-of-the-art open- and closed-source MLLMs show that fine-grained visual comparison remains brittle: models exhibit uneven performance across sources and change categories, with persistent failures on subtle low-level changes like noises and textures. For instance, three 7-8B-scale open-source MLLMs score 52.5-70.6% on semantic changes but only 8.7-33.3% on low-level changes like noise and texture, falsely assuming no changes between two image inputs. Surprisingly, despite strong performance of other closed-source commercial models, Grok 4.3 demonstrate remarkable performance drop on identifying noise and texture differences between images, falling significantly behind large open-source models like Kimi K2.5 and K3. Overall, VDiff-Bench provides a targeted diagnostic for evaluating comparative visual understanding in MLLMs, exposing failures that are not captured by standard single-image vision-language tasks.

Yixin Wan, Tianle Zheng, Kai-Wei Chang

Department of Computer Science University of California, Los Angeles

VLM Evaluation Vision-Language Models Fine-Grained Visual Perception

Ask PDF

Abstract

Fine-Grained Visual Perception

Explore similar work

DiCoBench: Benchmarking Multi-Image Fine-Grained Perception via Differential and Commonality Visual Cues

PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Models

VDiff-Bench: A Challenging Benchmark for Fine-Grained Image Difference Identification