What do MLLMs know about 'Waldo'? Evaluating Visual Reasoning under Multi-Instance Multi-Object Challenges

Saumya Mundra; Ajoy Mondal; Vineeth N Balasubramanian; C. V. Jawahar

cs.CVOct 28, 2026

What do MLLMs know about 'Waldo'? Evaluating Visual Reasoning under Multi-Instance Multi-Object Challenges

Authors: Saumya Mundra, Ajoy Mondal, Vineeth N Balasubramanian, C. V. Jawahar

Organizations: IIIT Hyderabad · Microsoft Research India · IIT Hyderabad

Abstract

Recent advances in Multimodal Large Language Models (MLLMs) have led to significant progress in visual reasoning capabilities. However, existing benchmarks curate image-question pairs pertaining to object categories with few instances, enabling trivial object referencing and limited visual ambiguity. In this work, we introduce MIMO-Bench, a benchmark for evaluating MLLMs in a Multi-Instance per Multi-Object categories (MIMO) setting. Each image-question pair in our benchmark pertains to instances belonging to object categories with multiple instances. The presence of such similar feature distractors causes hindrance in basic visual reasoning tasks, necessitating the referring expression to be composed of multiple visual subtasks to uniquely and unambiguously identify the target object. MIMO-Bench comprises 3,802 image-question pairs across attribute binding and numerical counting tasks. Despite impressive performances on existing benchmarks, both SOTA open-source (InternVL, QwenVL) and proprietary (Gemini, GPT-5) models exhibit substantially lower performance on MIMO-Bench, highlighting their limitations in the MIMO setting. The benchmark will be made publicly available.

Figures & tables

Figure 1: Existing vs. MIMO setting. MIMO introduces dense multi-instance scenes with more visual distractors and richer referring expressions.

Lineage	Benchmark(s)	Real/Syn	Template	Comp. REs	Image Res.	Distractor	Mean Distr.	# subtasks
Compositional	CLEVR ( Johnson et al., 2017 )	✗ / ✓	✓	✓	✗ / ✓	✗	1.74	7.2
Compositional	GQA ( Hudson and Manning, 2019 )	✓ / ✗	✓	✓	✗ / ✓	✗	1.67	3.1
Vision-centric (general)	MMStar, NaturalBench, VQAv2 ( Goyal et al., 2017 )	✓ / ✗	✗	✗	✗ / ✓	✗	2.49/2.11/1.46	3.9/2.7/2.6
Visual difficulty	V * Bench	✓ / ✗	✗	✗	✓ / ✗	$\sim$	0.12	2.4
Visual difficulty	HR-Bench, MME-RealWorld	✓ / ✗	✗	✗	✓ / ✗	$\sim$	2.01/6.54	4.7/3.9
Referring expression	Ref-Adv ( Dong et al., 2026 )	✓ / ✗	✗	✓	✗ / ✓	✓	4.01 †	6.6

Table 1: Positioning MIMO-Bench. Each row places a benchmark group along axes characterizing its construction: image source (real/synthetic), template-based question generation, compositional referring expressions (REs), image resolution (Hi/Lo), and if same-category distractor density is a controlled design axis ( ✓ ), incidental (

\sim

), or not considered ( ✗ ). # subtasks is the mean number of visual subtasks per query. Mean Distr. is the mean number of same-category instances per target, excluding the target itself; unless noted otherwise, values are computed using our SAM3-based protocol. † For Ref-Adv, we use the average same-category distractor count reported by its authors.

Figure 2: Dataset statistics for MIMO-Bench. Distributions of RE length and composition (left), and per-category instance counts over images containing each category (right).

Figure 3: MIMO-Bench Creation Pipeline : AB questions are generated by mapping instance annotations to the attribute dictionary, producing raw questions. NC questions are generated from REs describing groups. Human answering and verification retain only visually grounded and unambiguous QA pairs.

Figure 4: AB (left) and NC (right) tasks centered on a target instance or target group within an image.

Method	Variant	Attribute Binding (AB)				Numerical Counting (NC)
		Real		Syn.		Real			Syn.
		$\mathcal{I}_{\mathrm{ACC}}$	$\mathcal{G}_{\mathrm{ACC}}$	$\mathcal{I}_{\mathrm{ACC}}$	$\mathcal{G}_{\mathrm{ACC}}$	$\mathcal{I}_{\mathrm{MRA}}$	$\mathcal{L}_{\mathrm{MRA}}$	$\mathcal{G}_{\mathrm{MRA}}$	$\mathcal{I}_{\mathrm{MRA}}$	$\mathcal{L}_{\mathrm{MRA}}$	$\mathcal{G}_{\mathrm{MRA}}$
Chance & Human Baselines
Chance(Random)		15.8	15.6	15.8	15.3	–
Chance(Majority)		34.3	35.5	34.3	33.1	–
Human		89.3	86.7	85.7	87.5	87.7	86.4	90.1	89.6	87.4	92.0

Table 2: Evaluation results on MIMO-Bench, with AB and NC reported separately for real and synthetic splits; scores are percentages. Baselines : random chance is the mean

1/N

over answer options, and majority guess picks the most common answer per attribute type. “Low”, “medium” & “minimal” are reasoning efforts for the respective models. Abbreviations: Syn. = synthetic; – = NA

Model	MIMO Bench	Crop 768	Crop 512	Existing Avg. †
Q3-32B-I	47.5	55.0	48.9	81.4
Q3-30B-A3B-I	33.8	42.2	44.9	77.3
Q3-8B-I	40.3	48.9	44.4	75.6
Q2.5-72B-I	42.6	45.7	48.3	78.3
Q2.5-32B-I	39.3	48.5	45.0	74.7
Q2.5-7B-I	35.8	42.8	47.6	71.7

Table 3:

\mathcal{I}_{\mathrm{ACC}}

on MIMO-Bench and MIMO-Crop 512/768, and standard low-res benchmarks. † Mean of MMStar, RealWorldQA, and MM-Bench. I: Instruct variant.

Model	Thk.	$\mathcal{I}_{\mathrm{ACC}}$	$\mathcal{G}_{\mathrm{ACC}}$	$\mathcal{I}_{\mathrm{MRA}}$	$\mathcal{L}_{\mathrm{MRA}}$	$\mathcal{G}_{\mathrm{MRA}}$
Q3-8B	w/o	39.2	40.1	43.9	40.0	37.3
Q3-8B	w/	44.4	44.8	47.3	62.3	43.2
Q3-30B-A3B	w/o	35.2	32.3	34.0	39.7	30.5
Q3-30B-A3B	w/	37.0	37.3	35.4	40.1	31.3
Q3-32B	w/o	46.3	46.0	46.3	46.1	50.4
Q3-32B	w/	46.2	43.5	48.8	48.9	44.2

Table 4: Effect of thinking/reasoning. (“w/o”,“w/”) denotes (Instruct/Thinking) variant for Qwen and (Low/High) reasoning effort for Gemini.

Panel of Figure 5 — Figure 5: Visual prompting setup. Highlighting the target with a bounding box (a) instead of an RE consistently improves $\mathcal{I}_{\text{ACC}}$ (b).

Figure 6: Effect of visual prompting on

\mathcal{I}_{\mathrm{ACC}}

Panel of Figure 7 — Figure 7: Visual Distractors Effect: (left) Graying setup & (right) Performance trends as distractors increase.

Model	Zoom.	$\bm{\Delta}$
Q3-8B-Instruct	Naive	$+0.89$
Q2.5-7B-Instruct	Naive	$+1.50$
IVL3-8B	Naive	$+5.46$
IVL3-14B	Naive	$+16.90$
IVL3.5-8B	Naive	$+2.14$
IVL3.5-14B	Naive	$+11.75$

Table 5: Effect of zooming on model performance.

\Delta

denotes the change in

\mathcal{I}_{\text{ACC}}

relative to MIMO-Bench.

Figure 8: CoT on MIMO-Bench.

Appendix figures & tables44 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 9: Example prompts and corresponding generated synthetic images for MIMO-Bench. We consistently swap aesthetic details to gather varying images characterized by the MIMO setting.

Model	Syn. AB $\mathcal{I}_{\mathrm{ACC}}$
G2.5-Pro	47.7
GPT-5-medium	49.7
IVL3.5-241B-A28B	45.1
Q3-32B-Instruct	47.9

Appendix

Table 6: Self-preference check on Gemini-generated images. Attribute-binding instance accuracy on the synthetic split. G2.5-Pro does not exhibit a clear self-preference advantage despite being used to generate the synthetic images.

Figure 10: Qualitative comparison of synthetic images generated by Grok-4, Flux, and G2.5-Pro under the same MIMO-style prompts. G2.5-Pro produces more realistic, feature-diverse scenes with stronger prompt adherence. Grok-4 and Flux exhibit reduced realism, weaker feature diversity, and object-level artifacts, making them less suitable for constructing unambiguous MIMO-style QA pairs.

Figure 11: Pre-prompt used for generating attribute-binding dictionaries for different object categories (1/2).

Figure 12: Pre-prompt used for generating attribute-binding dictionaries for different object categories (2/2).

Panel of Figure 13 — Figure 13: Person attribute dictionary in JSON format.

Panel of Figure 14 — Figure 14: Plant attribute dictionary in JSON format.

Benchmark	Objects	Attr	Relations	# Subtasks
CLEVR	2.50	3.50	1.20	7.20
GQA	2.24	0.42	0.43	3.10
MMStar	3.36	0.08	0.46	3.90
NaturalBench	1.87	0.53	0.30	2.70
VQAv2	1.76	0.35	0.49	2.60
V * Bench	1.92	0.07	0.40	2.40

Appendix

Table 7: Referring-expression composition across benchmarks, parsed with the same benchmark-agnostic parser. Objects, Attr bindings and relations are the three terms of Eq. 4 ; same-category distractor density is reported in Table 1 .

Figure 17: Pre-prompt for parsing referring expressions into attributes/actions, relations, and plain objects (1/3).

Panel of Figure 19 — Figure 19: Human annotation interface examples used for referring-expression verification and QA filtering. Panels show target verification with and without bounding boxes, invalid and valid question cases, ambiguous questions, and non-answerable attributes.

Figure 20: Sample question dictionary of attribute binding getting modified from raw question answers to the final set of questions, having gone through human answering and verification.

Figure 21: Sample question dictionary for numerical counting task. A parent node will have a non-empty “children" list as seen for nodes $1$ and $6$ .

Panel of Figure 22 — Figure 22: MIMO-Crop examples. Left: a feasible crop where the target and referenced neighboring object are both retained. Right: a rejected case where no crop can preserve all objects mentioned in the referring expression.

Panel of Figure 23 — Figure 23: Prompt used for attribute-binding questions.

Model	Exact group (%)	Model	Exact group (%)
Gemini-2.5-Flash	33.33%	InternVL3.5-8B	3.92%
Qwen3-VL-8B-Thinking	15.83%	InternVL3.5-38B	3.61%
InternVL3.5-241B-A28B	11.75%	InternVL3-8B	3.55%
Qwen3-VL-32B-Thinking	9.78%	InternVL3-14B	3.41%
Gemini-2.5-Pro	9.39%	Qwen3-VL-8B-Instruct	3.13%
Qwen2.5-VL-72B-Instruct	6.94%	Qwen3-VL-30B-A3B-Thinking	2.70%

Appendix

Table 8: Exact Group Accuracy (%): the percentage of target groups for which all valid attribute questions are answered correctly.

Model	sign( $\Delta_{GL}$ )	G $>$ L (%)	L $>$ G (%)
G2.5-Flash	$+$	42.1	43.9
G2.5-Pro	$+$	61.4	29.8
GPT-5-medium	$+$	59.6	31.6
Q3-32B-Instruct	$-$	31.6	54.4
Q3-8B-Instruct	$-$	33.3	52.6
Q3-30B-A3B-Instruct	$-$	29.8	56.1

Appendix

Table 9: Parent-level comparison between global and localized counting on the parent groups. “G

>

L” and “L

>

G” give the percentage of parent groups where the global score is strictly greater or strictly smaller than the localized child average, respectively. sign(

\Delta_{GL}

) reports the direction of the gap

\mathcal{G}_{\mathrm{MRA}}-\mathcal{L}_{\mathrm{MRA}}

over the real and synthetic sets.

\sim

denotes a near-tie. A negative sign together with L

>

G % exceeding G

>

L % indicates that localized decomposition helps.

Model	Exact Cons. (%)	MAE	Bias
IVL3.5-241B-A28B	72.7	1.00	-1.00
Q3-32B-Thinking	50.0	0.75	-0.35
G3.1-Pro-Preview-Low	45.3	1.55	-0.19
G3.5-Flash-Low	45.3	1.34	-0.36
G3.5-Flash-High	43.4	1.58	-1.02
Q3-8B-Thinking	42.9	0.86	-0.86

Appendix

Table 10: Hierarchical consistency of numerical-counting predictions. Negative bias means the predicted parent count is smaller than the sum of predicted child counts.

GT	$\mathcal{MRA}$	Exact	Bias	MAE	Off-by-1	Off-by-2+
0	0.3333	0.3333	+1.0833	1.0833	0.6111	0.3889
1	0.4069	0.4069	+0.7430	0.9272	0.8223	0.1777
2	0.3555	0.3555	+0.4947	1.1089	0.6838	0.3162
3	0.4968	0.3541	+0.5889	1.4324	0.7109	0.2891
4	0.4263	0.2564	-0.5406	1.4124	0.5962	0.4038
5	0.5234	0.2888	-0.6231	1.6261	0.6231	0.3769

Appendix

Table 11: Full count-conditioned numerical-counting statistics aggregated across all available model predictions. Rows are grouped by ground-truth (GT) target-set size. We report mean

\mathcal{MRA}

, exact-match rate, signed bias

(\hat{y}-y)

, mean absolute error (MAE), off-by-one rate

(|\hat{y}-y|\leq 1)

, and off-by-two-or-more rate

(|\hat{y}-y|\geq 2)

. Positive bias indicates overcounting and negative bias indicates undercounting. Per-GT sample frequencies are intentionally omitted for compactness.

Evaluated MLLMs
Method	$512\times 512$	$1024\times 1024$	MIMO-Bench	WVP
	$\mathcal{I}_{\mathrm{ACC}}$	$\mathcal{I}_{\mathrm{ACC}}$	$\mathcal{I}_{\mathrm{ACC}}$	$\mathcal{I}_{\mathrm{ACC}}$
Qwen3-VL
32B	65.5	56.8	46.3	64.3
30B-A3B	52.9	45.2	35.2	59.2
8B	56.1	50.6	39.2	64.7

Appendix

Table 12: Comparison of MLLM performance on the subset used for visual-prompting and graying ablations. The

512\times 512

1024\times 1024

, and MIMO-Bench columns correspond to the graying setup with progressively more visible distractors, while WVP reports performance with visual prompting. † We could not obtain a WVP result for IVL3-78B because of API server issues.

Panel of Figure 25 — Figure 25: Compact view of how referring-expression composition relates to standard accuracy and absolute improvement under visual prompting.

Figure 26: Percentage gain with visual prompting.

Model	Grounding	$\mathbf{G^{+}A^{+}}$	$\mathbf{G^{+}A^{-}}$	$\mathbf{G^{-}A^{+}}$	$\mathbf{G^{-}A^{-}}$
Qwen3-VL-8B	Box-choice	32.6	35.8	12.9	18.7
	Coordinates	14.5	9.9	33.3	42.2
Qwen2.5-VL-7B	Box-choice	19.7	36.8	13.7	29.8
	Coordinates	2.6	3.2	34.5	59.7
InternVL3-8B	Box-choice	23.2	27.6	21.6	27.6
	Coordinates	2.5	2.2	45.8	49.4

Appendix

Table 13: Explicit RE-grounding outcomes and their correspondence with AB answers on matched examples. Cells are percentages.

G^{+}

G^{-}

denote correct/incorrect grounding and

A^{+}

A^{-}

correct/incorrect AB answers.

Model	Grounding	Isolated G-Acc.	$\mathbf{P(A^{+}\mid G^{+})}$	$\mathbf{P(A^{+}\mid G^{-})}$	$\bm{\Delta}$
Qwen3-VL-8B	Box-choice	59.3	47.6	40.8	+6.9
	Coordinates	25.1	59.4	44.1	+15.3
Qwen2.5-VL-7B	Box-choice	53.8	34.8	31.4	+3.4
	Coordinates	8.3	44.5	36.6	+7.9
InternVL3-8B	Box-choice	52.9	45.7	44.0	+1.7
	Coordinates	9.1	53.2	48.1	+5.1

Appendix

Table 14: Grounding-only accuracy and conditional AB accuracy on the same matched examples. Isolated G-Acc. comes from a separate grounding-only run.

\Delta=P(A^{+}\mid G^{+})-P(A^{+}\mid G^{-})

. Joint box-choice AB values use an overlaid image and are not directly comparable with the main benchmark

\mathcal{I}_{\mathrm{ACC}}

values.

Panel of Figure 27 — Figure 27: Graying experiment diagnostics. The first panel summarizes average distractor count; the second shows that the accuracy gain per removed distractor varies substantially across images and window sizes.

Model	Base	Gray 512	Blur 512	Inpaint 512
Qwen3-VL-8B	39.2	+16.9	+10.1	+0.0
InternVL3-14B	36.1	+17.1	+12.7	+18.4
InternVL3.5-14B	35.4	+7.1	+10.1	+12.1
InternVL3-8B	37.3	+6.7	+6.3	+3.5
InternVL3.5-8B	39.2	+7.9	+4.2	+5.5
Mean	37.4	+11.1	+8.7	+7.9

Appendix

Table 15: Distractor-reduction interventions at

512\times 512

on the 800-sample subset. Base is unmodified-image

\mathcal{I}_{\mathrm{ACC}}

; intervention entries are changes in percentage points. Qwen2.5-VL-7B was not part of the graying comparison and is omitted.

Model	Base	Gray 1024	Blur 1024	Inpaint 1024
Qwen3-VL-8B	39.2	+11.4	+6.6	+0.0
InternVL3-14B	36.1	+12.2	+9.9	+16.6
InternVL3.5-14B	35.4	+1.6	+6.5	+12.8
InternVL3-8B	37.3	+1.5	+1.9	+2.4
InternVL3.5-8B	39.2	+2.3	+4.1	+4.8
Mean	37.4	+5.8	+5.8	+7.3

Appendix

Table 16: Distractor-reduction interventions at

1024\times 1024

on the same subset. Entries are changes in percentage points from the same unmodified-image baseline.

Control	$\Delta$ vs. Base	Target visible	Target absent	Visibility $\Delta$
Fixed 1024	+1.0	49.7	34.1	+15.6
Random 1024	+0.8	53.5	31.9	+21.6
Fixed 512	-5.3	58.3	33.2	+25.1
Random 512	-8.7	49.0	31.3	+17.7

Appendix

Table 17: Fixed and random crop controls on the 800-sample intervention subset.

\Delta

is the change from the matched full-image baseline; visibility columns condition accuracy on whether the target remains in the crop.

Model	Base	MIMO-Crop 512
InternVL3-14B	37.2	50.7
InternVL3-8B	35.9	46.8
InternVL3.5-8B	40.5	50.7
Qwen3-VL-32B	47.5	48.9
Qwen3-VL-8B	40.3	44.4
Qwen2.5-VL-7B	35.8	47.6

Appendix

Table 18: Full-benchmark performance with the externally supplied MIMO-Crop 512 relevant region. This is not a matched baseline for the 800-sample predicted-crop analysis.

Model	Base	0–25	25–50	50–75	75–100
InternVL3-14B	36.1	48.8	51.6	56.5	52.0
InternVL3-8B	37.3	35.4	44.9	51.5	51.0
InternVL3.5-8B	39.2	37.2	42.7	36.4	45.8
Qwen3-VL-32B	46.3	41.2	45.5	49.0	52.9
Qwen3-VL-8B	39.2	39.3	45.5	39.0	50.8
Qwen2.5-VL-7B	35.8	29.7	40.5	28.6	39.2

Appendix

Table 19: Final-answer accuracy conditioned on predicted-crop IoU with the ground-truth target box. Base and IoU bins use the 800-sample intervention subset. All values are percentages.

Model	MIMO-Crop 512	Zoom $\Delta$
InternVL3-14B	50.7	+16.90
InternVL3-8B	46.8	+5.46
InternVL3.5-8B	50.7	+2.14
Qwen3-VL-32B	48.9	+3.0
Qwen3-VL-8B	44.4	+0.89
Qwen2.5-VL-7B	47.6	+1.50

Appendix

Table 20: Relevant-region reference and zooming gain for the same models. MIMO-Crop 512 is evaluated on the full benchmark and is therefore an external reference rather than a matched subset baseline. Zoom

\Delta

is the change under naive model-driven relevant-region extraction.

Model	$\Delta$
Qwen2.5-VL-7B	+1.0
Qwen3-VL-8B	+1.5
InternVL3-8B	+2.0
InternVL3.5-8B	+1.1
InternVL3-14B	-1.5
InternVL3.5-14B	+3.5

Appendix

Table 21: Two-pass no-crop control on the 800-sample intervention subset.

\Delta

is the change in

\mathcal{I}_{\mathrm{ACC}}

from the matched single-pass full-image condition.

Model	Sample	Object	Image
Gemini-3.5 Flash Low	54.7 [52.0, 57.3]	54.7 [51.6, 57.9]	54.7 [49.7, 62.3]
Gemini-3.1 Pro Preview Low	55.0 [52.3, 57.7]	55.0 [51.8, 58.3]	55.0 [50.0, 62.4]
Gemini-2.5 Flash	50.2 [48.1, 52.4]	50.2 [47.8, 52.7]	50.2 [47.3, 53.7]
Gemini-2.5 Pro	48.5 [46.7, 50.3]	48.5 [46.3, 50.6]	48.5 [45.5, 51.6]
GPT-5 minimal	30.7 [26.8, 35.2]	30.7 [28.9, 33.1]	30.7 [29.1, 32.2]
GPT-5 medium	51.8 [49.9, 53.6]	51.8 [49.6, 53.9]	51.8 [48.6, 55.0]

Appendix

Table 22: Attribute-binding

\mathcal{I}_{\mathrm{ACC}}

point estimates and 95% bootstrap intervals under sample-, object-, and image-level resampling.

Model	Group	Image
Gemini-3.5 Flash Low	62.3 [59.0, 65.6]	62.3 [56.0, 69.7]
Gemini-3.1 Pro Preview Low	61.8 [58.4, 65.3]	61.8 [54.1, 69.3]
Gemini-2.5 Flash	46.3 [43.7, 48.9]	46.3 [43.9, 51.4]
Gemini-2.5 Pro	46.9 [44.2, 49.5]	46.9 [43.7, 55.3]
GPT-5 minimal	27.2 [24.3, 31.2]	27.2 [23.1, 30.2]
GPT-5 medium	50.4 [47.9, 52.9]	50.4 [46.4, 54.4]

Appendix

Table 23: Attribute-binding

G_{\mathrm{ACC}}

point estimates and 95% bootstrap intervals under target-group and image-level resampling.

Model	Sample	Group	Image
Gemini-3.5 Flash Low	61.5 [56.3, 66.5]	61.5 [54.3, 68.3]	61.5 [49.7, 72.1]
Gemini-3.1 Pro Preview Low	61.3 [56.0, 66.5]	61.3 [54.4, 67.9]	61.3 [50.3, 71.6]
Gemini-2.5 Flash	40.8 [35.0, 46.5]	40.8 [34.2, 47.5]	40.8 [30.9, 51.2]
Gemini-2.5 Pro	49.0 [43.5, 54.5]	49.0 [42.0, 56.0]	49.0 [39.9, 57.9]
GPT-5 minimal	44.0 [38.6, 49.3]	44.0 [37.4, 50.6]	44.0 [34.5, 53.5]
GPT-5 medium	49.0 [43.8, 54.2]	49.0 [42.8, 55.3]	49.0 [41.1, 56.6]

Appendix

Table 24: Numerical-counting

\mathcal{I}_{\mathrm{MRA}}

point estimates and 95% bootstrap intervals under sample-, parent-group-, and image-level resampling.

Model	Group	Image
Gemini-3.5 Flash Low	60.8 [52.2, 68.9]	60.8 [47.6, 73.0]
Gemini-3.1 Pro Preview Low	60.5 [52.2, 68.3]	60.5 [51.0, 70.1]
Gemini-2.5 Flash	37.1 [28.9, 45.5]	37.1 [27.0, 48.0]
Gemini-2.5 Pro	44.0 [35.7, 52.6]	44.0 [32.2, 55.8]
GPT-5 minimal	42.9 [35.1, 50.7]	42.9 [32.8, 52.9]
GPT-5 medium	46.7 [40.0, 53.7]	46.7 [37.7, 55.9]

Appendix

Table 25: Numerical-counting

\mathcal{L}_{\mathrm{MRA}}

point estimates and 95% bootstrap intervals under parent-group and image-level resampling.

Model	Sample	Image
Gemini-3.5 Flash Low	60.7 [52.5, 68.5]	60.7 [50.1, 70.0]
Gemini-3.1 Pro Preview Low	63.8 [55.6, 71.7]	63.8 [53.2, 73.0]
Gemini-2.5 Flash	47.3 [37.1, 57.5]	47.3 [32.7, 61.4]
Gemini-2.5 Pro	66.4 [58.0, 74.5]	66.4 [56.8, 76.1]
GPT-5 minimal	43.8 [34.8, 52.8]	43.8 [31.8, 55.8]
GPT-5 medium	55.1 [46.1, 63.8]	55.1 [42.2, 67.2]

Appendix

Table 26: Numerical-counting

\mathcal{G}_{\mathrm{MRA}}

point estimates and 95% bootstrap intervals under sample- and image-level resampling.

Model	$\Delta$ $\mathcal{I}_{\mathrm{ACC}}$	95% CI of $\Delta$
Qwen3-VL-8B-Instruct	+25.5	[19.84, 31.59]
GLM-4.5V	+18.5	[13.42, 23.96]
Qwen3-VL-32B-Instruct	+18.0	[13.85, 22.12]
Qwen2.5-VL-32B-Instruct	+22.5	[18.11, 26.98]
Qwen3-VL-30B-A3B-Instruct	+24.0	[16.20, 29.15]
InternVL3-38B	+10.8	[6.95, 14.97]

Appendix

Table 27: Paired change in

\mathcal{I}_{\mathrm{ACC}}

under visual prompting on the matched 800-sample subset. All intervals exclude zero.

Model	$\Delta\mathcal{I}_{\mathrm{ACC}}$ (95% CI)	$\Delta G_{\mathrm{ACC}}$ (95% CI)
Q3-8B	+5.2 [+0.8, +9.9]	+4.7 [-2.7, +12.1]
Q3-30B-A3B	+1.8 [-0.9, +4.4]	+5.0 [+1.9, +7.6]
Q3-32B	-0.1 [-4.4, +7.4]	-2.5 [-9.5, +10.5]
G3.1-Pro-Preview	-0.6 [-2.5, +1.1]	+0.1 [-2.4, +2.4]
G3.5-Flash	+3.6 [+0.6, +5.7]	-1.9 [-4.8, +0.4]

Appendix

Table 28: Paired reasoning-mode changes and 95% bootstrap intervals for attribute binding. For Qwen models,

\Delta

is Thinking minus Instruct; for Gemini models, it is High minus Low reasoning effort. Bold entries have intervals excluding zero.

Model	$\Delta\mathcal{I}_{\mathrm{MRA}}$ (95% CI)	$\Delta\mathcal{L}_{\mathrm{MRA}}$ (95% CI)	$\Delta\mathcal{G}_{\mathrm{MRA}}$ (95% CI)
Q3-8B	+3.4 [-6.0, +13.5]	+22.3 [+11.2, +36.2]	+5.9 [-4.4, +15.3]
Q3-30B-A3B	+1.4 [-4.5, +3.1]	+0.4 [-3.9, +3.3]	+0.8 [-6.2, +7.6]
Q3-32B	+2.5 [-8.9, +11.9]	+2.8 [-10.4, +14.3]	-6.2 [-23.1, +7.6]
G3.1-Pro-Preview	-4.8 [-8.6, +1.3]	-5.9 [-12.4, +4.1]	+1.2 [-10.7, +11.9]
G3.5-Flash	+6.4 [+3.2, +10.6]	+13.3 [+9.3, +18.0]	+5.0 [+0.4, +12.3]

Appendix

Table 29: Paired reasoning-mode changes and 95% bootstrap intervals for numerical counting, using the same reasoning-mode comparison. Bold entries have intervals excluding zero.

Model	$\Delta$ ZS-CoT vs. Base (95% CI)	$\Delta$ SC-CoT vs. Base (95% CI)
IVL3-14B	+4.0 [+0.8, +7.5]	+6.0 [-2.3, +11.6]
IVL3-8B	-1.0 [-4.0, +2.3]	-1.0 [-5.1, +6.5]
IVL3.5-14B	+1.0 [-4.3, +5.4]	+0.0 [-14.2, +9.1]
IVL3.5-8B	+7.0 [+4.3, +9.6]	+3.0 [-4.8, +8.3]
QVL2.5-7B	+0.0 [-4.9, +6.0]	+1.0 [-14.8, +9.6]
QVL2.5-32B	-2.0 [-13.2, +6.7]	-3.0 [-10.1, +2.2]

Appendix

Table 30: Paired changes under zero-shot and self-consistency chain-of-thought prompting relative to the same base condition. Bold entries have 95% intervals excluding zero.

Metric	Human	95% CI
Real $\mathcal{I}_{\mathrm{ACC}}$	89.3	[81.7,93.2]
Real $G_{\mathrm{ACC}}$	86.7	[83.2,89.8]
Syn. $\mathcal{I}_{\mathrm{ACC}}$	85.7	[77.1,89.6]
Syn. $G_{\mathrm{ACC}}$	87.5	[84.9,90.1]
Real $\mathcal{I}_{\mathrm{MRA}}$	87.7	[85.1,88.2]
Real $\mathcal{L}_{\mathrm{MRA}}$	86.4	[83.2,88.5]

Appendix

Table 31: Human baseline point estimates and group-resampled 95% bootstrap intervals. The lowest lower bound (77.1%) remains above the strongest corresponding model result.

MLLM Evaluation

See all 40 papers

May 25, 2026

Can MLLMs Reason Beyond Language? VisReason: A Comprehensive Benchmark for Vision-Centric Reasoning

Recent multimodal large language models (MLLMs) achieve strong performance on visual reasoning benchmarks, yet it remains unclear to what extent such performance reflects reasoning directly grounded in visual evidence. We introduce VisReason, a benchmark for vision-centric reasoning in everyday scenarios where perception and inference are tightly coupled. VisReason contains 1,505 questions across 10 categories spanning perceptual, structural, and conceptual reasoning. Our evaluation shows that VisReason poses a qualitatively different challenge from existing benchmarks, exposing substantial gaps between humans and current MLLMs and revealing limited benefits from test-time reasoning strategies. VisReason offers a focused diagnostic for evaluating vision-centric reasoning beyond language.

Longteng Guo, Yifan Wang, Pengkang Huo +4

Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences

Visually Grounded Reasoning Visual Reasoning VLM Reasoning Multimodal Model Evaluation Multimodal Understanding Reasoning for VLMs+1

Ask PDF

Jun 4, 2026

WorldBench: A Challenging and Visually Diverse Multimodal Reasoning Benchmark

In real-world applications, models are expected to perform reliably across diverse settings. Yet, many existing multimodal benchmarks expand task types without capturing the visual diversity needed to handle open-ended visual inputs. We present WorldBench, a challenging and visually diverse reasoning benchmark to evaluate Multimodal Large Language Models (MLLMs). We build a taxonomy of thousands of visual concepts across multiple domains (e.g., living things). Guided by this taxonomy, we curate a broad collection of images from search engines and existing datasets to comprehensively represent the visual world. Through structured trial-and-error, we manually design challenging questions that frontier MLLMs fail to answer. On quantitative and human evaluations, WorldBench achieves higher visual diversity than any existing diverse benchmark. Evaluating 15 MLLMs on WorldBench reveals weaknesses in visual understanding: even the strongest model reaches only 64.0% accuracy, while some models perform marginally above chance-level. We hope our work highlights the importance of visual diversity in building multimodal benchmarks.

Yida Yin, Harish Krishnakumar, Chung Peng Lee +9

Princeton University · NYU · University of Waterloo +1

Multimodal Large Language Models Multimodal Reasoning Visually Grounded Reasoning Multimodal Model Evaluation Multimodal Understanding

Ask PDF

Feb 13, 2025

ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models

Large Multimodal Models (LMMs) exhibit shortfalls when interpreting images and, by some measures, have poorer spatial cognition than young children or animals. Despite this, they attain high scores on many popular visual benchmarks, with headroom rapidly eroded by model progress. This creates a need for difficult benchmarks that remain relevant for longer. We introduce ZeroBench - a lightweight visual reasoning benchmark curated using adversarial filtering to be "impossible" for frontier LMMs at its original release, with initial SotA scores of 0% pass@1 and pass^5. We track progress on ZeroBench over the subsequent year, observing SotA reaching 6% pass^5 and 19% pass@5, indicating the potential longevity of the benchmark. We evaluate 46 LMMs on ZeroBench, compare performance to a human baseline, analyse strengths and weaknesses, chart a year of progress in visual capabilities, and publicly release ZeroBench at https://zerobench.github.io.

Jonathan Roberts, Mohammad Reza Taesiri, Ansh Sharma +31

University of Cambridge

Visually Grounded Reasoning VLM Reasoning Multimodal Understanding Visual Reasoning Multimodal Foundation Models Multimodal Model Evaluation+1

Ask PDF

Abstract

MLLM Evaluation

Explore similar work

Can MLLMs Reason Beyond Language? VisReason: A Comprehensive Benchmark for Vision-Centric Reasoning

WorldBench: A Challenging and Visually Diverse Multimodal Reasoning Benchmark

ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models