VisualWorldBench: A Fine-Grained Multi-Task Benchmark for Evaluating Visual World Knowledge in MLLMs

Chen Chen; Zheming Liang; Jiaqi Wang

cs.CVOct 14, 2026

VisualWorldBench: A Fine-Grained Multi-Task Benchmark for Evaluating Visual World Knowledge in MLLMs

Authors: Chen Chen, Zheming Liang, Jiaqi Wang

Organizations: University of Science and Technology of China · Shanghai Innovation Institute

Abstract

Visual world knowledge is a core capability of multimodal large language models (MLLMs), underpinning knowledge-intensive visual question answering and fine-grained visual reasoning. However, existing benchmarks remain limited in two key respects. First, their category coverage is narrow: most benchmarks span only a small number of categories across a few domains, falling far short of the open-ended scale of real-world visual entities. Second, their task construction is often insufficiently rigorous: models can answer correctly using coarse-grained cues, vague associations, or elimination strategies without truly mastering precise concepts, making it difficult to accurately assess their knowledge of fine-grained entities. We introduce VisualWorldBench, a taxonomy-grounded multi-task benchmark built on a hierarchical taxonomy of 21,291 leaf-node classes across 8 visual domains, with 32,541 evaluation instances whose ground-truth labels cover 15,479 categories across four complementary tasks: closed-set recognition, contrastive selection, precise localization, and open-world naming, covering different directions of knowledge retrieval and varying degrees of answer openness to comprehensively evaluate visual world knowledge. Most questions are constructed from taxonomically proximate categories, forcing precise discrimination among fine-grained entities. Experiments reveal that the strongest closed-source model achieves only 80.4% average accuracy, exposing capability gaps hidden under conventional benchmarks. We further construct VisualWorldBench-50K, a training resource covering 50K+ categories. Fine-tuning on VisualWorldBench-50K yields over 21 percentage points improvement on VisualWorldBench, with consistent transfer to public fine-grained recognition and visual knowledge benchmarks.

Figures & tables

Figure 1: Overview of VisualWorldBench. The benchmark spans 8 visual domains —Animal, Plant, Fungus, Food, Scene, Transport, Artifact, and Other—with a taxonomy of 21,291 leaf-node classes and 32,541 evaluation instances covering 15,479 ground-truth categories . The benchmark evaluates four complementary tasks: closed-set recognition (MCQ), contrastive selection (Grid-Sel), spatial localization (Grid-Loc), and open-world naming (Open-QA).

Fine-grained recognition benchmarks
Benchmark	Year	Coverage	#Classes	#Eval	Tasks	Tax.	MLLM
CUB-200 [ 5 ]	2011	Bird	200	5.8K	Cls	✗	✗
Stanford Dogs [ 6 ]	2011	Dog	120	8.6K	Cls	✗	✗
FGVC-Aircraft [ 7 ]	2013	Aircraft	100	3.3K	Cls	✓	✗
iNat2021 [ 8 ]	2021	Bio	10K	500K	Cls	✓	✗
FG-BMK [ 16 ]	2026	FG	–	1.01M	QA+Ret+Cls	✗	✓

Table 1: Comparison with existing visual world knowledge and fine-grained recognition benchmarks. VisualWorldBench is the only benchmark that combines large-scale fine-grained coverage, multi-domain scope, and multiple complementary task formats in a single MLLM-oriented evaluation. For VisualWorldBench, #Classes counts distinct ground-truth categories across the four tasks; its underlying taxonomy contains 21,291 leaf-node classes. ∗ For InfoSeek-Human, 806 denotes the number of visual entities referenced by the questions, not distinct answer categories. Tax. : hierarchical taxonomy provided. FG : fine-grained.

Figure 2: Taxonomy-grounded construction and four-format evaluation design of VisualWorldBench. (a) Benchmark construction: a hierarchical taxonomy (illustrated with an example subtree), multi-format task generation, and two-stage quality control (model-assisted triage by three frontier models followed by human verification and release balancing). (b) The four evaluation formats—MCQ for closed-set recognition, Grid-Sel for contrastive selection, Grid-Loc for precise localization, and Open-QA for open-world naming—with their task setup, sample counts, and design choices.

Figure 3: Examples of the four VisualWorldBench tasks, all grounded in a shared 21,291-class taxonomy. MCQ (closed-set recognition): the model selects the correct category from 20 taxonomy-aware options (sibling, near, and random distractors). Grid-Sel / Grid-Loc (contrastive selection and precise localization): the model identifies or localizes the target category among visually similar alternatives in a composite grid. Grid-Sel requires selecting the correct cell, while Grid-Loc requires outputting a precise bounding box on the same grid layout. Open-QA (open-world naming): the model generates the fine-grained category name without any candidate set.

Closed-source models
		Sel			Loc
Model	MCQ	2 $\times$ 2	3 $\times$ 3	Avg	2 $\times$ 2	3 $\times$ 3	Avg	Open	Avg
Gemini 3.1 Pro	86.0	94.2	87.3	90.8	84.6	83.3	84.0	60.6	80.4
Doubao-Seed 2.0 Pro	75.1	77.9	71.6	74.8	71.4	69.7	70.6	74.5	73.8
Mimo-V2-Omni	65.5	76.7	68.7	72.7	47.3	44.2	45.8	40.0	56.0
GPT-5.4	67.9	76.4	65.8	71.1	35.1	22.0	28.6	37.4	51.2

Table 2: Main results on VisualWorldBench. MCQ : MCQ accuracy; Sel : Grid-Sel accuracy (2

\times

2 / 3

\times

3 / avg); Loc : Grid-Loc [email protected] (2

\times

2 / 3

\times

3 / avg); Open : Open-QA accuracy (EM+LLM Judge); Avg : arithmetic mean of MCQ, Sel Avg, Loc Avg, and Open-QA. † Trained on VisualWorldBench-50K (Section 3.4 ). Best result per column in bold .

	Public MCQ						iNat21
Model	Bird	Dog	Air.	Flwr.	Pet	Avg	Animal	Plant	Avg	WorldVQA
Qwen3-VL-2B	49.6	57.6	55.0	75.4	79.1	63.3	27.3	30.6	29.0	8.2
Qwen3-VL-4B	55.2	74.0	57.1	85.9	93.0	73.1	35.9	36.1	36.0	10.2
Qwen3-VL-8B	59.3	71.5	60.7	86.8	88.1	73.3	40.2	41.3	40.8	11.2
InternVL3.5-8B	41.7	55.4	42.2	54.1	79.8	54.6	63.0	50.3	56.7	4.8
Qwen3-VL-2B †	62.7 +13.1	75.4 +17.8	73.5 +18.5	89.3 +13.9	88.1 +9.0	77.8 +14.5	51.5 +24.2	38.9 +8.3	45.2 +16.2	13.4 +5.2

Table 3: Generalization to widely used public fine-grained benchmarks. Bird–Pet: 4-option MCQ accuracy on five public fine-grained benchmarks. iNat21 : leaf-level classification accuracy on the animal and plant splits of iNaturalist 2021, plus their average. WorldVQA : open-ended visual QA accuracy (LLM-judged, structured answer format). † Trained on VisualWorldBench-50K.

Figure 4: Capability profiles vary substantially across visual domains, taxonomy depth, and category exposure. (a) MCQ accuracy by visual domain (red

\diamond

: proprietary; blue

\circ

: open-source). Cross-domain gaps exceed 33 pp within individual models. (b) Grid-Sel accuracy by taxonomy depth for four open-source models (Qwen3.5-27B/9B/4B, Qwen3-VL-8B) at 2

\times

2 and 3

\times

3 resolutions. Performance degrades substantially at the deepest levels (7+). (c) Memorization vs. generalization (Qwen3-VL-2B): Base = instruct baseline (no SFT); Holdout = fine-tuned excluding records with held-out ground-truth labels; Full = fine-tuned on the complete SFT data. Selection and localization improve on categories held out from SFT target labels, with smaller gains on MCQ.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Task	Format	#Samples	#Cls	Metric
MCQ	20-option MC	14,717	11,466	Accuracy
Grid-Sel 2 $\times$ 2	Select cell	4,085	3,726	Accuracy
Grid-Sel 3 $\times$ 3	Select cell	4,125	3,785	Accuracy
Grid-Loc 2 $\times$ 2	Output bbox	3,959	3,613	[email protected]
Grid-Loc 3 $\times$ 3	Output bbox	3,446	3,204	[email protected]
Open-QA	Free-form text	2,209	2,187	EM + LLM Judge

Appendix

Table 4: Statistics of VisualWorldBench. #Cls : unique canonical ground-truth names in the released benchmark split. The benchmark is built on a taxonomy of 21,291 leaf-node classes; the released evaluation subset covers 15,479 unique ground-truth categories, while nearly all taxonomy classes appear as distractors. Grid-Loc is evaluated at IoU threshold

\tau{=}0.5

following the visual grounding standard. Grid tasks are evaluated at two spatial resolutions (2

\times

2 and 3

\times

3) to probe fine-grained discrimination at different spatial granularities.

Domain	MCQ
Animal	4,011
Artifact	2,535
Food	2,411
Plant	2,030
Fungus	1,696
Transport	1,664

Appendix

Table 5: Per-domain sample counts in the released benchmark split for MCQ. Grid tasks and Open-QA follow the same 8-domain taxonomy; per-domain breakdowns are omitted for brevity. The released split is intentionally not domain-balanced, so sparse domains such as Scene and especially Other should not be over-interpreted as standalone evaluation subsets.

Option type	Count	Ratio
Sibling	111,866	38.0%
Random	108,327	36.8%
Near	59,430	20.2%
GT	14,717	5.0%

Appendix

Table 6: MCQ option composition in the released benchmark split. Percentages use construction-time option types over all slots, including the ground-truth option; these types are omitted from public mcq.jsonl .

Domain	#Leaf classes	#GT classes	Max depth
Animal	6,620	5,323	13
Plant	2,198	1,786	8
Fungus	1,690	1,492	6
Artifact	4,819	2,753	14
Transport	2,597	1,859	8
Food	2,373	1,853	9

Appendix

Table 7: Per-domain taxonomy statistics. #Leaf classes : total leaf-node categories in the taxonomy. #GT classes : unique ground-truth categories appearing in the released benchmark across all tasks. Nearly all 21,291 leaf classes additionally appear as distractors in MCQ or Grid tasks, even when they are not used as ground-truth labels.

Task	Candidate pool	Release pool	Final	Retention
MCQ	47,697	22,356	14,717	30.9%
Grid-Sel 2 $\times$ 2	19,131	5,559	4,085	21.4%
Grid-Sel 3 $\times$ 3	18,064	5,512	4,125	22.8%
Grid-Loc 2 $\times$ 2	18,681	5,383	3,959	21.2%
Grid-Loc 3 $\times$ 3	15,335	4,606	3,446	22.5%
Open-QA	10,000	6,169	2,209	22.1%

Appendix

Table 8: Quality-control statistics. The Release pool column reports the count after the task-specific intermediate filtering stage (release balancing for MCQ and Grid-Sel, visual QC for Grid-Loc and Open-QA) but before the final image-quality filter. The Final column reports the count after the additional automated image-quality filter.

Task	Audited	Issue	Issue rate	Main issue types
MCQ	200	6	3.0%	ambiguity
Grid-Sel	200	8	4.0%	cell mismatch
Grid-Loc	200	8	4.0%	box misalignment
Open-QA	200	5	2.5%	alias ambiguity
Total	800	27	3.4%

Appendix

Table 9: Random audit of non-flagged samples. 200 samples per task are drawn uniformly at random from the non-flagged pool (seed = 42) and reviewed under the same criteria as the main verification pass. Issue : any sample requiring correction or removal because of label error, ambiguity, non-identifiability, leakage, or corrupted composition.

Model	Family	MCQ	Open	MCQ–Open	MCQ–Loc
Gemini 3.1 Pro	Closed-source	86.0	60.6	25.4	2.7
Qwen3.5-27B	Open-source	59.1	28.7	30.4	2.9
Gemma-4-31B	Open-source	60.5	27.2	33.3	24.4
InternVL3.5-8B	Open-source	35.5	10.3	25.2	28.1
Qwen3-VL-8B †	SFT	80.0	33.9	46.1	15.7

Appendix

Table 10: Sanity check: cross-task capability dissociations across model families. MCQ–Open : difference between closed-set MCQ accuracy and Open-QA accuracy. MCQ–Loc : difference between MCQ accuracy and Grid-Loc 3

\times

3 accuracy. Large gaps persist across all families, confirming that the dissociations are general rather than artifacts of any assisting model.

Format	#Records	#Classes	Primary signal
MCQ	$\sim$ 6.9M	48,170	Closed-set recognition
Reasoning-augmented MCQ	$\sim$ 0.6M	25,376	Fine-grained visual rationale
Bounding-box grounding	$\sim$ 0.4M	31,122	Spatial grounding
Open-ended QA	$\sim$ 0.4M	31,033	Category-name generation
Total	$\sim$ 7.7M	50K+	Multi-format fine-grained supervision

Appendix

Table 11: VisualWorldBench-50K composition. The resource is dominated by MCQ supervision; reasoning-augmented MCQ is a subset of the MCQ records rather than an additional disjoint pool.

Model	GPUs	Epochs	Global batch	Time	GPU-hours
Qwen3-VL-2B	8 $\times$ H200	3	1024	32h	256
Qwen3-VL-4B	16 $\times$ H200	3	1024	30h	480
Qwen3-VL-8B	16 $\times$ H200	3	1024	45h	720
InternVL3.5-8B	16 $\times$ H200	3	1024	41h	656

Appendix

Table 12: Compute resources for the main Qwen3-VL SFT, InternVL3.5 SFT runs. GPU-hours are computed as wall-clock hours multiplied by the number of GPUs.

Metric	Value
Audited samples	200
Human–judge agreement	94.0%
Precision (judge accept)	98.0%
Recall (judge accept)	90.7%
F1	94.2%
Cohen’s $\kappa$	0.880

Appendix

Table 13: Open-QA judge validation on Doubao-Seed-2.0-pro predictions. We sample 200 Gemini 2.5 Pro judge decisions (100 accept, 100 reject) on Open-QA predictions produced by Doubao-Seed-2.0-pro uniformly at random (seed = 42) and independently re-evaluate them with a human annotator. Precision : fraction of judge-accepted predictions that are truly correct. Recall : fraction of truly correct predictions that the judge accepts. Cohen’s

\kappa

follows the standard inter-rater agreement coefficient [ 46 ] .

Format	Before	After	Removed	Removal %
MCQ	6,872K	5,197K	1,675K	24.4%
Bounding-box	435K	305K	130K	30.0%
Open-ended QA	385K	262K	123K	31.9%
Total	7,692K	5,764K	1,928K	25.1%

Appendix

Table 14: Training data filtering for the holdout experiment. Records are removed when their ground-truth label belongs to the held-out category set.

Condition	MCQ	Sel 3 $\times$ 3	Loc 3 $\times$ 3
Instruct baseline	28.0	10.9	20.2
Holdout model	35.1	30.3	41.2
Full model	70.6	31.3	51.6
$\Delta_{\text{H-B}}$	$+$ 7.1	$+$ 19.4	$+$ 21.0

Appendix

Table 15: Holdout experiment results on held-out categories. All numbers are percentages evaluated on released-benchmark instances whose ground-truth labels belong to the held-out category set.

\Delta_{\text{H-B}}

: holdout model gain over instruct baseline.

Explore similar work

WorldBench: A Challenging and Visually Diverse Multimodal Reasoning Benchmark
Jun 4, 2026Yida Yin, Harish Krishnakumar, Chung Peng Lee +9Multimodal Large Language Models Multimodal Reasoning
Princeton University · NYU · University of Waterloo +1
VKnowU: Evaluating Visual Knowledge Understanding in Multimodal LLMs
Nov 25, 2025Tianxiang Jiang, Sheng Xia, Yicheng Xu +5VLM Evaluation Multimodal Large Language Models
University of Science and Technology of China · 2Shanghai AI Laboratory · 3Nanjing University +2
PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Models
Jul 27, 2026Zichao Lin, Yifeng Xie, Bowen Qu +30Object Hallucination in VLMs Multimodal Large Language Models
Moonshot AI

Jun 4, 2026J/K move · Enter open · S save

WorldBench: A Challenging and Visually Diverse Multimodal Reasoning Benchmark

Yida Yin, Harish Krishnakumar, Chung Peng Lee, Boya Zeng+8

Princeton University · NYU · University of Waterloo+1

Multimodal Large Language Models Multimodal Reasoning Visually Grounded Reasoning Multimodal Model Evaluation Multimodal Understanding

Open paper AskPDF arXiv

In real-world applications, models are expected to perform reliably across diverse settings. Yet, many existing multimodal benchmarks expand task types without capturing the visual diversity needed to handle open-ended visual inputs. We present WorldBench, a challenging and visually diverse reasoning benchmark to evaluate Multimodal Large Language Models (MLLMs). We build a taxonomy of thousands of visual concepts across multiple domains (e.g., living things). Guided by this taxonomy, we curate a broad collection of images from search engines and existing datasets to comprehensively represent the visual world. Through structured trial-and-error, we manually design challenging questions that frontier MLLMs fail to answer. On quantitative and human evaluations, WorldBench achieves higher visual diversity than any existing diverse benchmark. Evaluating 15 MLLMs on WorldBench reveals weaknesses in visual understanding: even the strongest model reaches only 64.0% accuracy, while some models perform marginally above chance-level. We hope our work highlights the importance of visual diversity in building multimodal benchmarks.

Abstract

Explore similar work

WorldBench: A Challenging and Visually Diverse Multimodal Reasoning Benchmark

VKnowU: Evaluating Visual Knowledge Understanding in Multimodal LLMs

PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Models