Organizations: Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · OPPO AI Center, OPPO Inc.
We propose VC-VQA, a large-scale benchmark for vision-centric knowledge reasoning, aiming to systematically evaluate the capabilities of Multimodal Large Language Models (MLLMs) in fine-grained visual knowledge acquisition and complex multi-stage reasoning. Unlike existing benchmarks that primarily treat visual information as entity recognition and rely on text-based external knowledge, VC-VQA explicitly models the role of visual information by considering both its source (internal vs. external images) and stage of involvement (pre-retrieval vs. post-retrieval). Specifically, VC-VQA requires models to retrieve external visual evidence, identify visual clues before retrieval, and localize, verify, or match retrieved knowledge against the internal image. We further construct a controlled multimodal knowledge base with traceable textual and visual evidence for each sample. We evaluate 21 representative MLLMs under four settings: no external knowledge, with provided evidence, knowledge base retrieval, and open-web search. Experimental results lead to three key findings: current models still struggle with vision-centric knowledge reasoning; fine-grained visual knowledge acquisition remains the primary bottleneck; and agentic retrieval highlights the value of multi-step planning and cross-modal knowledge integration. Overall, VC-VQA provides a more challenging benchmark for evaluating vision-centric knowledge reasoning in MLLMs. Our code and dataset are available at https://github.com/CASIA-IVA-Lab/VC-VQA.
Figures & tables
Figure 1: Comparison between previous work and our benchmark.
OKVQA
KVQA
InfoSeek
EVQA
Dyn-VQA
M 3 -VQA
MM-BrowseComp
VC-VQA
Fine-Grained Entity
×
✓
✓
✓
✓
✓
✓
✓
Multi-Hop
×
✓
×
✓
✓
✓
✓
✓
Visual Reasoning Stage
↪ Pre-retrieval
×
×
×
×
×
×
×
✓
↪ Post-retrieval
×
×
×
×
×
×
×
✓
Visual Information Source
Table 1: Comparison between our VC-VQA and previous VQA datasets.
Figure 2: Data Construction Pipeline. See Appendix B for more.
Statistic
Number
(I, Q, A) triplets
30,000
Ex triplets
10,000
In-pre triplets
10,000
In-post triplets
10,000
Unique I
26055
Unique Q
12395
Table 2: Statistics of VC-VQA.
Model
Closed Book
Description
Q-Only
Ex
In-pre
In-post
Overall
Overall
Overall
Closed-source Model
GPT-4o Hurst et al. (2024)
47.5
10.4
24.0
27.3
10.1
2.5
GPT-5.4 Singh et al. (2025)
56.6
23.8
32.0
37.5
12.5
2.1
Gemini-2.5-Flash Team et al. (2023)
42.8
41.8
45.6
43.4
8.0
5.6
Gemini-3.1-Flash-Lite-preview Team et al. (2023)
55.1
44.6
45.9
48.5
7.6
3.0
Table 3: Performance comparison under Closed Book, Description and Q-Only settings.
Model
Entity Name
Gold Evidence
Original
Ex
In-pre
In-post
Overall
Ex
In-pre
In-post
Overall
Ex
Closed-source Model
GPT-4o
53.3
70.4
82.7
68.8
87.8
86.8
90.6
88.4
91.3
GPT-5.4
57.7
77.0
81.9
72.2
85.6
88.9
90.6
88.4
93.0
Gemini-2.5-Flash
49.1
72.3
91.7
71.0
78.3
82.9
93.0
84.7
90.2
Gemini-3.1-Flash-Lite
56.9
87.6
83.4
76.0
88.4
90.1
91.8
90.1
93.8
Table 4: Performance comparison under Entity Name, Gold Evidence and Original settings.
Model
Closed
Heuristic
Agentic
Closed-source Model
GPT-4o
27.3
45.3
49.6
GPT-5.4
37.5
47.4
52.2
Gemini-2.5-Flash
43.4
47.7
50.2
Gemini-3.1-Flash-Lite
48.5
51.3
54.4
Gemini-3-Flash
56.3
57.4
60.2
Table 5: Performance comparison under KB setting.
Setting
Ex
In-pre
In-post
Overall
Closed Book
53.2
41.8
44.6
46.5
Gold Evidence
81.6
89.6
89.0
86.7
Agentic KB Retrieve
60.3
50.3
49.8
53.5
Agentic Web Search
62.2
50.7
51.5
54.8
- w/o Image Search
30.8
51.3
50.1
44.1
- w/o Reverse Image Search
56.4
43.8
45.7
48.7
Table 6: Performance comparison under Web setting.
Tool
Ex
In-pre
In-post
Text Search
26.7
29.1
33.6
Image Search
32.2
1.9
14.2
Reverse Image Search
12.5
21.0
8.3
Image Cropping
6.3
26.4
16.8
Webpage Browsing
22.2
21.7
27.1
Table 7: Proportion of calls to the five tools.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Input
Environment
Methods
Example Models
Provided Evidence
Knowledge Base
Closed Book
{I, Q}
-
End-to-end
Gemini-3.1-Pro
No
-
Oracle
{I, Q, E}
-
End-to-end
Gemini-3.1-Pro
Yes
-
KB
{I, Q}
KB
Pipeline
Agentic Retrieval
No
Wikipedia
Web
{I, Q}
Web
Pipeline
Agentic Search
No
Web
Appendix
Table 8: Comparison of different evaluation settings.
Gold Evidence
Name
KB
Web
Closed Book
Description
Q-Only
I
✓
✓
✓
✓
✓
Text
✗
Q
✓
✓
✓
✓
✓
✓
✓
Visual Entity Name
✓
✓
Retrieved
Searched
✗
✗
✗
Evidence
✓
✗
Retrieved
Searched
✗
✗
✗
Appendix
Table 9: Comparison of different benchmarking protocols.
Organization
Model
Release time
OpenAI
GPT-4o
May 2024
GPT-5.4
Mar. 2026
Google
Gemini-2.5-Flash
Jun. 2025
Gemini-3.1-Flash-Lite-preview
Mar. 2026
Gemini-3-Flash-preview
Dec. 2025
Gemini-3.1-Pro
Feb. 2026
Appendix
Table 10: Model information.
Model
Heuristic Retrieval
Agentic Retrieval
Ex
In-pre
In-post
Overall
Ex
In-pre
In-post
Overall
Closed-source Model
GPT-4o
55.3
42.1
38.4
45.3
59.2
48.6
40.9
49.6
GPT-5.4
57.9
46.7
37.6
47.4
61.4
52.8
42.3
52.2
Gemini-2.5-Flash
50.3
44.9
47.8
47.7
52.5
47.1
51.1
50.2
Gemini-3.1-Flash-Lite
57.7
47.5
48.5
51.3
60.9
49.6
52.7
54.4
Appendix
Table 11: Performance comparison between Heuristic Retrieval and Agentic Retrieval models under KB settings.
Model
Closed Book
KB-Heuristic
KB-Agentic
Name
Gold Evidence
Original
GPT-4o
47.5
55.3
59.2
53.3
87.8
91.3
GPT-5.4
56.6
57.9
61.4
57.7
85.6
93.0
Gemini-2.5-Flash
42.8
50.3
52.5
49.1
78.3
90.2
Gemini-3.1-Flash-Lite
55.1
57.7
60.9
56.9
88.4
93.8
Gemini-3-Flash
58.0
58.6
62.4
56.6
86.4
94.2
Gemini-3.1-Pro
57.0
58.1
62.8
57.5
86.4
94.8
Appendix
Table 12: Performance comparison on Ex questions.
Model
Closed Book
KB-Heuristic
KB-Agentic
Name
Gold Evidence
GPT-4o
10.4
42.1
48.6
70.4
86.8
GPT-5.4
23.8
46.7
52.8
77.0
88.9
Gemini-2.5-Flash
41.8
44.9
47.1
72.3
82.9
Gemini-3.1-Flash-Lite
44.6
47.5
49.6
87.6
90.1
Gemini-3-Flash
57.3
58.5
59.9
87.9
90.0
Gemini-3.1-Pro
60.2
59.1
63.4
88.4
89.9
Appendix
Table 13: Performance comparison on In-pre questions
Model
Closed Book
KB-Heuristic
KB-Agentic
Name
Gold Evidence
GPT-4o
24.0
38.4
40.9
82.7
90.6
GPT-5.4
32.0
37.6
42.3
81.9
90.6
Gemini-2.5-Flash
45.6
47.8
51.1
91.7
93.0
Gemini-3.1-Flash-Lite
45.9
48.5
52.7
83.4
91.8
Gemini-3-Flash
53.5
55.1
58.4
93.1
93.2
Gemini-3.1-Pro
54.1
54.9
59.6
93.2
93.4
Appendix
Table 14: Performance comparison on In-post questions.
Model
Closed Book
KB-Heuristic
KB-Agentic
Name
Gold Evidence
GPT-4o
27.3
45.3
49.6
68.8
88.4
GPT-5.4
37.5
47.4
52.2
72.2
88.4
Gemini-2.5-Flash
43.4
47.7
50.2
71.0
84.7
Gemini-3.1-Flash-Lite
48.5
51.3
54.4
76.0
90.1
Gemini-3-Flash
56.3
57.4
60.2
79.2
89.9
Gemini-3.1-Pro
57.1
57.4
61.9
79.7
89.9
Appendix
Table 15: Performance comparison on all questions.
Figure 3: Accuracy of models with different parameter quantities.
Model
Ex
In-pre
In-post
Heuristic Retrieval
38.4
33.1
26.5
Agentic Retrieval
46.5
37.7
34.6
Appendix
Table 16: Retrieval accuracy of the heuristic retrieval and agentic retrieval.
Tool
Ex
In-pre
In-post
Text Retrieval
17.2
62.7
68.7
Image Retrieval
76.4
11.0
7.5
Image Cropping
6.3
26.4
23.8
Appendix
Table 17: Proportion of calls to the three tools.
Model
Vision only
Name and Vision
Qwen3.5-9B
75.5
40.3
Qwen3.5-27B
81.4
44.8
Appendix
Table 18: Performance comparison under two settings.
Knowledge-based visual question answering combines image understanding with external or learned knowledge to answer questions requiring facts beyond the image.
We present M3-VQA, a novel knowledge-based Visual Question Answering (VQA) benchmark, to enhance the evaluation of multimodal large language models (MLLMs) in fine-grained multimodal entity understanding and complex multi-hop reasoning. Unlike existing VQA datasets that focus on coarse-grained categories and simple reasoning over single entities, M3-VQA introduces diverse multi-entity questions involving multiple distinct entities from both visual and textual sources. It requires models to perform both sequential and parallel multi-hop reasoning across multiple documents, supported by traceable, detailed evidence and a curated multimodal knowledge base. We evaluate 16 leading MLLMs under three settings: without external knowledge, with gold evidence, and with retrieval-augmented input. The poor results reveal significant challenges for MLLMs in knowledge acquisition and reasoning. Models perform poorly without external information but improve markedly when provided with precise evidence. Furthermore, reasoning-aware agentic retrieval surpasses heuristic methods, highlighting the importance of structured reasoning for complex multimodal understanding. M3-VQA presents a more challenging evaluation for advancing the multimodal reasoning capabilities of MLLMs. Our code and dataset are available at https://github.com/CASIA-IVA-Lab/M3VQA.
Jiatong Ma, Longteng Guo, Yuchen Liu +4
Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences
Knowledge-based Visual Question Answering (KB-VQA) requires Multimodal Large Language Models (MLLMs) to identify and combine fine-grained visual cues with retrieved textual evidence. However, retrieval often introduces noisy and partially relevant content, while images contain distracting visual regions, causing pretrained MLLMs to overlook the evidence that actually supports the answer. To address this, we introduce Look Twice (LoT), a training-free inference-time framework that turns the model's own internal attention into an explicit multimodal evidence-selection mechanism. LoT first leverages the model's internal attention patterns to identify query-relevant image regions and textual sentences, filters attention sinks and distracting content, and reformulates the input to explicitly highlight the selected evidence before answer generation. The method requires no parameter updates, auxiliary models, or architectural modifications. Across four KB-VQA benchmarks and ten off-the-shelf MLLMs ranging from 2B to 38B parameters, LoT improves every evaluated backbone, with average gains of up to +12.5 accuracy points. It also provides further gains when combined with established context-refinement strategies, yielding additional improvements over already refined inputs. These results establish LoT as a general and effective mechanism for enabling pretrained MLLMs to exploit available multimodal evidence more accurately. Source code is publicly available at https://aimagelab.github.io/LoT/.
Visual Question Answering (VQA) benchmarks have largely emphasized perception-based tasks that can be solved from visual content alone. In contrast, many real-world scenarios require external knowledge that is not directly observable in the image to answer correctly. We introduce WikiVQABench, a human-curated knowledge-grounded VQA benchmark constructed by systematically combining Wikipedia images, their associated article captions, and structured knowledge from Wikidata. Our pipeline uses large language models (LLMs) to generate candidate multiple-choice image-question-answer sets. All generated instances are subsequently reviewed and curated by human annotators to ensure factual correctness, visual-text consistency, and that each question requires external knowledge in addition to visual evidence for correct resolution. WikiVQABench comprises a substantial collection of Wikipedia images with curated multiple-choice questions designed to benchmark knowledge-aware vision-language models (VLMs). Evaluation of fifteen VLMs (256M-90B parameters) reveals a wide performance range (24.7%-75.6% accuracy), demonstrating that the benchmark effectively discriminates model capabilities on knowledge-intensive reasoning. The dataset and benchmarking code are publicly available.