Organizations: Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · OPPO AI Center, OPPO Inc.
We propose VC-VQA, a large-scale benchmark for vision-centric knowledge reasoning, aiming to systematically evaluate the capabilities of Multimodal Large Language Models (MLLMs) in fine-grained visual knowledge acquisition and complex multi-stage reasoning. Unlike existing benchmarks that primarily treat visual information as entity recognition and rely on text-based external knowledge, VC-VQA explicitly models the role of visual information by considering both its source (internal vs. external images) and stage of involvement (pre-retrieval vs. post-retrieval). Specifically, VC-VQA requires models to retrieve external visual evidence, identify visual clues before retrieval, and localize, verify, or match retrieved knowledge against the internal image. We further construct a controlled multimodal knowledge base with traceable textual and visual evidence for each sample. We evaluate 21 representative MLLMs under four settings: no external knowledge, with provided evidence, knowledge base retrieval, and open-web search. Experimental results lead to three key findings: current models still struggle with vision-centric knowledge reasoning; fine-grained visual knowledge acquisition remains the primary bottleneck; and agentic retrieval highlights the value of multi-step planning and cross-modal knowledge integration. Overall, VC-VQA provides a more challenging benchmark for evaluating vision-centric knowledge reasoning in MLLMs. Our code and dataset are available at https://github.com/CASIA-IVA-Lab/VC-VQA.
Figures & tables
Figure 1: Comparison between previous work and our benchmark.
OKVQA
KVQA
InfoSeek
EVQA
Dyn-VQA
M 3 -VQA
MM-BrowseComp
VC-VQA
Fine-Grained Entity
×
✓
✓
✓
✓
✓
✓
✓
Multi-Hop
×
✓
×
✓
✓
✓
✓
✓
Visual Reasoning Stage
↪ Pre-retrieval
×
×
×
×
×
×
×
✓
↪ Post-retrieval
×
×
×
×
×
×
×
✓
Visual Information Source
Table 1: Comparison between our VC-VQA and previous VQA datasets.
Figure 2: Data Construction Pipeline. See Appendix B for more.
Statistic
Number
(I, Q, A) triplets
30,000
Ex triplets
10,000
In-pre triplets
10,000
In-post triplets
10,000
Unique I
26055
Unique Q
12395
Table 2: Statistics of VC-VQA.
Model
Closed Book
Description
Q-Only
Ex
In-pre
In-post
Overall
Overall
Overall
Closed-source Model
GPT-4o Hurst et al. (2024)
47.5
10.4
24.0
27.3
10.1
2.5
GPT-5.4 Singh et al. (2025)
56.6
23.8
32.0
37.5
12.5
2.1
Gemini-2.5-Flash Team et al. (2023)
42.8
41.8
45.6
43.4
8.0
5.6
Gemini-3.1-Flash-Lite-preview Team et al. (2023)
55.1
44.6
45.9
48.5
7.6
3.0
Table 3: Performance comparison under Closed Book, Description and Q-Only settings.
Model
Entity Name
Gold Evidence
Original
Ex
In-pre
In-post
Overall
Ex
In-pre
In-post
Overall
Ex
Closed-source Model
GPT-4o
53.3
70.4
82.7
68.8
87.8
86.8
90.6
88.4
91.3
GPT-5.4
57.7
77.0
81.9
72.2
85.6
88.9
90.6
88.4
93.0
Gemini-2.5-Flash
49.1
72.3
91.7
71.0
78.3
82.9
93.0
84.7
90.2
Gemini-3.1-Flash-Lite
56.9
87.6
83.4
76.0
88.4
90.1
91.8
90.1
93.8
Table 4: Performance comparison under Entity Name, Gold Evidence and Original settings.
Model
Closed
Heuristic
Agentic
Closed-source Model
GPT-4o
27.3
45.3
49.6
GPT-5.4
37.5
47.4
52.2
Gemini-2.5-Flash
43.4
47.7
50.2
Gemini-3.1-Flash-Lite
48.5
51.3
54.4
Gemini-3-Flash
56.3
57.4
60.2
Table 5: Performance comparison under KB setting.
Setting
Ex
In-pre
In-post
Overall
Closed Book
53.2
41.8
44.6
46.5
Gold Evidence
81.6
89.6
89.0
86.7
Agentic KB Retrieve
60.3
50.3
49.8
53.5
Agentic Web Search
62.2
50.7
51.5
54.8
- w/o Image Search
30.8
51.3
50.1
44.1
- w/o Reverse Image Search
56.4
43.8
45.7
48.7
Table 6: Performance comparison under Web setting.
Tool
Ex
In-pre
In-post
Text Search
26.7
29.1
33.6
Image Search
32.2
1.9
14.2
Reverse Image Search
12.5
21.0
8.3
Image Cropping
6.3
26.4
16.8
Webpage Browsing
22.2
21.7
27.1
Table 7: Proportion of calls to the five tools.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Input
Environment
Methods
Example Models
Provided Evidence
Knowledge Base
Closed Book
{I, Q}
-
End-to-end
Gemini-3.1-Pro
No
-
Oracle
{I, Q, E}
-
End-to-end
Gemini-3.1-Pro
Yes
-
KB
{I, Q}
KB
Pipeline
Agentic Retrieval
No
Wikipedia
Web
{I, Q}
Web
Pipeline
Agentic Search
No
Web
Appendix
Table 8: Comparison of different evaluation settings.
Gold Evidence
Name
KB
Web
Closed Book
Description
Q-Only
I
✓
✓
✓
✓
✓
Text
✗
Q
✓
✓
✓
✓
✓
✓
✓
Visual Entity Name
✓
✓
Retrieved
Searched
✗
✗
✗
Evidence
✓
✗
Retrieved
Searched
✗
✗
✗
Appendix
Table 9: Comparison of different benchmarking protocols.
Organization
Model
Release time
OpenAI
GPT-4o
May 2024
GPT-5.4
Mar. 2026
Google
Gemini-2.5-Flash
Jun. 2025
Gemini-3.1-Flash-Lite-preview
Mar. 2026
Gemini-3-Flash-preview
Dec. 2025
Gemini-3.1-Pro
Feb. 2026
Appendix
Table 10: Model information.
Model
Heuristic Retrieval
Agentic Retrieval
Ex
In-pre
In-post
Overall
Ex
In-pre
In-post
Overall
Closed-source Model
GPT-4o
55.3
42.1
38.4
45.3
59.2
48.6
40.9
49.6
GPT-5.4
57.9
46.7
37.6
47.4
61.4
52.8
42.3
52.2
Gemini-2.5-Flash
50.3
44.9
47.8
47.7
52.5
47.1
51.1
50.2
Gemini-3.1-Flash-Lite
57.7
47.5
48.5
51.3
60.9
49.6
52.7
54.4
Appendix
Table 11: Performance comparison between Heuristic Retrieval and Agentic Retrieval models under KB settings.
Model
Closed Book
KB-Heuristic
KB-Agentic
Name
Gold Evidence
Original
GPT-4o
47.5
55.3
59.2
53.3
87.8
91.3
GPT-5.4
56.6
57.9
61.4
57.7
85.6
93.0
Gemini-2.5-Flash
42.8
50.3
52.5
49.1
78.3
90.2
Gemini-3.1-Flash-Lite
55.1
57.7
60.9
56.9
88.4
93.8
Gemini-3-Flash
58.0
58.6
62.4
56.6
86.4
94.2
Gemini-3.1-Pro
57.0
58.1
62.8
57.5
86.4
94.8
Appendix
Table 12: Performance comparison on Ex questions.
Model
Closed Book
KB-Heuristic
KB-Agentic
Name
Gold Evidence
GPT-4o
10.4
42.1
48.6
70.4
86.8
GPT-5.4
23.8
46.7
52.8
77.0
88.9
Gemini-2.5-Flash
41.8
44.9
47.1
72.3
82.9
Gemini-3.1-Flash-Lite
44.6
47.5
49.6
87.6
90.1
Gemini-3-Flash
57.3
58.5
59.9
87.9
90.0
Gemini-3.1-Pro
60.2
59.1
63.4
88.4
89.9
Appendix
Table 13: Performance comparison on In-pre questions
Model
Closed Book
KB-Heuristic
KB-Agentic
Name
Gold Evidence
GPT-4o
24.0
38.4
40.9
82.7
90.6
GPT-5.4
32.0
37.6
42.3
81.9
90.6
Gemini-2.5-Flash
45.6
47.8
51.1
91.7
93.0
Gemini-3.1-Flash-Lite
45.9
48.5
52.7
83.4
91.8
Gemini-3-Flash
53.5
55.1
58.4
93.1
93.2
Gemini-3.1-Pro
54.1
54.9
59.6
93.2
93.4
Appendix
Table 14: Performance comparison on In-post questions.
Model
Closed Book
KB-Heuristic
KB-Agentic
Name
Gold Evidence
GPT-4o
27.3
45.3
49.6
68.8
88.4
GPT-5.4
37.5
47.4
52.2
72.2
88.4
Gemini-2.5-Flash
43.4
47.7
50.2
71.0
84.7
Gemini-3.1-Flash-Lite
48.5
51.3
54.4
76.0
90.1
Gemini-3-Flash
56.3
57.4
60.2
79.2
89.9
Gemini-3.1-Pro
57.1
57.4
61.9
79.7
89.9
Appendix
Table 15: Performance comparison on all questions.
Figure 3: Accuracy of models with different parameter quantities.
Model
Ex
In-pre
In-post
Heuristic Retrieval
38.4
33.1
26.5
Agentic Retrieval
46.5
37.7
34.6
Appendix
Table 16: Retrieval accuracy of the heuristic retrieval and agentic retrieval.
Tool
Ex
In-pre
In-post
Text Retrieval
17.2
62.7
68.7
Image Retrieval
76.4
11.0
7.5
Image Cropping
6.3
26.4
23.8
Appendix
Table 17: Proportion of calls to the three tools.
Model
Vision only
Name and Vision
Qwen3.5-9B
75.5
40.3
Qwen3.5-27B
81.4
44.8
Appendix
Table 18: Performance comparison under two settings.
Knowledge-based visual question answering combines image understanding with external or learned knowledge to answer questions requiring facts beyond the image.