WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering

Yingjian Zhu; Xinming Wang; Kun Ding; Ying Wang; Bin Fan; Shiming Xiang

cs.CVApr 7, 2026

WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering

Authors: Yingjian Zhu, Xinming Wang, Kun Ding, Ying Wang, Bin Fan, Shiming Xiang

Organizations: School of Artificial Intelligence, University of Chinese Academy of Sciences · State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Science and Technology Beijing

Abstract

Multi-modal Retrieval-Augmented Generation (RAG) has emerged as a highly effective paradigm for Knowledge-Based Visual Question Answering (KB-VQA). Despite recent advancements, prevailing methods still primarily depend on images as the retrieval key, and often overlook or misplace the role of Vision-Language Models (VLMs), thereby failing to leverage their potential fully. In this paper, we introduce WikiSeeker, a novel multi-modal RAG framework that bridges these gaps by proposing a multi-modal retriever and redefining the role of VLMs. Rather than serving merely as answer generators, we assign VLMs two specialized agents: a Refiner and an Inspector. The Refiner utilizes the capability of VLMs to rewrite the textual query according to the input image, significantly improving the performance of the multimodal retriever. The Inspector facilitates a decoupled generation strategy by selectively routing reliable retrieved context to another LLM for answer generation, while relying on the VLM's internal knowledge when retrieval is unreliable. Extensive experiments on EVQA, InfoSeek, and M2KR demonstrate that WikiSeeker achieves state-of-the-art performance, with substantial improvements in both retrieval accuracy and answer quality. Our code will be released on https://github.com/zhuyjan/WikiSeeker.

Figures & tables

Figure 1: The overall architecture of WikiSeeker in comparison with previous methods. (A) Existing methods typically rely on visual-only retrieval and employ the VLM only as the answer generator. (B) Our proposed WikiSeeker conducts multi-modal retrieval and redefines the VLM as a Refiner and an Inspector.

Figure 2: The overall architecture of WikiSeeker. We employ VLMs as specialized agents rather than just generators. The pipeline consists of three stages: (1) Retrieval: A VLM-based Refiner expands the initial question with visual semantics. We then perform dense retrieval by concatenating visual and textual embeddings to match against a pre-constructed multi-modal knowledge base. (2) Rerank: A multi-modal reranker filters the top relevant candidate sections. (3) Generation: A VLM-based Inspector evaluates the sufficiency of the retrieved context. It dynamically routes the query: valid contexts are processed by Answer Generator (path “YES”), while insufficient contexts trigger the Inspector to answer using internal parametric knowledge (path “NO”).

Figure 3: The training pipeline of the VLM Refiner via Reinforcement Learning. The refined query is used to retrieve information from the multimodal knowledge base. VLM’s response is used to compute the structure-based format reward, while the retrieved context is used to compute the rank-based retrieval reward. The combined reward is finally used to update the model through the GRPO algorithm.

hit rank	$\left[1,5\right]$	$\left[6,10\right]$	$\left[11,20\right]$	$\left[21,50\right]$	$\left[51,100\right]$	$\left[101,200\right]$
reward	+4	+3.5	+3	+1	+0.5	+0.1

Table 1: The discrete mapping of retrieval rewards based on the hit rank.

Method	Ratio=0	Ratio=0.3	Ratio=0.7	Ratio=1.0
QwenVL(I+T)	22.75	41.26	68.99	88.46
QwenVL(T)	19.44	41.75	70.82	91.10
LLaMA	18.51	40.48	70.88	92.99
Qwen	19.77	42.52	70.80	93.45

Table 2: VQA performance comparison under different signal-to-noise ratios. The ratio indicates the proportion of correct sections in the input context. QwenVL(I+T) and QwenVL(T) denote Qwen2.5-VL with and without image input, respectively. More detailed experimental results can be found in Table 15 in Appendix C.2 .

Method	E-VQA				InfoSeek
Method	R@1	R@5	R@10	R@20	R@1	R@5	R@10	R@20
CLIP I-T	3.3	7.7	12.1	16.5	32.0	54.0	61.6	68.2
Wiki-LLaVA	3.3	-	9.9	13.2	36.9	-	66.1	71.9
LLM-RA	-	-	-	-	47.3	53.8	-	-
mR 2 AG	-	-	-	-	38.0	-	65.0	71.0
ReflectiVA	15.6	36.1	-	49.8	56.1	77.6	-	86.4

Table 3: Retrieval results on the E-VQA test set and InfoSeek validation set. “w/o. Refiner” and “w. Refiner” indicate whether the input query is expanded by Refiner. Best results are highlighted in bold.

Method	E-VQA (M2KR)					OKVQA-GS (M2KR)					OVEN (M2KR)
Method	R@1	R@5	R@10	PR@5	PR@20	R@1	R@5	R@10	PR@5	PR@20	R@1	R@5	R@10	R@20
CLIP	3.3	7.7	12.1	10.4	-	-	-	-	5.7	-	-	22.0	-	-
FLMR	-	-	-	-	-	-	-	-	68.1	-	-	40.5	-	-
PreFLMR
w/o. Refiner	40.4	62.5	70.2	72.2	78.0	13.8	30.5	39.3	67.5	86.6	31.1	64.2	76.5	85.2
w. Refiner	43.1	65.3	73.2	72.5	79.9	20.7	41.5	51.1	77.5	91.6	42.8	69.7	78.8	86.9

Table 4: Retrieval performance on M2KR benchmark. “w/o. Refiner” and “w. Refiner” indicate whether the input query is expanded by Refiner. Best results are highlighted in bold.

hit rank	$\left[1,5\right]$	$\left[6,10\right]$	$\left[11,20\right]$	$\left[21,200\right]$	miss
EVQA	500	1000	1000	2500	2000
InfoSeek	0	500	1000	2500	3000

Table 5: The sampling distribution based on retrieval hit rank used to construct the RL datasets.

Method	Generator Model	Gen. FT	Ret. FT	E-VQA	InfoSeek
Method	Generator Model	Gen. FT	Ret. FT	E-VQA	Unseen-Q	Unseen-E	Overall
RoRA-VLM	LLaVA-1.5-7B	✓	✗	20.29	27.34	25.10	26.9
Wiki-LLaVA	LLaVA-1.5-7B	✓	✗	21.8	30.1	27.8	28.9
LLM-RA	BLIP2-Flan-T5XL	✓	✓	-	26.12	20.90	23.14
EchoSight	Mistral-7B \| LLaMA3-8B	✗	✓	41.8	-	-	31.3
mR 2 AG	LLaVA-1.5-7B	✓	✓	-	40.6	39.8	40.2

Table 6: VQA accuracy comparison with the baselines. Gen. FT and Ret. FT indicate whether the generator and retriever of the method were fine-tuned, respectively. Best results are highlighted in bold.

Refiner	Inspector	E-VQA	InfoSeek
✗	✗	42.35	31.68
✗	✓	46.28	33.29
✓	✗	53.92	43.47
✓	✓	55.62	44.72

Table 7: The ablation study on the impact of Refiner and Inspector on the VQA results.

Method	R@1	R@5	R@10	R@20
Original Query	14.59	29.43	37.14	43.35
Image Caption	16.11	31.58	37.81	42.93
Zs Expansion	18.78	34.88	42.4	47.79
Zs Expansion+Caption	18.34	34.88	42.15	47.81
WikiSeeker Refiner	25.45	44.48	52.42	57.98

Table 8: Impact of different query formulation strategies on EVQA retrieval performance.

Setting	Generator	Strategy	Accuracy(%)
Zero-shot	VLM	VLM-only	51.81
	LLM	LLM-only	52.97
	VLM + LLM	Decoupled	54.48
Fine-tuned	VLM	VLM-only	52.59
	LLM	LLM-only	53.92
	VLM + LLM	Decoupled	56.34

Table 9: Oracle analysis of the decoupled generation strategy on the EVQA dataset.

Method	FT	EVQA	InfoSeek
VLM	✗	51.81	38.19
LLM	✗	52.97	35.46
VLM	✓	52.59	44.06
LLM	✓	53.92	43.47
LLM + Inspector	✓	55.62	44.72
Oracle Ensemble	✓	58.87	47.28

Table 10: Ablation study on the effectiveness of the WikiSeeker Inspector on EVQA and InfoSeek.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Impact of the multi-modal rerank stage weight $\beta_{1}$ on the final retrieval accuracy (R@1). The x-axis (log scale) represents the weight assigned to the initial retrieval similarity score, where $\beta_{1}=1.0$ indicates using only the retrieval score without reranking. The results show that a hybrid scoring mechanism yields the best performance, with optimal weights of 0.6 for EVQA and 0.8 for InfoSeek.

Figure 5: Impact of the modality weighting hyperparameter $\alpha$ on retrieval performance using refined queries. Left: Coarse-grained search across the full range $\alpha\in[0,1]$ , demonstrating that multi-modal retrieval outperforms uni-modal baselines. Right: Fine-grained search within the interval $[0.55,0.65]$ to identify the precise optimal weights. The central markers denote the average recall (mean of R@1, R@5, R@10, and R@20), while the shaded areas indicate the performance range between the lower bound (R@1) and the upper bound (R@20).

Figure 6: Impact of the textual rerank stage weight $\beta_{2}$ on the final context retrieval accuracy (Recall@1). The results demonstrate that a moderate incorporation of textual reranking scores ( $\beta_{2}=0.2$ ) yields the optimal performance for both EVQA and InfoSeek datasets.

Dataset	PASS Samples	FAIL Samples
EVQA	3,000	5,000
InfoSeek	15,000	15,000
Total	18,000	20,000

Appendix

Table 11: Distribution of the training data used for fine-tuning the Inspector module. "PASS" samples represent successful retrieval cases where the context is sufficient, while "FAIL" samples represent cases where the Inspector must rely on internal parametric knowledge.

Method	Pseudo Recall@K						Recall@K
Method	1	5	10	20	50	100	1	5	10	20	50	100
w/o. Refiner	51.20	72.24	78.72	83.44	88.37	90.93	40.40	62.53	70.24	78.03	86.05	91.09
w. Refiner	51.57	72.48	79.20	84.32	88.91	91.07	43.12	65.25	73.17	79.95	87.01	91.25

Appendix

Table 12: Detailed retrieval performance comparison on the EVQA split of the M2KR benchmark.

Method	Pseudo Recall@K						Recall@K
Method	1	5	10	20	50	100	1	5	10	20	50	100
w/o. Refiner	42.01	67.52	78.02	86.64	94.11	96.87	13.77	30.52	39.34	49.56	63.79	71.68
w. Refiner	57.91	77.55	85.32	91.58	96.43	98.18	20.69	41.48	51.15	61.57	72.65	79.45

Appendix

Table 13: Detailed retrieval performance comparison on the OKVQA split of the M2KR benchmark.

Method	Recall@K
Method	1	5	10	20	50	100	500
w/o. Refiner	31.09	64.18	76.46	85.25	92.52	95.21	97.97
w. Refiner	42.75	69.71	78.85	86.89	92.73	95.72	98.34

Appendix

Table 14: Detailed retrieval performance on the OVEN split of the M2KR benchmark.

Method	Signal-to-Noise Ratio
Method	0	0.1	0.2	0.3	0.4	0.5	0.6	0.7	0.8	0.9	Oracle
Qwen2.5-VL-7B (I+T)	22.75	28.27	35.45	41.26	48.27	55.10	61.68	68.99	74.99	81.79	88.46
Qwen2.5-VL-7B (Text-only)	19.44	27.18	34.91	41.75	49.20	56.13	62.93	70.82	77.33	84.42	91.10
LLaMA3-8B	18.51	25.81	33.90	40.48	48.80	55.92	63.12	70.88	78.08	85.37	92.99
Qwen2.5-7B	19.77	26.17	34.42	42.52	49.31	56.53	63.37	70.80	77.77	85.18	93.45

Appendix

Table 15: Full VQA performance comparison under different Signal-to-Noise Ratios. The ratio represents the proportion of correct sections mixed into the input context. "Qwen2.5-VL-7B (I+T)" denotes the standard multimodal setting, while "(Text-only)" denotes the same model with the image input removed. The best result for each ratio is highlighted in bold.

Method	Avg. Ret. Time	Avg. Inf. Time	VQA Result
LLaVA-1.5-7B	-	1.432	17.00
PreFLMR	0.984	2.196	54.45
WikiSeeker (Ours)	0.915	2.417	65.87

Appendix

Table 16: Inference efficiency comparison on the M2KR EVQA split.

	Ground Truth: PASS	Ground Truth: FAIL
Predicted: PASS	TP: 1,274 (26.82%)	FP: 264 (5.60%)
Predicted: FAIL	FN: 586 (12.34%)	TN: 2,626 (55.28%)

Appendix

Table 17: Confusion matrix of the Inspector’s routing decisions on the EVQA test set.

Panel of Figure 7 — Figure 7: Prompt of WikiSeeker Refiner.

Panel of Figure 8 — Figure 8: Prompt of WikiSeeker Inspector.

Panel of Figure 9 — Figure 9: Prompt of WikiSeeker Generator on EVQA.

Panel of Figure 10 — Figure 10: Prompt of WikiSeeker Generator on InfoSeek.

Panel of Figure 11 — Figure 11: Prompt for GT answer expansion.

Panel of Figure 12 — Figure 12: Prompt designed for summarizing extended Wikipedia sections.

Panel of Figure 13 — Figure 13: Prompt for image caption and zero-shot query expansion.

Figure 14: Qualitative examples demonstrating the effectiveness of the Refiner. For each case, the original user query fails to retrieve the correct entity (marked with ✗ ). The Refiner analyzes the image through a chain-of-thought reasoning process to generate a refined query, which successfully retrieves the ground-truth entity (marked with ✓ ).

Figure 15: Visualization of the decoupled generation strategy enabled by the Inspector. Left (Green Context): Scenarios where retrieval is accurate. The relevant evidence is underlined. The LLM accurately extracts the answer, whereas the VLM fails. The Inspector validates the context and routes the query to the LLM Generator (indicated by “Gen”). Right (Red Context): Scenarios where retrieval fails. The LLM cannot answer correctly due to missing information, but the VLM correctly answers using internal parametric knowledge. The Inspector detects the retrieval failure and provides the answer directly (indicated by “Ins”).

Knowledge-Based VQA

Knowledge-based visual question answering combines image understanding with external or learned knowledge to answer questions requiring facts beyond the image.

See all 42 papers

Learning to Search: A Decision-Based Agent for Knowledge-Based Visual Question Answering
Apr 8, 2026Zhuohong Chen, Zhenxian Wu, Yunyao Yu +6Visual Question Answering Tool Use in VLMs
Tsinghua University · Hefei University of Technology · University of Arizona
ProMSA:Progressive Multimodal Search Agents for Knowledge-Based Visual Question Answering
Jun 26, 2026ZhengXian Wu, Hangrui Xu, Kai Shi +8Multimodal RAG Multimodal Search Agents
OPPO AI Center, OPPO Inc. China · The Shenzhen International Graduate School, Tsinghua University · Nanyang Technological University, Singapore
Enhancing Visual Question Answering with Multimodal LLMs via Chain-of-Question Guided Retrieval-Augmented Generation
May 5, 2026Quanxing Xu, Ling Zhou, Xian Zhong +3Multimodal RAG Visual Question Answering
School of Computer Science and Engineering, Macau University of Science and Technology, Macao SAR 999078, China · Hubei Key Laboratory of Transportation Internet of Things, School of Computer Science and Artificial Intelligence, Wuhan University of Technology, Wuhan, Hubei 430070, China · State Key Laboratory of Maritime Technology and Safety, Wuhan University of Technology, Wuhan 430063, China +3

Apr 8, 2026J/K move · Enter open · S save

Learning to Search: A Decision-Based Agent for Knowledge-Based Visual Question Answering

Zhuohong Chen, Zhenxian Wu, Yunyao Yu, Hangrui Xu+5

Tsinghua University · Hefei University of Technology · University of Arizona

Visual Question Answering Tool Use in VLMs Agentic RAG Knowledge-Based VQA

Open paper AskPDF arXiv

Knowledge-based visual question answering (KB-VQA) requires vision-language models to understand images and use external knowledge, especially for rare entities and long-tail facts. Most existing retrieval-augmented generation (RAG) methods adopt a fixed pipeline that sequentially retrieves information, filters it, and then produces an answer. Such a design makes it difficult to adapt to diverse question types. Moreover, it separates retrieval from reasoning, making it hard for the model to decide when to search, how to refine queries, or when to stop. As a result, the retrieved evidence is often poorly aligned with the question. To address these limitations, we reformulate KB-VQA as a search-agent problem and model the solving process as a multi-step decision-making procedure. At each step, the agent selects one of four actions-Answer, Image Retrieval, Text Retrieval, and Caption-based on its current information state. We further design an automated pipeline to collect multi-step trajectories that record the agent's reasoning process, tool usage, and intermediate decisions. These trajectories are then used as supervision for fine-tuning. Experiments on InfoSeek and E-VQA demonstrate that our method achieves state-of-the-art performance, consistently outperforming prior baselines and confirming the effectiveness of our framework.

Abstract

Knowledge-Based VQA

Explore similar work

Learning to Search: A Decision-Based Agent for Knowledge-Based Visual Question Answering

ProMSA:Progressive Multimodal Search Agents for Knowledge-Based Visual Question Answering

Enhancing Visual Question Answering with Multimodal LLMs via Chain-of-Question Guided Retrieval-Augmented Generation