Boosting Knowledge-based Visual Question Answering with Structured Context Reasoning

Qiyou Liu; Yong Zhang; Jianjie Luo; Zhenguo Yang; Yi Yu

cs.CVAug 17, 2026

Boosting Knowledge-based Visual Question Answering with Structured Context Reasoning

Authors: Qiyou Liu, Yong Zhang, Jianjie Luo, Zhenguo Yang, Yi Yu

Organizations: School of Computer Science, Guangdong University of Technology, Guangzhou, China · School of Artificial Intelligence, Shenzhen University, Shenzhen, China · Graduate School of Advanced Science and Engineering, Hiroshima University, Hiroshima, Japan

Abstract

Knowledge-based Visual Question Answering aims to answer questions about an image by integrating external knowledge with visual and textual information. Recent approaches often rely on in-context learning to prompt Large Language Models (LLMs) with multimodal context in a zero-shot or few-shot manner. However, we observe that directly concatenating heterogeneous visual descriptions and retrieved knowledge into long, unstructured prompts often degrades reasoning performance, due to both excessive irrelevant context and the lack of explicit relational structure. In this paper, we propose an LLM-based Structured Context Reasoning (SCoRe) framework that infers both explicit and implicit relationships for prediction. SCoRe consists of three stages: Context Acquisition, which generates diverse visual notes and retrieves explicit knowledge via an efficient two-stage multimodal retrieval strategy; Context Selection, which filters relevant visual, explicit, and implicit knowledge using LLM-guided selection; and Context Compression, which performs Relational Logic Distillation (RLD) to transform raw text into explicit entity-relation triplets. These relational triplets serve as a concise and structured prompt for final answer prediction. Extensive experiments on the OK-VQA and A-OKVQA benchmarks demonstrate that SCoRe consistently outperforms state-of-the-art methods.

Figures & tables

Fig. 1: Two paradigms for LLM-based KB-VQA: (a) Unstructured context reliance, where raw evidence is concatenated into flat, redundant prompts; (b) Our structured context reasoning, which distills multimodal inputs into relational triplets through progressive acquisition, selection, and compression.

Fig. 2: The proposed LLM-based Structured Context Reasoning (SCoRe) method.

Fig. 3: The two-stage progressive multimodal retrieval strategy.

Method	QA	Shot	OK-VQA	A-OKVQA
Method	Model	Number	OK-VQA	A-OKVQA
LLM-based Zero-shot Methods
PICa [ 24 ]	GPT-3 175B	0	17.1	-
PNP-VQA [ 21 ]	UnifiedQA 11B	0	35.9	-
Img2LLM [ 6 ]	OPT 175B	0	45.6	42.9
LAMOC [ 5 ]	FLAN-T5-XXL 11B	0	40.3	37.9

TABLE I: Comparison with SOTA methods on the OK-VQA and A-OKVQA datasets. The best scores are highlighted in bold , and the second best scores are highlighted in underline .

Fig. 4: The Pseudo Recall of intermediate outputs.

VN	EKP	CS	CC	OK-VQA
✓				47.5
✓	✓			49.0
✓	✓	✓		51.2
✓	✓	✓	✓	52.1

TABLE II: Ablation study results of key modules on OK-VQA dataset.

top- $k$	$k_{1}$ =1	$k_{1}$ =10	$k_{1}$ =20	$k_{1}$ =50	$k_{1}$ =100	$k_{1}$ =200
$k_{2}$ =5 (PR@5)	55.02	56.41	58.77	63.58	63.58	64.62
$k_{2}$ =10 (PR@10)	66.72	68.64	68.90	73.71	74.14	75.19
$k_{2}$ =20 (PR@20)	76.76	78.07	77.99	81.74	82.88	82.88

TABLE III: The Pseudo Recall across different

k_{1}

and

k_{2}

	ALL	$k_{1}$ =1	$k_{1}$ =10	$k_{1}$ =20	$k_{1}$ =50	$k_{1}$ =100	$k_{1}$ =200
Num	21,015,324	8,746	72,969	135,835	294,726	560,324	988,434

TABLE IV: The number of candidate passages for different

k_{1}

Fig. 5: The qualitative comparison of KGenVQA with zero-shot settings (a-b) and EF-VQA with few-shot settings (c-d).

Fig. 6: Qualitative case study of SCoRe failure modes.

Knowledge-Based VQA

Knowledge-based visual question answering combines image understanding with external or learned knowledge to answer questions requiring facts beyond the image.

See all 42 papers

Apr 7, 2026

WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering

Multi-modal Retrieval-Augmented Generation (RAG) has emerged as a highly effective paradigm for Knowledge-Based Visual Question Answering (KB-VQA). Despite recent advancements, prevailing methods still primarily depend on images as the retrieval key, and often overlook or misplace the role of Vision-Language Models (VLMs), thereby failing to leverage their potential fully. In this paper, we introduce WikiSeeker, a novel multi-modal RAG framework that bridges these gaps by proposing a multi-modal retriever and redefining the role of VLMs. Rather than serving merely as answer generators, we assign VLMs two specialized agents: a Refiner and an Inspector. The Refiner utilizes the capability of VLMs to rewrite the textual query according to the input image, significantly improving the performance of the multimodal retriever. The Inspector facilitates a decoupled generation strategy by selectively routing reliable retrieved context to another LLM for answer generation, while relying on the VLM's internal knowledge when retrieval is unreliable. Extensive experiments on EVQA, InfoSeek, and M2KR demonstrate that WikiSeeker achieves state-of-the-art performance, with substantial improvements in both retrieval accuracy and answer quality. Our code will be released on https://github.com/zhuyjan/WikiSeeker.

Yingjian Zhu, Xinming Wang, Kun Ding +3

School of Artificial Intelligence, University of Chinese Academy of Sciences · State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Science and Technology Beijing

Knowledge-Based VQA Multimodal RAG Vision-Language Retrieval Query Reformulation

Ask PDF

May 5, 2026

Enhancing Visual Question Answering with Multimodal LLMs via Chain-of-Question Guided Retrieval-Augmented Generation

With advances in multimodal research and deep learning, Multimodal Large Language Models (MLLMs) have emerged as a powerful paradigm for a wide range of multimodal tasks. As a core problem in vision-language research, Visual Question Answering (VQA) has increasingly employed MLLMs to improve performance, particularly in open-domain settings where external knowledge is essential. In this work, we aim to further enhance retrieval-based VQA by more effectively integrating MLLMs with structured reasoning and knowledge acquisition. We introduce a logical prompting strategy that fuses Chain-of-Thought (CoT) reasoning with Visual Question Decomposition (VQD), termed CoVQD, to guide retrieval toward more accurate and relevant knowledge for MLLM inference. Building on this idea, we propose a new framework, CoVQD-guided RAG (CgRAG), which enables MLLMs to access more comprehensive and coherent external knowledge while benefiting from structured visual-text reasoning guidance, thereby improving generalization and reliability in complex cross-domain VQA scenarios. Extensive experiments on E-VQA, InfoSeek, and OKVQA benchmarks demonstrate the effectiveness of the proposed method.

Quanxing Xu, Ling Zhou, Xian Zhong +3

School of Computer Science and Engineering, Macau University of Science and Technology, Macao SAR 999078, China · Hubei Key Laboratory of Transportation Internet of Things, School of Computer Science and Artificial Intelligence, Wuhan University of Technology, Wuhan, Hubei 430070, China · State Key Laboratory of Maritime Technology and Safety, Wuhan University of Technology, Wuhan 430063, China +3

Multimodal RAG Visual Question Answering Multimodal Large Language Models Multimodal CoT Reasoning Knowledge-Based VQA

Ask PDF

Feb 17, 2026

MaS-VQA: A Mask-and-Select Framework for Knowledge-Based Visual Question Answering

Knowledge-based Visual Question Answering (KB-VQA) requires models to answer questions by integrating visual information with external knowledge. However, retrieved knowledge is often noisy, partially irrelevant, or misaligned with the visual content, while internal model knowledge is difficult to control and interpret. Naive aggregation of these sources limits reasoning effectiveness and reduces answer accuracy. To address this, we propose MaS-VQA, a selection-driven framework that tightly couples explicit knowledge filtering with implicit knowledge reasoning. MaS-VQA first retrieves candidate passages and applies a Mask-and-Select mechanism to jointly prune irrelevant image regions and weakly relevant knowledge fragments, producing compact, high-signal multimodal knowledge . This filtered knowledge then guides the activation of internal knowledge in a constrained semantic space, enabling complementary co-modeling of explicit and implicit knowledge for robust answer prediction. Experiments on Encyclopedic-VQA and InfoSeek demonstrate consistent performance gains across multiple MLLM backbones, and ablations verify that the selection mechanism effectively reduces noise and enhances knowledge utilization.

Xianwei Mao, Kai Ye, Sheng Zhou +4

Zhejiang University, Hangzhou, China · Alibaba Group, Hangzhou, China

Visual Question Answering Knowledge-Based VQA VLM Reasoning

Ask PDF

Abstract

Knowledge-Based VQA

Explore similar work

WikiSeeker: Rethinking the Role of Vision-Language Models in Knowledge-Based Visual Question Answering

Enhancing Visual Question Answering with Multimodal LLMs via Chain-of-Question Guided Retrieval-Augmented Generation

MaS-VQA: A Mask-and-Select Framework for Knowledge-Based Visual Question Answering