CrossModalQA: A Cross-modal and Multi-hop Benchmark for Multimodal Retrieval-augmented Generation

Jiacheng Cai; Zijin Hong; Zheng Yuan; Huachi Zhou; Qinggang Zhang; Xiao Huang

cs.CVAug 31, 2026

CrossModalQA: A Cross-modal and Multi-hop Benchmark for Multimodal Retrieval-augmented Generation

Authors: Jiacheng Cai, Zijin Hong, Zheng Yuan, Huachi Zhou, Qinggang Zhang, Xiao Huang

Organizations: The Hong Kong Polytechnic University · Jilin University

Abstract

Despite the strong capabilities of multimodal large language models (MLLMs), their parametric knowledge remains incomplete and difficult to update, motivating multimodal retrieval-augmented generation (RAG) to ground responses in external text and images. However, existing benchmarks face two major limitations: (i) they typically emphasize single-hop retrieval or reasoning over a small set of provided contexts rather than open-domain evidence discovery; and (ii) they provide fragmented coverage of cross-modal reasoning paths, leaving complex multi-hop and multi-image reasoning underexplored. In this paper, we introduce CrossModalQA, an open-domain benchmark for evaluating multimodal retrieval and reasoning over heterogeneous corpora. CrossModalQA contains 1,863 question-answer pairs constructed from 4,987 Wikipedia articles and 4,431 Wikimedia Commons images. It covers five complementary reasoning paths: vision-to-text, text-to-vision, vision-to-text-to-vision, multi-image intersection, and image-set reasoning. Every question requires retrieving and composing distributed textual and visual evidence, with an average reasoning depth of 3.50 hops. We construct the benchmark through multimodal knowledge graph-guided subgraph sampling and apply rule-based consistency checking and LLM verification to ensure multimodal dependence and traceable evidence. Extensive experiments demonstrate that existing multimodal RAG systems struggle to recover complete evidence chains and can underperform closed-book models when incomplete retrieval introduces distracting context. Further analysis reveals that complete cross-modal retrieval contributes more to answer accuracy than generator scaling, while multi-image retrieval and reasoning remain the primary bottlenecks limiting end-to-end performance.

Figures & tables

Method	Question Category
	V-to-T	T-to-V	V-to-T-to-V	Multi-image Intersection	Image-set
Text-only Questions
MultiModalQA ( Talmor et al., 2021 )	✓	✓	✗	✓	✓
WebQA ( Chang et al., 2022 )	✓	✓	✗	✗	✗
RetVQA ( Penamakuri et al., 2023 )	✗	✓	✗	✓	✗
Visual-RAG ( Wu et al., 2025 )	✗	✓	✗	✗	✗

Table 1: The five categories in CrossModalQA. “V” denotes vision while “T” denotes text. For text-only questions, V-to-T and V-to-T-to-V denote that the question describes visual features. For text-and-image questions, V-to-T and V-to-T-to-V denote that the question has attached images.

Figure 2: Distribution of QA instances by reasoning-hop count across the five question modes. Each cell reports the row-wise percentage of QA instances, while color intensity represents the corresponding absolute count. Most questions require three or four reasoning hops. Specifically, the Multi-image Intersection and Image-set mode exhibits a greater concentration of two- and three-hop questions, consistent with their design goal of evaluating a model’s ability to retrieve multiple relevant images from a large image corpus, with less emphasis on long reasoning chains. The average number of reasoning hops is 3.50.

Method	V to T		T to V		V to T to V		Multi-image Intersection		Image-set		Overall
	EM	F1	EM	F1	EM	F1	EM	F1	EM	F1	EM	F1
GPT 4o	20.2	20.9	29.8	32.1	37.2	39.3	21.7	21.5	8.3	23.9	24.3	27.6
GPT 4o + Gold texts	75.7	77.5	31.1	33.4	39.4	41.1	81.1	80.9	17.0	44.1	49.8	54.2
GPT 4o + Gold images	30.9	31.8	66.6	68.0	74.7	75.8	43.5	43.6	29.2	52.2	50.2	54.7
GPT 4o + Gold texts & images	77.3	79.1	64.2	66.4	70.1	70.8	82.9	83.4	29.2	56.6	66.1	70.3

Table 2: Performance of GPT-4o under different oracle-evidence settings. Gold texts and Gold images denote the provision of ground-truth evidence from the corresponding modality. We report Exact Match (EM) and micro-F1 for each of the five question modes, together with their overall average.

Method	V to T		T to V		V to T to V		Multi-image Intersection		Image-set		Overall
	EM	F1	EM	F1	EM	F1	EM	F1	EM	F1	EM	F1
Vanilla GPT 4o mini	13.7	14.6	27.4	28.8	30.4	31.7	17.4	17.4	6.7	22.7	19.8	23.2
Vanilla GPT 4o	20.2	20.9	29.8	32.1	37.2	39.3	21.7	21.5	8.3	23.9	24.3	27.6
M 2 RAG ( Liu et al., 2025b )	11.7	28.2	17.8	27.7	15.1	22.5	19.7	25.8	4.8	33.2	14.2	27.5
mKG-RAG (T2I) ( Yuan et al., 2026 )	14.8	15.3	21.7	21.5	28.0	28.4	20.7	20.3	6.7	15.6	19.0	20.1
mKG-RAG (T2T) ( Yuan et al., 2026 )	21.6	20.8	14.9	15.5	27.7	28.0	21.7	20.6	7.4	24.3	19.2	22.1

Table 3: Exact-match and F1 scores (%).

Method	V to T				T to V				V to T to V
	R@5	R@10	P@5	H@5	R@5	R@10	P@5	H@5	R@5	R@10	P@5	H@5
Image retrieval
mKG-RAG (T2I) ( Yuan et al., 2026 )	51.0	60.8	12.0	56.8	45.3	52.1	9.1	45.4	32.8	42.3	13.4	55.5
Multimodal retrieval
M 2 RAG ( Liu et al., 2025b )	36.9	45.1	19.3	70.2	49.0	55.3	24.4	85.1	23.9	30.0	16.7	58.6
Method	Multi-image Intersection				Image-set				Overall

Table 4: Retrieval performance. R@5, R@10, P@5, and H@5 denote Recall@5, Recall@10, Precision@5, and Hit@5, respectively. mKG-RAG (T2I) is image-only retrieval, while M 2 RAG retrieves both images and text rticles.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

ID	Category	Quota
A	Animals and animal taxa	2,000
B	Plants and plant taxa	2,000
C	Buildings and structures	2,000
D	Paintings	2,000
E	Sculptures	500
F	Other cultural relics	1,000

Appendix

Table 5: Category taxonomy used for domain-targeted article selection. The quota specifies the number of articles retained in each category. The total number of articles we select is 10,000.

Figure 3: Average reasoning hops of CrossModalQA by mode.

Figure 4: Exact Match (EM) by reasoning hop and QA mode in the GPT-4o + Gold texts & images setting. The line plot shows the overall EM for each reasoning hop, while the heatmap reports the per-mode EM. Values in parentheses indicate the number of QA instances in each cell. We exclude the 1-hop and 7-hop cases because of insufficient sample sizes for reliable evaluation. EM scores are reported on a 0–100 scale and rounded to the nearest integer.

Figure 5: Micro-F1 by reasoning hop and QA mode in the GPT-4o + Gold texts & images setting. The line plot shows the overall Micro-F1 for each reasoning hop, while the heatmap reports the per-mode Micro-F1. Values in parentheses indicate the number of QA instances in each cell. We exclude the 1-hop and 7-hop cases because of insufficient sample sizes for reliable evaluation. F1 scores are reported on a 0–100 scale and rounded to the nearest integer.

Method	V to T		T to V		V to T to V		Multi-image Intersection		Image-set		Overall
	EM	F1	EM	F1	EM	F1	EM	F1	EM	F1	EM	F1
GPT-4o mini	13.7	14.6	27.4	28.8	30.4	31.7	17.4	17.4	6.7	22.7	19.8	23.2
GPT-4o mini + Gold texts	71.3	74.0	23.5	26.8	31.4	34.0	73.7	75.1	14.7	42.3	43.7	49.1
GPT-4o mini + Gold images	23.5	26.1	65.0	65.9	66.4	66.7	33.0	33.2	21.8	49.7	43.2	48.8
GPT-4o mini + Gold texts & images	71.6	73.4	59.8	60.9	59.4	61.1	74.2	75.1	22.4	52.3	58.8	63.5

Appendix

Table 6: Performance of GPT-4o mini under different oracle-evidence settings. Gold texts and Gold images denote the provision of ground-truth evidence from the corresponding modality. We report Exact Match (EM) and micro-F1 for each of the five question modes, together with their overall average.

Figure 6: Example of prompts used in question generation stage.

Figure 7: Example of prompts used in question generation stage.

Explore similar work

Mar 5, 2026

MultiHaystack: Benchmarking Multimodal Retrieval and Reasoning over 40K Images, Videos, and Documents

Multimodal large language models (MLLMs) achieve strong performance on benchmarks that evaluate text, image, or video understanding separately. However, these settings do not assess a critical real-world requirement, which involves retrieving relevant evidence from large, heterogeneous multimodal corpora prior to reasoning. Most existing benchmarks restrict retrieval to small, single-modality candidate sets, substantially simplifying the search space and overstating end-to-end reliability. To address this gap, we introduce MultiHaystack, the first benchmark designed to evaluate both retrieval and reasoning under large-scale, cross-modal conditions. MultiHaystack comprises over 46,000 multimodal retrieval candidates across documents, images, and videos, along with 747 open yet verifiable questions. Each question is grounded in a unique validated evidence item within the retrieval pool, requiring evidence localization across modalities and fine-grained reasoning. In our study, we find that models perform competitively when provided with the corresponding evidence, but their performance drops sharply when required to retrieve that evidence from the full corpus. Additionally, even the strongest retriever, E5-V, achieves only 40.8% Recall@1, while state-of-the-art MLLMs such as GPT-5 experience a significant drop in reasoning accuracy from 80.86% when provided with the corresponding evidence to 51.4% under top-5 retrieval. These results indicate that multimodal retrieval over heterogeneous pools remains a primary bottleneck for MLLMs, positioning MultiHaystack as a valuable testbed that highlights underlying limitations obscured by small-scale evaluations and promotes retrieval-centric advances in multimodal systems.

Dannong Xu, Zhongyu Yang, Jun Chen +6

INSAIT, Sofia University “St. Kliment Ohridski” · Lanzhou University · King Abdullah University of Science and Technology +3

Multimodal IR Multimodal Model Evaluation Cross-Modal Retrieval Multimodal Large Language Models Multimodal Reasoning

Ask PDF

Apr 28, 2026

M $^3$ -VQA: A Benchmark for Multimodal, Multi-Entity, Multi-Hop Visual Question Answering

We present M

^3

-VQA, a novel knowledge-based Visual Question Answering (VQA) benchmark, to enhance the evaluation of multimodal large language models (MLLMs) in fine-grained multimodal entity understanding and complex multi-hop reasoning. Unlike existing VQA datasets that focus on coarse-grained categories and simple reasoning over single entities, M

^3

-VQA introduces diverse multi-entity questions involving multiple distinct entities from both visual and textual sources. It requires models to perform both sequential and parallel multi-hop reasoning across multiple documents, supported by traceable, detailed evidence and a curated multimodal knowledge base. We evaluate 16 leading MLLMs under three settings: without external knowledge, with gold evidence, and with retrieval-augmented input. The poor results reveal significant challenges for MLLMs in knowledge acquisition and reasoning. Models perform poorly without external information but improve markedly when provided with precise evidence. Furthermore, reasoning-aware agentic retrieval surpasses heuristic methods, highlighting the importance of structured reasoning for complex multimodal understanding. M

^3

-VQA presents a more challenging evaluation for advancing the multimodal reasoning capabilities of MLLMs. Our code and dataset are available at https://github.com/CASIA-IVA-Lab/M3VQA.

Jiatong Ma, Longteng Guo, Yuchen Liu +4

Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences

Multimodal RAG Visual Question Answering Multimodal QA Knowledge-Based VQA Multimodal Model Evaluation Multimodal Understanding

Ask PDF

Jun 24, 2026

MKG-RAG-Bench: Benchmarking Retrieval in Multimodal Knowledge Graph-Augmented Generation

Retrieval-augmented generation (RAG) over knowledge graphs has emerged as a promising approach for grounding large language models, yet existing benchmarks largely overlook the challenges of retrieval in multimodal knowledge graph RAG (MKG-RAG). In practice, retrieval is a critical bottleneck: multimodal knowledge is heterogeneous, difficult to align across modalities, and often poorly served by retrievers designed for unstructured corpora. To address this gap, we introduce MKG-RAG-Bench, a cross-domain benchmark explicitly designed to evaluate retrieval in MKG-RAG. MKG-RAG-Bench is constructed from two multimodal knowledge graphs spanning general and medical domains, and includes carefully aligned question-answering datasets that support controlled evaluation of both retrieval and downstream generation. The benchmark is built using an LLM-based curation pipeline that filters low-utility knowledge, generates structurally grounded queries with exact supervision, and systematically covers diverse modality configurations. Through extensive experiments across representative retriever families and modality settings, we show that effective multimodal retrieval remains challenging yet crucial for end-to-end MKG-RAG performance, and that retrieval quality strongly determines generation outcomes. By isolating retrieval as a first-class evaluation target, MKG-RAG-Bench provides a principled foundation for diagnosing current limitations and advancing multimodal knowledge graph RAG systems.

Xiaochen Wang, Bao Hoang, Han Liu +2

The Pennsylvania State University University Park, Pennsylvania, USA · Michigan State University East Lansing, Michigan, USA · Dalian University of Technology Dalian, Liaoning, China +1

Multimodal RAG Retrieval-Augmented Generation Graph Retrieval-Augmented Generation RAG Evaluation Multimodal KGs

Ask PDF

Abstract

Explore similar work

MultiHaystack: Benchmarking Multimodal Retrieval and Reasoning over 40K Images, Videos, and Documents

M3^33-VQA: A Benchmark for Multimodal, Multi-Entity, Multi-Hop Visual Question Answering

MKG-RAG-Bench: Benchmarking Retrieval in Multimodal Knowledge Graph-Augmented Generation

M $^3$ -VQA: A Benchmark for Multimodal, Multi-Entity, Multi-Hop Visual Question Answering