CRIT: Graph-Based Automatic Data Synthesis to Enhance Cross-Modal Multi-Hop Reasoning

Junyoung Sung; Seungwoo Lyu; Minjun Kim; Sumin An; Arsha Nagrani; Paul Hongsuck Seo

cs.LGApr 2, 2026

CRIT: Graph-Based Automatic Data Synthesis to Enhance Cross-Modal Multi-Hop Reasoning

Authors: Junyoung Sung, Seungwoo Lyu, Minjun Kim, Sumin An, Arsha Nagrani, Paul Hongsuck Seo

Organizations: Dept. of CSE, Korea University · Google DeepMind

Abstract

Real-world reasoning often requires combining information across modalities, connecting textual context with visual cues in a multi-hop process. Yet, most multimodal benchmarks fail to capture this ability: they typically rely on single images or set of images, where answers can be inferred from a single modality alone. This limitation is mirrored in the training data, where interleaved image-text content rarely enforces complementary, multi-hop reasoning. As a result, Vision-Language Models (VLMs) frequently hallucinate and produce reasoning traces poorly grounded in visual evidence. To address this gap, we introduce CRIT, a new dataset and benchmark built with a graph-based automatic pipeline for generating complex cross-modal reasoning tasks. CRIT consists of diverse domains ranging from natural images, videos, and text-rich sources, and includes a manually verified test set for reliable evaluation. Experiments on this benchmark reveal that even state-of-the-art models struggle on such reasoning tasks. Models trained on CRIT show significant gains in cross-modal multi-hop reasoning, including strong improvements on SPIQA and other standard multimodal benchmarks.

Figures & tables

Figure 1: Example from CRIT. The CRIT dataset is designed to train and evaluate cross-modal multi-hop reasoning over interleaved image-text contexts. Each example combines multiple textual and visual sources that must be jointly interpreted to answer a compositional question. The colored highlights in the text indicate distinct evidence segments contributing to the reasoning chain, while the numbered boxes in the images and text correspond to the multi-step inference process outlined in the Chain of Thoughts panel. Together, they illustrate how CRIT requires connecting dispersed clues across modalities to reach a grounded answer.

Figure 2: Overall Process of Cross-Modal Multi-Hop QA Generation. The procedure consists of three main stages. (1) Multimodal Content Graph Construction: images annotated with scene graphs are sampled (a), unique entities are filtered and merged (b), and text nodes connected with edges are generated via LLM prompting (c, d). (2) Textual Context Generation: subgraphs are extracted from the multimodal content graph (e) to produce complementary textual descriptions (f). (3) Question–Answer Generation: subgraphs are further sampled (g) to generate QA pairs requiring cross-modal multi-hop reasoning (h). Orange and yellow circular nodes represent visual entities originating from different images, while blue square nodes denote text nodes. Entities with identical names are distinguished by numerical subscripts (e.g., apple_1 , apple_2 ).

Split	Statistic	NI	VF	SP
Train	# Samples	49,159	8,022	27,018
	Avg. Images/Sample	3.8	3.5	3.7
	Avg. Text Tokens/Sample	2,442	649	5,443
	# QA	153,781	16,071	99,615
	# 2-hop	109,735	8,061	58,429
	# 3-hop	12,271	6,042	28,385

Table 1: CRIT Dataset Statistics. CRIT comprises three domains: natural image (NI), video frame (VF), and scientific paper (SP). This table reports the key statistics for each domain.

		NI		VF		SP
Model	# Prm	EM	F1	EM	F1	EM	F1
Proprietary Models
Gemini 2.0 Flash [ 56 ]	-	30.4	32.7	30.6	35.3	11.5	13.7
GPT-4o [ 25 ]	-	35.1	37.7	32.0	38.9	8.4	14.0
GPT-4o-mini [ 25 ]	-	24.7	25.5	28.7	33.3	9.7	13.1
Open-Source Models

Table 2: Results on CRIT. Performance comparison of proprietary, open-source, and fine-tuned VLMs across natural image (NI), video frame (VF), and scientific paper (SP). Models fine-tuned on CRIT are highlighted with a gray background. Open-source models are grouped by model size for clearer comparison. # Prm denotes the number of model parameters.

		SPIQA				VEGA		MMQA		FCMR
Fine-tuned Dataset	# Samples	METEOR	ROUGE-L	CIDEr	BERTScore F1	ROUGE-L	BLEU	EM	F1	F1
No Fine-tuning	-	1.13	1.10	1.03	28.23	31.4	6.7	27.4	31.7	40.5
Mantis-Instruct	837k	3.60	10.17	23.83	42.83	29.5	5.1	27.3	30.9	44.9
Mantis-Instruct+CRIT	1005k	10.53	22.77	67.93	55.80	35.1	7.1	30.0	33.8	50.5

Table 3: Performance on Cross-Modal Reasoning Benchmarks. Performance comparison of zero-shot Idefics2-8B and the same model fine-tuned on Mantis-Instruct alone or jointly with CRIT, where Mantis-Instruct and CRIT have a 5:1 dataset size ratio. # Samples indicates the total number of training samples in the corresponding fine-tuned dataset.

Fine-tuned Dataset	# Samples	MuirBench	BLINK	MP-DocVQA	MMStar	MME	SeedBench	ChartQA
No Fine-tuning	-	26.2	45.2	46.7	30.2	1792	26.4	26.4
Mantis-Instruct	837k	33.1	45.7	48.2	47.2	1909	67.3	63.8
Mantis-Instruct+CRIT	1005k	32.6	47.6	49.8	48.3	1910	67.3	64.9

Table 4: Performance on Standard Multimodal Benchmarks. Performance comparison of zero-shot Idefics2-8B and the same model fine-tuned on Mantis-Instruct alone or jointly with CRIT, where Mantis-Instruct and CRIT have a 5:1 dataset size ratio. # Samples indicates the total number of training samples in the corresponding fine-tuned dataset.

		NI		VF		SP
Fine-tuned Dataset	# Samples	EM	F1	EM	F1	EM	F1
No Fine-tuning	–	28.3	29.1	24.0	27.8	6.8	9.6
CRIT	84k	58.6	59.5	38.8	42.2	15.9	22.5
CRIT Augmented	210K	62.6	63.7	45.6	49.1	16.7	22.8

Table 5: Results on CRIT Extension. Performance comparison of zero-shot Qwen2.5-VL-7B and the same model fine-tuned on the original CRIT training set and an extended version augmented with additional samples. # Samples indicates the total number of training samples in the corresponding fine-tuned dataset.

Figure 3: Error categories and their distribution across 75 randomly sampled GPT-4o responses on CRIT.

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

	SPIQA				VEGA		MMQA		CRIT
Image	M	R-L	C	B-F1	R-L	B	EM	F1	EM	F1
✓	17.3	30.8	88.0	57.8	40.1	12.3	48.8	56.0	28.0	32.3
✗	11.8	19	36.6	49.9	40.3	12.7	42.9	49.6	10.5	12.5

Appendix

Table 6: Ablation on Image Input. We evaluate GPT-4o on benchmarks formulated as interleaved image–text inputs with and without images. Performance remains strong on existing benchmarks but drops sharply on CRIT, indicating that CRIT is designed to require cross-modal reasoning. M: METEOR, R-L: ROUGE-L, C: CIDEr, B-F1: BERTScore F1, B: BLEU.

Figure 4: Image Distribution

Figure 5: Text Distribution

Dataset	Interleaved Image-Text	Multi Hop	Diverse Data Source	Data Generation	Reasoning Trace	Num QA
ManyModalQA [ 19 ]	✗	✗	✗	Human experts	✗	10.1k
CT2C-QA [ 71 ]	✗	✗	✗	LLMs + Human experts	✗	10k
MME-Finance [ 17 ]	✗	✗	✗	LLMs + Human experts	✗	2.3k
WebQA [ 8 ]	✗	✓	✗	Human experts	✗	46.7k
MuMuQA [ 50 ]	✗	✓	✗	Human experts	✗	1.4k
MMQA [ 55 ]	✗	✓	✗	Template + Human experts	✗	30k

Appendix

Table 7: Comparison of our dataset with prior cross-modal reasoning works. CRIT targets cross-modal multi-hop reasoning over interleaved image-text inputs, constructed through a graph-based automatic pipeline spanning diverse data sources, and provides 271k QA pairs together with explicit reasoning traces.

		SPIQA				VEGA		MMQA		FCMR	MuirBench	BLINK	MP-DocVQA
Fine-tuned Dataset	# Samples	M	R-L	C	B-F1	R-L	B	EM	F1	F1	Acc	Acc	Acc
No Fine-tuning	-	1.13	1.10	1.03	28.23	31.4	6.7	27.4	31.7	40.5	26.2	45.2	46.7
Mantis-Instruct	837k	3.60	10.17	23.83	42.83	29.5	5.1	27.3	30.9	44.9	33.1	45.7	48.2
Mantis-Instruct+CRIT	1005k	10.53	22.77	67.93	55.80	35.1	7.1	30.0	33.8	50.5	32.6	47.6	49.8
Mantis-Instruct+SPIQA	1005k	10.0	22.47	71.7	54.57	28.4	4.0	29.2	32.3	47.5	34.9	46.4	48.8

Appendix

Table 8: Ablation of Fine-Tuning Data Composition Performance comparison of zero-shot Idefics2-8B and the same model fine-tuned on different datasets. We compare fine-tuning on Mantis-Instruct alone and jointly with CRIT or SPIQA. Mantis-Instruct and CRIT follow a 5:1 sampling ratio, and all joint settings contain the same total number of training samples. Joint training with CRIT provides the strongest overall improvements, while size-matched SPIQA yields notably smaller gains, underscoring the contribution of CRIT. M: METEOR, R-L: ROUGE-L, C: CIDEr, B-F1: BERTScore F1, B: BLEU, Acc: Accuracy.

Hyperparameter	Qwen2.5-VL CRIT	Idefics2 CRIT
Epochs	1	1
Batch size	64	128
Learning rate	2e-5	2e-5
Optimizer	AdamW	AdamW
Warmup ratio	0.05	0.05
Scheduler	cosine	cosine

Appendix

Table 9: Training hyperparameters for Qwen2.5-VL and Idefics2.

		NI		VF		SP
Model	# Prm	EM	F1	EM	F1	EM	F1
Qwen2.5-VL ${}_{\text{{CRIT}}}$	7B	58.6	59.5	38.8	42.2	15.9	22.5
Intern3-VL [ 73 ]	8B	33.1	34.3	35.2	41.3	7.3	12.1
Qwen3-VL-Thinking [ 6 ]	8B	31.8	32.9	33.1	37.1	7.3	9.6
GLM-4.1V-Thinking [ 20 ]	9B	12.0	17.7	11.2	12.1	5.2	10.4
Kimi-VL-Thinking [ 58 ]	16B	33.8	35.3	33.9	38.1	8.1	12.3

Appendix

Table 10: Results on CRIT including reasoning models. Performance comparison of reasoning models and fine-tuned VLM across natural image (NI), video frame (VF), and scientific paper (SP). Models fine-tuned on CRIT are highlighted with a gray background. # Prm denotes the number of model parameters.

		NI		VF		SP
Model	#Prm	EM	F1	EM	F1	EM	F1
Proprietary Models
GPT-4o [ 25 ]	-	21.0	21.4	32.8	39.2	10.2	13.0
GPT-4o-mini [ 25 ]	-	24.9	26.0	31.7	36.0	9.4	13.3
Open-Source Models
Phi3.5-Vision [ 2 ]	4B	24.1	25.4	29.5	31.8	5.5	9.5

Appendix

Table 11: Results on CRIT with Direct Answer. Performance comparison of proprietary, open-source, and fine-tuned VLMs across natural image (NI), video frame (VF), and scientific paper (SP). Models fine-tuned on CRIT are highlighted with a gray background. Open-source models are grouped by model size for clearer comparison. # Prm denotes the number of model parameters.

		Overall		2-hop		3-hop		4-hop		5-hop
Model	# Prm	EM	F1	EM	F1	EM	F1	EM	F1	EM	F1
Proprietary Models
Gemini 2.0 Flash [ 56 ]	-	26.0	28.8	26.2	28.6	22.2	26.3	27.9	29.7	30.5	34.3
GPT-4o [ 25 ]	-	28.0	32.3	28.4	32.2	25.1	31.1	27.2	29.3	32.5	38.2
GPT-4o-mini [ 25 ]	-	22.0	24.3	22.0	23.4	19.5	24.6	22.1	24.1	27.8	30.0
Open-Source Models

Appendix

Table 12: Results on CRIT by Hops. Performance comparison of proprietary, open-source, and fine-tuned VLMs across natural image. Models fine-tuned on CRIT are highlighted with a gray background. Open-source models are grouped by model size for clearer comparison. # Prm denotes the number of model parameters.

Figure 6: Failure Case I: Generating Cross-Modal Multi-Hop QA with Direct Prompting

Figure 7: Failure Case II: Generating Cross-Modal Multi-Hop QA with Direct Prompting

Figure 8: Illustration of Visual Perception Error Case.

Figure 9: Illustration of Evidence Localization Error Case.

Figure 10: Illustration of Textual Comprehension Error Case.

Figure 11: Illustration of Answer Generation Error Case.

Figure 12: Qualitative Example of CRIT in Video Frame Domain.

Figure 13: Qualitative Example of CRIT in Scientific Paper Domain.

Panel of Figure 14 — Figure 14: VLM-as-a-Judge Prompt for Pairwise QA Evaluation.

Panel of Figure 15 — Figure 15: Text Node Generation Prompt Type 1.

Panel of Figure 16 — Figure 16: Text Node Generation Prompt Type 2.

Panel of Figure 17 — Figure 17: Text Node Generation Prompt Type 3.

Panel of Figure 18 — Figure 18: Edge Generation Prompt.

Panel of Figure 19 — Figure 19: Textual Context Generation Prompt.

Panel of Figure 20 — Figure 20: Question-Answer Pair Generation Prompt.

Panel of Figure 21 — Figure 21: CoT Response Generation Prompt.

Panel of Figure 22 — Figure 22: Video Caption to Graph Conversion Prompt.

Panel of Figure 23 — Figure 23: Paragraph(without figure reference) to Graph Transformation Prompt.

Panel of Figure 24 — Figure 24: Paragraph(with figure reference) to Graph Transformation Prompt.

Multimodal Reasoning

Multimodal reasoning studies how AI systems combine text with images, audio, video, and other signals to solve evidence-dependent tasks.

See all 508 papers

CrossModalQA: A Cross-modal and Multi-hop Benchmark for Multimodal Retrieval-augmented Generation
Aug 31, 2026Jiacheng Cai, Zijin Hong, Zheng Yuan +3Multimodal RAG Multimodal QA
The Hong Kong Polytechnic University · Jilin University
HopChain: Multi-Hop Data Synthesis for Generalizable Vision-Language Reasoning
Mar 17, 2026Shenzhi Wang, Shixuan Liu, Jing Zhou +8Reasoning for VLMs VLM Reasoning
Qwen Team, Alibaba Inc. · LeapLab, Tsinghua University
Genesis: Visual Augmentation Primitives for Multimodal Chain-of-Thought Reasoning
Oct 11, 2026Fan Yang, Lifeng Chen, Yipu Wang +14VLM Reasoning Multimodal CoT Reasoning
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences · University of Chinese Academy of Sciences · Beijing Jiaotong University +4

Aug 31, 2026J/K move · Enter open · S save

CrossModalQA: A Cross-modal and Multi-hop Benchmark for Multimodal Retrieval-augmented Generation

Jiacheng Cai, Zijin Hong, Zheng Yuan, Huachi Zhou+2

The Hong Kong Polytechnic University · Jilin University

Multimodal RAG Multimodal QA Cross-Modal Retrieval

Open paper AskPDF arXiv

Despite the strong capabilities of multimodal large language models (MLLMs), their parametric knowledge remains incomplete and difficult to update, motivating multimodal retrieval-augmented generation (RAG) to ground responses in external text and images. However, existing benchmarks face two major limitations: (i) they typically emphasize single-hop retrieval or reasoning over a small set of provided contexts rather than open-domain evidence discovery; and (ii) they provide fragmented coverage of cross-modal reasoning paths, leaving complex multi-hop and multi-image reasoning underexplored. In this paper, we introduce CrossModalQA, an open-domain benchmark for evaluating multimodal retrieval and reasoning over heterogeneous corpora. CrossModalQA contains 1,863 question-answer pairs constructed from 4,987 Wikipedia articles and 4,431 Wikimedia Commons images. It covers five complementary reasoning paths: vision-to-text, text-to-vision, vision-to-text-to-vision, multi-image intersection, and image-set reasoning. Every question requires retrieving and composing distributed textual and visual evidence, with an average reasoning depth of 3.50 hops. We construct the benchmark through multimodal knowledge graph-guided subgraph sampling and apply rule-based consistency checking and LLM verification to ensure multimodal dependence and traceable evidence. Extensive experiments demonstrate that existing multimodal RAG systems struggle to recover complete evidence chains and can underperform closed-book models when incomplete retrieval introduces distracting context. Further analysis reveals that complete cross-modal retrieval contributes more to answer accuracy than generator scaling, while multi-image retrieval and reasoning remain the primary bottlenecks limiting end-to-end performance.

Abstract

Multimodal Reasoning

Explore similar work

CrossModalQA: A Cross-modal and Multi-hop Benchmark for Multimodal Retrieval-augmented Generation

HopChain: Multi-Hop Data Synthesis for Generalizable Vision-Language Reasoning

Genesis: Visual Augmentation Primitives for Multimodal Chain-of-Thought Reasoning