MMKG-RDS: Reasoning Data Synthesis via Deep Mining of Multimodal Knowledge Graphs

Lun Zhan; Feng Xiong; Huanyong Liu; Feng Zhang; Yuhui Yin

cs.AIFeb 27, 2026

MMKG-RDS: Reasoning Data Synthesis via Deep Mining of Multimodal Knowledge Graphs

Authors: Lun Zhan, Feng Xiong, Huanyong Liu, Feng Zhang, Yuhui Yin

Organizations: AI Research Institute, Qihoo 360

Abstract

Synthesizing high-quality training data is crucial for enhancing domain models' reasoning abilities. Existing methods face limitations in long-tail knowledge coverage, effectiveness verification, and interpretability. Knowledge-graph-based approaches still fall short in functionality, granularity, customizability, and evaluation. To address these issues, we propose MMKG-RDS, a flexible framework for reasoning data synthesis that leverages multimodal knowledge graphs. It supports fine-grained knowledge extraction, customizable path sampling, and multidimensional data quality scoring. We validate MMKG-RDS with the MMKG-RDS-Bench dataset, covering five domains, 17 task types, and 14,950 samples. Experimental results show fine-tuning Qwen3 models (0.6B/8B/32B) on a small number of synthesized samples improves reasoning accuracy by 9.2%. The framework also generates distinct data, challenging existing models on tasks involving tables and formulas, useful for complex benchmark construction. The dataset and code are available at https://github.com/360AILAB-NLP/MMKG-RDS

Figures & tables

Framework	Function				KG Data			User Design		Analysis
Framework	Doc Parse	KG Builder	Path Sample	QA Filter	Node Type	Relation Type	Modal	Schema	Task	Dim	Statistic Display	Dataset
Easy Dataset ( Miao et al., 2025 )	✓	✗	✗	✓	✗	✗		✗	1	1	✗	✗
DataFlow ( Liang et al., 2025 )	✓	✗	✗	✓	✗	✗		✗	1	6	✗	✗
GraphGen ( Chen et al., 2025b )	✓	✓	✓	✓	1	1		✗	3	3	✗	✗
Deepdive ( Lu et al., 2025 )	✗	✗	✓	✓	1	1		✗	1	1	✗	✗
Medreason ( Wu et al., 2025 )	✗	✗	✓	✓	1	1		✗	1	1	✗	✓

Table 1: Comparison of existing Framework with MMKG-RDS.

Figure 1: High-level architecture of MMKG-RDS.The system consists of five layers: Preprocessing, KG Builder, Storage, Synthesis, and Analysis. MMKG-RDS can dynamically adjust various components and task configurations based on user input, enabling the generation of high-quality, task-specific data for downstream LLM applications.

Figure 2: MMKG-RDS comprises: (a) Document parsing: extracting and structuring multimodal documents; (b) Knowledge graph building: a five-stage construction process; (c) Data generation: synthesizing data via customizable sampling; (d) Quality filtering: deduplication, quality control, and scoring for data distribution tuning.

Node Type	Symbol	Fileds
Document	$\mathcal{V}_{1}$	<id, content, title, path, schema>
Chunk	$\mathcal{V}_{2}$	<id, content, doc_id>
Entity	$\mathcal{V}_{3}$	<id, name, desc, attr, src_id_list>
Assertion	$\mathcal{V}_{4}$	<id, head, relation, tail, desc, src_id_list>
Image	$\mathcal{V}_{5}$	<id, content, caption, path, desc>
Table	$\mathcal{V}_{6}$	<id, content, caption, path, desc>

Table 2: 7 types of node definitions and fileds.

	simple	medium	hard
weak_model	✓	✗	-
strong_model	✓	✓	✗

Table 3: Definition of difficulty levels, where - denotes ✓ or ✗ .

Figure 3: Statistical Information of MMKG-RDS-Bench: (a) Distribution of MMKG-RDS-Bench Across Different Domains; (b) Distribution of MMKG-RDS-Bench Across Different Tasks; (c) Distribution of Question Lengths in MMKG-RDS-Bench.

	Model	Task
	Model	Entity	SISC	STSC	SFSC	SFMC	MC	MI	MIMC	MT	MTMC
LLM	Qwen3-0.6B	79.6	65.6	36.6	23.8	30.5	76.5	48.8	52.1	10.8	26.3
	Qwen3-1.7B	88.3	71.6	41.2	29.8	43.7	85.8	57.9	52.9	13.0	34.4
	Qwen3-4B	93.8	81.0	53.8	43.9	55.9	90.4	66.9	63.6	20.0	42.3
	Qwen3-8B	93.8	79.5	53.9	44.2	60.8	91.2	68.6	60.1	21.8	44.9
	Qwen3-14B	92.9	80.7	56.5	47.7	62.4	88.9	72.9	66.9	25.1	49.9

Table 4: Accuracy(%) on different tasks across the Qwen3 full-series models. All results are averages within each task category with reasoning mode disabled.

Figure 4: (left) Accuracy(%) on different tasks across the Qwen3-series models. (right) Accuracy(%) on different tasks across the Qwen3-VL-series models.

Model	Input
Model	text	img	img+text
Qwen3-VL-2B	51.5	49.0	51.1
Qwen3-VL-4B	60.4	56.8	60.3
Qwen3-VL-8B	61.3	58.4	60.8
Qwen3-VL-32B	64.0	61.4	63.8
Qwen3-VL-30B-A3B	61.5	57.7	61.1

Table 5: Accuracy(%) on different input across the Qwen3-VL full-series models. With reasoning mode disabled, all results are averages for tasks that involve image inputs within the synthetic dataset.

Model	Mode
Model	no_think	think	SFT(think)
Qwen3-0.6B	36.4	39.7	51.5 (+11.8)
Qwen3-8B	53.4	59.0	65.6 (+6.6)
Qwen3-32B	55.2	58.7	67.9 (+9.2)

Table 6: Performance comparison of Qwen3 models under different modes (no_think, think, and SFT) on the synthesized dataset. Results are reported in accuracy scores.

Model	Difficulty
Model	simple+medium	hard
Qwen3-0.6B	52.0	54.7 (+2.7)
Qwen3-8B	67.8	70.5 (+2.7)
Qwen3-32B	69.4	72.7 (+3.3)

Table 7: Accuracy(%) after SFT on 1 k samples selected from different difficulty in the training set.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Neo4j-based Knowledge Graph in the Chemistry Domain

Node	Domain
Node	law	report	paper	history	chemistry
Document	1	10	10	1	1
Chunk	9961	697	1748	2594	7847
Assertion	30091	12804	24164	32867	58703
Entity	15183	5097	10140	12788	19568
Table	0	98	104	7	43

Appendix

Table 8: Statistics on the number of knowledge graph nodes in different domains.

Edge	Domain
Edge	law	report	history	paper	chemistry
chk $\to$ doc	9961	697	2594	1748	7847
chk $\to$ fml	0	0	0	5	0
chk $\to$ img	0	0	0	1	22
chk $\to$ tbl	0	5	0	85	54
ent $\to$ chk	35589	4909	23684	13382	31231

Appendix

Table 9: Counts of edge types between different node categories in the multimodal knowledge graph. Node category abbreviations are: doc (Document), chk (Chunk), ent (Entity), ass (Assertion), fml (Formula), img (Image), tbl (Table).

Modality Category	Task Types
Table-centric QA	Single-Table QA
	Single-Table Multi-hop QA
	Single-Table & Single-Chunk (Text) QA
	Single-Table & Multi-Chunk (Text) QA
	Multi-Table QA
	Multi-Table Multi-hop QA

Appendix

Table 10: Taxonomy of predefined Question Answering Tasks by Modality.

Figure 6: Case 1 from MMKG-RDS-bench

Figure 7: Case 2 from MMKG-RDS-bench

Figure 8: Distribution of Complexity on MMKG-RDS-bench.

Figure 9: Distribution of Difficulty on MMKG-RDS-bench.

	Model	Difficulty			Complexity
	Model	simple	medium	hard	3	4	5
LLM	Qwen3-0.6B	73.5	28.9	13.1	47.3	41.6	40.2
	Qwen3-1.7B	97.0	12.5	12.1	47.3	47.3	47.0
	Qwen3-4B	93.9	66.3	22.3	64.5	58.4	58.9
	Qwen3-8B	94.7	68.5	23.1	63.4	59.0	60.1
	Qwen3-14B	95.7	78.3	25.5	62.4	61.7	63.3

Appendix

Table 11: Accuracy(%) on different difficulty and complexity across the Qwen3 full-series models. All results are averages within each task category with reasoning mode disabled. Among these models, Qwen3-1.7B and Qwen3-235B-A22B participated in the difficulty level evaluation.

Synthetic Data Generation

Synthetic data generation creates artificial examples, simulations, and datasets to train, test, or protect machine-learning systems.

See all 625 papers

Learning from Synthetic Data Improves Multi-hop Reasoning
Mar 2, 2026Anmol Kabra, Yilun Yin, Albert Gong +6RL for Language Model Reasoning Multi-Hop Reasoning
Cornell University · University of Cambridge · Stanford University
Learning to Pose Problems: Reasoning-Driven and Solver-Adaptive Data Synthesis
Nov 13, 2025Yongxian Wei, Yilin Zhao, Li Shen +7Synthetic Data Generation Synthetic Data Generation for Language Models
Tsinghua University · Tencent · Sun Yat-sen University +1
CHIMERA: Compact Synthetic Data for Generalizable LLM Reasoning
Mar 1, 2026Xinyu Zhu, Yihao Feng, Yanchao Sun +5LLM Reasoning Synthetic Data Generation for Language Models
Department of Computer Science, University of Virginia, Charlottesville, VA, USA · Apple, Cupertino, CA, USA · Department of Computer Science, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA

Mar 2, 2026J/K move · Enter open · S save

Learning from Synthetic Data Improves Multi-hop Reasoning

Anmol Kabra, Yilun Yin, Albert Gong, Kamilė Stankevičiūtė+5

Cornell University · University of Cambridge · Stanford University

RL for Language Model Reasoning Multi-Hop Reasoning Compositional Reasoning

Open paper AskPDF arXiv

Reinforcement Learning (RL) has been shown to significantly boost reasoning capabilities of large language models (LLMs) in math, coding, and multi-hop reasoning tasks. However, RL fine-tuning requires abundant high-quality verifiable data, often sourced from human annotations, generated from frontier LLMs, or scored by LLM-based verifiers. All three have considerable limitations: human-annotated datasets are small and expensive to curate, LLM-generated data is hallucination-prone and costly, and LLM-based verifiers are inaccurate and slow. In this work, we investigate a cheaper alternative: RL fine-tuning on rule-generated synthetic data for multi-hop reasoning tasks. We discover that LLMs fine-tuned on synthetic data perform significantly better on popular real-world question-answering benchmarks, despite the synthetic data containing only fictional knowledge. On stratifying performance by question difficulty, we find that synthetic data teaches LLMs to compose knowledge -- a fundamental and generalizable reasoning skill. Our work highlights rule-generated synthetic reasoning data as a free and scalable resource to improve LLM reasoning capabilities.

Abstract

Synthetic Data Generation

Explore similar work

Learning from Synthetic Data Improves Multi-hop Reasoning

Learning to Pose Problems: Reasoning-Driven and Solver-Adaptive Data Synthesis

CHIMERA: Compact Synthetic Data for Generalizable LLM Reasoning