MEDMKG: Benchmarking Medical Knowledge Exploitation with Multimodal Knowledge Graph

Xiaochen Wang; Yuan Zhong; Lingwei Zhang; Lisong Dai; Ting Wang; Fenglong Ma

cs.AIMay 22, 2025

MEDMKG: Benchmarking Medical Knowledge Exploitation with Multimodal Knowledge Graph

Authors: Xiaochen Wang, Yuan Zhong, Lingwei Zhang, Lisong Dai, Ting Wang, Fenglong Ma

Organizations: Pennsylvania State University, USA · Renmin Hospital of Wuhan University, China · Stony Brook University, USA

Abstract

Medical deep learning models depend heavily on domain-specific knowledge to perform well on knowledge-intensive clinical tasks. Prior work has primarily leveraged unimodal knowledge graphs, such as the Unified Medical Language System (UMLS), to enhance model performance. However, integrating multimodal medical knowledge graphs remains largely underexplored, mainly due to the lack of resources linking imaging data with clinical concepts. To address this gap, we propose MEDMKG, a Medical Multimodal Knowledge Graph that unifies visual and textual medical information through a multi-stage construction pipeline. MEDMKG fuses the rich multimodal data from MIMIC-CXR with the structured clinical knowledge from UMLS, utilizing both rule-based tools and large language models for accurate concept extraction and relationship modeling. To ensure graph quality and compactness, we introduce Neighbor-aware Filtering (NaF), a novel filtering algorithm tailored for multimodal knowledge graphs. We evaluate MEDMKG across three tasks under two experimental settings, benchmarking twenty-four baseline methods and four state-of-the-art vision-language backbones on six datasets. Results show that MEDMKG not only improves performance in downstream medical tasks but also offers a strong foundation for developing adaptive and robust strategies for multimodal knowledge integration in medical artificial intelligence.

Figures & tables

Figure 1. The overview of MedMKG construction pipeline and evaluation methods.

Statistic	Count
Total Number of Edges	35,387
Number of Concepts	3,149
Number of Images	4,868
Number of Relations	262
Number of Cross-modality Edges	20,705
Number of Intra-modality Edges	14,682

Table 1. Data Statistics Summary

Model	Head Prediction				Relation Prediction				Tail Prediction
Model	MR $\downarrow$	Hits@3 $\uparrow$	Hits@5 $\uparrow$	Hits@10 $\uparrow$	MR $\downarrow$	Hits@3 $\uparrow$	Hits@5 $\uparrow$	Hits@10 $\uparrow$	MR $\downarrow$	Hits@3 $\uparrow$	Hits@5 $\uparrow$	Hits@10 $\uparrow$
TransR	1505.66	1.80	3.71	7.50	106.01	5.84	10.55	19.98	887.07	3.34	6.65	13.11
TransD	1219.92	3.84	7.56	12.22	53.49	27.49	35.52	46.36	586.61	5.25	10.90	18.67
TransE	1248.36	3.36	6.33	9.93	39.74	20.13	28.79	42.45	544.79	4.99	9.20	15.14
TransH	1263.25	3.29	6.16	9.99	37.40	22.02	30.21	43.41	561.39	5.05	9.67	16.07
RotatE	1560.49	1.35	2.69	5.04	129.84	0.77	1.42	3.25	739.95	1.54	3.47	6.82

Table 2. Performance of 17 approaches on three link prediction tasks. Bold indicates the best value among baselines.

Figure 2. Human assessment results.

Methods	OpenI						MIMIC-CXR
	Precision @ $K$ $\uparrow$			Recall @ $K$ $\uparrow$			Precision @ $K$ $\uparrow$			Recall @ $K$ $\uparrow$
	10	20	100	10	20	100	10	20	100	10	20	100
CLIP	1.17	1.00	0.56	11.10	19.24	53.48	1.11	0.98	0.58	11.11	19.52	58.26
$+$ FashionKLIP	1.29	1.16	0.63	12.64	22.75	60.46	1.19	0.99	0.56	11.91	19.82	56.06
$+$ KnowledgeCLIP	2.63	1.99	0.79	25.56	38.83	76.16	2.33	1.73	0.74	23.32	34.53	74.37
PubMedCLIP	1.17	0.98	0.51	10.81	18.47	48.46	0.69	0.65	0.43	6.91	13.01	42.79

Table 3. Results (%) on text-image retrieval task for OpenI and MIMIC-CXR datasets. Notable augmentation with MedMKG is observed, especially for KnowledgeCLIP.

Methods	VQA-RAD				SLAKE				PathVQA
Methods	Acc $\uparrow$	Prec $\uparrow$	Rec $\uparrow$	F1 $\uparrow$	Acc $\uparrow$	Prec $\uparrow$	Rec $\uparrow$	F1 $\uparrow$	Acc $\uparrow$	Prec $\uparrow$	Rec $\uparrow$	F1 $\uparrow$
CLIP	64.94	62.71	62.71	62.71	65.07	62.09	74.86	67.88	81.89	88.37	76.54	82.03
$+$ KRISP	73.71	78.89	60.17	68.27	56.90	55.00	69.14	61.27	84.21	89.83	79.79	84.51
$+$ MKBN	70.12	70.87	61.86	66.06	70.14	73.47	61.71	67.08	84.68	89.35	81.33	85.15
$+$ K-PathVQA	66.14	62.79	68.64	65.59	69.30	73.57	58.86	65.40	84.15	85.74	84.75	85.24
$+$ EKGRL	67.73	65.04	67.80	66.39	70.70	71.01	68.57	69.77	84.77	86.38	85.24	85.81

Table 4. Results (%) on knowledge-augmented visual question answering. Best results within each backbone block are in bold .

Ablation	Accuracy	Precision	Recall	F1
MedMKG w. NaF	84.2	89.8	79.8	84.5
MedMKG w. Random Filtering	80.7	96.9	66.5	78.4
MedMKG w/o Filtering	79.4	98.1	62.6	76.4

Table 5. Ablation study results.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 3 — Figure 3. Prompt for concept disambiguation and relation extraction.

Panel of Figure 4 — Figure 4. Entity & relation extraction comparison across LLMs.

Panel of Figure 5 — Figure 5. Distribution of entities involved in MedMKG . The top 10 semantic types are shown individually, and rare types are grouped as “Others.”

Methods	VQA-RAD				SLAKE				PathVQA
Methods	Acc $\uparrow$	Prec $\uparrow$	Rec $\uparrow$	F1 $\uparrow$	Acc $\uparrow$	Prec $\uparrow$	Rec $\uparrow$	F1 $\uparrow$	Acc $\uparrow$	Prec $\uparrow$	Rec $\uparrow$	F1 $\uparrow$
MedCSPCLIP	68.13	61.59	85.59	71.63	66.20	83.95	38.86	53.12	77.72	73.37	92.24	81.73
$+$ KRISP	80.08	84.00	71.19	77.06	70.70	91.76	44.57	60.00	83.19	94.71	72.96	82.43
$+$ MKBN	69.72	65.44	75.42	70.08	67.32	75.21	50.29	60.27	85.37	86.17	86.84	86.51
$+$ K-PathVQA	67.73	75.34	46.61	57.59	71.55	74.03	65.14	69.30	85.31	89.35	82.65	85.87
$+$ EKGRL	76.10	73.39	77.12	75.21	69.30	78.95	51.43	62.28	84.92	92.75	78.19	84.85

Appendix

Table 6. Results (%) on knowledge-augmented visual question answering with PubMedCLIP as the backbone.

Multimodal KGs

Multimodal knowledge graphs connect entities and relations with text, images, video, time, and other evidence for structured reasoning.

See all 42 papers

Aligning Multimodal Patient Evidence with Biomedical Knowledge Graphs for Clinical LLMs
Oct 5, 2026Jiawen Du, Arshan Ali Khan, Chenhao Zhang +7KG Construction Graph-Based Retrieval
University of North Carolina at Chapel Hill · Siddaganga Institute of Technology · Foci Labs +1
Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning
Jun 8, 2025Weiwen Xu, Hou Pong Chan, Long Li +15Medical Report Generation Multimodal Pretraining
LASA Team · DAMO Academy, Alibaba Group
M2G-LLM: Enhancing Clinical Prediction via Multimodal Graph Reasoning and LLM Context Injection
Sep 18, 2026Inyoung Choi, Sukwon Yun, Jiayi Xin +3Multimodal Large Language Models Clinical Prediction
University of Pennsylvania, Philadelphia, PA, USA · University of North Carolina at Chapel Hill, Chapel Hill, NC, USA

Oct 5, 2026J/K move · Enter open · S save

Aligning Multimodal Patient Evidence with Biomedical Knowledge Graphs for Clinical LLMs

Jiawen Du, Arshan Ali Khan, Chenhao Zhang, Zachary Plotkin+6

University of North Carolina at Chapel Hill · Siddaganga Institute of Technology · Foci Labs+1

KG Construction Graph-Based Retrieval Knowledge Graph Question Answering Multimodal KGs Medical LLMs Biomedical KGs

Open paper AskPDF arXiv

Clinical questions often depend on linking a patient's multimodal evidence to external biomedical knowledge, yet existing predictive systems rarely represent such links explicitly, so they can neither be traced to their evidence sources nor removed to measure their contributions. We present MM-KG (Multimodal Knowledge Graph), which represents heterogeneous, multimodal patient observations and biomedical concepts as separate layers in one typed graph, joined by explicit alignment edges. First, modality-specific harmonizers convert EHR text, imaging, genomic, and biospecimen data into typed observations mapped to UMLS concepts, which a route-prioritized aligner links to a biomedical knowledge graph. Query-conditioned retrieval then selects a compact subgraph for downstream use by a large language model or a graph neural network. We build MM-KGs for MIMIC-IV and ADNI, and evaluate them with a 2x2 design that separates patient evidence, biomedical knowledge, and their interaction. On questions that require both sources, neither source alone performs far above chance, whereas their combination yields a drug-controlled AUROC interaction of +0.194 on MIMIC and +0.299 on ADNI. On held-out five-candidate ranking, MM-KG outperforms MindMap by +0.131 Hits@1 and leads an adapted GraphCare on the items that require consulting the patient, and deleting the single answer-bearing relation from the retrieved packet returns Hits@1 to the no-knowledge baseline. Finally, query-conditioned retrieval reaches 0.731 AUROC with 6.8x less context than the strongest generic policy, whereas static knowledge graph context gives no consistent gain on ordinary outcome prediction. Knowledge graphs thus benefit clinical LLMs not as background context but as explicit links between multimodal patient evidence and the relation a question requires, and MM-KG makes these links retrievable, traceable, and testable.

Abstract

Multimodal KGs

Explore similar work

Aligning Multimodal Patient Evidence with Biomedical Knowledge Graphs for Clinical LLMs

Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning

M2G-LLM: Enhancing Clinical Prediction via Multimodal Graph Reasoning and LLM Context Injection