VL-KGE: Vision-Language Models Meet Knowledge Graph Embeddings

Athanasios Efthymiou; Stevan Rudinac; Monika Kackovic; Nachoem Wijnberg; Marcel Worring

cs.AIMar 2, 2026

VL-KGE: Vision-Language Models Meet Knowledge Graph Embeddings

Authors: Athanasios Efthymiou, Stevan Rudinac, Monika Kackovic, Nachoem Wijnberg, Marcel Worring

Organizations: University of Amsterdam, Amsterdam, The Netherlands

Abstract

Real-world multimodal knowledge graphs (MKGs) are inherently heterogeneous, modeling entities that are associated with diverse modalities. Traditional knowledge graph embedding (KGE) methods excel at learning continuous representations of entities and relations, yet they are typically designed for unimodal settings. Recent approaches extend KGE to multimodal settings but remain constrained, often processing modalities in isolation, resulting in weak cross-modal alignment, and relying on simplistic assumptions such as uniform modality availability across entities. Vision-Language Models (VLMs) offer a powerful way to align diverse modalities within a shared embedding space. We propose Vision-Language Knowledge Graph Embeddings (VL-KGE), a framework that integrates cross-modal alignment from VLMs with structured relational modeling to learn unified multimodal representations of knowledge graphs. Experiments on WN9-IMG and two novel fine art MKGs, WikiArt-MKG-v1 and WikiArt-MKG-v2, demonstrate that VL-KGE consistently improves over traditional unimodal and multimodal KGE methods in link prediction tasks. Our results highlight the value of VLMs for multimodal KGE, enabling more robust and structured reasoning over large-scale heterogeneous knowledge graphs.

Figures & tables

Panel of Figure 1 — Figure 1. Example triples from the WN9-IMG dataset ( Xie et al., 2017 ) . Entities correspond to ImageNet ( Deng et al., 2009 ) synsets, represented by sets of images (shown in red ) and their WordNet ( Miller, 1995 ) textual definitions (shown in cyan ), connected by semantic relations.

Panel of Figure 2 — Figure 2. Example subgraphs from the WikiArt-MKGs. Artworks are represented visually, while associated entities (e.g., artists, styles, genres, locations) are represented textually. WikiArt-MKG-v1 (inner dashed box) captures core artwork-level relations, whereas WikiArt-MKG-v2 (outer dashed box) extends the graph with additional entity types and richer semantic links.

MKG	Visual	Textual	Entities	Relations	Train	Val	Test
WN9-IMG	6,555	6,555	6,555	9	11,741	1,337	1,319
WikiArt-MKG-v1	75,921	837	76,758	4	299,968	34,020	19,695
WikiArt-MKG-v2	216,564	7,602	224,166	22	7,877,220	208,513	208,368

Table 1. Statistics of MKGs. Visual and Textual denote the number of visual and textual entities, Entities the total number of entities, Relations the number of relation types, and Train , Val , and Test the number of triples in each split.

Method	MRR	Hits@1	Hits@3	Hits@10
MMKRL	0.913	0.905	0.917	0.932
OTKGE	0.923	0.911	0.930	0.947
TransE	0.904	0.894	0.909	0.922
VB-TransE	0.910	0.890	0.923	0.944
VL-TransE (base: BLIP)	0.910	0.894	0.921	0.940
VL-TransE (base: CLIP)	0.913	0.890	0.928	0.950

Table 2. Results on WN9-IMG. MMKRL and OTKGE results are as reported in ( Cao et al., 2022 ) . For all metrics, higher is better. For each KGE, best results are in bold, second best underlined .

Method	WikiArt-MKG-v1				WikiArt-MKG-v2
	MRR	Hits@1	Hits@3	Hits@10	MRR	Hits@1	Hits@3	Hits@10
Zero-shot BLIP	0.403	0.246	0.492	0.721	0.175	0.096	0.193	0.334
Zero-shot CLIP	0.510	0.357	0.609	0.798	0.237	0.139	0.263	0.442
TransE	0.245	0.116	0.282	0.511	0.206	0.130	0.222	0.365
VB-TransE	0.489	0.325	0.584	0.833	0.309	0.186	0.356	0.558
VL-TransE (base: BLIP)	0.681	0.537	0.794	0.924	0.419	0.307	0.470	0.641

Table 3. Results on WikiArt-MKGs. For all metrics, higher is better. For each KGE, best results are in bold, second best underlined .

Panel of Figure 3 — Figure 3. Qualitative comparison of zero-shot CLIP and VL-ComplEx (base: CLIP) on WikiArt-MKG-v2. Given an artwork (top rows) or an artist (bottom rows) as a query, we show the top-5 predicted entities for selected relations. For artist queries, we use only textual input representations. Correctly retrieved entities are shown in bold.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Head	Relation	Tail
143,102	isAssociatedWithTag	1,937
212,649	isCreatedByArtist	4,201
197,335	belongsToGenre	67
158,160	isCreatedInYear	620
81,277	isCreatedWithMedium	149
134,939	hasStyle	20

Appendix

Table 4. Number of unique head and tail entities per relation in WikiArt-MKG-v2. The top section shows artwork-to-attribute, artist-to-attribute, and artist-to-artist relations, while the bottom section shows the two relatedness relations.

Method	AP				mAP	ILD-A	ILD-V
	Style	Genre	Tag	Date
Zero-shot CLIP	0.521	0.615	0.299	0.579	0.503	0.638	0.767
VL-TransE	0.523	0.613	0.299	0.581	0.504	0.638	0.768
VL-DistMult	0.539	0.601	0.283	0.592	0.504	0.645	0.779
VL-ComplEx	0.539	0.602	0.285	0.592	0.504	0.645	0.779
VL-RotatE	0.445	0.594	0.277	0.505	0.455	0.666	0.761

Appendix

Table 5. Results for the isRelatedToArtwork relation. We report average precision (AP) for each attribute, mean AP (mAP), and intra-list diversity (ILD) based on attributes (ILD-A) and visual features (ILD-V). Best results for AP and mAP are in bold , second best underlined .

Method	AP			mAP	ILD-A	ILD-T
	Movement	School	Date
Zero-shot CLIP	0.292	0.217	0.524	0.344	0.845	0.462
VL-TransE	0.486	0.445	0.782	0.571	0.702	0.466
VL-DistMult	0.356	0.334	0.695	0.462	0.794	0.438
VL-ComplEx	0.336	0.278	0.673	0.429	0.822	0.442
VL-RotatE	0.440	0.409	0.729	0.526	0.728	0.467

Appendix

Table 6. Results for the isRelatedToArtist relation. We report average precision (AP) for each attribute, mean AP (mAP), and intra-list diversity (ILD) based on attributes (ILD-A) and textual features (ILD-T). Best results for AP and mAP are in bold , second best underlined .

Method	WikiArt-v1				WikiArt-v2
	Style	Artist	Timeframe	Tags	Style	Artist	Date $\ddagger$	Tags
ViT	0.560	0.445	0.599	0.576	0.514	0.395	0.104	0.174
BLIP	0.611	0.412	0.655	0.611	0.586	0.430	0.142	0.280
CLIP	0.737	0.701	0.773	0.690	0.698	0.647	0.153	0.364

Appendix

Table 7. Linear-probe results on WikiArt-v1 and WikiArt-v2. Style, Artist, and Timeframe report accuracy; Tags reports mAP; and Date

\ddagger

reports cumulative accuracy with

\theta=10

(

\pm

10 years). For all metrics, higher is better. Best results are in bold , second best underlined .

Fine-tuning		WikiArt-MKG-v1				WikiArt-MKG-v2
Visual	Textual	MRR	Hits@1	Hits@3	Hits@10	MRR	H@1	H@3	H@10
✗	✗	0.784	0.661	0.892	0.974	0.577	0.463	0.642	0.794
✓	✗	0.753	0.630	0.854	0.959	0.566	0.453	0.631	0.783
✗	✓	0.787	0.667	0.893	0.974	0.575	0.461	0.641	0.792
✓	✓	0.753	0.632	0.852	0.958	0.566	0.453	0.629	0.781

Appendix

Table 8. Results using VL-ComplEx (base: CLIP) on the WikiArt-MKG-v1 and WikiArt-MKG-v2 validation sets for fine-tuning different modalities. For all metrics, higher is better. Best results are in bold , second best underlined .

Modality		Average				Concatenation				Weighted
Visual	Textual	MRR	Hits@1	Hits@3	Hits@10	MRR	Hits@1	Hits@3	Hits@10	MRR	Hits@1	Hits@3	Hits@10
✗	✗	0.898	0.896	0.898	0.899	0.898	0.896	0.898	0.899	0.898	0.896	0.898	0.899
✓	✗	0.921	0.913	0.924	0.939	0.910	0.905	0.909	0.918	0.921	0.912	0.925	0.938
✗	✓	0.919	0.909	0.923	0.938	0.909	0.905	0.909	0.917	0.919	0.910	0.923	0.936
✓	✓	0.924	0.913	0.927	0.950	0.907	0.905	0.907	0.909	0.922	0.914	0.924	0.939

Appendix

Table 9. Results using VL-DistMult (base: CLIP) on the WN9-IMG validation set under different modality configurations and fusion strategies. For all metrics, higher is better. Best results are in bold , second best underlined .

Figure 4. Per-relation mean reciprocal rank (MRR) on the WikiArt-MKG-v2 validation set for zero-shot CLIP and VL-KGEs.

Explore similar work

GraphVLM: Benchmarking Vision Language Models for Multimodal Graph Learning
Mar 9, 2026Jiajin Liu, Dongzhe Fan, Chuanhao Ji +2Multimodal Graph Learning Multimodal Model Evaluation
NYU Shanghai · New York University · East China Normal University +1
ViSR-KGC: Visual Subgraph Reasoning with Vision-Language Models for Multimodal Knowledge Graph Completion
Aug 6, 2026Jiafan Li, Mengxue Yang, Jiaqi Zhu +3VLM Reasoning Multimodal KGs
Institute of Software, Chinese Academy of Sciences, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · CITIC Securities, Beijing, China +1
When Modalities Remember: Continual Learning for Multimodal Knowledge Graphs
Apr 3, 2026Linyu Li, Zhi Jin, Yichi Zhang +5Multimodal KGs Continual Learning
Key Laboratory of High Confidence Software Technologies (PKU), Ministry of Education School of Computer Science, Peking University Beijing, China · School of Computer Science Zhejiang University Hangzhou, China · School of Cyber Science and Engineering Wuhan University Wuhan, China +1

Mar 9, 2026J/K move · Enter open · S save

GraphVLM: Benchmarking Vision Language Models for Multimodal Graph Learning

Jiajin Liu, Dongzhe Fan, Chuanhao Ji, Daochen Zha+1

NYU Shanghai · New York University · East China Normal University+1

Multimodal Graph Learning Multimodal Model Evaluation

Open paper AskPDF arXiv

Vision-Language Models (VLMs) have demonstrated remarkable capabilities in aligning and understanding multimodal signals, yet their potential to reason over structured data, where multimodal entities are connected through explicit relational graphs, remains largely underexplored. Unlocking this capability is crucial for real-world applications such as social networks, recommendation systems, and scientific discovery, where multimodal information is inherently structured. To bridge this gap, we present GraphVLM, a systematic benchmark designed to evaluate and harness the capabilities of VLMs for multimodal graph learning (MMGL). GraphVLM investigates three complementary paradigms for integrating VLMs with graph reasoning: (1) VLM-as-Encoder, which enriches graph neural networks through multimodal feature fusion; (2) VLM-as-Aligner, which bridges modalities in latent or linguistic space to facilitate LLM-based structured reasoning; and (3) VLM-as-Predictor, which directly employs VLMs as multimodal backbones for graph learning tasks. Extensive experiments across six datasets from diverse domains demonstrate that VLMs enhance multimodal graph learning via all three roles. Among these paradigms, VLM-as-Predictor achieves the most substantial and consistent performance gains, revealing the untapped potential of vision-language models as a new foundation for multimodal graph learning. The benchmark code is publicly available at https://github.com/oamyjin/GraphVLM.

Abstract

Explore similar work

GraphVLM: Benchmarking Vision Language Models for Multimodal Graph Learning

ViSR-KGC: Visual Subgraph Reasoning with Vision-Language Models for Multimodal Knowledge Graph Completion

When Modalities Remember: Continual Learning for Multimodal Knowledge Graphs