Multi-Modal Knowledge Graphs (MMKGs) benefit from visual information, yet large-scale image collection is hard to curate and often excludes ambiguous but relevant visuals (e.g., logos, symbols, abstract scenes). We present Beyond Images, an automatic data-centric enrichment pipeline with optional human auditing. This pipeline operates in three stages: (1) large-scale retrieval of additional entity-related images, (2) conversion of all visual inputs into textual descriptions to ensure that ambiguous images contribute usable semantics rather than noise, and (3) fusion of multi-source descriptions using a large language model (LLM) to generate concise, entity-aligned summaries. These summaries replace or augment the text modality in standard MMKG models without changing their architectures or loss functions. Across three public MMKG datasets and multiple baseline models, we observe consistent gains (up to 7% Hits@1 overall). Furthermore, on a challenging subset of entities with visually ambiguous logos and symbols, converting images into text yields large improvements (201.35% MRR and 333.33% Hits@1). Additionally, we release a lightweight Text-Image Consistency Check Interface for optional targeted audits, improving description quality and dataset reliability. Our results show that scaling image coverage and converting ambiguous visuals into text is a practical path to stronger MMKG completion. Code, datasets, and supplementary materials are available at https://github.com/pengyu-zhang/Beyond-Images.
Figures & tables
Figure 1: Original MMKG Dataset : the entity “ Amsterdam ” with its photo and a textual description. Ambiguous yet Relevant Images : additional visuals such as the three red Saint Andrew’s crosses and an abstract cityscape, whose semantics may be unclear when incorporated directly as visual features. Textual Descriptions of the Additional Ambiguous Images : our framework converts these images into concise, entity-aligned text, expanding semantic coverage while mitigating noise from ambiguous visual embeddings.
Figure 2: Overview of our Beyond Images framework. Given an MMKG entity, (1) the Modality Extension Module retrieves additional web images from search engines; (2) the Semantic Generation Module produces per-image textual descriptions for both original and newly retrieved images; (3) the LLM-based Semantic Fusion Module consolidates valid descriptions into a single, rich summary paragraph via an explicit prompt. The summary becomes an enhanced textual view of the entity and is stored in the new enriched dataset for downstream MMKG completion.
MKG-W
MKG-Y
DB15K
Entity
15,000
15,000
12,842
Relation
169
28
279
Train
34,196
21,310
79,222
Validation
4,276
2,665
9,902
Test
4,274
2,663
9,904
Text
14,123
12,305
9,078
Table 1: Statistics of the three public MMKG datasets used in our experiments. Original Images are the image statistics provided by the source MMKGs. New Images are the additional images retrieved by our framework to extend coverage. “Avg Img” denotes the average number of images per entity; “Entity w/ Img” is the number of entities with at least one image.
MKG-W
MKG-Y
DB15K
MRR
H@1
H@3
H@10
MRR
H@1
H@3
H@10
MRR
H@1
H@3
H@10
MMRNS [ 31 ]
Baseline
35.03
28.59
37.49
47.47
35.93
30.53
39.07
45.47
32.68
23.01
37.86
51.01
G(o)
35.73
29.65
38.37
48.69
36.59
31.78
40.19
46.43
33.57
24.04
39.13
52.71
G(n)
36.13
29.93
38.58
49.02
36.93
31.96
40.33
46.58
33.37
23.78
39.02
52.40
G(o+n)
36.26
30.08
38.70
49.19
37.03
32.12
40.46
46.70
33.67
24.16
39.27
52.89
Table 2: Link prediction results on three MMKG datasets with four models (MMRNS, MyGO, NativE, AdaMF). Rows denote input settings: Baseline (original MMKG only), G(o) (textual descriptions from original images), G(n) (textual descriptions from newly retrieved images), G(o+n) (concatenation of G(o) and G(n) ), and Fusion (an LLM summary generated from all image descriptions). Columns report MRR and Hits@ K . Bold numbers indicate the best setting per model and dataset. The last row, improv.(%) , gives the relative gain of Fusion over the Baseline for each metric.
MRR
H@1
H@3
H@10
Baseline
13.89
7.50
15.00
27.50
Fusion
41.87
32.50
47.50
57.50
improv.(%)
+ 201.35
+ 333.33
+ 216.67
+ 109.09
Table 3: Performance on a subset of entities whose image sets include logos or symbols . Baseline uses the original MMKG without converting these images. Fusion applies our framework to convert the logo-like images into textual descriptions. Results show that our framework recovers usable semantics, yielding large gains in MRR and Hits@ K .
Figure 3: Hits@1 comparison on the MKG-W dataset across four models under three modality settings: I+T (image embeddings + textual descriptions), T+G (textual descriptions + image-generated descriptions), and I+T+G (all). Legend: G(o) denotes descriptions from original images, G(n) from newly retrieved images, G(o+n) their concatenation, and LLM Fusion an LLM summary over all descriptions. Bars report Hits@1 (higher is better).
MRR
H@1
H@3
H@10
Image Only
31.70
24.47
32.34
41.68
Text Only
35.93
28.41
36.75
45.09
Image + Text
36.10
29.78
38.54
47.75
Table 4: Modality ablation on the MKG-W dataset. We compare Image Only , Text Only , and Image + Text . Text alone outperforms image alone across all metrics, while fusing both modalities yields the best MRR and Hits@ K .
MRR
H@1
H@3
H@10
Image-to-Text Models (no LLM Fusion)
Baseline
36.10
29.78
38.54
47.75
Git-large-coco
36.59
30.61
38.57
48.02
Llava-v1.5-7b
36.21
30.15
38.40
47.83
Blip2-flan-t5-xxl
37.42
31.42
39.88
49.35
LLM Fusion (using Blip2 for image-to-text)
Table 5: Performance comparison of different pre-trained image-to-text generation models on the MKG-W dataset using the MyGO model. “ H@ K ” stands for “Hits at K .”