KARAMEL: Knowledge-Aware Ranking for Multimodal Entity Linking in the Arts via Heterogeneous Graphs

Raffaele Scaringi; Alejandro Sierra-Múnera; Gennaro Vessio; Giovanna Castellano; Ralf Krestel

cs.CVOct 12, 2026

KARAMEL: Knowledge-Aware Ranking for Multimodal Entity Linking in the Arts via Heterogeneous Graphs

Authors: Raffaele Scaringi, Alejandro Sierra-Múnera, Gennaro Vessio, Giovanna Castellano, Ralf Krestel

Organizations: Department of Computer Science, University of Bari Aldo Moro, Italy · Hasso Plattner Institute, University of Potsdam, Germany · ZBW - Leibniz Information Centre for Economics, University of Kiel, Germany

Abstract

Discovering and disambiguating entities from multimodal and heterogeneous data is a key challenge in discovery science, where relevant knowledge is often implicit and distributed across modalities. We present KARAMEL, a framework for multimodal entity linking (MEL) - the task of aligning visual and textual mentions to entities in a knowledge base - in the cultural heritage domain. Unlike prior vision-language models that primarily rely on surface-level multimodal alignment, our approach incorporates sociohistorical and contextual knowledge through a knowledge-aware architecture that fuses a vision-language encoder with a heterogeneous graph neural network. This design enables contextual reasoning over complex relational structures and supports robust entity disambiguation under ambiguous or underspecified evidence. To benchmark multimodal discovery in this setting, we introduce WikiMuSA (Wikidata-based Multimodal Semantic data for Art), a new dataset that links artworks to Wikidata entities via images, textual descriptions, and structured contextual knowledge. Experimental results show that KARAMEL surpasses strong baselines, particularly on implicit or abstract mentions, while ablation studies highlight the critical role of domain-specific, structured context in learning expressive representations. While instantiated in the cultural heritage domain, KARAMEL addresses a general knowledge discovery problem and applies to other scientific settings involving multimodal observations and heterogeneous background knowledge. Both the dataset and source code are made publicly available.

Figures & tables

Figure 1: Comparison between the standard multimodal entity linking setting (a), where entity mentions are explicitly highlighted in the text, and our proposed setting (b), where mentions must be inferred from raw Wikipedia articles without predefined annotations.

Figure 2: Logical schema of $\mathcal{W}$ iki $\mathcal{M}$ u $\mathcal{SA}$ . Each artwork is associated with metadata, including author, genre, and movement. Depicted entities are linked through enriched, contextual relationships that enhance expressiveness and graph connectivity.

Figure 3: Architecture of KARAMEL. The image-text cross-encoder and the GNN encoder are jointly trained to extract complementary information from artworks and depicted entities. The cross-modal ranker is then trained to rank candidate entities effectively for a given input mention.

Model	R@1	R@10	R@50	R@100	MAP	MRR
BERT [ 43 ]	2.33 ( $\pm$ 0.88)	10.56 ( $\pm$ 2.53)	24.52 ( $\pm$ 3.61)	32.41 ( $\pm$ 3.2)	5.93 ( $\pm$ 1.53)	11.94 ( $\pm$ 2.36)
CLIP [ 41 ]	0.24 ( $\pm$ 0.08)	2.68 ( $\pm$ 0.68)	13.12 ( $\pm$ 2.9)	22.57 ( $\pm$ 4.12)	1.25 ( $\pm$ 0.09)	3.0 ( $\pm$ 0.19)
ViLT [ 47 ]	0.62 ( $\pm$ 0.1)	5.01 ( $\pm$ 0.58)	16.4 ( $\pm$ 0.51)	23.71 ( $\pm$ 1.27)	1.08 ( $\pm$ 0.22)	2.43 ( $\pm$ 0.5)
MIMIC [ 12 ]	27.22 ( $\pm$ 1.19)	60.93 ( $\pm$ 1.32)	77.26 ( $\pm$ 0.75)	82.52 ( $\pm$ 0.62)	47.0 ( $\pm$ 1.66)	65.72 ( $\pm$ 1.88)
KARAMEL ( Ours )	27.26 ( $\pm$ 0.47)	62.31 ( $\pm$ 0.57)	78.48 ( $\pm$ 0.67)	83.65 ( $\pm$ 0.45)	48.45 ( $\pm$ 0.7)	66.88 ( $\pm$ 0.51)

Table 1: Ranking metrics for the baselines and KARAMEL. Each cell reports the mean and standard deviation (in parentheses). All metrics are expressed in the range [0, 100]. The best performance per metric is shown in bold .

Model	R@1	R@10	R@50	R@100	MAP	MRR
MIMIC [ 12 ]	50.07 ( $\pm$ 1.98)	83.0 ( $\pm$ 1.47)	91.76 ( $\pm$ 1.28)	94.73 ( $\pm$ 0.91)	70.08 ( $\pm$ 1.96)	80.44 ( $\pm$ 1.67)
KARAMEL	51.07 ( $\pm$ 1.62)	82.91 ( $\pm$ 2.35)	91.09 ( $\pm$ 2.09)	94.25 ( $\pm$ 1.7)	71.34 ( $\pm$ 1.92)	81.34 ( $\pm$ 1.57)

Table 2: Ranking metrics for MIMIC and KARAMEL on MELArt [ 10 ] . Each cell reports the mean and standard deviation (in parentheses). All metrics are expressed in the range [0, 100]. The best performance per metric is shown in bold .

Figure 4: Average rank difference of KARAMEL on MIMIC [ 12 ] . Bars below the x-axis indicate improved ranking of KARAMEL. Numbers within each bar denote the number of ( artwork , depicted entity ) pairs supporting the corresponding feature. Blue bars denote statistically significant effects, whereas red bars indicate non-significant effects.

Image & Text		MIMIC	KARAMEL
	1	Oar. water-borne propulsion	Polyphemus giant son of Poseidon.
	2	Polyphemus . giant son of Poseidon.	Odysseus . Greek king of Ithaca.
The artwork “Odysseus and Polyphemus” represents a pivotal moment from Homer’s epic poem, where Odysseus taunts the one-eyed Cyclops Polyphemus as he prepares to hurl a rock at the escaping boat of Odysseus and his crew.	3	wind wave . waves that occur on water	oar . water-borne propulsion
	4	cyclops . giant in Greek mythology.	cyclops . giant in Greek mythology.
Ground Truth: Polyphemus . giant son of Poseidon. Odysseus . Greek king of Ithaca.

Table 3: Top-4 ranking comparison for “ Odysseus and Polyphemus ” ( Q20537891 ), a painting by Arnold Böcklin (1896), between KARAMEL and MIMIC [ 12 ] . Ground truth entities are highlighted.

Full Graph	Encoding	R@10	MAP	MRR
✗	Multimodal	60.82 ( $\pm$ 1.42)	47.11 ( $\pm$ 1.80)	65.44 ( $\pm$ 2.17)
✗	Hybrid	60.44 ( $\pm$ 2.32)	46.96 ( $\pm$ 3.39)	65.64 ( $\pm$ 3.55)
✗	Textual	61.42 ( $\pm$ 1.92)	47.87 ( $\pm$ 2.72)	66.74( $\pm$ 3.05)
✓	Multimodal	62.05 ( $\pm$ 0.38)	48.28 ( $\pm$ 0.83)	67.53 ( $\pm$ 1.09)
✓	Hybrid	61.87 ( $\pm$ 0.62)	48.81 ( $\pm$ 0.69)	67.93 ( $\pm$ 1.04)
✓	Textual	62.31 ( $\pm$ 0.57)	48.45 ( $\pm$ 0.70)	66.88 ( $\pm$ 0.51)

Table 4: Ablation study on the effect of both contextual factors (i.e., Full Graph) and node feature encoding variations (i.e., Textual, Hybrid, and Multimodal). Numbers in bold indicate the best results.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Distribution of artwork Wikipedia articles (

x

-axis) by original language (

y

-axis).

	Rouge1	Rouge2	RougeL
mean	0.745735	0.381481	0.530465
std	0.158000	0.138832	0.134143

Appendix

Table 5: ROUGE precision metrics

Figure 6: Length in words of the original articles and their summaries. The color represents the ROUGE-1 precision for each article-summary pair.

Figure 7: Distribution of ROUGE precision between the summaries and the full articles.

Cultural Heritage

Cultural heritage uses AI to document, analyze, preserve, and interpret artifacts, sites, archives, and living traditions.

See all 57 papers

Multi-Perspective Evidence Synthesis and Reasoning for Unsupervised Multimodal Entity Linking
Apr 22, 2026Mo Zhou, Jianwei Wang, Kai Wang +3
The University of New South Wales Sydney, Australia · Shanghai Jiao Tong University Shanghai, China · University of Technology Sydney Sydney, Australia
Think Before You Link: Rarity, Reasoning, and Retrieval in Multilingual Entity Linking
Sep 9, 2026Parinthapat Pengpun, Simran Khanuja, Graham NeubigMultimodal Grounding VLM Reasoning
Carnegie Mellon University
VL-KGE: Vision-Language Models Meet Knowledge Graph Embeddings
Mar 2, 2026Athanasios Efthymiou, Stevan Rudinac, Monika Kackovic +2KG Embedding Multimodal KGs
University of Amsterdam, Amsterdam, The Netherlands

Apr 22, 2026J/K move · Enter open · S save

Multi-Perspective Evidence Synthesis and Reasoning for Unsupervised Multimodal Entity Linking

Mo Zhou, Jianwei Wang, Kai Wang, Helen Paik+2

The University of New South Wales Sydney, Australia · Shanghai Jiao Tong University Shanghai, China · University of Technology Sydney Sydney, Australia

Open paper AskPDF arXiv

Multimodal Entity Linking (MEL) is a fundamental task in data management that maps ambiguous mentions with diverse modalities to the multimodal entities in a knowledge base. However, most existing MEL approaches primarily focus on optimizing instance-centric features and evidence, leaving broader forms of evidence and their intricate interdependencies insufficiently explored. Motivated by the observation that human expert decision-making process relies on multi-perspective judgment, in this work, we propose MSR-MEL, a Multi-perspective Evidence Synthesis and Reasoning framework with Large Language Models (LLMs) for unsupervised MEL. Specifically, we adopt a two-stage framework: (1) Offline Multi-Perspective Evidence Synthesis constructs a comprehensive set of evidence. This includes instance-centric evidence capturing the instance-centric multimodal information of mentions and entities, group-level evidence that aggregates neighborhood information, lexical evidence based on string overlap ratio, and statistical evidence based on simple summary statistics. A core contribution of our framework is the synthesis of group-level evidence, which effectively aggregates vital neighborhood information by graph. We first construct LLM-enhanced contextualized graphs. Subsequently, different modalities are jointly aligned through an asymmetric teacher-student graph neural network. (2) Online Multi-Perspective Evidence Reasoning leverages the power of LLM as a reasoning module to analyze the correlation and semantics of the multi-perspective evidence to induce an effective ranking strategy for accurate entity linking without supervision. Extensive experiments on widely used MEL benchmarks demonstrate that MSR-MEL consistently outperforms state-of-the-art unsupervised methods. The source code of this paper was available at: https://anonymous.4open.science/r/MSR-MEL-C21E/.

Abstract

Cultural Heritage

Explore similar work

Multi-Perspective Evidence Synthesis and Reasoning for Unsupervised Multimodal Entity Linking

Think Before You Link: Rarity, Reasoning, and Retrieval in Multilingual Entity Linking

VL-KGE: Vision-Language Models Meet Knowledge Graph Embeddings