Knowing When to Trust Images: Reliability-Aware Multi-modal Entity Alignment

Chenxiao Li; Yunhe Feng; Dongfang Liu; Dong Nie; Yan Huang; Heng Fan

cs.CLSep 20, 2026

Knowing When to Trust Images: Reliability-Aware Multi-modal Entity Alignment

Authors: Chenxiao Li, Yunhe Feng, Dongfang Liu, Dong Nie, Yan Huang, Heng Fan

Organizations: University of North Texas · Purdue University · Meta Inc

Abstract

The visual modality, i.e., images, plays a key role in multi-modal entity alignment (MMEA). Existing approaches often directly fuse the image with other modalities to align different entities. Although simple, such strategies overlook the potential noise in the images and their semantic misalignment with corresponding entities, resulting in suboptimal fusion and degraded performance. Addressing this, we propose a novel Reliability-Aware framework for MMEA (RA-MMEA), which assesses visual reliability and adaptively improves unreliable visual representations for robust entity alignment. The core lies in two modules, including dependency-aware visual reliability prediction (DA-VRP) and stability-regularized visual embedding generation (SR-VEG). The former aims to estimate the reliability of an image by leveraging multi-modal dependency within the entity, while the latter focuses on producing alternative visual representation conditioned on semantics encoded in textual modalities for multi-modal fusion. Compared to current methods, RA-MMEA enables more reliable visual representations for modality fusion, thereby improving performance. In extensive experiments, RA-MMEA achieves state-of-the-art results, verifying the importance of reliable visual modality for entity alignment and the effectiveness of RA-MMEA. The code and results will be released.

Figures & tables

Figure 1: Comparison of existing methods directly fusing image and other modalities in (a) and RA-MMEA that first assesses the reliability of the image and then performs fusion in (b). Best viewed in pdf for all figures .

Figure 2: Illustration of the proposed RA-MMEA, which assesses visual modality reliability and adaptively improves unreliable visual representations through generated feature for MMEA. (a) Inputs with image, graph, relation, and attribute; (b) Multi-modal Embedding; (c) Dependency-Aware Visual Reliability Prediction; (d) Stability-Regularized Visual Embedding Generation; and (e) Multi-modal Fusion.

Setting	Models	Venue	FB15K-DB15K			FB15K-YG15K
Setting	Models	Venue	Hits@1 $\uparrow$	Hits@10 $\uparrow$	MRR $\uparrow$	Hits@1 $\uparrow$	Hits@10 $\uparrow$	MRR $\uparrow$
Non-iterative	MMEA ( Chen et al., 2020 )	KSEM’20	0.265	0.541	0.357	0.234	0.480	0.317
	EVA ( Liu et al., 2021 )	AAAI’21	0.199	0.448	0.283	0.153	0.361	0.224
	MSNEA ( Chen et al., 2022 )	KDD’22	0.114	0.296	0.175	0.103	0.249	0.153
	MCLEA ( Lin et al., 2022 )	COLING’22	0.295	0.582	0.393	0.254	0.484	0.332
	MoAlign ( Li et al., 2023a )	EMNLP’23	0.318	0.564	0.409	0.296	0.525	0.378

Table 1: Results on two cross-KG datasets using 20% of the reference entity alignments for training. The best results are shown in bold and the second best results are underlined .

Figure 3: Performance of strong baselines across visual noise levels on FB15K-DB15K.

Noise Ratio	Ratio=20%	Ratio=50%	Ratio=80%
Detection Rate (%)	75.24	74.87	75.08

Table 2: Detection rate of DA-VRP under varying visual noise ratios.

	DA-VRP	SR-VEG	Hits@1 $\uparrow$	Hits@10 $\uparrow$	MRR $\uparrow$
❶	✗	✗	0.682	0.860	0.733
❷	✓	✗	0.798	0.886	0.855
❸	✗	✓	0.701	0.873	0.752
❹	✓	✓	0.849	0.948	0.879

Table 3: Ablation studies on DA-VRP and SR-VEG.

		Hits@1 $\uparrow$	Hits@10 $\uparrow$	MRR $\uparrow$
❶	DA-VRP w/ BS	0.503	0.762	0.590
❷	DA-VRP w/ GS	0.849	0.948	0.879

Table 4: Ablation studies on Gumbel-Softmax (GS) and Bernoulli Sampling (BS) in DA-VRP.

		Hits@1 $\uparrow$	Hits@10 $\uparrow$	MRR $\uparrow$
❶	SR-VEG w/o $\mathcal{L}_{p}$	0.833	0.941	0.874
❷	SR-VEG w/ $\mathcal{L}_{p}$	0.849	0.948	0.879

Table 5: Ablation studies on

\mathcal{L}_{p}

for SR-VEG.

		Hits@1 $\uparrow$	Hits@10 $\uparrow$	MRR $\uparrow$
❶	generation w/ $\hat{w}_{k}^{m}$	0.830	0.938	0.867
❷	generation w/ $w_{k}^{m}$	0.849	0.948	0.879

Table 6: Ablation studies on using

\hat{w}_{k}^{m}

and using

w_{k}^{m}

for hybrid representation in SR-VEG.

		Hits@1 $\uparrow$	Hits@10 $\uparrow$	MRR $\uparrow$
❶	$\gamma=1$	0.834	0.942	0.874
❷	$\gamma=5$	0.836	0.943	0.875
❸	$\gamma=10$	0.845	0.946	0.877
❹	$\gamma=15$	0.849	0.948	0.879
❺	$\gamma=20$	0.840	0.945	0.879

Table 7: Ablation studies on the parameter

\gamma

for

\mathcal{L}_{p}

Models	Hits@1 $\uparrow$	Hits@10 $\uparrow$	MRR $\uparrow$
MEAformer ( Chen et al., 2023a )	0.417	0.715	0.518
MEAformer (+)	0.506	0.758	0.594
Improvement (%)	8.9	4.3	7.6
RICEA Li et al. (2025a)	0.471	0.720	0.557
RICEA (+)	0.501	0.755	0.589
Improvement (%)	3.0	3.5	3.2

Table 8: Plug-and-play evaluation of DA-VRP and SR-VEG across representative MMEA frameworks. "+" denotes their variants.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	KG	Ent.	Rel.	Attr.	Rel. Tr.	Attr. Tr.	Image	EA pairs
FB15K-DB15K	FB15K	14,951	1,345	116	592,213	29,395	14,836	12,846
FB15K-DB15K	DB15K	12,842	279	225	89,197	48,080	12,812	12,846
FB15K-YG15K	FB15K	14,951	1,345	116	592,213	29,395	14,836	11,199
FB15K-YG15K	YG15K	15,404	32	7	122,886	23,532	11,172	11,199
DBP15K ZH-EN	ZH	19,388	1,701	8,111	70,414	248,035	15,912	15,000
DBP15K ZH-EN	EN	19,572	1,323	7,173	95,142	343,218	14,125	15,000

Appendix

Table 9: Statistics of MMEA datasets, with EA pairs representing the reference entity alignments.

Setting	Models	FB15K-DB15K			FB15K-YG15K
Setting	Models	Hits@1 $\uparrow$	Hits@10 $\uparrow$	MRR $\uparrow$	Hits@1 $\uparrow$	Hits@10 $\uparrow$	MRR $\uparrow$
Non-iterative	MMEA ( Chen et al., 2020 )	0.417	0.703	0.512	0.403	0.645	0.486
	EVA ( Liu et al., 2021 )	0.334	0.589	0.422	0.311	0.534	0.388
	MSNEA ( Chen et al., 2022 )	0.288	0.590	0.388	0.320	0.589	0.413
	MCLEA ( Lin et al., 2022 )	0.555	0.784	0.637	0.501	0.705	0.574
	MoAlign ( Li et al., 2023a )	0.576	0.749	0.634	0.550	0.713	0.617

Appendix

Table 10: Results on two cross-KG datasets using 50% of the reference entity alignments for training. The best results are shown in bold and the second best results are underlined .

Setting	Models	FB15K-DB15K			FB15K-YG15K
Setting	Models	Hits@1 $\uparrow$	Hits@10 $\uparrow$	MRR $\uparrow$	Hits@1 $\uparrow$	Hits@10 $\uparrow$	MRR $\uparrow$
Non-iterative	MMEA ( Chen et al., 2020 )	0.590	0.869	0.685	0.598	0.839	0.682
	EVA ( Liu et al., 2021 )	0.484	0.696	0.563	0.491	0.692	0.565
	MSNEA ( Chen et al., 2022 )	0.518	0.779	0.613	0.531	0.778	0.620
	MCLEA ( Lin et al., 2022 )	0.735	0.890	0.790	0.667	0.824	0.722
	MoAlign ( Li et al., 2023a )	0.699	0.882	0.773	0.689	0.884	0.769

Appendix

Table 11: Results on two cross-KG datasets using 80% of the reference entity alignments for training. The best results are shown in bold and the second best results are underlined .

Non-iterative
Models	$\textbf{DBP15K}_{\emph{ZH-EN}}$			$\textbf{DBP15K}_{\emph{JA-EN}}$			$\textbf{DBP15K}_{\emph{FR-EN}}$
Models	Hits@1 $\uparrow$	Hits@10 $\uparrow$	MRR $\uparrow$	Hits@1 $\uparrow$	Hits@10 $\uparrow$	MRR $\uparrow$	Hits@1 $\uparrow$	Hits@10 $\uparrow$	MRR $\uparrow$
EVA ( Liu et al., 2021 )	0.891	0.961	0.917	0.941	0.986	0.958	0.970	0.996	0.982
MSNEA ( Chen et al., 2022 )	0.859	0.936	0.887	0.921	0.970	0.939	0.954	0.989	0.968
MCLEA ( Lin et al., 2022 )	0.860	0.950	0.893	0.914	0.975	0.938	0.953	0.990	0.967
MEAformer ( Chen et al., 2023a )	0.909	0.974	0.933	0.950	0.990	0.965	0.972	0.997	0.983

Appendix

Table 12: Unsupervised results on bilingual datasets are presented.

Figure 4: Performance of strong baselines with fewer seed alignments on FB15K-DB15K.

Models	Hits@1 $\uparrow$	Hits@10 $\uparrow$	MRR $\uparrow$
RA-MMEA-R	0.708	0.824	0.745
RA-MMEA-A	0.732	0.845	0.811
RA-MMEA-V	0.798	0.886	0.855

Appendix

Table 13: Performance of applying our reliability prediction method to textual modalities.

		Hits@1 $\uparrow$	Hits@10 $\uparrow$	MRR $\uparrow$
❶	Concatenation	0.831	0.941	0.873
❷	Weighted-Concatenation	0.835	0.944	0.876
❸	Ours	0.849	0.948	0.879

Appendix

Table 14: Comparison of strategies for handling unreliable visual representations.

Figure 5: Case study of visual reliability on DBP15K, showing representative images predicted as unreliable (left) and reliable (right) by DA-VRP based on their cross-modal dependency with the corresponding entity semantics. The entity associated with each image is shown above the image.

Models	Train. Time	Learnable Paras.	GPU Mem.
MEAformer ( Chen et al., 2023a ) *	22min	10M	13.6G
RICEA Li et al. (2025a) *	22min	10M	13.2G
RA-MMEA (Ours)	24min	13M	16.0G

Appendix

Table 15: Comparison of complexity on FB15K-DB15K. * indicates our reproduced results.

Cross-Modal Alignment

Cross-modal alignment maps representations from different data types into compatible spaces so models can compare, transfer, and combine information.

See all 440 papers

Silence and Noise: Resolving Suppression and Overvaluation in Multimodal Entity Alignment
Oct 12, 2026Xizhe Zhang, Jingfeng Zhang, Meng-Fen ChiangCross-Modal Alignment Multimodal KGs
School of Computer Science, The University of Auckland, New Zealand · College of Intelligent Robotics and Advanced Manufacturing/Institute of Trustworthy Embodied AI, Fudan University, China · College of Electrical and Computer Engineering, National Yang Ming Chiao Tung University, Taiwan
SAVER: Selective As-Needed Vision Evidence for Multimodal Information Extraction
May 20, 2026Miaobo Hu, Shuhao Hu, Bokun Wang +5Relation Extraction Named Entity Recognition
Institute of Information Engineering, Chinese Academy of Sciences, Beijing 100092, China · School of Cyber Security, University of Chinese Academy of Sciences, Beijing 100049, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China
MiMIC: Mitigating Visual Modality Collapse in Universal Multimodal Retrieval While Avoiding Semantic Misalignment
Apr 23, 2026Juan Li, Chuanghao Ding, Xujie Zhang +1Cross-Modal Learning Cross-Modal Alignment
State Key Laboratory for Novel Software Technology, Nanjing University

Oct 12, 2026J/K move · Enter open · S save

Silence and Noise: Resolving Suppression and Overvaluation in Multimodal Entity Alignment

Xizhe Zhang, Jingfeng Zhang, Meng-Fen Chiang

School of Computer Science, The University of Auckland, New Zealand · College of Intelligent Robotics and Advanced Manufacturing/Institute of Trustworthy Embodied AI, Fudan University, China · College of Electrical and Computer Engineering, National Yang Ming Chiao Tung University, Taiwan

Cross-Modal Alignment Multimodal KGs Missing-Modality Learning

Open paper AskPDF arXiv

Multimodal entity alignment (MMEA) is fundamental to integrating heterogeneous knowledge graphs (KGs), yet existing paradigms remain critically vulnerable to incomplete modalities and cross-graph structural asymmetries. This paper exposes a fundamental oversight in current MMEA research: treating feature missingness and topological discrepancies as isolated preprocessing or denoising challenges, which overlooks their destructive coupling during multimodal fusion. We systematically formulate and evaluate two newly identified failure modes: 1) Detail Missing (DM)-induced overvaluation, where the absence of local modality details causes models to overemphasize uninformative, blurred representations; and 2) Context Imbalance (CI)-induced suppression, where topological discrepancies across KGs allow dense structural contexts to overwhelm weaker but highly discriminative modal signals, triggering high-level semantic drift. To break this cycle of degradation, we introduce ALMEA, a generative and calibration framework that establishes a bidirectional feedback loop between low-level feature reconstruction and high-level latent space alignment. Specifically, Cross-Modality Imputation (CMI) synthesizes missing modality details to mitigate overvaluation, while Latent Semantic Calibration (LSC) aligns cross-graph joint embeddings to suppress topology-driven semantic drift. Under sparse supervision, we develop Controlled Iterative Selection (CIS) to adaptively acquire representative and diverse pseudo-labeled pairs, optimizing latent calibration under constrained budgets. Extensive evaluations on standard benchmarks demonstrate that ALMEA consistently outperforms state-of-the-art baselines. At a 20% supervision rate, ALMEA yields substantial absolute improvements of up to 5.20% in MRR and 6.40% in Hits@1, maintaining robust gains across higher supervision thresholds.

Abstract

Cross-Modal Alignment

Explore similar work

Silence and Noise: Resolving Suppression and Overvaluation in Multimodal Entity Alignment

SAVER: Selective As-Needed Vision Evidence for Multimodal Information Extraction

MiMIC: Mitigating Visual Modality Collapse in Universal Multimodal Retrieval While Avoiding Semantic Misalignment