More Perspectives, Stronger Signals: Multi-Perspective Enhancement and Progressive Fusion for Multimodal Entity Representation Learning

Chenyi Xiong; Yan Zhang; Jing Hu; Ziyue Qin; Kui Xiao; Xiaopan Lyu; Xiaoju Hou; Zhifei Li

cs.AIAug 29, 2026

More Perspectives, Stronger Signals: Multi-Perspective Enhancement and Progressive Fusion for Multimodal Entity Representation Learning

Authors: Chenyi Xiong, Yan Zhang, Jing Hu, Ziyue Qin, Kui Xiao, Xiaopan Lyu, Xiaoju Hou, Zhifei Li

Organizations: School of Computer Science, Hubei University Wuhan, China · School of Cyber Science and Technology, Hubei University Wuhan, China · Institute of Vocational Education, Guangdong Industry Polytechnic University Guangzhou, China · School of Computer Science, Hubei Key Laboratory of Big Data Intelligent Analysis and Application, Hubei University Wuhan, China

Abstract

Learning effective multimodal entity representations is fundamental for reasoning tasks such as multimodal knowledge graph completion (MMKGC). However, existing methods often suffer from semantic over-smoothing within modalities and ineffective noise filtration across modalities, particularly under sparse or ambiguous conditions. To overcome these limitations, we propose PrismF, a unified framework that synergizes multi-perspective enhancement with progressive fusion to extract stronger signals from diverse inputs. PrismF enhances fine-grained intra-modal semantics through a multi-perspective mechanism that decomposes each modality into complementary views and constrains them with a decoupling loss to reduce representation collapse. Furthermore, it improves cross-modal integration through a progressive fusion strategy that dynamically calibrates inter-modal interactions, enabling the model to emphasize informative signals while suppressing noisy or unreliable ones. Extensive experiments on three public benchmarks show that PrismF achieves the strongest overall performance, including relative improvements of 4.04% in MRR and 11.17% in Hits@1 on KVC16K. Our code can be found at https://github.com/HubuKG/PrismF.

Figures & tables

Figure 1. Illustration of reasoning challenges in MMKGC. When querying ( Thunderstruck, has_cast, ? ), textual ambiguity and visual similarity between entities with similar names ( Kevin Durant vs. Durand Scott ) lead to reasoning confusion.

Figure 2. Overview of the PrismF framework. (a) Multi-Perspective Enhancement: decomposes each modality into multiple learnable perspectives with noise modeling to capture intra-modal variability and enhance fine-grained representations. (b) Progressive Modality Fusion: dynamically integrates cross-modal features through confidence calibration, mutual gating, and relation-scaled attention to achieve multimodal fusion. (c) Link Prediction: leverages fused and modality-specific representations with Tucker scoring for robust link prediction in KGs.

Dataset	#Entity	#Relation	#Train	#Valid	#Test	Image		Text
Dataset	#Entity	#Relation	#Train	#Valid	#Test	Num.	Dim.	Num.	Dim.
DB15K	12,842	279	79,222	9,902	9,904	12,818	4,096	9,078	768
MKG-Y	15,000	28	21,310	2,665	2,663	14,244	383	12,305	384
KVC16K	16,015	4	180,190	22,523	22,525	14,822	768	14,822	768

Table 1. Statistics of the multimodal knowledge graph datasets in our experiments. The image and text modality features are provided by the original datasets and are kept the same for all baselines.

Model	DB15K				MKG-Y				KVC16K
	MRR	Hits			MRR	Hits			MRR	Hits
	MRR	@1	@3	@10	MRR	@1	@3	@10	MRR	@1	@3	@10
Unimodal KGC Models
TransE (NeurIPS’13)	24.86	12.78	31.48	47.07	30.73	23.45	35.18	43.37	8.54	0.64	10.97	23.42
DistMult (ICLR’15)	23.03	14.78	26.28	39.59	25.04	19.33	27.80	35.95	6.37	3.03	6.11	12.61
ComplEx (ICML’16)	27.48	18.37	31.57	45.37	28.71	22.26	32.12	40.93	12.85	7.48	13.79	23.18

Table 2. Performance comparison of different knowledge graph completion models. The optimal results are highlighted in bold , while the second optimal results are underlined .

Figure 3. The MRR and Hits@1 results of removing each modality on DB15K.

Setting	DB15K		MKG-Y		KVC16K
Setting	MRR	Hits@1	MRR	Hits@1	MRR	Hits@1
w/o MuPE	37.98	31.41	38.13	34.98	16.05	9.32
w/o loss $\mathcal{L}_{pd}$	38.18	31.64	38.31	35.17	16.27	9.61
w/o noise $\epsilon^{i}_{m}$	38.72	32.17	38.23	35.01	16.09	9.39
w/o PMF	37.91	31.22	37.36	33.96	16.32	9.78
w/ concat as $\mathbf{g}$	38.03	30.62	37.98	34.82	16.58	10.12

Table 3. Performance comparison of different settings. We explored the impact of each key component.

Figure 4. Hyperparameter performance analysis across all datasets.

Figure 5. Performance comparison on the DB15K under two challenging scenarios: modality missing and data sparsity.

Figure 6. The visualization of modality contributions for PrismF. Here, we present the contributions of the structural, visual, textual, and fused multimodal during the reasoning process across all datasets.

Figure 7. Case examples. PrismF assigns larger weights to the more informative modality based on content reliability, effectively mitigating noise and ambiguity.

Multimodal Learning

Multimodal learning studies how models combine and reason over different data types, such as images, text, audio, signals, and structured records.

See all 338 papers

Silence and Noise: Resolving Suppression and Overvaluation in Multimodal Entity Alignment
Oct 12, 2026Xizhe Zhang, Jingfeng Zhang, Meng-Fen ChiangCross-Modal Alignment Multimodal KGs
School of Computer Science, The University of Auckland, New Zealand · College of Intelligent Robotics and Advanced Manufacturing/Institute of Trustworthy Embodied AI, Fudan University, China · College of Electrical and Computer Engineering, National Yang Ming Chiao Tung University, Taiwan
Time Imprint: Learning Time-Aware Representations in Multi-Modal Knowledge Graphs
Jul 8, 2026Pengyu Zhang, Klim Zaporojets, Congfeng Cao +2Multimodal Graph Learning Multimodal Contrastive Learning
University of Amsterdam Amsterdam, The Netherlands · Aarhus University Aarhus, Denmark
Progressive Representation Learning for Multimodal Sentiment Analysis with Incomplete Modalities
Mar 10, 2026Jindi Bao, Jianjun Qian, Mengkai Yan +1Multimodal Sentiment Analysis Missing-Modality Learning
Nanjing University of Science and Technology, Nanjing 210094, China · Hohai University, Nanjing 211100, China

Oct 12, 2026J/K move · Enter open · S save

Silence and Noise: Resolving Suppression and Overvaluation in Multimodal Entity Alignment

Xizhe Zhang, Jingfeng Zhang, Meng-Fen Chiang

School of Computer Science, The University of Auckland, New Zealand · College of Intelligent Robotics and Advanced Manufacturing/Institute of Trustworthy Embodied AI, Fudan University, China · College of Electrical and Computer Engineering, National Yang Ming Chiao Tung University, Taiwan

Cross-Modal Alignment Multimodal KGs Missing-Modality Learning

Open paper AskPDF arXiv

Multimodal entity alignment (MMEA) is fundamental to integrating heterogeneous knowledge graphs (KGs), yet existing paradigms remain critically vulnerable to incomplete modalities and cross-graph structural asymmetries. This paper exposes a fundamental oversight in current MMEA research: treating feature missingness and topological discrepancies as isolated preprocessing or denoising challenges, which overlooks their destructive coupling during multimodal fusion. We systematically formulate and evaluate two newly identified failure modes: 1) Detail Missing (DM)-induced overvaluation, where the absence of local modality details causes models to overemphasize uninformative, blurred representations; and 2) Context Imbalance (CI)-induced suppression, where topological discrepancies across KGs allow dense structural contexts to overwhelm weaker but highly discriminative modal signals, triggering high-level semantic drift. To break this cycle of degradation, we introduce ALMEA, a generative and calibration framework that establishes a bidirectional feedback loop between low-level feature reconstruction and high-level latent space alignment. Specifically, Cross-Modality Imputation (CMI) synthesizes missing modality details to mitigate overvaluation, while Latent Semantic Calibration (LSC) aligns cross-graph joint embeddings to suppress topology-driven semantic drift. Under sparse supervision, we develop Controlled Iterative Selection (CIS) to adaptively acquire representative and diverse pseudo-labeled pairs, optimizing latent calibration under constrained budgets. Extensive evaluations on standard benchmarks demonstrate that ALMEA consistently outperforms state-of-the-art baselines. At a 20% supervision rate, ALMEA yields substantial absolute improvements of up to 5.20% in MRR and 6.40% in Hits@1, maintaining robust gains across higher supervision thresholds.

Abstract

Multimodal Learning

Explore similar work

Silence and Noise: Resolving Suppression and Overvaluation in Multimodal Entity Alignment

Time Imprint: Learning Time-Aware Representations in Multi-Modal Knowledge Graphs

Progressive Representation Learning for Multimodal Sentiment Analysis with Incomplete Modalities