More Perspectives, Stronger Signals: Multi-Perspective Enhancement and Progressive Fusion for Multimodal Entity Representation Learning
Authors: Chenyi Xiong, Yan Zhang, Jing Hu, Ziyue Qin, Kui Xiao, Xiaopan Lyu, Xiaoju Hou, Zhifei Li
Organizations: School of Computer Science, Hubei University Wuhan, China · School of Cyber Science and Technology, Hubei University Wuhan, China · Institute of Vocational Education, Guangdong Industry Polytechnic University Guangzhou, China · School of Computer Science, Hubei Key Laboratory of Big Data Intelligent Analysis and Application, Hubei University Wuhan, China
Learning effective multimodal entity representations is fundamental for reasoning tasks such as multimodal knowledge graph completion (MMKGC). However, existing methods often suffer from semantic over-smoothing within modalities and ineffective noise filtration across modalities, particularly under sparse or ambiguous conditions. To overcome these limitations, we propose PrismF, a unified framework that synergizes multi-perspective enhancement with progressive fusion to extract stronger signals from diverse inputs. PrismF enhances fine-grained intra-modal semantics through a multi-perspective mechanism that decomposes each modality into complementary views and constrains them with a decoupling loss to reduce representation collapse. Furthermore, it improves cross-modal integration through a progressive fusion strategy that dynamically calibrates inter-modal interactions, enabling the model to emphasize informative signals while suppressing noisy or unreliable ones. Extensive experiments on three public benchmarks show that PrismF achieves the strongest overall performance, including relative improvements of 4.04% in MRR and 11.17% in Hits@1 on KVC16K. Our code can be found at https://github.com/HubuKG/PrismF.
Figures & tables
Figure 1. Illustration of reasoning challenges in MMKGC. When querying ( Thunderstruck, has_cast, ? ), textual ambiguity and visual similarity between entities with similar names ( Kevin Durant vs. Durand Scott ) lead to reasoning confusion.
Figure 2. Overview of the PrismF framework. (a) Multi-Perspective Enhancement: decomposes each modality into multiple learnable perspectives with noise modeling to capture intra-modal variability and enhance fine-grained representations. (b) Progressive Modality Fusion: dynamically integrates cross-modal features through confidence calibration, mutual gating, and relation-scaled attention to achieve multimodal fusion. (c) Link Prediction: leverages fused and modality-specific representations with Tucker scoring for robust link prediction in KGs.
Dataset
#Entity
#Relation
#Train
#Valid
#Test
Image
Text
Num.
Dim.
Num.
Dim.
DB15K
12,842
279
79,222
9,902
9,904
12,818
4,096
9,078
768
MKG-Y
15,000
28
21,310
2,665
2,663
14,244
383
12,305
384
KVC16K
16,015
4
180,190
22,523
22,525
14,822
768
14,822
768
Table 1. Statistics of the multimodal knowledge graph datasets in our experiments. The image and text modality features are provided by the original datasets and are kept the same for all baselines.
Model
DB15K
MKG-Y
KVC16K
MRR
Hits
MRR
Hits
MRR
Hits
@1
@3
@10
@1
@3
@10
@1
@3
@10
Unimodal KGC Models
TransE (NeurIPS’13)
24.86
12.78
31.48
47.07
30.73
23.45
35.18
43.37
8.54
0.64
10.97
23.42
DistMult (ICLR’15)
23.03
14.78
26.28
39.59
25.04
19.33
27.80
35.95
6.37
3.03
6.11
12.61
ComplEx (ICML’16)
27.48
18.37
31.57
45.37
28.71
22.26
32.12
40.93
12.85
7.48
13.79
23.18
Table 2. Performance comparison of different knowledge graph completion models. The optimal results are highlighted in bold , while the second optimal results are underlined .
Figure 3. The MRR and Hits@1 results of removing each modality on DB15K.
Setting
DB15K
MKG-Y
KVC16K
MRR
Hits@1
MRR
Hits@1
MRR
Hits@1
w/o MuPE
37.98
31.41
38.13
34.98
16.05
9.32
w/o loss Lpd
38.18
31.64
38.31
35.17
16.27
9.61
w/o noise ϵmi
38.72
32.17
38.23
35.01
16.09
9.39
w/o PMF
37.91
31.22
37.36
33.96
16.32
9.78
w/ concat as g
38.03
30.62
37.98
34.82
16.58
10.12
Table 3. Performance comparison of different settings. We explored the impact of each key component.
Figure 4. Hyperparameter performance analysis across all datasets.
Figure 5. Performance comparison on the DB15K under two challenging scenarios: modality missing and data sparsity.
Figure 6. The visualization of modality contributions for PrismF. Here, we present the contributions of the structural, visual, textual, and fused multimodal during the reasoning process across all datasets.
Figure 7. Case examples. PrismF assigns larger weights to the more informative modality based on content reliability, effectively mitigating noise and ambiguity.
Multimodal entity alignment (MMEA) is fundamental to integrating heterogeneous knowledge graphs (KGs), yet existing paradigms remain critically vulnerable to incomplete modalities and cross-graph structural asymmetries. This paper exposes a fundamental oversight in current MMEA research: treating feature missingness and topological discrepancies as isolated preprocessing or denoising challenges, which overlooks their destructive coupling during multimodal fusion. We systematically formulate and evaluate two newly identified failure modes: 1) Detail Missing (DM)-induced overvaluation, where the absence of local modality details causes models to overemphasize uninformative, blurred representations; and 2) Context Imbalance (CI)-induced suppression, where topological discrepancies across KGs allow dense structural contexts to overwhelm weaker but highly discriminative modal signals, triggering high-level semantic drift. To break this cycle of degradation, we introduce ALMEA, a generative and calibration framework that establishes a bidirectional feedback loop between low-level feature reconstruction and high-level latent space alignment. Specifically, Cross-Modality Imputation (CMI) synthesizes missing modality details to mitigate overvaluation, while Latent Semantic Calibration (LSC) aligns cross-graph joint embeddings to suppress topology-driven semantic drift. Under sparse supervision, we develop Controlled Iterative Selection (CIS) to adaptively acquire representative and diverse pseudo-labeled pairs, optimizing latent calibration under constrained budgets. Extensive evaluations on standard benchmarks demonstrate that ALMEA consistently outperforms state-of-the-art baselines. At a 20% supervision rate, ALMEA yields substantial absolute improvements of up to 5.20% in MRR and 6.40% in Hits@1, maintaining robust gains across higher supervision thresholds.
Xizhe Zhang, Jingfeng Zhang, Meng-Fen Chiang
School of Computer Science, The University of Auckland, New Zealand · College of Intelligent Robotics and Advanced Manufacturing/Institute of Trustworthy Embodied AI, Fudan University, China · College of Electrical and Computer Engineering, National Yang Ming Chiao Tung University, Taiwan
Multi-Modal Knowledge Graphs (MMKGs) enrich entities with multiple modalities such as text and images, yet entities with highly similar multi-modal features remain difficult to distinguish. Temporal information of an entity can serve as an additional modality to disambiguate such entities, but existing approaches rarely treat time as a separate modality alongside text and images due to two major challenges: (1) sparse temporal semantics, which hinder alignment with richer modalities, and (2) multiple timestamps, which introduce noise or reduce robustness in representation learning. To address these challenges, we propose Time Imprint, a framework that treats time as an entity-level modality and jointly aligns temporal, textual, and visual representations via a three-view contrastive objective. Additionally, to mitigate multi-timestamp ambiguity, Time Imprint studies a compact timestamp subset selection design space and aggregates the selected timestamps into a discriminative temporal embedding with attention pooling, balancing temporal specificity and robustness. Experiments on three MMKG benchmarks demonstrate that Time Imprint achieves state-of-the-art link prediction performance, improving Hits@1 by up to 6.07% overall and yielding up to 58% gains on the subset of the top-1% ambiguity samples. We further examine different fusion strategies and the sensitivity to timestamp availability and quality, clarifying when and why time-as-modality is most beneficial, while adding only modest training overhead. We release our code at https://anonymous.4open.science/r/Time-Imprint.
Pengyu Zhang, Klim Zaporojets, Congfeng Cao +2
University of Amsterdam Amsterdam, The Netherlands · Aarhus University Aarhus, Denmark
Multimodal Sentiment Analysis (MSA) seeks to infer human emotions by integrating textual, acoustic, and visual cues. However, existing approaches often rely on all modalities are completeness, whereas real-world applications frequently encounter noise, hardware failures, or privacy restrictions that result in missing modalities. There exists a significant feature misalignment between incomplete and complete modalities, and directly fusing them may even distort the well-learned representations of the intact modalities. To this end, we propose PRLF, a Progressive Representation Learning Framework designed for MSA under uncertain missing-modality conditions. PRLF introduces an Adaptive Modality Reliability Estimator (AMRE), which dynamically quantifies the reliability of each modality using recognition confidence and Fisher information to determine the dominant modality. In addition, the Progressive Interaction (ProgInteract) module iteratively aligns the other modalities with the dominant one, thereby enhancing cross-modal consistency while suppressing noise. Extensive experiments on CMU-MOSI, CMU-MOSEI, and SIMS verify that PRLF outperforms state-of-the-art methods across both inter- and intra-modality missing scenarios, demonstrating its robustness and generalization capability.
Jindi Bao, Jianjun Qian, Mengkai Yan +1
Nanjing University of Science and Technology, Nanjing 210094, China · Hohai University, Nanjing 211100, China