Organizations: Natural Language Processing and Knowledge Graph Lab, School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China · Centre for Learning, Teaching and Technology, The Education University of Hong Kong, Hong Kong SAR, China
Accurate representation of multimodal knowledge is crucial for event forecasting in real-world scenarios. However, existing studies have largely focused on static settings, overlooking the dynamic acquisition and fusion of multimodal knowledge. 1) At the knowledge acquisition level, how to learn time-sensitive information of different modalities, especially the dynamic structural modality. Existing dynamic learning methods are often limited to shallow structures across heterogeneous spaces or simple unispaces, making it difficult to capture deep relation-aware geometric features. 2) At the knowledge fusion level, how to learn evolving multimodal fusion features. Existing knowledge fusion methods based on static coattention struggle to capture the varying historical contributions of different modalities to future events. To this end, we propose DyMRL, a Dynamic Multispace Representation Learning approach to efficiently acquire and fuse multimodal temporal knowledge. 1) For the former issue, DyMRL integrates time-specific structural features from Euclidean, hyperbolic, and complex spaces into a relational message-passing framework to learn deep representations, reflecting human intelligences in associative thinking, high-order abstracting, and logical reasoning. Pretrained models endow DyMRL with time-sensitive visual and linguistic intelligences. 2) For the latter concern, DyMRL incorporates advanced dual fusion-evolution attention mechanisms that assign dynamic learning emphases equally to different modalities at different timestamps in a symmetric manner. To evaluate DyMRL's event forecasting performance through leveraging its learned multimodal temporal knowledge in history, we construct four multimodal temporal knowledge graph benchmarks. Extensive experiments demonstrate that DyMRL outperforms state-of-the-art dynamic unimodal and static multimodal baseline methods.
Figures & tables
Figure 1. Illustration of multimodal temporal knowledge.
Figure 2. Architecture of DyMRL model.
Datasets
∣E∣
∣V∣
∣L∣
∣R∣
#Historical
#Current
#Future
#Time granularity
#Time range
#Timestamp
GDELT-IMG-TXT
7,691
59,196
7,691
240
1,734,399
238,765
305,241
15 mins
2018/1/1–2018/1/31
2,751
ICE14-IMG-TXT
7,128
46,089
7,128
230
74,845
8,514
7,371
24 hours
2014/1/1–2014/12/31
365
ICE0515-IMG-TXT
10,488
70,001
10,488
251
368,868
46,302
46,159
24 hours
2005/1/1–2015/12/31
4,017
ICE18-IMG-TXT
23,033
111,624
23,033
256
373,018
45,995
49,545
24 hours
2018/1/1–2018/10/31
304
Table 1. Statistical information of the constructed multimodal temporal KG datasets.
Model
GDELT-IMG-TXT
ICE14-IMG-TXT
ICE0515-IMG-TXT
ICE18-IMG-TXT
MRR
H@1
H@10
MRR
H@1
H@10
MRR
H@1
H@10
MRR
H@1
H@10
Static multimodal
TransAE ( IJCNN’19 )
13.31 ∗
4.42 ∗
29.93 ∗
24.34 ∗
11.76 ∗
49.62 ∗
26.79 ∗
13.86 ∗
52.18 ∗
19.66 ∗
9.97 ∗
39.16 ∗
MoSE ( EMNLP’22 )
16.76 ∗
10.01 ∗
30.00 ∗
27.18 ∗
15.59 ∗
51.29 ∗
29.71 ∗
18.21 ∗
53.45 ∗
20.88 ∗
11.25 ∗
41.09 ∗
OTKGE ( NeurIPS’22 )
17.01 ∗
10.09 ∗
30.50 ∗
31.36 ∗
21.03 ∗
51.57 ∗
31.36 ∗
20.36 ∗
53.42 ∗
22.51 ∗
12.70 ∗
42.68 ∗
IMF ( WWW’23 )
17.97 ∗
10.94 ∗
31.81 ∗
33.89 ∗
24.28 ∗
52.69 ∗
34.78 ∗
24.41 ∗
55.71 ∗
26.57 ∗
16.96 ∗
45.48 ∗
DySarl ( MM’24 )
30.81 ∗
18.27 ∗
46.40 ∗
38.14 ∗
21.75 ∗
57.80 ∗
47.28 ∗
29.65 ∗
69.67 ∗
33.52 ∗
21.14 ∗
52.00 ∗
Table 2. Comparison of multimodal future forecasting performance (in percentage) against static multimodal and dynamic unimodal baseline methods across four multimodal temporal KG datasets using time-aware filtered evaluation metrics.
Figure 3. Study on the dynamic effects of different modalities at different historical timestamps on future forecasting.
Datasets (STRUC-IMG-TXT)
GDELT
ICE14
ICE0515
ICE18
w/o Euclidean msg
67.67
54.97
71.26
56.15
w/o Hyperbolic msg
56.99
54.80
69.53
54.60
w/o Complex msg
72.02
55.51
71.29
56.98
w/o Multilayer msg propagation
31.36
52.98
62.45
50.51
w/o Update module
78.02
61.97
74.65
63.45
w/o Structural information
26.43
51.77
58.80
47.54
Table 3. Ablation results across all the MTKG datasets.
Figure 7
Notation
Description
Notation
Description
R
Euclidean space
B
Hyperbolic space
C
Complex space
G
A multimodal temporal KG composed of events
T
Set of timestamps in G
E
Set of entities in G
R
Set of relations in G
V
Set of images in G
L
Set of texts in G
T
# of timestamps ( t∈{0,⋯,T−1} )
N
# of entities (size of E )
R
# of relations (size of R )
Table 1. Set of notations used in the DyMRL model.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 1. Study on the multimodal event forecasting time.
Real-world multimodal knowledge graphs (MMKGs) are dynamic, with new entities, relations, and multimodal knowledge emerging over time. Existing continual knowledge graph reasoning (CKGR) methods focus on structural triples and cannot fully exploit multimodal signals from new entities. Existing multimodal knowledge graph reasoning (MMKGR) methods, however, usually assume static graphs and suffer catastrophic forgetting as graphs evolve. To address this gap, we present a systematic study of continual multimodal knowledge graph reasoning (CMMKGR). We construct several continual multimodal knowledge graph benchmarks from existing MMKG datasets and propose MRCKG, a new CMMKGR model. Specifically, MRCKG employs a multimodal-structural collaborative curriculum to schedule progressive learning based on the structural connectivity of new triples to the historical graph and their multimodal compatibility. It also introduces a cross-modal knowledge preservation mechanism to mitigate forgetting through entity representation stability, relational semantic consistency, and modality anchoring. In addition, a multimodal contrastive replay scheme with a two-stage optimization strategy reinforces learned knowledge via multimodal importance sampling and representation alignment. Experiments on multiple datasets show that MRCKG preserves previously learned multimodal knowledge while substantially improving the learning of new knowledge.
Linyu Li, Zhi Jin, Yichi Zhang +5
Key Laboratory of High Confidence Software Technologies (PKU), Ministry of Education School of Computer Science, Peking University Beijing, China · School of Computer Science Zhejiang University Hangzhou, China · School of Cyber Science and Engineering Wuhan University Wuhan, China +1
Multi-Modal Knowledge Graphs (MMKGs) enrich entities with multiple modalities such as text and images, yet entities with highly similar multi-modal features remain difficult to distinguish. Temporal information of an entity can serve as an additional modality to disambiguate such entities, but existing approaches rarely treat time as a separate modality alongside text and images due to two major challenges: (1) sparse temporal semantics, which hinder alignment with richer modalities, and (2) multiple timestamps, which introduce noise or reduce robustness in representation learning. To address these challenges, we propose Time Imprint, a framework that treats time as an entity-level modality and jointly aligns temporal, textual, and visual representations via a three-view contrastive objective. Additionally, to mitigate multi-timestamp ambiguity, Time Imprint studies a compact timestamp subset selection design space and aggregates the selected timestamps into a discriminative temporal embedding with attention pooling, balancing temporal specificity and robustness. Experiments on three MMKG benchmarks demonstrate that Time Imprint achieves state-of-the-art link prediction performance, improving Hits@1 by up to 6.07% overall and yielding up to 58% gains on the subset of the top-1% ambiguity samples. We further examine different fusion strategies and the sensitivity to timestamp availability and quality, clarifying when and why time-as-modality is most beneficial, while adding only modest training overhead. We release our code at https://anonymous.4open.science/r/Time-Imprint.
Pengyu Zhang, Klim Zaporojets, Congfeng Cao +2
University of Amsterdam Amsterdam, The Netherlands · Aarhus University Aarhus, Denmark
Dynamic text-attributed graphs (DyTAGs) provide a powerful framework for modeling evolving systems in which node semantics and time-dependent interactions are tightly coupled. Recently, multimodal learning has emerged as a promising yet underexplored direction for enhancing DyTAG representation learning. However, existing methods typically rely on rigid modality partitions and one-shot fusion strategies, which limit their ability to capture the intrinsic and evolving dependencies between node semantics and interaction behaviors. To address these limitations, we propose PRISM, an iterative cross-modal posterior refinement framework for DyTAG representation learning. PRISM organizes DyTAG information into semantic and behavioral modalities, providing a more intrinsic alternative to carrier-level modality partitions. Instead of fusing the two modalities in a single step, PRISM learns a refinement trajectory that progressively transforms semantic priors into behavior-conditioned posterior states through cross-modal interaction with behavioral evidence. Extensive experiments on DTGB benchmark datasets show that PRISM achieves strong performance on temporal link prediction and destination node retrieval tasks. Further ablation studies validate the effectiveness of semantic--behavioral modeling and iterative posterior refinement.
Trimble Chang, Yihang Liu, Mingjing Han +1
College of Artificial Intelligence Nankai University