Organizations: State Key Laboratory for Novel Software Technology, Nanjing University · Nanjing Forest University, College of Information Science and Technology · Zhejiang Provincial Seaport Investment & Operation Group Co. Ltd
Multimodal graphs, where nodes contain heterogeneous features such as images and text, are increasingly common in real-world applications. Effectively learning on such graphs requires both adaptive intra-modal message passing and efficient inter-modal aggregation. However, most existing approaches to multimodal graph learning are typically extended from conventional graph neural networks and rely on static structures or dense attention, which limit flexibility and expressive node embedding learning. In this paper, we propose a novel multimodal graph representation learning framework with Dynamic information Pathways (DiP). By introducing modality-specific pseudo nodes, DiP enables dynamic message routing within each modality via proximity-guided pseudo-node interactions and captures inter-modality dependence through efficient information pathways in a shared state space. This design achieves adaptive, expressive, and sparse message propagation across modalities with linear complexity. We conduct the link prediction and node classification tasks to evaluate performance and carry out full experimental analyses. Extensive experiments across multiple benchmarks demonstrate that DiP consistently outperforms baselines.
Figures & tables
Figure 1: A multimodal ego-graph example from a recommendation system, where its node is attributed with multimodal raw data (i.e., image and text), and the link indicates the complex relations between nodes.
Figure 2: The overview framework of DiP. DiP first encodes the raw images and texts with frozen modality encoders for multimodal graph nodes. The recursive L -steps multimodal message passing mechanism is the key component of DiP, which consists of the Intra-Modal Diffusion Pathway and Inter-Modal Aggregation Pathway modules, which output expressiveness node representations incorporating the intra- and inter-modal message passing. Finally, the learned multimodal embeddings are fed to the task heads for link prediction and node classification training.
Encoder
Model
Amazon-Sports
Amazon-Cloth
Goodreads-LP
MRR ↑
Hit @1↑
Hit @10↑
MRR ↑
Hit @1↑
Hit @10↑
MRR ↑
Hit @1↑
Hit @10↑
Ev=CLIPEt=CLIP
MLP
28.22 ± 0.09
14.54 ± 0.16
59.40 ± 0.08
21.10 ± 0.04
10.70 ± 0.03
42.77 ± 0.05
11.03 ± 0.06
4.87 ± 0.04
21.61 ± 0.11
GCN
31.38 ± 0.08
16.58 ± 0.13
66.14 ± 0.08
22.28 ± 0.05
11.83 ± 0.04
43.52 ± 0.10
25.34 ± 0.06
13.81 ± 0.12
50.36 ± 0.14
SAGE
33.83 ± 0.08
17.57 ± 0.14
71.90 ± 0.07
24.58 ± 0.18
12.16 ± 0.11
51.12 ± 0.09
44.10 ± 1.37
32.32 ± 1.38
69.07 ± 1.19
BUDDY
31.55 ± 0.13
15.05 ± 0.43
70.92 ± 0.25
23.44 ± 0.26
11.06 ± 0.20
51.08 ± 0.50
43.25 ± 0.23
31.84 ± 0.35
67.93 ± 0.03
MMGCN
31.96 ± 0.10
16.35 ± 0.11
68.46 ± 0.08
22.20 ± 0.05
10.76 ± 0.1
46.62 ± 0.12
31.84 ± 0.09
18.63 ± 0.31
59.85 ± 0.19
Table 1: Link prediction results on Amazon-Sports, Amazon-Cloth, and Goodreads-LP. Bold is the best result.
Dataset
Ev
Et
MLP
GCN
SAGE
MMGCN
MMGAT
UniGraph2
DiP (ours)
Ele-Fashion
CLIP
CLIP
85.16 ± 0.03
79.83 ± 0.03
87.10 ± 0.02
86.10 ± 0.50
84.66 ± 0.29
85.93 ± 0.15
87.93 ± 0.04
ViT
T5
84.98 ± 0.05
79.63 ± 0.07
84.41 ± 0.09
82.39 ± 0.30
84.01 ± 0.08
85.47 ± 0.20
87.19 ± 0.06
ImageBind
ImageBind
88.73 ± 0.01
80.35 ± 0.02
87.71 ± 0.13
86.21 ± 0.84
86.12 ± 0.08
86.52 ± 0.31
89.50 ± 0.07
DINOv2
T5
84.87 ± 0.01
79.37 ± 0.04
85.31 ± 0.09
85.53 ± 0.33
84.54 ± 0.27
86.18 ± 0.09
87.46 ± 0.05
Goodreads-NC
CLIP
CLIP
72.29 ± 0.02
81.61 ± 0.01
83.30 ± 0.02
83.29 ± 0.20
76.48 ± 0.59
82.56 ± 0.35
85.37 ± 0.12
ViT
T5
67.82 ± 0.02
81.67 ± 0.01
83.30 ± 0.02
81.85 ± 0.22
75.43 ± 0.76
81.93 ± 0.41
86.64 ± 0.07
Table 2: Node classification results on Goodreads-NC and Ele-Fashion. Bold indicates the best performance.
Figure 3: (a) DiP with adaptive pathways can maintain the performance as the model depth increases. (b) Message passing pathways. The proximities between sampled graph nodes and pseudo nodes among two modalities. Some pseudo-nodes show link activation (bright rows), which may be a clustering pattern of nodes from different classes.
Ablation
Ele-Fashion
Amazon-Sports
Acc ↑
F1 ↑
MRR ↑
Hit @1↑
Hit @10↑
w/o. P(v)
84.62
78.48
31.24
17.26
69.85
w/o. P(t)
85.34
77.68
31.59
17.42
70.16
w/o. Local.
84.62
77.81
30.26
16.85
61.34
w/o. Global.
83.16
76.45
29.48
16.37
67.48
w/o. P(v)↔P(t)
82.47
74.16
30.17
16.03
65.27
Table 3: The ablation studies of modules in DiP.
Model
Ele-Fashion
Amazon-Sports
Time(s)
Mem(MB)
Time(s)
Mem(MB)
GCN
0.253
1605.20
208.527
1962.37
SAGE
0.326
1312.31
224.361
2043.06
MMGCN
1.237
2030.45
346.217
2570.52
MGAT
1.524
2340.30
382.425
2803.25
Ours
0.531
462.34
213.134
660.24
Table 4: The model complexity analysis. Time (s) and memory (MB) consumed per epoch, the lower is better.
Figure 4: Ablation on the number of visual ( npv ) and textual ( npt ) pseudo nodes.
Figure 5: The T-SNE plots for node embedding from Ele-Fashion dataset.
Figure 6: Visualization of multimodal graph benchmarks. All nodes of these benchmarks have both visual and text features. (a) Amazon-Sports: The image and text come from the original image and title of the sports equipment. (b) Goodreads-LP: The images correspond to book covers. We do not show the text features of Goodreads-LP since the book description is very long. (c) Ele-fashion: The images and texts correspond to the original image and title of the fashion product, respectively.
Name
Text Features
Visual Features
Task
Metrics
Scale
Split Ratio
Amazon-Sports
Product Titles
Product Images
LP
MRR, Hits@K
Small
8/1/1
Amazon-Cloth
Product Titles
Product Images
LP
MRR, Hits@K
Medium
8/1/1
Goodreads-LP
Book Description
Book Images
LP
MRR, Hits@K
Large
6/1/3
Ele-Fashion
Fashion Titles
Fashion Images
NC
Accuracy
Medium
6/1/3
Goodreads-NC
Book Description
Book Images
NC
Accuracy
Large
6/1/3
Table 5: Overview of the multimodal graph datasets. We present five multimodal graph datasets with varying scale and tasks. All datasets are split at random. Abbreviations: LP = Link Prediction. NC = Node Classification. OR = Original.
Name
Nodes
Edges
Average Degree
Average CC
Average RA
Transitivity
Edge Homophily
Amazon-Sports
50,250
356,202
14.18
0.4002
0.3377
0.2658
N/A
Amazon-Cloth
125,839
951,271
15.12
0.2940
0.2588
0.1846
N/A
Goodreads-LP
636,502
3,437,017
10.79
0.1102
0.0685
0.0348
N/A
Goodreads-NC
685,294
7,235,084
21.11
0.1614
0.1056
0.0498
0.6667
Ele-Fashion
97,766
199,602
4.08
0.1730
0.1467
0.0560
0.7675
Table 6: Detailed graph-based statistics of datasets. CC = Cluster Coefficient. RA = Resource Allocation. N/A = Not Applicable (Nodes do not have class labels).
Multimodal graph learning requires jointly training over graph structure and heterogeneous node attributes, yet existing methods largely decouple these processes: prior multimodal graph neural networks (GNNs) focus on aligning modalities in a shared embedding space while operating on fixed or weakly adapted graph structures, and graph structure learning approaches infer topology from unimodal node representations without accounting for multimodal interactions. This separation fundamentally limits the ability of GNNs to capture semantically meaningful relationships in multimodal settings, where observed edges are often noisy, incomplete, or misaligned with underlying semantics. We propose ReCoG (Reciprocal Co-Evolution for Multimodal Graph Learning), a new learning paradigm that tightly couples graph structure learning and multimodal representation learning through end-to-end reciprocal interaction. Concretely, ReCoG integrates (i) a multimodal graph refiner that infers and corrects edges using cross-modal semantic evidence, and (ii) a coupled cross-modal message passing mechanism that performs joint intra- and inter-modality propagation over the refined graph. This unified design yields greater expressiveness than decoupled or two-stage formulations and allows dynamic interaction between topology and representation learning. Across diverse benchmarks for node classification and link prediction, ReCoG consistently outperforms strong multimodal graph structure learning baselines, including graph foundation models. Our results demonstrate that reciprocal co-evolution of structure and semantics is important for effective multimodal graph learning, challenging the prevailing separation between topology and representation learning.
Rui Xue, Tianfu Wu
Department of Electrical and Computer Engineering (ECE) North Carolina State University, Raleigh, NC 27695, USA
Dynamic text-attributed graphs (DyTAGs) provide a powerful framework for modeling evolving systems in which node semantics and time-dependent interactions are tightly coupled. Recently, multimodal learning has emerged as a promising yet underexplored direction for enhancing DyTAG representation learning. However, existing methods typically rely on rigid modality partitions and one-shot fusion strategies, which limit their ability to capture the intrinsic and evolving dependencies between node semantics and interaction behaviors. To address these limitations, we propose PRISM, an iterative cross-modal posterior refinement framework for DyTAG representation learning. PRISM organizes DyTAG information into semantic and behavioral modalities, providing a more intrinsic alternative to carrier-level modality partitions. Instead of fusing the two modalities in a single step, PRISM learns a refinement trajectory that progressively transforms semantic priors into behavior-conditioned posterior states through cross-modal interaction with behavioral evidence. Extensive experiments on DTGB benchmark datasets show that PRISM achieves strong performance on temporal link prediction and destination node retrieval tasks. Further ablation studies validate the effectiveness of semantic--behavioral modeling and iterative posterior refinement.
Trimble Chang, Yihang Liu, Mingjing Han +1
College of Artificial Intelligence Nankai University
Multimodal Graph Neural Networks (MGNNs) have shown strong potential for learning from multimodal attributed graphs, yet most existing approaches rely on tightly coupled architectures that suffer from prohibitive computational overhead. In this paper, we present a systematic empirical analysis showing that decoupled MGNNs are substantially more efficient and scalable for large-scale graph learning. However, we identify a critical bottleneck in existing decoupled pipelines, namely modal conflict, which arises in both the propagation and aggregation stages. Specifically, independent multi-hop diffusion causes cross-modal semantic divergence during propagation, while naive fusion fails to align multi-hop feature trajectories during aggregation, jointly limiting effective representation learning. To address this challenge, we propose CAMPA, a Cross-modal Aligned Multimodal Propagation & Aggregation framework for decoupled multimodal graph learning. Concretely, CAMPA introduces a two-stage alignment mechanism: (1) cross-modal aligned propagation, which injects cross-modal similarity priors into message passing to preserve semantic consistency without additional parameter overhead; (2) trajectory aligned aggregation, which leverages trajectory-level self-attention and cross-attention to capture and align long-range dependencies across modalities and hops. Extensive experiments on diverse benchmark datasets and tasks demonstrate that CAMPA consistently outperforms strong coupled and decoupled baselines while preserving the efficiency advantages of the decoupled paradigm.