Multimodal Graph Representation Learning with Dynamic Information Pathways

Xiaobin Hong; Mingkai Lin; Xiaoli Wang; Chaoqun Wang; Wenzhong Li

cs.CVMar 10, 2026

Multimodal Graph Representation Learning with Dynamic Information Pathways

Authors: Xiaobin Hong, Mingkai Lin, Xiaoli Wang, Chaoqun Wang, Wenzhong Li

Organizations: State Key Laboratory for Novel Software Technology, Nanjing University · Nanjing Forest University, College of Information Science and Technology · Zhejiang Provincial Seaport Investment & Operation Group Co. Ltd

Abstract

Multimodal graphs, where nodes contain heterogeneous features such as images and text, are increasingly common in real-world applications. Effectively learning on such graphs requires both adaptive intra-modal message passing and efficient inter-modal aggregation. However, most existing approaches to multimodal graph learning are typically extended from conventional graph neural networks and rely on static structures or dense attention, which limit flexibility and expressive node embedding learning. In this paper, we propose a novel multimodal graph representation learning framework with Dynamic information Pathways (DiP). By introducing modality-specific pseudo nodes, DiP enables dynamic message routing within each modality via proximity-guided pseudo-node interactions and captures inter-modality dependence through efficient information pathways in a shared state space. This design achieves adaptive, expressive, and sparse message propagation across modalities with linear complexity. We conduct the link prediction and node classification tasks to evaluate performance and carry out full experimental analyses. Extensive experiments across multiple benchmarks demonstrate that DiP consistently outperforms baselines.

Figures & tables

Figure 1: A multimodal ego-graph example from a recommendation system, where its node is attributed with multimodal raw data (i.e., image and text), and the link indicates the complex relations between nodes.

Encoder

Model

Amazon-Sports

Amazon-Cloth

Goodreads-LP

MRR

\uparrow

Hit

@1\uparrow

Hit

@10\uparrow

MRR

\uparrow

Hit

@1\uparrow

Hit

@10\uparrow

MRR

\uparrow

Hit

@1\uparrow

Hit

@10\uparrow

E_{v}=\text{CLIP}

E_{t}=\text{CLIP}

MLP

28.22

\pm

0.09

14.54

\pm

0.16

59.40

\pm

0.08

21.10

\pm

0.04

10.70

\pm

0.03

42.77

\pm

0.05

11.03

\pm

0.06

4.87

\pm

0.04

21.61

\pm

0.11

GCN

31.38

\pm

0.08

16.58

\pm

0.13

66.14

\pm

0.08

22.28

\pm

0.05

11.83

\pm

0.04

43.52

\pm

0.10

25.34

\pm

0.06

13.81

\pm

0.12

50.36

\pm

0.14

SAGE

33.83

\pm

0.08

17.57

\pm

0.14

71.90

\pm

0.07

24.58

\pm

0.18

12.16

\pm

0.11

51.12

\pm

0.09

44.10

\pm

1.37

32.32

\pm

1.38

69.07

\pm

1.19

BUDDY

31.55

\pm

0.13

15.05

\pm

0.43

70.92

\pm

0.25

23.44

\pm

0.26

11.06

\pm

0.20

51.08

\pm

0.50

43.25

\pm

0.23

31.84

\pm

0.35

67.93

\pm

0.03

MMGCN

31.96

\pm

0.10

16.35

\pm

0.11

68.46

\pm

0.08

22.20

\pm

0.05

10.76

\pm

0.1

46.62

\pm

0.12

31.84

\pm

0.09

18.63

\pm

0.31

59.85

\pm

0.19

Table 1: Link prediction results on Amazon-Sports, Amazon-Cloth, and Goodreads-LP. Bold is the best result.

Dataset	$E_{v}$	$E_{t}$	MLP	GCN	SAGE	MMGCN	MMGAT	UniGraph2	DiP (ours)
Ele-Fashion	CLIP	CLIP	85.16 $\pm$ 0.03	79.83 $\pm$ 0.03	87.10 $\pm$ 0.02	86.10 $\pm$ 0.50	84.66 $\pm$ 0.29	85.93 $\pm$ 0.15	87.93 $\pm$ 0.04
	ViT	T5	84.98 $\pm$ 0.05	79.63 $\pm$ 0.07	84.41 $\pm$ 0.09	82.39 $\pm$ 0.30	84.01 $\pm$ 0.08	85.47 $\pm$ 0.20	87.19 $\pm$ 0.06
	ImageBind	ImageBind	88.73 $\pm$ 0.01	80.35 $\pm$ 0.02	87.71 $\pm$ 0.13	86.21 $\pm$ 0.84	86.12 $\pm$ 0.08	86.52 $\pm$ 0.31	89.50 $\pm$ 0.07
	DINOv2	T5	84.87 $\pm$ 0.01	79.37 $\pm$ 0.04	85.31 $\pm$ 0.09	85.53 $\pm$ 0.33	84.54 $\pm$ 0.27	86.18 $\pm$ 0.09	87.46 $\pm$ 0.05
Goodreads-NC	CLIP	CLIP	72.29 $\pm$ 0.02	81.61 $\pm$ 0.01	83.30 $\pm$ 0.02	83.29 $\pm$ 0.20	76.48 $\pm$ 0.59	82.56 $\pm$ 0.35	85.37 $\pm$ 0.12
	ViT	T5	67.82 $\pm$ 0.02	81.67 $\pm$ 0.01	83.30 $\pm$ 0.02	81.85 $\pm$ 0.22	75.43 $\pm$ 0.76	81.93 $\pm$ 0.41	86.64 $\pm$ 0.07

Table 2: Node classification results on Goodreads-NC and Ele-Fashion. Bold indicates the best performance.

Panel of Figure 3 — Figure 3: (a) DiP with adaptive pathways can maintain the performance as the model depth increases. (b) Message passing pathways. The proximities between sampled graph nodes and pseudo nodes among two modalities. Some pseudo-nodes show link activation (bright rows), which may be a clustering pattern of nodes from different classes.

Ablation	Ele-Fashion		Amazon-Sports
Ablation	Acc $\uparrow$	F1 $\uparrow$	MRR $\uparrow$	Hit $@1\uparrow$	Hit $@10\uparrow$
w/o. $\mathcal{P}^{(v)}$	84.62	78.48	31.24	17.26	69.85
w/o. $\mathcal{P}^{(t)}$	85.34	77.68	31.59	17.42	70.16
w/o. Local.	84.62	77.81	30.26	16.85	61.34
w/o. Global.	83.16	76.45	29.48	16.37	67.48
w/o. $\mathcal{P}^{(v)}\leftrightarrow\mathcal{P}^{(t)}$	82.47	74.16	30.17	16.03	65.27

Table 3: The ablation studies of modules in DiP.

Model	Ele-Fashion		Amazon-Sports
Model	Time(s)	Mem(MB)	Time(s)	Mem(MB)
GCN	0.253	1605.20	208.527	1962.37
SAGE	0.326	1312.31	224.361	2043.06
MMGCN	1.237	2030.45	346.217	2570.52
MGAT	1.524	2340.30	382.425	2803.25
Ours	0.531	462.34	213.134	660.24

Table 4: The model complexity analysis. Time (s) and memory (MB) consumed per epoch, the lower is better.

Panel of Figure 4 — Figure 4: Ablation on the number of visual ( $n_{p_{v}}$ ) and textual ( $n_{p_{t}}$ ) pseudo nodes.

Panel of Figure 5 — Figure 5: The T-SNE plots for node embedding from Ele-Fashion dataset.

Figure 6: Visualization of multimodal graph benchmarks. All nodes of these benchmarks have both visual and text features. (a) Amazon-Sports: The image and text come from the original image and title of the sports equipment. (b) Goodreads-LP: The images correspond to book covers. We do not show the text features of Goodreads-LP since the book description is very long. (c) Ele-fashion: The images and texts correspond to the original image and title of the fashion product, respectively.

Name	Text Features	Visual Features	Task	Metrics	Scale	Split Ratio
Amazon-Sports	Product Titles	Product Images	LP	MRR, Hits@K	Small	8/1/1
Amazon-Cloth	Product Titles	Product Images	LP	MRR, Hits@K	Medium	8/1/1
Goodreads-LP	Book Description	Book Images	LP	MRR, Hits@K	Large	6/1/3
Ele-Fashion	Fashion Titles	Fashion Images	NC	Accuracy	Medium	6/1/3
Goodreads-NC	Book Description	Book Images	NC	Accuracy	Large	6/1/3

Table 5: Overview of the multimodal graph datasets. We present five multimodal graph datasets with varying scale and tasks. All datasets are split at random. Abbreviations: LP = Link Prediction. NC = Node Classification. OR = Original.

Name	Nodes	Edges	Average Degree	Average CC	Average RA	Transitivity	Edge Homophily
Amazon-Sports	50,250	356,202	14.18	0.4002	0.3377	0.2658	N/A
Amazon-Cloth	125,839	951,271	15.12	0.2940	0.2588	0.1846	N/A
Goodreads-LP	636,502	3,437,017	10.79	0.1102	0.0685	0.0348	N/A
Goodreads-NC	685,294	7,235,084	21.11	0.1614	0.1056	0.0498	0.6667
Ele-Fashion	97,766	199,602	4.08	0.1730	0.1467	0.0560	0.7675

Table 6: Detailed graph-based statistics of datasets. CC = Cluster Coefficient. RA = Resource Allocation. N/A = Not Applicable (Nodes do not have class labels).

Explore similar work

ReCoG: Reciprocal Co-Evolution for Multimodal Graph Learning
Aug 24, 2026Rui Xue, Tianfu WuMultimodal Graph Learning Graph Structure Learning
Department of Electrical and Computer Engineering (ECE) North Carolina State University, Raleigh, NC 27695, USA
PRISM: Iterative Cross-Modal Posterior Refinement for Dynamic Text-Attributed Graphs
May 7, 2026Trimble Chang, Yihang Liu, Mingjing Han +1Cross-Modal Representation Learning Temporal Link Prediction
College of Artificial Intelligence Nankai University
CAMPA: Efficient and Aligned Multimodal Graph Learning via Decoupled Propagation and Aggregation
May 12, 2026Daohan Su, Hao Liu, Xunkai Li +6Efficient Multimodal Inference Cross-Modal Alignment

Aug 24, 2026J/K move · Enter open · S save

ReCoG: Reciprocal Co-Evolution for Multimodal Graph Learning

Rui Xue, Tianfu Wu

Department of Electrical and Computer Engineering (ECE) North Carolina State University, Raleigh, NC 27695, USA

Multimodal Graph Learning Graph Structure Learning Cross-Modal Representation Learning

Open paper AskPDF arXiv

Multimodal graph learning requires jointly training over graph structure and heterogeneous node attributes, yet existing methods largely decouple these processes: prior multimodal graph neural networks (GNNs) focus on aligning modalities in a shared embedding space while operating on fixed or weakly adapted graph structures, and graph structure learning approaches infer topology from unimodal node representations without accounting for multimodal interactions. This separation fundamentally limits the ability of GNNs to capture semantically meaningful relationships in multimodal settings, where observed edges are often noisy, incomplete, or misaligned with underlying semantics. We propose ReCoG (Reciprocal Co-Evolution for Multimodal Graph Learning), a new learning paradigm that tightly couples graph structure learning and multimodal representation learning through end-to-end reciprocal interaction. Concretely, ReCoG integrates (i) a multimodal graph refiner that infers and corrects edges using cross-modal semantic evidence, and (ii) a coupled cross-modal message passing mechanism that performs joint intra- and inter-modality propagation over the refined graph. This unified design yields greater expressiveness than decoupled or two-stage formulations and allows dynamic interaction between topology and representation learning. Across diverse benchmarks for node classification and link prediction, ReCoG consistently outperforms strong multimodal graph structure learning baselines, including graph foundation models. Our results demonstrate that reciprocal co-evolution of structure and semantics is important for effective multimodal graph learning, challenging the prevailing separation between topology and representation learning.

Abstract

Explore similar work

ReCoG: Reciprocal Co-Evolution for Multimodal Graph Learning

PRISM: Iterative Cross-Modal Posterior Refinement for Dynamic Text-Attributed Graphs

CAMPA: Efficient and Aligned Multimodal Graph Learning via Decoupled Propagation and Aggregation