TMTE: Effective Multimodal Graph Learning with Task-aware Modality and Topology Co-evolution

Yinlin Zhu; Xunkai Li; Di Wu; Wang Luo; Miao Hu; Guocong Quan

cs.LGMar 29, 2026

TMTE: Effective Multimodal Graph Learning with Task-aware Modality and Topology Co-evolution

Authors: Yinlin Zhu, Xunkai Li, Di Wu, Wang Luo, Miao Hu, Guocong Quan

Organizations: Sun Yat-sen University, Guangzhou, China · Beijing Institute of Technology, Beijing, China

Abstract

Multimodal-attributed graphs (MAGs) are a fundamental data structure for multimodal graph learning (MGL), enabling both graph-centric and modality-centric tasks. However, our empirical analysis reveals inherent topology quality limitations in real-world MAGs, including noisy interactions, missing connections, and task-agnostic relational structures. A single graph derived from generic relationships is therefore unlikely to be universally optimal for diverse downstream tasks. To address this challenge, we propose Task-aware Modality and Topology co-Evolution (TMTE), a novel MGL framework that jointly and iteratively optimizes graph topology and multimodal representations toward the target task. TMTE is motivated by the bidirectional coupling between modality and topology: multimodal attributes induce relational structures, while graph topology shapes modality representations. Concretely, TMTE casts topology evolution as multi-perspective metric learning over modality embeddings with an anchor-based approximation, and formulates modality evolution as smoothness-regularized fusion with cross-modal alignment, yielding a closed-loop task-aware co-evolution process. Extensive experiments on 9 MAG datasets and 1 non-graph multimodal dataset across 6 graph-centric and modality-centric tasks show that TMTE consistently achieves state-of-the-art performance. Our code is available at https://anonymous.4open.science/r/TMTE-1873.

Figures & tables

Figure 1. Experimental results of our empirical study . We compare the proposed TMTE and MM-GCN (representative MGL baseline) on the Toys dataset. (a) Node classification performance under three topology settings. (b) G2Image performance under three topology settings. All results are presented as percentages.

Figure 2. Overview of TMTE, which jointly and evolutionarily optimizes the topology and modality toward the downstream task.

Panel of Table 1 — Table 1. Performance comparison on three graph-centric downstream tasks. The best, second best and third best results are highlighted in red , blue and orange , respectively.

Panel of Table 2 — Table 2. Performance comparison on three modality-centric downstream tasks. The best, second best and third best results are highlighted in red , blue and orange , respectively.

Panel of Table 3 — Table 3. Ablation study on 6 MAG datasets with graph-centric and modality-centric tasks.

Figure 3. Experimental results of our robustness analysis . We investigate two types of topological noise, including noise interactions (i.e., randomly adding edges), which are presented in (a) and (b); and missing interactions (i.e., randomly removing edges), which are presented in (c) and (d).

Figure 4. Accuracy curves under the MVSA dataset.


Method	E-Train. (s)	E-Infer. (s)	Param.
MMGCN	$\mathbf{0.0891}$	$\mathbf{0.0352}$	$15.8$ M
GraphGPT-O	$1.8257$	$0.5167$	$48.7$ M
Graph4MM	$0.1599$	$0.5311$	$33.2$ M
InstructG2I	$24.9133$	$0.8211$	$62.3$ M
DMGC	$\mathbf{0.1362}$	$0.0441$	$\mathbf{0.85}$ M

Table 4. Per-epoch efficiency on Movies.

Figure 5. Hyperparameter analysis for $\alpha$ and $T$ on four datasets and tasks.


$K$	ACC	F1-Score
$1$	$\mathbf{57.59_{\pm 0.17}}$	$\mathbf{48.23_{\pm 0.35}}$
$4$	$\mathbf{60.31_{\pm 0.24}}$	$\mathbf{53.48_{\pm 0.18}}$
$8$	$\mathbf{59.40_{\pm 0.32}}$	$\mathbf{51.26_{\pm 0.53}}$
$16$	$57.38_{\pm 0.29}$	$47.96_{\pm 0.33}$

Table 5. Impact of

K

on Movies.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Table 6 — Table 6. Additional performance comparison on Grocery, Bili Dance and RedditS datasets. The best, second best and third best results are highlighted in red , blue and orange , respectively.

Datasets	# Modalities	# Nodes	# Edges	# Classes	Domain
Movies	Text, Image	16,672	218,390	20	Movie Network
Grocery	Text, Image	17,074	171,340	20	E-Commerce Network
DY	Text, Image	8,299	35,627	-	Video Network
Bili Dance	Text, Image	2,307	9,127	-	Video Network
Toys	Text, Image	20,695	126,886	18	E-Commerce Network
RedditS	Text, Image	15,894	566,160	20	Social Network

Appendix

Table 7. The statistical information of the experimental datasets.

Explore similar work

Context-aware Modality-Topology Co-Alignment for Multimodal Attributed Graphs
Jun 12, 2026Sirui Zhang, Xu Wang, Zhengyu Wu +2Cross-Modal Alignment Multimodal Graph Learning
ReCoG: Reciprocal Co-Evolution for Multimodal Graph Learning
Aug 24, 2026Rui Xue, Tianfu WuMultimodal Graph Learning Graph Structure Learning
Department of Electrical and Computer Engineering (ECE) North Carolina State University, Raleigh, NC 27695, USA
Toward Omni Multimodal Graph Foundation Model: A Topology-Driven Binding Approach
Oct 2, 2026Xunkai Li, Chenxi Wan, Yinlin Zhu +4Graph Foundation Models Multimodal Graph Learning
Beijing Institute of Technology · Sun Yat-sen University

Jun 12, 2026J/K move · Enter open · S save

Context-aware Modality-Topology Co-Alignment for Multimodal Attributed Graphs

Sirui Zhang, Xu Wang, Zhengyu Wu, Xunkai Li+1

Cross-Modal Alignment Multimodal Graph Learning

Open paper AskPDF arXiv

Multimodal Attributed Graphs (MAGs) model real-world entities by coupling graph topology with heterogeneous attributes such as text and images. They support graph-centric tasks requiring structural and class-discriminative representations, and modality-centric tasks requiring fine-grained cross-modal correspondence. However, existing MAG methods often rely on fixed graph contexts or uniformly fused representations, causing task-agnostic propagation and over-compressed fusion that hinder diverse task requirements and modality-specific evidence preservation. To address this, we propose CoMAG, a unified MAG backbone that learns task-adaptive reliable contexts and modality-preserving alignment within them. CoMAG first conducts Reliable Context Learning by estimating edge reliability from multimodal semantic consistency, complementing raw topology with semantic neighbors, and selecting context components through a task-aware gate. It then performs Modality-preserving Hop-token Alignment by maintaining modality-specific multi-hop trajectories, matching modality-hop tokens across modalities, and decoupling shared and private representations. Thus, CoMAG produces graph and modality representations from one forward pass while retaining modality-specific cues. We further analyze stable propagation, over-smoothing mitigation, and modality-collapse control. Experiments on nine OpenMAG datasets compare CoMAG with feature-only, graph-only, multimodal, and unified MAG baselines across graph-level prediction, modality matching, and graph-conditioned generation. Results show that CoMAG achieves the best reported performance, demonstrating that task-adaptive reliable contexts and modality-preserving alignment improve structural prediction, cross-modal matching, and graph-conditioned generation while retaining sparse edge-linear complexity.

Abstract

Explore similar work

Context-aware Modality-Topology Co-Alignment for Multimodal Attributed Graphs

ReCoG: Reciprocal Co-Evolution for Multimodal Graph Learning

Toward Omni Multimodal Graph Foundation Model: A Topology-Driven Binding Approach