Coupling Perception and Reasoning in Federated Multimodal Graph Foundation Models

Zekai Chen; Xun Wu; Hailin Zhang; Xunkai Li; Yu Liu; Kairui Yang; Muyan Huang; Xuaner Chen

cs.LGSep 24, 2026

Coupling Perception and Reasoning in Federated Multimodal Graph Foundation Models

Authors: Zekai Chen, Xun Wu, Hailin Zhang, Xunkai Li, Yu Liu, Kairui Yang, Muyan Huang, Xuaner Chen,

Organizations: Beijing Institute of Technology · Shandong University · Institute of Information Engineering · University of Science and Technology Beijing

Abstract

Federated multimodal graph foundation models (GFMs) aim to adapt pretrained multimodal models to decentralized graph data, where each client owns a private multimodal graph and cannot share raw information. These models typically combine a multimodal Encoder that extracts semantic evidence from heterogeneous modalities and a graph neural network (GNN) that performs relational reasoning over graph structures. However, existing federated GFM adaptation methods mainly update graph-side modules while keeping the multimodal Encoder frozen, limiting adaptation to how information is propagated while fixing what information is extracted. Through empirical studies, we reveal that Encoder and GNN adaptations are not independent: Encoder adaptation is affected by graph relations, while cross-client module swapping reveals substantial pairing sensitivity between separately parameterized Encoder and GNN updates. Motivated by this observation, we propose FedCORE, a federated adaptation framework that represents Encoder and GNN updates through a shared low-dimensional latent state. FedCORE jointly optimizes this core from multimodal and structural signals and performs federated evolution directly in the shared state space, preserving compatibility between perception and reasoning adaptations. Extensive experiments demonstrate that FedCORE reduces the Encoder--GNN pairing gap from $30.6$ to $5.9$ , corresponding to an $80.7\%$ reduction over independent joint adaptation.

Figures & tables

Figure 1: Observation 1: Stronger GNNs cannot replace Encoder adaptation. Performance of graph-only adaptation with increasing GNN capacity compared with Encoder-adapted models.

Figure 2: Observation 2: Relational context conditions multimodal representation adaptation. Encoder-LoRA update trajectories under graph interventions and semantic-preserving perturbations.

Figure 3: Observation 3: Independently learned module updates can exhibit cross-client combination incompatibility. Cross-client Encoder–GNN pairing results on Toys.

Method	Node Classification (Acc)		Link Prediction (AUC)		Modal Retrieval (R@5)		G2Text (ROUGE-L)	G2Image (CLIP-S)
Method	Toys	Grocery	Bili Music	DY	QB	Bili Cartoon	Flickr30k	SemArt
FedGAMMA	$76.37_{\scriptstyle\pm 0.65}$	$73.73_{\scriptstyle\pm 0.54}$	$75.87_{\scriptstyle\pm 0.91}$	$72.25_{\scriptstyle\pm 0.26}$	$86.52_{\scriptstyle\pm 2.44}$	$80.22_{\scriptstyle\pm 0.78}$	$31.29_{\scriptstyle\pm 0.09}$	$70.87_{\scriptstyle\pm 0.23}$
STAGE	$79.19_{\scriptstyle\pm 0.02}$	$81.04_{\scriptstyle\pm 0.42}$	$69.41_{\scriptstyle\pm 1.74}$	$69.19_{\scriptstyle\pm 0.49}$	$94.91_{\scriptstyle\pm 3.65}$	$79.77_{\scriptstyle\pm 1.39}$	$30.87_{\scriptstyle\pm 0.47}$	$71.86_{\scriptstyle\pm 0.53}$
FedGFM+	$80.21_{\scriptstyle\pm 0.28}$	$83.76_{\scriptstyle\pm 0.15}$	$76.24_{\scriptstyle\pm 0.02}$	$74.30_{\scriptstyle\pm 0.11}$	$92.06_{\scriptstyle\pm 0.83}$	$84.57_{\scriptstyle\pm 4.38}$	$31.09_{\scriptstyle\pm 0.31}$	$71.58_{\scriptstyle\pm 0.26}$
Fed-PLANET	$80.58_{\scriptstyle\pm 0.07}$	$83.93_{\scriptstyle\pm 0.48}$	$65.29_{\scriptstyle\pm 0.90}$	$64.08_{\scriptstyle\pm 0.70}$	$89.89_{\scriptstyle\pm 2.64}$	$81.20_{\scriptstyle\pm 5.54}$	$31.35_{\scriptstyle\pm 0.23}$	$71.38_{\scriptstyle\pm 0.21}$
Frozen Enc-GNN	$78.93_{\scriptstyle\pm 0.14}$	$80.63_{\scriptstyle\pm 0.08}$	$74.50_{\scriptstyle\pm 0.70}$	$72.60_{\scriptstyle\pm 0.21}$	$86.45_{\scriptstyle\pm 2.56}$	$76.66_{\scriptstyle\pm 5.00}$	$31.07_{\scriptstyle\pm 0.35}$	$70.55_{\scriptstyle\pm 0.09}$

Table 1: Overall performance comparison. Best results are in bold and second-best are underlined .

Variant	Node Classification (Acc)	Link Prediction (AUC)	Modal Retrieval (R@5)	G2Text (ROUGE-L)	G2Image (CLIP-S)
	Toys	DY	QB	Flickr30k	SemArt
Full FedCORE	$\textbf{81.67}_{\scriptstyle\pm\textbf{0.05}}$	$\textbf{77.72}_{\scriptstyle\pm\textbf{0.32}}$	$\textbf{95.28}_{\scriptstyle\pm\textbf{0.61}}$	$\textbf{33.21}_{\scriptstyle\pm\textbf{0.76}}$	$\textbf{73.26}_{\scriptstyle\pm\textbf{0.21}}$
w/o Shared Latent State	$79.96_{\scriptstyle\pm 0.21}$	$74.86_{\scriptstyle\pm 0.43}$	$92.94_{\scriptstyle\pm 0.46}$	$30.71_{\scriptstyle\pm 0.94}$	$71.75_{\scriptstyle\pm 0.36}$
w/o SVD Bases	$80.87_{\scriptstyle\pm 0.12}$	$75.93_{\scriptstyle\pm 0.32}$	$94.18_{\scriptstyle\pm 0.73}$	$32.48_{\scriptstyle\pm 0.85}$	$72.59_{\scriptstyle\pm 0.29}$
w/o Geometry-Balanced Update	$81.24_{\scriptstyle\pm 0.08}$	$76.31_{\scriptstyle\pm 0.29}$	$94.76_{\scriptstyle\pm 0.66}$	$32.91_{\scriptstyle\pm 0.75}$	$72.94_{\scriptstyle\pm 0.24}$
Indep. Adapters + FedAvg	$80.51_{\scriptstyle\pm 0.06}$	$76.66_{\scriptstyle\pm 0.27}$	$95.05_{\scriptstyle\pm 0.15}$	$32.19_{\scriptstyle\pm 0.40}$	$72.24_{\scriptstyle\pm 0.21}$

Table 2: Ablation study. Best results are in bold and second-best results are underlined .

Figure 5: Relational core–update correspondence. Each point is one client pair. The x-axis shows shared-core similarity, and the y-axis shows Encoder/GNN update similarity.

Panel of Figure 6 — Figure 6: Robustness to client fragmentation and hyperparameter choices.

Figure 7: Robustness to modality corruption and missing modalities.

Method	Trainable Params.	Comm. Payload (Scalars/Round)	Total Ops (FLOPs)	Space Cost (MB)	Time (s/round)
FedGAMMA	$2.32\times 10^{5}$	$4.64\times 10^{5}$	$2.69\times 10^{11}$	$\mathbf{242.0}$	$2.933$
STAGE	$2.49\times 10^{5}$	$4.98\times 10^{5}$	$\mathbf{2.69\times 10^{10}}$	$469.6$	$1.795$
Fed-PLANET	$8.43\times 10^{5}$	$1.686\times 10^{6}$	$2.73\times 10^{11}$	$528.0$	$2.088$
Frozen Encoder + Trainable GNN + FedAvg	$2.49\times 10^{5}$	$4.98\times 10^{5}$	$\mathbf{2.69\times 10^{10}}$	$443.6$	$\mathbf{0.676}$
Independent Joint LoRA + FedAvg	$2.32\times 10^{5}$	$4.64\times 10^{5}$	$2.90\times 10^{10}$	$438.8$	$0.752$
FedCORE (Ours)	$\mathbf{2.56\times 10^{2}}$	$\mathbf{5.12\times 10^{2}}$	$3.12\times 10^{10}$	$441.3$	$1.155$

Table 3: Communication and resource efficiency of FedCORE and baseline methods.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Performance under progressive structural randomization. Absolute accuracy of Frozen Encoder, Independent Joint LoRA, and FedCORE as the structural integrity ratio

\rho

increases from a randomized graph to the original graph.

Panel of Figure 9 — Figure 9: Additional convergence and scalability analysis. (a) Test accuracy on Toys node classification across communication rounds. (b) Performance–communication trade-off on Grocery node classification as the adaptation scale increases.

Figure 10: Robustness to client heterogeneity. The left panel varies client feature drift

\alpha

, while the right panel varies the cross-community edge-injection ratio

\rho

before the final client partition. The node set, labels, and data splits remain fixed.

Multimodal Federated Learning

Multimodal federated learning trains models across data silos with different data types while keeping raw data local.

See all 51 papers

Toward Federated Multimodal Graph Foundation Models: A Topology-Aware Multimodal Alignment Framework
Jul 17, 2026Xunkai Li, Guohao Fu, Yuming Ai +4Multimodal Federated Learning Cross-Modal Alignment
Beijing Institute of Technology, Beijing, 100811, China
FedLAB: Traceable Semantic Codebooks for Federated Multimodal Graph Foundation Learning
Jun 30, 2026Zekai Chen, Kairui Yang, Xuaner Chen +4Multimodal Federated Learning Representation Learning
Beijing Institute of Technology, Beijing, China
STAGE: Tackling Semantic Drift in Multimodal Federated Graph Learning
May 12, 2026Zekai Chen, Xun Wu, Xunkai Li +3Multimodal Federated Learning Cross-Modal Representation Learning
Beijing Institute of Technology, Beijing, 100081, China · Minzu University of China, Beijing, 100081, China

Jul 17, 2026J/K move · Enter open · S save

Toward Federated Multimodal Graph Foundation Models: A Topology-Aware Multimodal Alignment Framework

Xunkai Li, Guohao Fu, Yuming Ai, Zhengyu Wu+3

Beijing Institute of Technology, Beijing, 100811, China

Multimodal Federated Learning Cross-Modal Alignment Multimodal Pretraining Federated Graph Learning Prompt Tuning Graph Foundation Models+1

Open paper AskPDF arXiv

Multimodal-attributed graphs (MAGs), whose nodes carry modalities such as images and text alongside topological structure, now pervade applications including social platforms, e-commerce, and biomedical networks, offering richer semantic signals than single-modality graphs. In practice, such graphs are fragmented across privacy-restricted silos owned by different platforms and institutions, so learning a broadly transferable model over them demands collaborative training that never exposes raw data. This places the task at the intersection of multimodal graph learning and federated learning, yet existing methods cover only one side of it. To address the challenges from these two perspectives, we propose FedGAMMA, casting federated multimodal graph foundation learning as a two-stage semantic-structural alignment problem of federated pre-training and prompt-based fine-tuning. During pre-training, a shared-private semantic enhancer disentangles cross-modal commonality from modality-specific information, aligning it through optimal transport, a topology-aware graph fusion module decouples semantic and structural views via semantic residual graphs and dual positional encodings, and a dual-channel affinity-aware aggregation mechanism estimates client similarity from feature and graph centroids without exposing raw data. During fine-tuning, FedGAMMA adapts the pretrained encoder through lightweight graph-aware prompts, a shared prompt pool with controlled exploration, and channel-wise prompt synchronization. Experiments on twelve multimodal graph datasets show FedGAMMA consistently surpassing a broad range of baselines across downstream tasks, with gains of up to 12.96%. FedGAMMA further outperforms competitive baselines accross multi-domain datasets on multiple tasks with up to 5.71% under few-shot learning scenario.

Abstract

Multimodal Federated Learning

Explore similar work

Toward Federated Multimodal Graph Foundation Models: A Topology-Aware Multimodal Alignment Framework

FedLAB: Traceable Semantic Codebooks for Federated Multimodal Graph Foundation Learning

STAGE: Tackling Semantic Drift in Multimodal Federated Graph Learning