M3D-BFS: a Multi-stage Dynamic Fusion Strategy for Sample-Adaptive Multi-Modal Brain Network Analysis

Rui Dong; Xiaotong Zhang; Jiaxing Li; Yueying Li; Jiayin Wei; Youyong Kong

cs.AIApr 2, 2026

M3D-BFS: a Multi-stage Dynamic Fusion Strategy for Sample-Adaptive Multi-Modal Brain Network Analysis

Authors: Rui Dong, Xiaotong Zhang, Jiaxing Li, Yueying Li, Jiayin Wei, Youyong Kong

Organizations: School of Computer Science and Engineering, Southeast University

Abstract

Multi-modal fusion is of great significance in neuroscience which integrates information from different modalities and can achieve better performance than uni-modal methods in downstream tasks. Current multi-modal fusion methods in brain networks, which mainly focus on structural connectivity (SC) and functional connectivity (FC) modalities, are static in nature. They feed different samples into the same model with identical computation, ignoring inherent difference between input samples. This lack of sample adaptation hinders model's further performance. To this end, we innovatively propose a multi-stage dynamic fusion strategy (M3D-BFS) for sample-adaptive multi-modal brain network analysis. Unlike other static fusion methods, we design different mixture-of-experts (MoEs) for uni- and multi-modal representations where modules can adaptively change as input sample changes during inference. To alleviate issue of MoE where training of experts may be collapsed, we divide our method into 3 stages. We first train uni-modal encoders respectively, then pretrain single experts of MoEs before finally finetuning the whole model. A multi-modal disentanglement loss is designed to enhance the final representations. To the best of our knowledge, this is the first work for dynamic fusion for multi-modal brain network analysis. Extensive experiments on different real-world datasets demonstrates the superiority of M3D-BFS.

Figures & tables

Figure 1: Illustration of M 3 D-BFS. Compared with traditional methods, M 3 D-BFS can adaptively adjust for different input samples.

Figure 2: The overall framework of our proposed M 3 D-BFS. It consists of 3 stages: (1) stage 1 : training for uni-modal encoders. (2) stage 2 : pretraining of M 3 D-BFS by using single expert. (3) stage 3 : Finetuning M 3 D-BFS via finetuing MoE blocks. $\mathcal{L}_{disen}$ is designed to enhance final embeddings. Structures of MoEs (uni/multi-modal) are also plotted.

Datasets	Methods	Modality	ACC	SEN	SPE	F1	AUC
HCP	GCN	FC	64.03 $\pm$ 1.21	55.01 $\pm$ 2.88	71.75 $\pm$ 2.37	57.15 $\pm$ 2.49	63.38 $\pm$ 1.25
	GCN	SC	68.23 $\pm$ 1.76	66.22 $\pm$ 4.79	72.27 $\pm$ 4.53	67.91 $\pm$ 2.71	69.61 $\pm$ 1.75
	BrainNPT	FC	67.78 $\pm$ 1.53	60.67 $\pm$ 1.34	73.98 $\pm$ 2.83	59.15 $\pm$ 3.70	66.97 $\pm$ 2.44
	BrainNPT	SC	70.74 $\pm$ 1.49	68.95 $\pm$ 4.82	70.02 $\pm$ 5.67	68.16 $\pm$ 2.13	70.11 $\pm$ 1.73
	SVM	FC, SC	74.49 $\pm$ 2.97	71.18 $\pm$ 4.95	77.32 $\pm$ 3.30	71.96 $\pm$ 3.49	74.25 $\pm$ 3.05
	Random Forest	FC, SC	68.24 $\pm$ 2.94	54.88 $\pm$ 5.78	79.64 $\pm$ 4.16	61.31 $\pm$ 4.44	67.26 $\pm$ 3.04

Table 1: Comparison results on two different datasets (HCP, ZDXX). We run 10 times and record average acc

\pm

std (%). The best results are marked bold , the second best underline and the third best gray .

Figure 3: Multi-modal brain networks construction.

Ablation types	Stage 1	Stage 2		Stage 3	Datasets
Ablation types	KD	pretrain	$\mathcal{L}_{contrast}$	$\mathcal{L}_{disen}$	HCP	zhongdaxinxiang
$S_{1}.A_{1}$	✗	✗	✗	✗	76.84 $\pm$ 1.92	77.28 $\pm$ 4.71
$S_{2}.A_{1}$	✓	✗	✗	✗	78.16 $\pm$ 1.57	79.67 $\pm$ 4.42
$S_{2}.A_{2}$	✓	✓	✗	✗	78.57 $\pm$ 1.06	77.00 $\pm$ 2.52
$S_{3}.A_{1}$	✓	✓	✓	✗	77.80 $\pm$ 1.22	78.68 $\pm$ 4.06
M 3 D-BFS	✓	✓	✓	✓	81.21 $\pm$ 0.85	80.85 $\pm$ 2.44

Table 2: Ablation studies on sub-stages and sub-modules.

Panel of Figure 4 — Figure 4: Sensitivity analysis experiments.

Panel of Figure 5 — Figure 5: Visualization for uni-modal percentage (SC:FC) in each fusion MoE block.

Panel of Figure 6 — Figure 6: Visualization for percentage of experts in different MoEs. Here each MoE consists of 4 experts.

Multimodal Fusion

Multimodal fusion combines information from different data types to improve prediction, perception, and decision-making.

See all 771 papers

Supervised Deep Multimodal Matrix Factorization for Interpretable Brain Network Analysis
May 13, 2026Amjad Seyedi, Lifang He, Songlin Zhao +2Neural Network Interpretability Multimodal Graph Learning
Dept. of Mathematics & Operational Research University of Mons, Mons, Belgium · Dept. of Computer Science & Engineering Lehigh University, Bethlehem, PA, USA · Dept. of Industrial & Systems Engineering Lehigh University, Bethlehem, PA, USA
Brain-OF: An Omnifunctional Foundation Model for fMRI, EEG and MEG
Feb 26, 2026Hanning Guo, Hanwen Bi, Farah Abdellatif +4Functional Magnetic Resonance Imaging Multimodal Foundation Models
INM-4, Forschungszentrum Jülich, Germany · Department of Computer Science 3 - Software Engineering, RWTH Aachen University, Germany · INM-7, Forschungszentrum Jülich, Germany +6
NeuroAlign: Hierarchical Multimodal Fusion of Dynamic and Structural Neuroimaging for MCI Analysis
May 31, 2026Xiongri Shen, Zhenxi Song, Jiaqi wang +13Feature Attribution Functional Connectivity
Department of Computer Science and Technology, Harbin Institute of Technology (Shenzhen), Shenzhen, 518055, China · School of Intelligence Science and Engineering, College of Artificial Intelligence, Harbin Institute of Technology, Shenzhen, 518055, China · School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing, China +5

May 13, 2026J/K move · Enter open · S save

Supervised Deep Multimodal Matrix Factorization for Interpretable Brain Network Analysis

Amjad Seyedi, Lifang He, Songlin Zhao, Akwum Onwunta+1

Dept. of Mathematics & Operational Research University of Mons, Mons, Belgium · Dept. of Computer Science & Engineering Lehigh University, Bethlehem, PA, USA · Dept. of Industrial & Systems Engineering Lehigh University, Bethlehem, PA, USA

Neural Network Interpretability Multimodal Graph Learning Multimodal Model Interpretability Neuroimaging Multimodal Fusion Matrix Factorization+1

Open paper AskPDF arXiv

We present Supervised Deep Multimodal Matrix Factorization (SD3MF), an interpretable framework for integrative brain network analysis that generalizes Symmetric Nonnegative Matrix Tri-Factorization (SNMTF) from unsupervised single-graph clustering to supervised prediction over populations of multimodal graphs. SD3MF learns deep hierarchical factorizations for each modality together with a shared latent representation that aligns subjects across views. An encoder-decoder formulation jointly optimizes graph reconstruction and supervised prediction, while adaptive weights enable data-driven multimodal fusion. By representing each subject through community-level interaction matrices, the model yields interpretable and discriminative features. Experiments on multimodal connectome datasets show that SD3MF consistently outperforms strong deep learning baselines such as CNNs and GNNs, while enabling biologically interpretable insights. Code for reproducibility is available at: https://github.com/amjadseyedi/SD3MF.

Abstract

Multimodal Fusion

Explore similar work

Supervised Deep Multimodal Matrix Factorization for Interpretable Brain Network Analysis

Brain-OF: An Omnifunctional Foundation Model for fMRI, EEG and MEG

NeuroAlign: Hierarchical Multimodal Fusion of Dynamic and Structural Neuroimaging for MCI Analysis