Brain-Behavior Modeling from Multimodal MRI using Opportunistic Pretraining and Common-Cohort Adaptation

Md. Tanvir Rahman; Taslima Ferdaus Shuva; Nabil Anan Orka; Asaduzzaman Khan; Mohammad Ali Moni

cs.CVOct 28, 2026

Brain-Behavior Modeling from Multimodal MRI using Opportunistic Pretraining and Common-Cohort Adaptation

Authors: Md. Tanvir Rahman, Taslima Ferdaus Shuva, Nabil Anan Orka, Asaduzzaman Khan, Mohammad Ali Moni

Organizations: School of Health and Rehabilitation Sciences, The University of Queensland, QLD, 4072, Australia · Department of Information and Communication Technology, Mawlana Bhashani Science and Technology University, Tangail, 1902, Bangladesh · Department of Computer Science and Engineering, Daffodil International University, Daffodil Smart City (DSC), Birulia, Savar, Dhaka, 1216, Bangladesh · Graduate School of Biomedical Sciences, Icahn School of Medicine at Mount Sinai, New York, 10029, NY, USA ·

Abstract

Incomplete imaging coverage limits sample sizes for multimodal cognitive prediction. We evaluated a two-stage opportunistic deep ensemble using baseline Adolescent Brain Cognitive Development Study data to predict fluid, crystallized, and total cognition. Separate neural models processed structural/diffusion, resting-state, and task-based MRI data with covariates. Models were pretrained on eligible modality cohorts (up to 9,686 participants), adapted to common training cohorts, and fused via Ridge regression (trimodal cohort: 4,216 participants). Five-fold family-aware cross-validation compared this approach against constituent and covariate-only models. Two-stage training increased mean test coefficient of determination (R^2) in 24 of 27 comparisons (median gain: 0.0081). Fusion increased mean R^2 by 0.0051-0.0333 over the strongest constituent and reduced mean absolute error by 0.0067-0.0976. Task-plus-resting-state and trimodal ensembles reduced absolute error across all targets (nominal p <= 0.0108). Task-plus-resting-state fusion achieved the highest mean R^2 for fluid (0.2222) and total cognition (0.3665), while structural-plus-resting-state led for crystallized cognition (0.3706), yielding R^2 gains of 0.0805, 0.1180, and 0.0890 over covariate-only references. Ablations showed expanded pretraining drove most improvements. Gradient-based attribution identified target-dependent input patterns. These findings support using partially observed cohorts for model development to achieve modest fusion gains. Pending external validation, this approach may support MRI-informed cognitive phenotyping.

Figures & tables

Characteristic	Struct	Task	Rest	Struct+Task	Struct+Rest	Task+Rest	All Modalities
Sample Information
Subjects ( $n$ )	9,686	7,287	5,555	6,845	5,344	4,356	4,216
Families ( $n$ )	8,238	6,405	4,901	6,015	4,712	3,921	3,789
Demographics & Anthropometrics
Age (months)	119.1 $\pm$ 7.5	119.5 $\pm$ 7.5	119.8 $\pm$ 7.5	119.6 $\pm$ 7.5	119.8 $\pm$ 7.5	120.1 $\pm$ 7.5	120.1 $\pm$ 7.5
BMI	18.7 $\pm$ 4.1	18.6 $\pm$ 3.9	18.3 $\pm$ 3.7	18.5 $\pm$ 3.9	18.3 $\pm$ 3.7	18.2 $\pm$ 3.7	18.2 $\pm$ 3.6

Table 1: Participant demographics, MRI characteristics, and cognitive scores following fold-based imputation.

Figure 1: Modality-specific architectures for cognitive prediction. (a) IntelliGNN processes a population-level graph with 94 regional nodes, target-specific structural and diffusion feature masks, edge attributes, and learned node embeddings. Covariates are incorporated before attention pooling and again at the regression head. (b) SPRN processes selected resting-state connectivity and regional variance features through projected residual layers, followed by covariate concatenation at the output head. (c) MAC-Net processes regional N-back and MID measurements and concatenates covariates at the output head. Each branch is fitted separately for $G_{f}$ , $G_{c}$ , and $G_{t}$ to predict a transformed cognitive target; outputs are inverse-transformed before score-scale evaluation and fusion. Models are trained independently and do not share a latent representation.

Figure 2: Overview of the dynamic multimodal late-fusion pipeline. Structural (sMRI), resting-state (rsfMRI), and task (tfMRI) modalities are processed independently using dedicated deep learning architectures (IntelliGNN, SPRN, and MAC-Net). Following modality-specific two-stage training, predictions for the common validation participants are extracted to calibrate the fusion weights. Prediction-level fusion is performed using a RidgeCV meta-regressor, which employs an internal 5-fold cross-validation strategy on the validation set. This internal loop selects the

L_{2}

regularization penalty while keeping family members together within each split. The validation partition is also used for neural checkpoint selection, so the fusion procedure constitutes validation-set blending. The dynamically regularized coefficients are then applied to the hold-out test predictions to generate the final unified cognitive prediction.

Configuration	Common train	Common validation	Common test	IntelliGNN train	MAC-Net train	SPRN train
sMRI+rsfMRI	3,426.2	849.0	1,068.8	7,578.0	—	3,629.6
sMRI+tfMRI	4,381.6	1,094.4	1,369.0	7,054.0	4,805.4	—
tfMRI+rsfMRI	2,793.6	691.2	871.2	—	5,616.6	3,925.0
sMRI+tfMRI+rsfMRI	2,698.6	674.2	843.2	7,953.8	5,657.0	3,966.2

Table 2: Mean participant counts per outer fold. Common training, validation, and test participants belong to the configuration-specific intersection cohort. Architecture-specific Stage 1 pools include eligible common training participants and additional modality-eligible participants after exclusion of held-out families. Values are averages across five folds and are reported to one decimal place.

Target	Cohort	Ridge	Lasso	Elastic Net	Random forest	MLP
$G_{f}$	sMRI	0.1899 $\pm$ 0.0153	0.1888 $\pm$ 0.0144	0.1854 $\pm$ 0.0126	0.1686 $\pm$ 0.0089	0.1857 $\pm$ 0.0144
	rsfMRI	0.1710 $\pm$ 0.0191	0.1726 $\pm$ 0.0185	0.1720 $\pm$ 0.0182	0.1569 $\pm$ 0.0165	0.1622 $\pm$ 0.0207
	tfMRI	0.1535 $\pm$ 0.0053	0.1522 $\pm$ 0.0055	0.1509 $\pm$ 0.0043	0.1303 $\pm$ 0.0138	0.1495 $\pm$ 0.0066
	sMRI+rsfMRI	0.1730 $\pm$ 0.0191	0.1734 $\pm$ 0.0188	0.1728 $\pm$ 0.0187	0.1554 $\pm$ 0.0174	0.1672 $\pm$ 0.0275
	sMRI+tfMRI	0.1448 $\pm$ 0.0158	0.1479 $\pm$ 0.0187	0.1485 $\pm$ 0.0182	0.1268 $\pm$ 0.0229	0.1411 $\pm$ 0.0177
	tfMRI+rsfMRI	0.1411 $\pm$ 0.0265	0.1417 $\pm$ 0.0261	0.1403 $\pm$ 0.0251	0.1175 $\pm$ 0.0226	0.1322 $\pm$ 0.0287

Table 3: Covariate-only

R^{2}

across targets and eligibility cohorts.

Target	Branch	$R^{2}$	Pearson r	MSE	MAE	$\Delta R^{2}$ (cov.)
$G_{f}$	IntelliGNN	0.2023 $\pm$ 0.0153	0.4530 $\pm$ 0.0134	86.8566 $\pm$ 2.3671	7.3535 $\pm$ 0.0929	+0.0124
	SPRN	0.1978 $\pm$ 0.0295	0.4521 $\pm$ 0.0297	81.3778 $\pm$ 5.2820	7.1445 $\pm$ 0.2368	+0.0252
	MAC-Net	0.2007 $\pm$ 0.0208	0.4535 $\pm$ 0.0186	73.2790 $\pm$ 4.1989	6.7683 $\pm$ 0.2165	+0.0472
$G_{c}$	IntelliGNN	0.3166 $\pm$ 0.0176	0.5635 $\pm$ 0.0162	33.1018 $\pm$ 0.8867	4.4649 $\pm$ 0.0607	+0.0146
	SPRN	0.3576 $\pm$ 0.0143	0.6030 $\pm$ 0.0159	28.5155 $\pm$ 1.9414	4.1737 $\pm$ 0.1500	+0.0729
	MAC-Net	0.3114 $\pm$ 0.0113	0.5591 $\pm$ 0.0103	29.5441 $\pm$ 1.3299	4.2313 $\pm$ 0.0841	+0.0486

Table 4: Unimodal brain-plus-covariate performance. Bold values indicate the best-performing architecture for a given metric within each cognitive target (highest for

R^{2}

, Pearson r, and

\Delta R^{2}

; lowest for MSE and MAE).

Cohort	Base model	$G_{f}$	$G_{c}$	$G_{t}$
sMRI+rsfMRI	IntelliGNN	0.1741 $\rightarrow$ 0.1822	0.2801 $\rightarrow$ 0.2934	0.2861 $\rightarrow$ 0.2950
sMRI+rsfMRI	SPRN	0.2075 $\rightarrow$ 0.2119	0.3573 $\rightarrow$ 0.3609	0.3513 $\rightarrow$ 0.3513
sMRI+tfMRI	IntelliGNN	0.1517 $\rightarrow$ 0.1551	0.2596 $\rightarrow$ 0.2620	0.2565 $\rightarrow$ 0.2695
sMRI+tfMRI	MAC-Net	0.2052 $\rightarrow$ 0.2036	0.3065 $\rightarrow$ 0.3100	0.3331 $\rightarrow$ 0.3358
tfMRI+rsfMRI	SPRN	0.1664 $\rightarrow$ 0.1747	0.3260 $\rightarrow$ 0.3242	0.3045 $\rightarrow$ 0.3155
tfMRI+rsfMRI	MAC-Net	0.1744 $\rightarrow$ 0.1977	0.2892 $\rightarrow$ 0.3086	0.3167 $\rightarrow$ 0.3345

Table 5: Mean test

R^{2}

for common-only neural training and the complete two-stage strategy within each multimodal cohort. Each comparison uses the same configuration-specific test participants. The two-stage results include opportunistic pretraining, common-cohort adaptation, and validation-based checkpoint selection; the comparison does not isolate adaptation alone.

Target	Cohort	$R^{2}$	Pearson r	MSE	MAE
$G_{f}$	sMRI+rsfMRI	0.2216 $\pm$ 0.0146	0.4773 $\pm$ 0.0185	78.3221 $\pm$ 2.6582	7.0037 $\pm$ 0.1642
	sMRI+tfMRI	0.2095 $\pm$ 0.0175	0.4581 $\pm$ 0.0195	71.7956 $\pm$ 3.5191	6.7300 $\pm$ 0.1362
	tfMRI+rsfMRI	0.2222 $\pm$ 0.0202	0.4743 $\pm$ 0.0211	68.6384 $\pm$ 4.2938	6.5746 $\pm$ 0.2131
	sMRI+tfMRI+rsfMRI	0.2211 $\pm$ 0.0174	0.4719 $\pm$ 0.0169	68.7092 $\pm$ 3.1316	6.5962 $\pm$ 0.1703
$G_{c}$	sMRI+rsfMRI	0.3706 $\pm$ 0.0162	0.6108 $\pm$ 0.0140	27.6907 $\pm$ 1.5092	4.1229 $\pm$ 0.1054
	sMRI+tfMRI	0.3167 $\pm$ 0.0179	0.5632 $\pm$ 0.0163	29.2300 $\pm$ 1.5637	4.2356 $\pm$ 0.0979

Table 6: Multimodal ensemble performance using validation-set Ridge blending with internal family-grouped penalty selection.

Target	Cohort	Best base	$\Delta R^{2}$	$\Delta r$	$\Delta$ MSE	$\Delta$ MAE	$p$ -value
$G_{f}$	sMRI+rsfMRI	SPRN	+0.0097	+0.0146	-0.9566	-0.0547	0.0012
	sMRI+tfMRI	MAC-Net	+0.0059	+0.0040	-0.5293	-0.0138	0.0802
	tfMRI+rsfMRI	MAC-Net	+0.0245	+0.0252	-2.1959	-0.0914	¡0.0001
	sMRI+tfMRI+rsfMRI	MAC-Net	+0.0160	+0.0159	-1.4104	-0.0526	0.0108
$G_{c}$	sMRI+rsfMRI	SPRN	+0.0097	+0.0089	-0.4382	-0.0182	0.0626
	sMRI+tfMRI	MAC-Net	+0.0067	+0.0042	-0.2874	-0.0116	0.0597

Table 7: Within-cohort fusion gains and family-aware statistical significance testing.

\Delta

values represent the ensemble mean minus the strongest constituent base mean. Positive values for

R^{2}

and Pearson

r

indicate predictive improvement; negative values for MSE and MAE indicate error reduction. The

p

-values are from one-sided family-aware sign-flipping tests of paired absolute-error differences and are interpreted as nominal evidence across the 12 comparisons.

Figure 3: Mean predictive performance across training and fusion procedures for selected bimodal configurations: sMRI+rsfMRI for fluid and crystallized cognition and tfMRI+rsfMRI for total cognition. Panels show

R^{2}

, Pearson correlation, MSE, and MAE. The isolated unimodal points were obtained from the corresponding modality-eligible cohorts and therefore involve different participants. Common-only, two-stage, and ensemble points within a configuration share its test partitions. The two-stage points represent the complete opportunistic training and checkpoint-selection procedure, not an isolated adaptation effect. Dashed lines show the corresponding covariate benchmark. Differences are descriptive means across five folds; complete variability estimates are provided in the tables.

Target	Cohort	sMRI	rsfMRI	tfMRI
$G_{f}$	sMRI+rsfMRI	0.438	0.644	—
	sMRI+tfMRI	0.240	—	0.764
	tfMRI+rsfMRI	—	0.397	0.683
	sMRI+tfMRI+rsfMRI	0.085	0.402	0.634
$G_{c}$	sMRI+rsfMRI	0.374	0.737	—
	sMRI+tfMRI	0.271	—	0.758

Table 8: Mean signed RidgeCV coefficients for prediction-level fusion. Coefficients act on score-scale predictions without additional standardization.

Figure 4: Explainability and late-fusion summaries for the selected multimodal pipelines across fluid ( $G_{f}$ ), crystallized ( $G_{c}$ ), and total ( $G_{t}$ ) cognition. Column 1 displays the top six neuroanatomical and connectomic features ranked using cross-fold composite gradient-based attribution scores. Feature scores were normalized separately within each constituent branch from Integrated Gradients (IG), signed Saliency, and Input $\times$ Gradient; their displayed magnitudes do not quantify relative importance across modalities. Column 2 displays grouped covariate scores, obtained by summing categorical indicator scores and min–max normalizing the grouped values; scanner identity has the highest score in the displayed configurations. Column 3 presents agreement among method-specific attribution magnitudes, with high correlation ( $r>0.95$ ) between IG and Input $\times$ Gradient, and distinct patterns for Saliency. This agreement describes method-specific consistency rather than independent validation of the extracted features. Column 4 displays the cross-fold mean signed RidgeCV coefficients acting on score-scale modality-specific predictions, rounded to two decimal places. These coefficients are not normalized proportions and do not represent percentages of explained variance.

Target	Modality	Model	Adaptation Regime	$R^{2}$	Pearson $r$	MSE	MAE
$G_{f}$	tfMRI+rsfMRI	MAC-Net	S1 Only	0.2000 $\pm$ 0.0177	0.4509 $\pm$ 0.0191	70.6238 $\pm$ 4.7386	6.6545 $\pm$ 0.2285
			S2 Unfrozen	0.2001 $\pm$ 0.0184	0.4506 $\pm$ 0.0198	70.6216 $\pm$ 4.8065	6.6581 $\pm$ 0.2366
			S2 Frozen (Prop.)	0.1977 $\pm$ 0.0177	0.4491 $\pm$ 0.0187	70.8343 $\pm$ 4.8362	6.6660 $\pm$ 0.2357
		SPRN	S1 Only	0.1713 $\pm$ 0.0376	0.4297 $\pm$ 0.0286	73.0565 $\pm$ 4.2562	6.8135 $\pm$ 0.2146
			S2 Unfrozen	0.1724 $\pm$ 0.0356	0.4295 $\pm$ 0.0286	72.9624 $\pm$ 4.1268	6.8093 $\pm$ 0.2070
			S2 Frozen (Prop.)	0.1747 $\pm$ 0.0346	0.4305 $\pm$ 0.0281	72.7745 $\pm$ 4.2558	6.7952 $\pm$ 0.2165

Table 9: Ablation analysis across optimal target configurations under 5-fold cross-validation. Metrics reflect unweighted means

\pm

standard deviations across outer test folds. Bold values indicate the best-performing metric within each ablation group for its modality.

Architecture	Atlas	Regions
IntelliGNN & MAC-Net	Desikan-Killiany	Cortical (68 nodes, Bilateral): Banks Superior Temporal Sulcus, Caudal Anterior Cingulate, Caudal Middle Frontal, Cuneus, Entorhinal, Fusiform, Inferior Parietal, Inferior Temporal, Isthmus Cingulate, Lateral Occipital, Lateral Orbitofrontal, Lingual, Medial Orbitofrontal, Middle Temporal, Parahippocampal, Paracentral, Pars Opercularis, Pars Orbitalis, Pars Triangularis, Pericalcarine, Postcentral, Posterior Cingulate, Precentral, Precuneus, Rostral Anterior Cingulate, Rostral Middle Frontal, Superior Frontal, Superior Parietal, Superior Temporal, Supramarginal, Frontal Pole, Temporal Pole, Transverse Temporal, Insula.
	ASEG	Subcortical (26 nodes, Bilateral): Cerebral White Matter, Lateral Ventricle, Inferior Lateral Ventricle, Cerebellar White Matter, Cerebellar Cortex, Thalamus Proper, Caudate, Putamen, Pallidum, Hippocampus, Amygdala, Accumbens Area, Ventral Diencephalon.
SPRN	Gordon 333	Cortical Networks (333 nodes): Default, Somatomotor (Hand and Mouth), Visual, Fronto-Parietal, Auditory, Cingulo-Parietal, Retrosplenial Temporal, Cingulo-Opercular, Ventral Attention, Dorsal Attention, Salience, Unassigned.
	ASEG	Subcortical (19 nodes): Bilateral: Cerebellum, Thalamus, Caudate, Putamen, Pallidum, Hippocampus, Amygdala, Accumbens, Ventral Diencephalon. Central: Brain Stem.

Table S1: Brain regions and parcellations utilized within each modality-specific architecture.

Task	Category	Conditions / Contrasts
N-back	Conditions	(1) 0-back, (2) 2-back, (3) Place, (4) Emotion.
	Contrasts	(5) 2-back vs 0-back, (6) Face vs Place, (7) Emotion vs Neutral, (8) Negative vs Neutral, (9) Positive vs Neutral.
MID	Contrasts	(10) Anticipation Large vs Small Loss, (11) Anticipation Loss vs Neutral, (12) Anticipation Large Reward vs Neutral, (13) Anticipation Large Loss vs Neutral, (14) Reward Positive vs Negative Feedback, (15) Loss Positive vs Negative Feedback, (16) Anticipation Small Reward vs Neutral, (17) Anticipation Small Loss vs Neutral, (18) Anticipation Reward vs Neutral, (19) Anticipation Large vs Small Reward.

Table S2: Task conditions and behavioral contrasts provided in exact model input order for MAC-Net (tfMRI).

Cohort	Model	$R^{2}$	$r$	MSE	MAE
S	Ridge	0.3020 $\pm$ 0.0146	0.5507 $\pm$ 0.0140	33.8094 $\pm$ 0.7981	4.5102 $\pm$ 0.0575
S	Lasso	0.3015 $\pm$ 0.0162	0.5504 $\pm$ 0.0154	33.8346 $\pm$ 0.8892	4.5128 $\pm$ 0.0603
S	Elastic Net	0.2983 $\pm$ 0.0155	0.5482 $\pm$ 0.0151	33.9870 $\pm$ 0.8115	4.5235 $\pm$ 0.0549
S	RF	0.2819 $\pm$ 0.0171	0.5317 $\pm$ 0.0162	34.7746 $\pm$ 0.5975	4.5808 $\pm$ 0.0377
S	MLP	0.3012 $\pm$ 0.0128	0.5506 $\pm$ 0.0126	33.8519 $\pm$ 0.8714	4.5121 $\pm$ 0.0479
T	Ridge	0.2619 $\pm$ 0.0116	0.5145 $\pm$ 0.0125	31.6688 $\pm$ 1.4650	4.3692 $\pm$ 0.0863

Table S3a: Covariate-only baseline predictive performance for crystallized cognition (

G_{c}

Cohort	Model	$R^{2}$	$r$	MSE	MAE
S	Ridge	0.1899 $\pm$ 0.0153	0.4390 $\pm$ 0.0147	88.2052 $\pm$ 2.4446	7.3982 $\pm$ 0.1028
S	Lasso	0.1888 $\pm$ 0.0144	0.4379 $\pm$ 0.0138	88.3249 $\pm$ 2.4258	7.4017 $\pm$ 0.0972
S	Elastic Net	0.1854 $\pm$ 0.0126	0.4343 $\pm$ 0.0121	88.7022 $\pm$ 2.3919	7.4180 $\pm$ 0.0962
S	RF	0.1686 $\pm$ 0.0089	0.4145 $\pm$ 0.0072	90.5318 $\pm$ 2.2640	7.5029 $\pm$ 0.1030
S	MLP	0.1857 $\pm$ 0.0144	0.4354 $\pm$ 0.0125	88.6616 $\pm$ 2.2901	7.4264 $\pm$ 0.1027
T	Ridge	0.1535 $\pm$ 0.0053	0.3943 $\pm$ 0.0074	77.5943 $\pm$ 3.6491	6.9554 $\pm$ 0.1677

Table S3b: Covariate-only baseline predictive performance for fluid cognition (

G_{f}

Cohort	Model	$R^{2}$	$r$	MSE	MAE
S	Ridge	0.3016 $\pm$ 0.0110	0.5510 $\pm$ 0.0087	55.8309 $\pm$ 1.7076	5.8780 $\pm$ 0.0782
S	Lasso	0.3009 $\pm$ 0.0113	0.5504 $\pm$ 0.0088	55.8858 $\pm$ 1.7216	5.8813 $\pm$ 0.0742
S	Elastic Net	0.2949 $\pm$ 0.0101	0.5453 $\pm$ 0.0081	56.3657 $\pm$ 1.6291	5.9041 $\pm$ 0.0694
S	RF	0.2784 $\pm$ 0.0128	0.5297 $\pm$ 0.0120	57.6646 $\pm$ 0.8868	5.9743 $\pm$ 0.0464
S	MLP	0.2986 $\pm$ 0.0130	0.5488 $\pm$ 0.0100	56.0791 $\pm$ 2.0593	5.9009 $\pm$ 0.1016
T	Ridge	0.2620 $\pm$ 0.0117	0.5131 $\pm$ 0.0116	48.6221 $\pm$ 2.0294	5.5149 $\pm$ 0.1286

Table S3c: Covariate-only baseline predictive performance for total cognition (

G_{t}

Cohort	Model	Strategy	$R^{2}$	$r$	MSE	MAE
S	IntelliGNN	U	0.3166 $\pm$ 0.0176	0.5635 $\pm$ 0.0162	33.1018 $\pm$ 0.8867	4.4649 $\pm$ 0.0607
S+R	IntelliGNN	C	0.2801 $\pm$ 0.0139	0.5330 $\pm$ 0.0127	31.6613 $\pm$ 1.3799	4.3985 $\pm$ 0.1109
S+R	IntelliGNN	TS	0.2934 $\pm$ 0.0140	0.5427 $\pm$ 0.0128	31.0790 $\pm$ 1.4264	4.3582 $\pm$ 0.1035
S+T	IntelliGNN	C	0.2596 $\pm$ 0.0101	0.5111 $\pm$ 0.0106	31.6634 $\pm$ 1.2773	4.3706 $\pm$ 0.0722
S+T	IntelliGNN	TS	0.2620 $\pm$ 0.0130	0.5148 $\pm$ 0.0113	31.5654 $\pm$ 1.4542	4.3659 $\pm$ 0.0968
S+T+R	IntelliGNN	C	0.2412 $\pm$ 0.0111	0.4937 $\pm$ 0.0111	30.2460 $\pm$ 0.8376	4.2986 $\pm$ 0.0775

Table S4a: Brain-plus-covariate predictive performance for crystallized cognition (

G_{c}

). U: unimodal; C: common-cohort-only training; TS: two-stage opportunistic training.

Cohort	Model	Strategy	$R^{2}$	$r$	MSE	MAE
S	IntelliGNN	U	0.2023 $\pm$ 0.0153	0.4530 $\pm$ 0.0134	86.8566 $\pm$ 2.3671	7.3535 $\pm$ 0.0929
S+R	IntelliGNN	C	0.1741 $\pm$ 0.0154	0.4217 $\pm$ 0.0176	83.1039 $\pm$ 2.7670	7.1735 $\pm$ 0.1629
S+R	IntelliGNN	TS	0.1822 $\pm$ 0.0150	0.4288 $\pm$ 0.0178	82.3039 $\pm$ 3.0911	7.1439 $\pm$ 0.1790
S+T	IntelliGNN	C	0.1517 $\pm$ 0.0207	0.3905 $\pm$ 0.0265	77.0592 $\pm$ 3.9948	6.9630 $\pm$ 0.1751
S+T	IntelliGNN	TS	0.1551 $\pm$ 0.0209	0.3968 $\pm$ 0.0252	76.7451 $\pm$ 3.9991	6.9424 $\pm$ 0.1693
S+T+R	IntelliGNN	C	0.1369 $\pm$ 0.0127	0.3740 $\pm$ 0.0133	76.1356 $\pm$ 3.2643	6.9173 $\pm$ 0.1685

Table S4b: Brain-plus-covariate predictive performance for fluid cognition (

G_{f}

). U: unimodal; C: common-cohort-only training; TS: two-stage opportunistic training.

Cohort	Model	Strategy	$R^{2}$	$r$	MSE	MAE
S	IntelliGNN	U	0.3137 $\pm$ 0.0130	0.5622 $\pm$ 0.0102	54.8532 $\pm$ 1.3464	5.8370 $\pm$ 0.0694
S+R	IntelliGNN	C	0.2861 $\pm$ 0.0233	0.5371 $\pm$ 0.0213	51.1280 $\pm$ 2.1220	5.6338 $\pm$ 0.1220
S+R	IntelliGNN	TS	0.2950 $\pm$ 0.0241	0.5449 $\pm$ 0.0226	50.4800 $\pm$ 1.7229	5.6106 $\pm$ 0.0839
S+T	IntelliGNN	C	0.2565 $\pm$ 0.0245	0.5074 $\pm$ 0.0244	48.4818 $\pm$ 2.2729	5.5119 $\pm$ 0.1070
S+T	IntelliGNN	TS	0.2695 $\pm$ 0.0252	0.5201 $\pm$ 0.0237	47.6388 $\pm$ 2.3303	5.4603 $\pm$ 0.1111
S+T+R	IntelliGNN	C	0.2498 $\pm$ 0.0122	0.5021 $\pm$ 0.0119	45.9180 $\pm$ 2.4034	5.3753 $\pm$ 0.1149

Table S4c: Brain-plus-covariate predictive performance for total cognition (

G_{t}

). U: unimodal; C: common-cohort-only training; TS: two-stage opportunistic training.

Target	Cohort	Model	Rank	Covariate	Sign	Score
Cryst	T+R	MACNET	1	MRI Device Serial Number	-	1.0000
Cryst	T+R	MACNET	2	Sex	-	0.1363
Cryst	T+R	MACNET	3	Age	+	0.1199
Cryst	T+R	MACNET	4	Highest Parental Education	+	0.0872
Cryst	T+R	MACNET	5	Total Intracranial Brain Volume	+	0.0545
Cryst	T+R	MACNET	6	Combined Family Income	+	0.0520

Table S5: Grouped covariate importance rankings across models and targets. Directions (signs) denote the sums of raw mean Integrated Gradients (IG); overall group magnitude scales with categorical count.

Target	Cohort	Model	Rank	Feature	Sign	Score
Cryst	S+R	INTELLIGNN	1	L. Fusiform [Sulcal Depth]	-	1.0000
Cryst	S+R	INTELLIGNN	2	L. Fusiform [T2 Gray/White Contrast]	+	0.7050
Cryst	S+R	INTELLIGNN	3	L. Fusiform [Volume]	-	0.5475
Cryst	S+R	INTELLIGNN	4	R. Fusiform [Sulcal Depth]	-	0.4257
Cryst	S+R	INTELLIGNN	5	L. Cerebellar Cortex [Volume]	-	0.3824
Cryst	S+R	INTELLIGNN	6	L. Fusiform [T1 White]	-	0.3459

Table S6a: Top 10 specific structural and diffusion-derived (sMRI) features ranked by composite importance score.

Target	Cohort	Model	Rank	Node	Sign	Score
Cryst	S+R	INTELLIGNN	1	L. Fusiform	-	3.3790
Cryst	S+R	INTELLIGNN	2	R. Fusiform	-	1.6270
Cryst	S+R	INTELLIGNN	3	L. Superior Temporal	-	0.8183
Cryst	S+R	INTELLIGNN	4	R. Inferior Temporal	-	0.6921
Cryst	S+R	INTELLIGNN	5	R. Superior Temporal	-	0.6838
Cryst	S+R	INTELLIGNN	6	L. Inferior Temporal	-	0.6415

Table S6b: Top 10 structural (sMRI) nodes, aggregated across multiple regional morphometric and microstructural features.

Target	Cohort	Model	Rank	Edge	Sign	Score
Cryst	T+R	SPRN	1	R. Visual $\leftrightarrow$ R. Retrosplenial Temporal	+	0.8647
Cryst	T+R	SPRN	2	L. SM Hand $\leftrightarrow$ L. None	-	0.7789
Cryst	T+R	SPRN	3	R. Ventral Attention $\leftrightarrow$ R. Visual	+	0.7747
Cryst	T+R	SPRN	4	R. None $\leftrightarrow$ R. Retrosplenial Temporal	+	0.7502
Cryst	T+R	SPRN	5	L. SM Hand $\leftrightarrow$ R. SM Hand	-	0.7348
Cryst	T+R	SPRN	6	L. Default $\leftrightarrow$ L. Default	+	0.7235

Table S7a: Top 10 global resting-state functional connectivity (rsfMRI) edges (highways) ranked by composite importance score.

Target	Cohort	Model	Rank	Node	Sign	Score
Cryst	T+R	SPRN	1	L. Default	+	15.5405
Cryst	T+R	SPRN	2	L. Ventral Attention	-	14.4449
Cryst	T+R	SPRN	3	R. Visual	+	13.3295
Cryst	T+R	SPRN	4	L. Dorsal Attention	-	12.7716
Cryst	T+R	SPRN	5	R. Default	+	11.8552
Cryst	T+R	SPRN	6	L. Cingulo-Parietal	+	11.4609

Table S7b: Top 10 resting-state (rsfMRI) nodes (hubs), aggregated across all incident connectivity edges and regional variance.

Target	Cohort	Model	Rank	Feature	Sign	Score
Cryst	T+R	SPRN	1	FC Edge: R. Visual $\leftrightarrow$ R. Retrosplenial Temporal	+	0.8647
Cryst	T+R	SPRN	2	FC Edge: L. SM Hand $\leftrightarrow$ L. None	-	0.7789
Cryst	T+R	SPRN	3	FC Edge: R. Ventral Attention $\leftrightarrow$ R. Visual	+	0.7747
Cryst	T+R	SPRN	4	FC Edge: R. None $\leftrightarrow$ R. Retrosplenial Temporal	+	0.7502
Cryst	T+R	SPRN	5	FC Edge: L. SM Hand $\leftrightarrow$ R. SM Hand	-	0.7348
Cryst	T+R	SPRN	6	FC Edge: L. Default $\leftrightarrow$ L. Default	+	0.7235

Table S7c: Top 10 specific resting-state (rsfMRI) features, including individual functional connectivity edges and regional signal variance.

Target	Cohort	Model	Rank	Connection	Sign	Score
Cryst	T+R	SPRN	1	L. Default $\leftrightarrow$ R. Default	+	0.3931
Cryst	T+R	SPRN	2	L. Ventral Attention $\leftrightarrow$ R. Dorsal Attention	-	0.3831
Cryst	T+R	SPRN	3	L. Ventral Attention $\leftrightarrow$ L. Dorsal Attention	-	0.3343
Cryst	T+R	SPRN	4	R. Dorsal Attention $\leftrightarrow$ R. Default	+	0.1868
Cryst	T+R	SPRN	5	L. Dorsal Attention $\leftrightarrow$ R. Dorsal Attention	-	0.1811
Cryst	T+R	SPRN	6	L. Ventral Attention $\leftrightarrow$ L. Visual	-	0.1255

Table S7d: Top intrinsic resting-state (rsfMRI) functional connections restricted strictly to the top 10 highest-ranked nodes.

Target	Cohort	Model	Rank	Node	Sign	Score
Cryst	T+R	MACNET	1	R. Precentral	-	4.4897
Cryst	T+R	MACNET	2	L. Pars Opercularis	-	3.3357
Cryst	T+R	MACNET	3	L. Superior Parietal	-	3.2978
Cryst	T+R	MACNET	4	L. Precentral	-	3.2268
Cryst	T+R	MACNET	5	L. Lingual	+	3.1179
Cryst	T+R	MACNET	6	L. Banks Superior Temporal Sulcus	-	2.9984

Table S8a: Top 10 functional task (tfMRI) nodes, aggregated across all conditions and contrasts.

Target	Cohort	Model	Rank	Feature	Sign	Score
Cryst	T+R	MACNET	1	R. Precentral [MID: Ant. Loss vs Neut]	-	0.9530
Cryst	T+R	MACNET	2	R. Precentral [MID: Ant. Small Loss vs Neut]	-	0.9477
Cryst	T+R	MACNET	3	R. Pars Triangularis [N-back: 0-back]	-	0.6803
Cryst	T+R	MACNET	4	R. Caudal Anterior Cingulate [N-back: 2-back vs 0-back]	+	0.6771
Cryst	T+R	MACNET	5	L. Pars Opercularis [N-back: 2-back]	-	0.6415
Cryst	T+R	MACNET	6	L. Superior Parietal [N-back: 2-back]	-	0.6284

Table S8b: Top 10 specific functional task (tfMRI) features, defined by distinct region-contrast pairings.

Abbreviation	Definition
ABCD	Adolescent Brain Cognitive Development
ASEG	Automated Segmentation (FreeSurfer parcellation)
BMI	Body Mass Index
C	Common-cohort-only training
DK	Desikan-Killiany (Cortical parcellation)
FA	Fractional Anisotropy

Table 31

Multimodal Fusion

Multimodal fusion combines information from different data types to improve prediction, perception, and decision-making.

See all 800 papers

MAC-Net: A Multi-Task Deep Learning Framework for Modeling Cognitive Function From Task-Based fMRI
Sep 27, 2026Md. Tanvir Rahman, Nabil Anan Orka, Asaduzzaman Khan +1Late Fusion Multi-Task Learning
School of Health and Rehabilitation Sciences, The University of Queensland, QLD 4072, Australia · Department of Information and Communication Technology, Mawlana Bhashani Science and Technology University, Tangail 1902, Bangladesh
The Variance Brain Foundation Models Forgot: Third-Order Statistics Predict Cognition Where Billion-Parameter Models Fail
May 29, 2026Giovanni Marraffini, Gabriel Mahuas, Trinidad Borrell +2Medical Imaging Foundation Models Functional Connectivity
Inria Saclay Île-de-France, CEA, Université Paris-Saclay, Palaiseau, France · Sigma Nova · Sorbonne Université, Institut du Cerveau - Paris Brain Institute - ICM +1
M3D-BFS: a Multi-stage Dynamic Fusion Strategy for Sample-Adaptive Multi-Modal Brain Network Analysis
Apr 2, 2026Rui Dong, Xiaotong Zhang, Jiaxing Li +3Multimodal Fusion Brain Network Analysis
School of Computer Science and Engineering, Southeast University

Sep 27, 2026J/K move · Enter open · S save

MAC-Net: A Multi-Task Deep Learning Framework for Modeling Cognitive Function From Task-Based fMRI

Md. Tanvir Rahman, Nabil Anan Orka, Asaduzzaman Khan, Mohammad Ali Moni

School of Health and Rehabilitation Sciences, The University of Queensland, QLD 4072, Australia · Department of Information and Communication Technology, Mawlana Bhashani Science and Technology University, Tangail 1902, Bangladesh

Late Fusion Multi-Task Learning Neuroimaging

Open paper AskPDF arXiv

Objective cognitive assessment from neural signals supports neurorehabilitation, but individual-level prediction from task-based fMRI (tfMRI) remains difficult because neural features coexist with substantial demographic and scanner-related variation. We present the Multi-task Activation and Contrast Network (MAC-Net), a covariate-aware deep learning framework for modeling individual cognitive function from regional tfMRI. By isolating tfMRI features into a dedicated neural pathway and restricting participant variables to a terminal late-fusion pathway, MAC-Net prevents dominant covariates from suppressing high-dimensional clinical representations during feature learning. Evaluating baseline data from 6,500 Adolescent Brain Cognitive Development Study participants under family-aware cross-validation, MAC-Net was benchmarked against linear models, random forests, and alternative deep architectures. The N-back plus Monetary Incentive Delay configuration achieved

R^{2}

values of 0.174, 0.238, and 0.277 for fluid, crystallized, and total cognition, outperforming covariate-only baselines (0.178) and alternative deep models (0.217). N-back was the most informative paradigm, whereas incorporating the Stop Signal Task marginally degraded performance. Feature attributions via Integrated Gradients, DeepLIFT, and Input Gradient were highly concordant, localizing working-memory-related frontal, parietal, and cingulate regions. These findings demonstrate that covariate-aware multi-task modeling yields reproducible cognitive-function estimations, establishing a robust neural engineering framework for clinical translation.

Abstract

Multimodal Fusion

Explore similar work

MAC-Net: A Multi-Task Deep Learning Framework for Modeling Cognitive Function From Task-Based fMRI

The Variance Brain Foundation Models Forgot: Third-Order Statistics Predict Cognition Where Billion-Parameter Models Fail

M3D-BFS: a Multi-stage Dynamic Fusion Strategy for Sample-Adaptive Multi-Modal Brain Network Analysis