Evidence Before Accuracy: A MRI-PET Fusion Network for Alzheimer Disease Classification with Causal Regional Validation

Saeid Firouzi Daghigh; Saeed Ayat

cs.CVSep 28, 2026

Evidence Before Accuracy: A MRI-PET Fusion Network for Alzheimer Disease Classification with Causal Regional Validation

Authors: Saeid Firouzi Daghigh, Saeed Ayat

Organizations: Department of Computer Engineering and Information Technology, Payame Noor University, Tehran, Iran

Abstract

Deep learning models for Alzheimer disease (AD) classification routinely report near-perfect discrimination, yet few are shown to rest on AD-relevant neurobiology rather than on dataset artifacts, subject-level leakage, or non-brain image content. We present a fusion network combining T1 MRI and FDG PET across axial, coronal, and sagittal planes, trained on ADNI consists of 554 paired subjects. The fusion model reaches AUC 0.962, accuracy 0.909, and F1 0.891, competitive with recent 3D CNN and multimodal transformer systems at substantially lower cost. We first quantify how much modality, plane and slice geometry matter. A validation-only search over slice centres and neighbour spacings moves AUC by 0.180 for MRI and 0.078 for PET, selecting narrow spacing for MRI and wide spacing for PET, with the chosen coronal centres falling on the hippocampal body and on the posterior cingulate respectively. The contribution, however, is the evidence layer built around that number. Shortcut controls collapse the model to AUC 0.622 (silhouette), 0.608 (exterior), and 0.500 (blank), and a label-permutation null yields 0.456. Forward region-of-interest (ROI) ablation shows that masking medial temporal cortex in MRI and the posterior default-mode network (DMN) in PET produces the largest shift in the AD logit, while area-matched controls remain indistinguishable from that null. Reverse ROI ablation shows that the medial temporal lobe alone retains 89.2% of above-chance discrimination in MRI and the posterior DMN alone retains 79.0% in PET. A quantitative comparison of attribution methods shows occlusion sensitivity reaching 3.5-5.0* enrichment inside a priori AD regions against 0.10-0.43* in controls. Ablation and attribution independently establish a biologically correct double dissociation: hippocampal evidence is carried by MRI, posterior cingulate evidence by PET.

Figures & tables

Group	MRI – PET series	Paired subjects	Train – Val – Test
CN	12,198 – 4,348	324	208 – 52 – 64
AD	5,916 – 2,424	230	148 – 36 – 46
All	18,114 – 6,772	554	356 – 88 – 110

TABLE I: ADNI Cohort Composition

Fig. 1: Preprocessing pipeline. MRI and PET are converted from DICOM with dcm2niix . MRI receives N4 bias correction and brain extraction; PET is kept native, with no bias correction and no skull stripping. PET is rigidly registered to the subject’s T1 (6 DOF, mutual information), the T1 is nonlinearly registered to the MNI152 1.5 mm template with ANTs SyN, and the two transforms are composed so each volume is resampled exactly once. MRI is z-scored within the brain mask; PET is converted to SUVR against an eroded pons reference. Fixed MNI coordinates then define the extracted $224\times 224$ slices.

Fig. 2: Representative CN (a) and AD (b) subjects before and after preprocessing, shown in all three planes for both modalities. The AD subject shows the expected ventricular enlargement and cortical thinning on MRI, and posterior hypometabolism on PET.

Axial			Coronal			Sagittal
$c$	$\Delta$	AUC	$c$	$\Delta$	AUC	$c$	$\Delta$	AUC
$+20$	0	0.704	$-10$	0	0.884	$-10$	0	0.758
$+20$	4	0.782	$-10$	4	0.871	$-10$	4	0.781
$+20$	8	0.753	$-10$	8	0.814	$-10$	8	0.826
$+20$	14	0.742	$-10$	14	0.848	$-10$	14	0.798
$0$	0	0.741	$-25$	0	0.809	$-25$	0	0.848

TABLE II: MRI Slice Search, Validation AUC

Axial			Coronal			Sagittal
$c$	$\Delta$	AUC	$c$	$\Delta$	AUC	$c$	$\Delta$	AUC
$+20$	0	0.875	$-40$	0	0.923	$-6$	0	0.925
$+20$	6	0.908	$-40$	6	0.926	$-6$	6	0.928
$+20$	10	0.901	$-40$	10	0.947	$-6$	10	0.928
$+20$	16	0.905	$-40$	16	0.943	$-6$	16	0.931
$+32$	0	0.916	$-55$	0	0.893	$-25$	0	0.901

TABLE III: PET Slice Search, Validation AUC

Fig. 3: Six-stream fusion architecture. Each of the six inputs (MRI and PET $\times$ three planes) passes through an independent VGG16-BN convolutional trunk initialised from ImageNet, with the first 34 layers frozen. Global average pooling produces a 512-dimensional descriptor per stream. The six descriptors are concatenated into a 3072-dimensional vector, followed by dropout ( $p=0.5$ ) and a single linear layer to two logits.

Mode	AUC	Accuracy	F1
Axial	0.877	0.736	0.695
Coronal	0.891	0.764	0.735
Sagittal	0.897	0.845	0.813
Fuse 3-plane	0.909	0.818	0.778
Fuse 3-plane single-plane init	0.944	0.873	0.844

TABLE IV: MRI Performance, Test Set

Mode	AUC	Accuracy	F1
Axial	0.913	0.836	0.809
Coronal	0.900	0.818	0.800
Sagittal	0.918	0.827	0.796
Fuse 3-plane	0.928	0.827	0.796
Fuse 3-plane single-plane init	0.936	0.855	0.830

TABLE V: PET Performance, Test Set

Mode	AUC	Accuracy	F1
MRI 3-plane	0.944	0.873	0.844
PET 3-plane	0.936	0.855	0.830
Fusion MRI + PET	0.962	0.909	0.891

TABLE VI: Multimodal Fusion, Test Set

Method	Modality	AUC	BACC
3D-ResNet [ 30 ]	M	0.936	0.865
3D-ResNet [ 30 ]	P	0.951	0.890
3D-ViT [ 31 ]	M	0.936	0.862
3D-ViT [ 31 ]	P	0.935	0.887
ResNet early fusion [ 32 ]	M + P	0.855	0.826
ResNet middle fusion [ 30 ]	M + P	0.881	0.826

TABLE VII: Comparison with Published Models

	Actual AD	Actual CN
Predicted AD	TP $=41$	FP $=5$
Predicted CN	FN $=5$	TN $=59$

TABLE VIII: Confusion Matrix, Fusion Model, Test Set

Mode	AUC
Without MRI	0.932
Without PET	0.938
All	0.962

TABLE IX: Modality Ablation on the Fusion Model

Mode	AUC
MRI - with preprocessing	0.944
MRI - without preprocessing	0.860
PET - with preprocessing	0.936
PET - without preprocessing	0.930
Fusion - with preprocessing	0.962
Fusion - without preprocessing	0.934

TABLE X: Preprocessing Ablation

Condition	AUC
Intact	0.962
Silhouette	0.622
Exterior	0.608
Blank	0.500

TABLE XI: Brain-Removal Shortcut Controls

Condition	AUC
Intact	0.962
Label-permutation null	0.456

TABLE XII: Label Permutation Null

ROI masked out	Area (px)	$\Delta$ AUC	$\Delta$ logit AD	$Z$ vs. null
Medial temporal	4,361	0.038	0.283	$+20.3$
Hippocampus	1,844	0.015	0.087	$+5.4$
Amygdala	1,011	0.012	0.057	$+3.0$
Parahippocampal	1,551	0.006	0.078	$+4.7$
Control composite	4,297	0.005	0.000	$-1.3$
Precentral gyrus	3,218	0.002	0.002	$-1.2$

TABLE XIII: Forward Ablation, MRI Streams

ROI masked out	Area (px)	$\Delta$ AUC	$\Delta$ logit AD	$Z$ vs. null
Posterior DMN	6,672	0.060	0.446	$+25.9$
Precuneus	1,878	0.016	0.199	$+11.1$
Posterior cingulate	1,728	0.011	0.274	$+15.6$
Inferior parietal	3,066	0.005	0.126	$+6.8$
Control composite	3,088	0.005	0.036	$+1.4$
Precentral gyrus	1,801	$-0.001$	0.023	$+0.7$

TABLE XIV: Forward Ablation, PET Streams

ROI retained	Area (px)	AUC	Retained	$\rho$
Medial temporal	4,361	0.912	89.2%	$+0.80$
Amygdala	1,011	0.877	81.8%	$+0.73$
Hippocampus	1,844	0.852	76.2%	$+0.73$
Parahippocampal	1,551	0.789	62.7%	$+0.57$
Control composite	4,297	0.476	$-5.2$ %	$+0.01$
Control composite XL	8,637	0.646	31.6%	$+0.26$

TABLE XV: Reverse Ablation, MRI Streams (ROI Retained, All Else Masked)

ROI retained	Area (px)	AUC	Retained	$\rho$
Posterior DMN	6,672	0.865	79.0%	$+0.77$
Precuneus	1,878	0.833	72.0%	$+0.66$
Posterior cingulate	1,728	0.816	68.4%	$+0.66$
Inferior parietal	3,066	0.784	61.5%	$+0.67$
Control composite	3,088	0.593	20.1%	$+0.12$
Control composite XL	4,716	0.561	13.2%	$+0.09$

TABLE XVI: Reverse Ablation, PET Streams (ROI Retained, All Else Masked)

Fig. 4: Attribution maps for one AD subject across all six streams and four methods, overlaid on the MNI template, with hippocampus (cyan), posterior cingulate (green) and precuneus (orange) outlined. Occlusion and Layer-CAM concentrate inside the outlined regions in the corresponding modality; Grad-CAM and Grad-CAM++ concentrate on the cortical rim and image border.

Stream	Medial temporal (hipp / amyg / parahipp)	Posterior DMN (post-cing / precuneus / inf-par)	Control (precentral / occipital / lingual)
MRI axial	1.43 / 2.94 / 1.17	n/a	— / 0.10 / 0.35
MRI coronal	3.54 / 4.17 / 3.54	n/a	0.34 / — / —
MRI sagittal	3.22 / 2.99 / 2.65	n/a	0.78 / 0.55 / 1.69
PET axial	n/a	2.46 / 3.83 / 0.70	0.43 / 0.29 / —
PET coronal	0.38 / — / 0.61	4.99 / 3.96 / 1.15	— / — / 0.18
PET sagittal	1.96 / 1.20 / 1.51	n/a	0.11 / 1.08 / 3.26

TABLE XVII: Occlusion Enrichment by Stream (AD Subjects;

1.0=

Chance)

Method	Peak enrichment, AD regions	Enrichment, control regions	Pointing game (AD, best stream)	Usable
Occlusion	$3.5-5.0$	$0.10-0.43$	54.3% (chance 16.7%)	Yes - primary evidence
Layer-CAM	$2.0-3.0$	$0.22-0.62$	69.6% (chance 6.2%)	Directionally; weak AD–CN separation
Grad-CAM	$\approx 1.0-2.3$	$0.36-2.00$	21.7% (chance 4.8%)	No
Grad-CAM++	$\approx 1.0$	$0.59-1.32$	8.7% (chance 16.7%)	No

TABLE XVIII: Attribution Method Comparison

Method	Stream	AD	CN	Chance
Occlusion	MRI axial	39.1%	17.2%	6.7%
Occlusion	MRI coronal	52.2%	29.7%	6.2%
Occlusion	MRI sagittal	54.3%	46.9%	4.7%
Occlusion	PET axial	54.3%	40.6%	16.7%
Occlusion	PET coronal	54.3%	23.4%	13.1%
Layer-CAM	MRI coronal	69.6%	59.4%	6.2%

TABLE XIX: Pointing Game: Peak Attribution Inside the A Priori AD ROI Set

Explore similar work

Jun 18, 2026cs.LG

Alzheimer's Disease Diagnosis using a Multimodal Approach with 3D MRI and PET

Alzheimer's disease (AD) is an irreversible neurodegenerative disorder and a leading cause of death worldwide. Early diagnosis plays an important part especially at the Mild Cognitive Impairment stage, where timely intervention can help slow its progression before it advances to AD. Neuroimaging data, like Magnetic Resonance Imaging (MRI) and Positron Emission Tomography (PET) scans, can help detect brain changes early by providing structural and functional brain changes related to the disease. Yet, many multimodal models still fuse MRI and PET with static concatenation and apply identical computation to all subjects, which limits robustness to patient/site heterogeneity and can waste computation. To address these limitations, we present the first study of combining 3D convolutional feature extractors with three fusion strategies - concatenation, Gated Multimodal Unit (GMU), and gated self-attention - and a sparsely gated Mixture-of-Experts (MoE) classifier that performs input-adaptive routing, activating only the most informative experts per case. Finally, we utilize Grad-CAM to visualize disease-related regions, ensuring model interpretability. Experiments are performed across three binary classification tasks (NC vs. MCI, MCI vs. AD, and NC vs. AD). Results show that GMU achieves accuracies of 80.46 % (NC vs. MCI) and 95.47 % (NC vs. AD), while gated self-attention attains 82.08 % on MCI vs. AD. Ablations show that removing the MoE consistently degrades accuracy across all tasks. These findings underscore the value of input-adaptive, multimodal modeling for AD diagnosis by leveraging the complementary nature of MRI and PET.

Loukas Ilias, Anthi-Maria Vozinaki, Christos Ntanos +1

DSS Lab, School of ECE, NTUA, 15780 Athens, Greece

Alzheimer Multimodal Neuroimaging Positron Emission Tomography Magnetic Resonance Imaging

Ask PDF

Sep 14, 2026cs.CV

Anatomical Grounding and Leakage-Aware Multimodal Contrastive Learning for Alzheimer's Disease Classification from Structural MRI

Deep networks trained on structural MRI for Alzheimer's disease (AD) staging often reach reasonable accuracy while attending to anatomically irrelevant regions, and multimodal models that add clinical tables frequently rely on variables that were used to assign the diagnostic label in the first place. We study both issues with a deliberately lightweight slice-based encoder (ResNet18 with a one-layer Transformer over slices) on 1,075 baseline T1-weighted scans from ADNI-1. First, we use FastSurfer segmentations as an anatomical reference: YOLOv8 models trained on segmentation-derived labels localize Alzheimer-relevant structures with mAP_50 above 0.96, and a Grad-CAM comparison shows that the image-only classifier frequently attends to the skull, orbits and background. Second, we adapt a CLIP-style image - tabular contrastive framework and organize ADNIMERGE variables along a label-leakage spectrum. Fusion with cognitive scores yields 87.3% three-way accuracy, which we treat as a leakage-driven upper bound rather than an imaging result; fusion with regional volumes yields 73.0%. We observe that the choice of contrastive target changes what the image encoder learns: on MCI vs. CN, the image-only head reaches 52.4% when the encoder is aligned to cognitive scores and 73.8% when aligned to volumes, although no tabular input is used at inference. Third, restricting the input to a per-subject crop of the medial temporal lobe raises image-only three-way accuracy from 58.7% to 65.1%. All results come from single runs on a small balanced test set, and we report confidence intervals and the protocol differences that prevent direct comparison with published numbers.

Paul-Gabriel Nicolae, Irina Georgiana Mocanu

Computer Science and Engineering Department, Faculty of Automatic Control and Computers, National University of Science and Technology POLITEHNICA Bucharest, Romania

Alzheimer Multimodal Contrastive Learning Disease Anatomy Deep Network

Ask PDF

Jul 8, 2026cs.CV

AT-Attn: Temporal-Aware Cross-Attention for Longitudinal Multimodal Alzheimer's Disease Diagnosis

In longitudinal Alzheimer's disease (AD) diagnosis support, clinical and imaging information is often collected at irregular visits. Integrating these multimodal observations may improve diagnostic assessment, but naive fusion can degrade performance when MRI is noisy or intermittently unavailable. We propose AT-Attn, a temporal-aware multimodal framework that combines Change-and-Time encoding, time-biased asymmetric cross-attention, and gated fusion to integrate MRI with longitudinal clinical information. We evaluate AT-Attn on an MRI-retained ADNI cohort of 1,520 patients using structural MRI, six cognitive-scale trajectories, and seven static clinical variables under patient-level five-fold cross-validation. The main asymmetric AT-Attn model achieves accuracy 0.719+/-0.024, macro F1 0.721+/-0.023, ROC-AUC 0.873+/-0.013, and PR-AUC 0.783+/-0.018, outperforming unimodal and naive multimodal fusion baselines while remaining competitive with strong tabular baselines. These results suggest that a temporal-aware and constrained fusion strategy can help structural MRI contribute clinically relevant complementary information for patient-level AD diagnosis support.

Xinyue Du, Yibo Liu, Zhenglei Zhou +3

School of Information Science and Engineering, East China University of Science and Technology, China · School of Computer Science, Shanghai Jiao Tong University, China · Tencent, China

Alzheimer Multimodal Clinical Data Disease Temporal Attention Cross-Attention

Ask PDF