How Robust Are CNNs and Vision Transformers for Alzheimer's MRI Classification Across Datasets?

Hafija Aktar; Molly Croes; Frank S. Menniti; Abdeltawab Hendawi

cs.CVOct 24, 2026

How Robust Are CNNs and Vision Transformers for Alzheimer's MRI Classification Across Datasets?

Authors: Hafija Aktar, Molly Croes, Frank S. Menniti, Abdeltawab Hendawi

Organizations: Department of Computer Science, University of Rhode Island, Kingston, RI 02881, USA · Department of Biomedical and Pharmaceutical Sciences, George & Anne Ryan Institute for Neuroscience, University of Rhode Island, Kingston, RI 02881, USA

Abstract

Accurate assessment of Alzheimer's disease stages from MRI is important for understanding disease progression and supporting the development of computer-aided analysis tools. Deep learning models have shown promising results for Alzheimer's disease classification from MRI. However, when evaluated on single datasets, it is difficult to determine whether the observed performance reflects consistent architectural behavior or dataset-specific effects. In this work, we present a multi-dataset benchmarking framework with standardized preprocessing and evaluation. We compare five architectures: ResNet-50, ReXNet-150, Swin Transformer, ViT-Base, and ViT-Large. The models are evaluated across three MRI datasets representing severe class imbalance, train-test distribution mismatch, and large-scale augmentation. The results show broadly consistent performance patterns across datasets, with ViT-Large achieving the best performance on all three datasets and reaching up to 99.84% accuracy and 99.86% macro F1. Performance differences between architectures are most pronounced under distribution mismatch and considerably smaller in the augmented-data setting. Across datasets, lower class-level performance is observed primarily among adjacent early-stage categories, particularly Very Mild and Mild impairment. These findings highlight the importance of evaluating deep learning architectures across multiple data conditions rather than relying on single-dataset performance.

Figures & tables

Paper	Year	Data	Model	Key finding	Relevance
He et al. [ 11 ]	2016	ImageNet	ResNet	Residual learning enables deep, transferable CNNs.	CNN baseline
Han et al. [ 12 ]	2021	ImageNet	ReXNet	Efficient CNN design reduces representational bottlenecks.	Efficient CNN
Dosovitskiy et al. [ 8 ]	2021	ImageNet	ViT	Patch-based transformers achieve strong performance at scale.	ViT basis
Liu et al. [ 13 ]	2021	ImageNet	Swin	Hierarchical shifted-window attention improves efficiency and representation.	Swin baseline
Basaia et al. [ 2 ]	2019	ADNI, clinical	3D CNN	Single structural MRI scans can support AD classification.	CNN AD study
Wen et al. [ 1 ]	2020	ADNI, AIBL, OASIS	CNN, SVM	Highlights reproducibility issues and limited cross-dataset generalization.	Benchmarking gap

TABLE I: Representative studies on MRI-based Alzheimer’s disease classification, covering CNN, transformer, and hybrid models, along with multi-dataset benchmarking and dataset shift.

Fig. 1: Representative axial MRI slices illustrating the four diagnostic categories used in this study.

Dataset	Total Images	Data Type	Key Property	Evaluation Focus
Dataset I	6,400	Real MRI images	Severe class imbalance	Impact of severe class imbalance on model performance
Dataset II	11,519	Real and synthetic MRI images	Balanced train, imbalanced test	Robustness under train-test distribution mismatch
Dataset III	33,984	Augmented MRI images	Large-scale augmentation with moderate imbalance	Effect of large-scale augmentation on classification performance

TABLE II: Key characteristics and evaluation focus of the three MRI datasets.

	Dataset I			Dataset II			Dataset III
Class	Train	Val	Test	Train	Val	Test	Train	Val	Test
Non / No Impairment	2,286	457	457	2,042	518	640	6,720	1,440	1,440
Very Mild	1,568	320	320	2,041	519	448	6,272	1,344	1,344
Mild	672	128	128	2,048	512	179	6,272	1,344	1,344
Moderate	46	9	9	2,061	499	12	4,524	970	970
Split Total	4,572	914	914	8,192	2,048	1,279	23,788	5,098	5,098

TABLE III: Per-class distribution across all three datasets after stratified splitting.

Fig. 2: Overview of the experimental pipeline for Alzheimer’s disease MRI stage classification. MRI datasets are preprocessed and augmented before being used to train multiple deep learning models. The models are fine-tuned and evaluated using classification metrics to predict Alzheimer’s disease stages.

Model	Architecture Type	Depth	Patch Size	Attention Heads	Parameters
ResNet-50	CNN	50	–	–	25M
ReXNet-150	CNN	150	–	–	16M
Swin Transformer	Hierarchical Transformer	24	$4\times 4$	3 / 6 / 12 / 24	88M
ViT-Base	Vision Transformer	12	$16\times 16$	12	86M
ViT-Large	Vision Transformer	24	$16\times 16$	16	307M

TABLE IV: Architectural characteristics of the evaluated deep learning models.

	Dataset I		Dataset II		Dataset III
Model	Learning Rate (Backbone / Head)	Weight Decay	Learning Rate	Weight Decay	Learning Rate (Backbone / Head)	Weight Decay
ResNet-50	$3{\times}10^{-4}$ / $1{\times}10^{-3}$	$10^{-4}$	$3{\times}10^{-4}$	$10^{-4}$	$1{\times}10^{-4}$	$10^{-4}$
ReXNet-150	$3{\times}10^{-4}$ / $1{\times}10^{-3}$	$10^{-4}$	$1{\times}10^{-4}$	$10^{-4}$	$1{\times}10^{-4}$	$10^{-4}$
Swin Transformer	$5{\times}10^{-5}$ / $1{\times}10^{-4}$	$0.05$	$5{\times}10^{-5}$	$10^{-4}$	$5{\times}10^{-5}$	$10^{-4}$
ViT-Base	$5{\times}10^{-5}$ / $1{\times}10^{-4}$	$0.05$	$1{\times}10^{-4}$	$10^{-4}$	$5{\times}10^{-6}$ / $5{\times}10^{-5}$	$10^{-4}$
ViT-Large	$3{\times}10^{-5}$ / $1{\times}10^{-4}$	$0.05$	$1{\times}10^{-4}$	$10^{-4}$	$2{\times}10^{-6}$ / $5{\times}10^{-5}$	$10^{-4}$

TABLE V: Model-specific optimization settings across the three datasets. A single learning rate indicates that the same rate is applied to the backbone and classification head.

Hyperparameter	Value
Input Resolution	$224\times 224$
Mini-Batch Size	32
Max Epochs	20
Loss Function	Cross-Entropy (Label Smoothing)
Gradient Clipping	1.0 (Datasets I & III)

TABLE VI: Common training hyperparameters used across the evaluated models.

Setting	Dataset I	Dataset II	Dataset III
LR Scheduler	Cosine Annealing	ReduceLROnPlateau	ReduceLROnPlateau
Early Stop Monitor	Balanced Accuracy	Macro F1	Macro F1
Early Stop Patience	4	5	5
Label Smoothing	0.02	0.10	0.10
Imbalance Strategy	WeightedRandomSampler ( $\alpha{=}0.5$ )	Class-weighted loss	None
Gradient Accumulation	2 steps	None	4 steps

TABLE VII: Dataset-specific training configurations. ReduceLROnPlateau reduces the learning rate by a factor of 0.5. WeightedRandomSampler applies tempered inverse-frequency weights with

\alpha=0.5

	Dataset I		Dataset II		Dataset III
Model	Acc (%)	Macro F1 (%)	Acc (%)	Macro F1 (%)	Acc (%)	Macro F1 (%)
ViT-Large	98.47	98.85	97.81	98.30	99.84	99.86
Swin Transformer	98.36	98.75	97.19	97.60	99.57	99.60
ReXNet-150	97.81	98.38	95.70	95.80	99.41	99.46
ResNet-50	97.48	97.79	94.06	93.00	98.53	98.64
ViT-Base	97.16	98.15	91.30	86.00	98.35	98.50

TABLE VIII: Performance comparison of the five benchmark architectures across the three datasets. Macro F1 is reported to account for class imbalance.

	Dataset I				Dataset II				Dataset III
Model	Non Demented	Very Mild Demented	Mild Demented	Moderate Demented	No Impairment	Very Mild Impairment	Mild Impairment	Moderate Impairment	Non Demented	Very Mild Demented	Mild Demented	Moderate Demented
ViT-Large	0.9881	0.9779	0.9881	1.0000	0.9820	0.9710	0.9800	1.0000	0.9976	0.9978	0.9989	1.0000
Swin Transformer	0.9857	0.9798	0.9846	1.0000	0.9790	0.9660	0.9600	1.0000	0.9934	0.9914	0.9989	1.0000
ReXNet-150	0.9827	0.9684	0.9841	1.0000	0.9600	0.9470	0.9690	0.9570	0.9903	0.9899	0.9981	1.0000
ResNet-50	0.9837	0.9666	0.9612	1.0000	0.9600	0.9200	0.9200	0.9200	0.9799	0.9758	0.9903	0.9995
ViT-Base	0.9723	0.9616	0.9922	1.0000	0.9300	0.8900	0.9100	0.7200	0.9730	0.9700	0.9960	1.0000

TABLE IX: Per-class F1-scores of the evaluated models across the three datasets.

Fig. 3: Accuracy decline from Dataset I to Dataset II for each model. Models are ordered by the magnitude of performance degradation under the evaluated train-test distribution mismatch. ViT-Large shows the smallest decrease (0.66 points), while ViT-Base shows the largest (5.86 points).

Alzheimer's Disease Classification

Alzheimer’s disease classification uses clinical, behavioral, and brain-imaging data to distinguish disease stages and predict progression.

See all 92 papers

Aug 12, 2026

A comparison of CNN architectures for Alzheimer's disease detection in single-view MRI scans

Alzheimer's disease is a leading cause of death with no cure. Therefore, early detection is critical to slow progression and preserve quality of life. Diagnosis relies on medical history, cognitive tests, physical exams, and MRI brain scans, making deep learning suitable for Alzheimer's classification. This work proposes a benchmark that evaluates ten different convolutional neural network (CNN) architectures (including ResNet, DenseNet, MobileNet, EfficientNet, and VGG family models) under the same held-out test split protocol. A two-stage transfer learning and full fine-tuning pipeline is introduced to perform training using a class-balanced subset (3,900 images) derived from the OASIS medical imaging dataset, comprising 86,437 single-view MRI brain scans labeled into four classifications of Alzheimer's disease: Non-Demented, Very Mild Dementia, Mild Dementia, and Moderate Dementia. The best results were achieved by VGG16, with a 0.9637 validation accuracy and a 0.9533 test accuracy score. A key finding documented in this work is the difficulty of classifying the transition from Non-Demented to Very Mild Demented stages, observed consistently across all ten architectures.

Hiram Zuniga, Ulises Orozco-Rosas, Kenia Picos

CETYS Universidad, Ave. CETYS Universidad No. 4, Fracc. El Lago, C.P. 22210, Tijuana, Baja California, Mexico

Alzheimer's Disease Classification Convolutional Neural Networks Medical Image Benchmarks

Ask PDF

Feb 28, 2026

Multiple Inputs and Mixwd data for Alzheimer's Disease Classification Based on 3D Vision Transformer

The current methods for diagnosing Alzheimer Disease using Magnetic Resonance Imaging (MRI) have significant limitations. Many previous studies used 2D Transformers to analyze individual brain slices independently, potentially losing critical 3D contextual information. Region of interest-based models often focus on only a few brain regions despite Alzheimer's affecting multiple areas. Additionally, most classification models rely on a single test, whereas diagnosing Alzheimer's requires a multifaceted approach integrating diverse data sources for a more accurate assessment. This study introduces a novel methodology called the Multiple Inputs and Mixed Data 3D Vision Transformer (MIMD-3DVT). This method processes consecutive slices together to capture the feature dimensions and spatial information, fuses multiple 3D ROI imaging data inputs, and integrates mixed data from demographic factors, cognitive assessments, and brain imaging. The proposed methodology was experimentally evaluated using a combined dataset that included the Alzheimer's Disease Neuroimaging Initiative (ADNI), the Australian Imaging, Biomarker, and Lifestyle Flagship Study of Ageing (AIBL), and the Open Access Series of Imaging Studies (OASIS). Our MIMD-3DVT, utilizing single or multiple ROIs, achieved an accuracy of 97.14%, outperforming the state-of-the-art methods in distinguishing between Normal Cognition and Alzheimer's Disease.

Juan A. Castro-Silva, Maria N. Moreno Garcia, Diego H. Peluffo-Ordoñez

Faculty of Engineering, Universidad Surcolombiana, Neiva 410002, Colombia · College of Computing, Mohammed VI Polytechnic University, Lot 660, Hay Moulay Rachid, Ben Guerir 43150, Morocco

Alzheimer's Disease Classification Multimodal Classification 3D ViTs Neuroimaging

Ask PDF

Oct 23, 2026

EigenGradCAM with AugSmooth: Enhancing Explainability and Precision in Multi-Stage Alzheimer's Disease Detection

Alzheimer's disease (AD) is a progressive neurological disorder associated with memory loss, cognitive decline, and behavioral disturbances. Early and accurate diagnosis is essential, yet MRI-based deep learning models often lack interpretability, limiting their clinical use. This study presents an explainable deep learning approach for four-class AD staging using 2D axial MRI images. The method combines eigenvector-based Grad-CAM with gradient smoothing to improve saliency localization and heatmap stability. Principal component analysis is applied to gradient-weighted convolutional layers, while Gaussian noise is added to saliency weights before decomposition. To address class imbalance in the Moderate Dementia category, Wasserstein Generative Adversarial Network with Gradient Penalty (WGAN-GP) generates synthetic samples to balance the four-stage training distribution. Custom convolutional neural networks are trained on the Best Alzheimer MRI Dataset, achieving 97.2% test accuracy, 97.0% balanced accuracy, and a Matthews correlation coefficient of 0.944. The area under the ROC curve ranges from 0.96 to 0.99 across classes. Generated heatmaps highlight the hippocampus and entorhinal cortex, regions associated with Alzheimer's disease biomarkers. These findings suggest that the proposed approach combines strong classification performance with improved model interpretability, although further clinical validation is needed.

Arsham Azam, Sagar Ashraf, Tayyaba Farhat +2

Faculty of Computer Science and Information Technology, The Superior University, Lahore 54600, Pakistan · Intelligent Data Visual Computing Research (IDVCR), Lahore 54600, Pakistan · HospiceMD, Inc., West Covina, California, USA +1

Alzheimer's Disease Classification Explainable Medical Image Analysis Alzheimer's Disease Gradient-Based Attribution

Ask PDF

Abstract

Alzheimer's Disease Classification

Explore similar work

A comparison of CNN architectures for Alzheimer's disease detection in single-view MRI scans

Multiple Inputs and Mixwd data for Alzheimer's Disease Classification Based on 3D Vision Transformer

EigenGradCAM with AugSmooth: Enhancing Explainability and Precision in Multi-Stage Alzheimer's Disease Detection