cs.CVOct 24, 2026

How Robust Are CNNs and Vision Transformers for Alzheimer's MRI Classification Across Datasets?

Authors: Hafija Aktar, Molly Croes, Frank S. Menniti, Abdeltawab Hendawi

Organizations: Department of Computer Science, University of Rhode Island, Kingston, RI 02881, USA · Department of Biomedical and Pharmaceutical Sciences, George & Anne Ryan Institute for Neuroscience, University of Rhode Island, Kingston, RI 02881, USA

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Accurate assessment of Alzheimer's disease stages from MRI is important for understanding disease progression and supporting the development of computer-aided analysis tools. Deep learning models have shown promising results for Alzheimer's disease classification from MRI. However, when evaluated on single datasets, it is difficult to determine whether the observed performance reflects consistent architectural behavior or dataset-specific effects. In this work, we present a multi-dataset benchmarking framework with standardized preprocessing and evaluation. We compare five architectures: ResNet-50, ReXNet-150, Swin Transformer, ViT-Base, and ViT-Large. The models are evaluated across three MRI datasets representing severe class imbalance, train-test distribution mismatch, and large-scale augmentation. The results show broadly consistent performance patterns across datasets, with ViT-Large achieving the best performance on all three datasets and reaching up to 99.84% accuracy and 99.86% macro F1. Performance differences between architectures are most pronounced under distribution mismatch and considerably smaller in the augmented-data setting. Across datasets, lower class-level performance is observed primarily among adjacent early-stage categories, particularly Very Mild and Mild impairment. These findings highlight the importance of evaluating deep learning architectures across multiple data conditions rather than relying on single-dataset performance.

Figures & tables

PaperYearDataModelKey findingRelevance
He et al. [ 11 ]2016ImageNetResNetResidual learning enables deep, transferable CNNs.CNN baseline
Han et al. [ 12 ]2021ImageNetReXNetEfficient CNN design reduces representational bottlenecks.Efficient CNN
Dosovitskiy et al. [ 8 ]2021ImageNetViTPatch-based transformers achieve strong performance at scale.ViT basis
Liu et al. [ 13 ]2021ImageNetSwinHierarchical shifted-window attention improves efficiency and representation.Swin baseline
Basaia et al. [ 2 ]2019ADNI, clinical3D CNNSingle structural MRI scans can support AD classification.CNN AD study
Wen et al. [ 1 ]2020ADNI, AIBL, OASISCNN, SVMHighlights reproducibility issues and limited cross-dataset generalization.Benchmarking gap
TABLE I: Representative studies on MRI-based Alzheimer’s disease classification, covering CNN, transformer, and hybrid models, along with multi-dataset benchmarking and dataset shift.
Fig. 1: Representative axial MRI slices illustrating the four diagnostic categories used in this study.
DatasetTotal ImagesData TypeKey PropertyEvaluation Focus
Dataset I6,400Real MRI imagesSevere class imbalanceImpact of severe class imbalance on model performance
Dataset II11,519Real and synthetic MRI imagesBalanced train, imbalanced testRobustness under train-test distribution mismatch
Dataset III33,984Augmented MRI imagesLarge-scale augmentation with moderate imbalanceEffect of large-scale augmentation on classification performance
TABLE II: Key characteristics and evaluation focus of the three MRI datasets.
 Dataset IDataset IIDataset III
ClassTrainValTestTrainValTestTrainValTest
Non / No Impairment2,2864574572,0425186406,7201,4401,440
Very Mild1,5683203202,0415194486,2721,3441,344
Mild6721281282,0485121796,2721,3441,344
Moderate46992,061499124,524970970
Split Total4,5729149148,1922,0481,27923,7885,0985,098
TABLE III: Per-class distribution across all three datasets after stratified splitting.
Fig. 2: Overview of the experimental pipeline for Alzheimer’s disease MRI stage classification. MRI datasets are preprocessed and augmented before being used to train multiple deep learning models. The models are fine-tuned and evaluated using classification metrics to predict Alzheimer’s disease stages.
ModelArchitecture TypeDepthPatch SizeAttention HeadsParameters
ResNet-50CNN50––25M
ReXNet-150CNN150––16M
Swin TransformerHierarchical Transformer244×44\times 43 / 6 / 12 / 2488M
ViT-BaseVision Transformer1216×1616\times 161286M
ViT-LargeVision Transformer2416×1616\times 1616307M
TABLE IV: Architectural characteristics of the evaluated deep learning models.
 Dataset IDataset IIDataset III
ModelLearning Rate (Backbone / Head)Weight DecayLearning RateWeight DecayLearning Rate (Backbone / Head)Weight Decay
ResNet-503×10−43{\times}10^{-4} / 1×10−31{\times}10^{-3}10−410^{-4}3×10−43{\times}10^{-4}10−410^{-4}1×10−41{\times}10^{-4}10−410^{-4}
ReXNet-1503×10−43{\times}10^{-4} / 1×10−31{\times}10^{-3}10−410^{-4}1×10−41{\times}10^{-4}10−410^{-4}1×10−41{\times}10^{-4}10−410^{-4}
Swin Transformer5×10−55{\times}10^{-5} / 1×10−41{\times}10^{-4}0.050.055×10−55{\times}10^{-5}10−410^{-4}5×10−55{\times}10^{-5}10−410^{-4}
ViT-Base5×10−55{\times}10^{-5} / 1×10−41{\times}10^{-4}0.050.051×10−41{\times}10^{-4}10−410^{-4}5×10−65{\times}10^{-6} / 5×10−55{\times}10^{-5}10−410^{-4}
ViT-Large3×10−53{\times}10^{-5} / 1×10−41{\times}10^{-4}0.050.051×10−41{\times}10^{-4}10−410^{-4}2×10−62{\times}10^{-6} / 5×10−55{\times}10^{-5}10−410^{-4}
TABLE V: Model-specific optimization settings across the three datasets. A single learning rate indicates that the same rate is applied to the backbone and classification head.
HyperparameterValue
Input Resolution224×224224\times 224
Mini-Batch Size32
Max Epochs20
Loss FunctionCross-Entropy (Label Smoothing)
Gradient Clipping1.0 (Datasets I & III)
TABLE VI: Common training hyperparameters used across the evaluated models.
SettingDataset IDataset IIDataset III
LR SchedulerCosine AnnealingReduceLROnPlateauReduceLROnPlateau
Early Stop MonitorBalanced AccuracyMacro F1Macro F1
Early Stop Patience455
Label Smoothing0.020.100.10
Imbalance StrategyWeightedRandomSampler ( α=0.5\alpha{=}0.5 )Class-weighted lossNone
Gradient Accumulation2 stepsNone4 steps
TABLE VII: Dataset-specific training configurations. ReduceLROnPlateau reduces the learning rate by a factor of 0.5. WeightedRandomSampler applies tempered inverse-frequency weights with α=0.5\alpha=0.5 .
 Dataset IDataset IIDataset III
ModelAcc (%)Macro F1 (%)Acc (%)Macro F1 (%)Acc (%)Macro F1 (%)
ViT-Large98.4798.8597.8198.3099.8499.86
Swin Transformer98.3698.7597.1997.6099.5799.60
ReXNet-15097.8198.3895.7095.8099.4199.46
ResNet-5097.4897.7994.0693.0098.5398.64
ViT-Base97.1698.1591.3086.0098.3598.50
TABLE VIII: Performance comparison of the five benchmark architectures across the three datasets. Macro F1 is reported to account for class imbalance.
 Dataset IDataset IIDataset III
ModelNon DementedVery Mild DementedMild DementedModerate DementedNo ImpairmentVery Mild ImpairmentMild ImpairmentModerate ImpairmentNon DementedVery Mild DementedMild DementedModerate Demented
ViT-Large0.98810.97790.98811.00000.98200.97100.98001.00000.99760.99780.99891.0000
Swin Transformer0.98570.97980.98461.00000.97900.96600.96001.00000.99340.99140.99891.0000
ReXNet-1500.98270.96840.98411.00000.96000.94700.96900.95700.99030.98990.99811.0000
ResNet-500.98370.96660.96121.00000.96000.92000.92000.92000.97990.97580.99030.9995
ViT-Base0.97230.96160.99221.00000.93000.89000.91000.72000.97300.97000.99601.0000
TABLE IX: Per-class F1-scores of the evaluated models across the three datasets.
Fig. 3: Accuracy decline from Dataset I to Dataset II for each model. Models are ordered by the magnitude of performance degradation under the evaluated train-test distribution mismatch. ViT-Large shows the smallest decrease (0.66 points), while ViT-Base shows the largest (5.86 points).
More in this topic

Alzheimer's Disease Classification

Alzheimer’s disease classification uses clinical, behavioral, and brain-imaging data to distinguish disease stages and predict progression.

See all 92 papers

Explore similar work

CardsList

  1. A comparison of CNN architectures for Alzheimer's disease detection in single-view MRI scans

    Aug 12, 2026Hiram Zuniga, Ulises Orozco-Rosas, Kenia PicosAlzheimer's Disease ClassificationConvolutional Neural Networks

    CETYS Universidad, Ave. CETYS Universidad No. 4, Fracc. El Lago, C.P. 22210, Tijuana, Baja California, Mexico

  2. Multiple Inputs and Mixwd data for Alzheimer's Disease Classification Based on 3D Vision Transformer

    Feb 28, 2026Juan A. Castro-Silva, Maria N. Moreno Garcia, Diego H. Peluffo-OrdoñezAlzheimer's Disease ClassificationMultimodal Classification

    Faculty of Engineering, Universidad Surcolombiana, Neiva 410002, Colombia · College of Computing, Mohammed VI Polytechnic University, Lot 660, Hay Moulay Rachid, Ben Guerir 43150, Morocco

  3. EigenGradCAM with AugSmooth: Enhancing Explainability and Precision in Multi-Stage Alzheimer's Disease Detection

    Oct 23, 2026Arsham Azam, Sagar Ashraf, Tayyaba Farhat +2Alzheimer's Disease ClassificationExplainable Medical Image Analysis

    Faculty of Computer Science and Information Technology, The Superior University, Lahore 54600, Pakistan · Intelligent Data Visual Computing Research (IDVCR), Lahore 54600, Pakistan · HospiceMD, Inc., West Covina, California, USA +1