Comparative Study of Anatomical and Learned Features in AI Models for Structural Brain MRI
Authors: Boyang Yu, Miquel Lopez Escoriza, Long Chen, Arjun V. Masurkar, Narges Razavian, Carlos Fernandez-Granda
Organizations: Center for Data Science, New York University, NYU, 60 Fifth Ave, New York, 10011, New York, United States. · Department of Neurology, NYU Grossman School of Medicine, 550 1st Ave, New York, 10016, New York, United States. · Courant Institute of Mathematical Sciences, NYU, 251 Mercer St, New York, 10012, New York, United States.
In this work, we comprehensively evaluate three popular feature-extraction paradigms in AI-based neuroimaging modeling: (1) computation of anatomical surfaces and volumes, (2) supervised learning with convolutional neural networks (CNNs), and (3) unsupervised pretraining of vision transformer (ViT) foundation models, followed by supervised finetuning. Our study is based on 18 publicly available datasets containing 3D structural T1-weighted MRI scans from approximately 80,000 participants across seven distinct clinical tasks. We observe that a linear model based on anatomical features matches the diagnostic performance of complex nonlinear features learned by sophisticated AI frameworks, including foundation models trained on thousands of scans. Conversely, CNNs and pretrained ViTs learn features that implicitly capture relevant anatomical information, bypassing the need for explicit feature extraction. Building upon these insights, we propose Anatomy Segmentation Pretraining (ASP), a novel method to incorporate anatomical information during foundation-model pretraining, which outperforms existing models in biological age estimation.
Figures & tables
Figure 1: Overview of the study . (a) Cohorts used in the study. Cohorts used for pretraining, downstream training, internal testing, and external evaluation. Reported n values denote MRI scans; all splits were performed at the participant level, such that all scans from the same participant were assigned to the same split. External cohorts were drawn from independent data sources. (b) AI models included in the study . These represent the main feature-extraction approaches for neuroimaging data: linear models based on anatomical features, convolutional neural networks (CNNs) and vision transformers (ViTs) pretrained on large general-population cohort. For the ViTs, in addition to standard pretraining based on masked autoencoding, we propose a novel scheme called Anatomical Segmentation Pretraining (ASP).
Dataset
Number of scans
Number of subjects
Age (years) (mean ± std)
Gender (female %)
Brain volume in mL (mean ± std)
UKBB
67294
62665
64.93 ± 7.69
52.81%
1156 ± 114
HCP
1350
1350
38.66 ± 25.76
54.97%
1179 ± 137
CamCAN
653
653
54.2 ± 18.5
49.35%
1156 ± 128
IXI
581
581
48.65 ± 16.47
53.87%
1137 ± 131
NACC
6956
4926
71.41 ± 11.25
53.34%
1034 ± 185
ADNI
6471
1817
74.68 ± 7.64
47.45%
1072 ± 116
Table 1: Datasets used to build the pretraining cohort, and the training and evaluation cohorts for each task in the study.
Figure 2: Performance comparison across diagnostic and segmentation tasks. Alzheimer’s disease diagnosis was evaluated across an internal ADNI test cohort and seven external cohorts (NACC, AIBL, MIRIAD, and OASIS 1-4), where ASP ViT achieved the highest macro-AUC in most cohorts, though statistical significance over the CNN and anatomical logistic regression was only reached for OASIS 2. In the diagnostics of Parkinson’s disease (PPMI) and Epilepsy (OpenNeuro), the Pretrained ViT reached the highest AUC, showing a statistically significant improvement only over BrainIAC. Estimated probabilities for Epilepsy from ASP ViT are also shown for EPISURG, an additional evaluation set with only epilepsy cases. For the identification of brain tumor IDH mutations (UCSF-PDGM) ASP ViT yielded the highest AUC, though these differences were not statistically significant. CNN could not be applied to the epilepsy task due to limited training data. In segmentation tasks, a U-Net achieved the highest Dice score for adult brain gliomas (BraTS 2023), while the Pretrained ViT performed best for stroke lesion segmentation (ATLAS R2.0); in both cases, the differences were only statistically significant relative to UNETR. Overall, no significant performance differences were observed between the pretrained ViTs, CNNs, and anatomical logistic regression models across the evaluated tasks.
Figure 3: Age estimation performance across internal and external cohorts. The ASP ViT model demonstrates superior performance, achieving a mean absolute error (MAE) of 2.54 years in the internal test cohort (a) and 4.19 years in both the NACC (b) and ADNI (c) external test cohorts. While the Pretrained ViT maintains competitive performance on the internal cohort with an MAE only 0.35 years greater than the ASP ViT, this performance gap widens to over 1 year in the external cohorts. Logistic regression underperforms relative to the ViT-based approaches by a margin of 1 to 2 years, though it consistently outperforms the CNN. These performance comparisons remain stable when the data is stratified by sex and age. Age-based stratification reveals more pronounced errors for individuals over 80, a demographic less represented in the training and pretraining datasets. When the best-performing model (ASP ViT) is applied to clinical populations, age estimates for dementia patients from the NACC (b) and ADNI (c) datasets are consistently higher than their biological age across most age intervals.
Figure 4: a. Prompt-based segmentation architecture . The input consists of an image and a brain region index, which is used to retrieve the corresponding brain region embedding. This embedding is added to the image representation, and the combined features are passed through a lightweight decoder to predict the segmentation mask for the specified region. b. Linear‑probe accuracy/ R2 on six target variables—two global attributes (age, gender) and four region‑specific volumetric measures—tracked over pre‑training epochs. The differences in overall accuracy across regions arise because some regions are inherently easier to predict than others. For example, cortex volume tends to be more accurately segmented due to its larger size and higher contrast, compared to smaller and less distinct regions like the hippocampus. For demographic statistics the same happens. Gender is easier to infer than age since it is a binary attribute, while age requires finer-grained regression. c–d. Validation IoU by Brain Region. Regional performance varies based on anatomical complexity; for instance, the superior frontal cortex’s small, irregular geometry results in lower IoU, whereas the smoother, more consistent morphology of the pallidum allows for higher segmentation accuracy. e. ASP training ablation. Age-estimation performance for different ASP training variants on the external NACC cohort. The full ASP model achieves the lowest error, while removing alternating MAE updates, removing encoder freezing, using MAE alone, or training from scratch reduces performance. Detailed results are provided in Appendix Section 3.3 .
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
# images
# subjects
Age in yrs (mean ± std)
Gender (female %)
Brain volume in mL (mean ± std)
UKBB
67294
62665
64.93 ± 7.69
52.81%
1156 ± 114
HCP
1350
1350
38.66 ± 25.76
54.97%
1179 ± 137
CamCAN
653
653
54.2 ± 18.5
49.35%
1156 ± 128
IXI
581
581
48.65 ± 16.47
53.87%
1137 ± 131
NACC
6956
4926
71.41 ± 11.25
53.34%
1034 ± 185
ADNI
6471
1817
74.68 ± 7.64
47.45%
1072 ± 116
Appendix
Table 2: Dataset distribution in BrainATLAS. We include a total of 18 distinct neuroimaging dataset from diverse populations.
Workflow
Explanation
Time (mean ± std, hours)
-autorecon1
steps 1-5
Initial processing of the MRI data
0.36 ± 0.06
-autorecon2
steps 6-23
Reconstruction of the brain’s cortical surface and refinement of subcortical structures
3.78 ± 0.76
-autorecon3
steps 24-31
Refinement of segmentation and cortical surfaces
1.97 ± 0.39
Appendix
Table 3: FreeSurfer main workflow explanation and processing time
Category
Feature Name
Volume
Right-VentralDC_Volume_mm3
Volume
CC_Posterior_Volume_mm3
Volume
3rd-Ventricle_Volume_mm3
Volume
inferiortemporal_GrayVol
Volume
lateraloccipital_GrayVol
Volume
lateralorbitofrontal_GrayVol
Appendix
Table 4: Brain anatomical features categorized by Volume, Surface Area, and Cortex Thickness.
Region
Number or MRIs
Percentage Brain Pixels
Label
2
66709
19.67
Left-Cerebral-White-Matter
4
66709
1.22
Left-Lateral-Ventricle
5
66706
0.04
Left-Inf-Lat-Vent
7
66709
1.29
Left-Cerebellum-White-Matter
8
66709
4.64
Left-Cerebellum-Cortex
10
66709
0.62
Left-Thalamus-Proper
Appendix
Table 5: Brain region segmentation statistics across all labeled regions before pre-processing.
Normalized Label
Percentage Brain Pixels
Region
Accumbens-area
0.08
[26, 58]
Amygdala
0.28
[18, 54]
Brain-Stem
1.88
[16]
CC
0.33
[251, 252, 253, 254, 255]
Caudate
0.59
[11, 50]
Cerebellum-Cortex
9.43
[8, 47]
Appendix
Table 6: Grouped brain region segmentation statistics after pre-processing.
Dataset
Linear 10%
Non-Linear 10%
Linear 30%
Non-Linear 30%
Linear 100%
Non-Linear 100%
ADNI
0.6882 [0.6566, 0.7159]
0.7055 [0.6719, 0.7395]
0.7470 [0.7165, 0.7798]
0.7398 [0.7144, 0.7721]
0.7586 [0.7279, 0.7874]
0.7529 [0.7203, 0.7834]
NACC
0.7221 [0.6723, 0.7693]
0.7510 [0.7187, 0.7901]
0.7931 [0.7527, 0.8463]
0.8061 [0.7546, 0.8516]
0.7954 [0.7566, 0.8531]
0.8216 [0.7741, 0.8661]
AIBL
0.7111 [0.6710, 0.7526]
0.7582 [0.7162, 0.8010]
0.8004 [0.7576, 0.8468]
0.7933 [0.7421, 0.8424]
0.8105 [0.7682, 0.8522]
0.8261 [0.7847, 0.8651]
OASIS1
0.8328 [0.7867, 0.8751]
0.7967 [0.7421, 0.8356]
0.8168 [0.7673, 0.8538]
0.8150 [0.7637, 0.8671]
0.8066 [0.7590, 0.8486]
0.8245 [0.7706, 0.8772]
OASIS2
0.6796 [0.6234, 0.7394]
0.6825 [0.6219, 0.7313]
0.7085 [0.6393, 0.7522]
0.7038 [0.6429, 0.7598]
0.7378 [0.6778, 0.7867]
0.7710 [0.7180, 0.8173]
OASIS3
0.6621 [0.5885, 0.7348]
0.6847 [0.6018, 0.7619]
0.7124 [0.6396, 0.7831]
0.7009 [0.6135, 0.7890]
0.7258 [0.6488, 0.7949]
0.7192 [0.6310, 0.7955]
Appendix
Table 7: Model comparison on dementia detection task with anatomical features. External cohort performance (95% CI) per dataset is also included.
Dataset
Label Encoding
AD vs CN
AD vs MCI
MCI vs CN
MACRO
ADNI
Soft
0.8873 [0.8551, 0.9201]
0.7445 [0.7099, 0.7742]
0.6926 [0.6532, 0.7389]
0.7586 [0.7279, 0.7874]
ADNI
Rigid
0.8442 [0.8165, 0.8762]
0.7184 [0.6887, 0.7412]
0.6565 [0.6092, 0.6997]
0.7397 [0.7088, 0.7703]
ADNI
Multi-class
0.7107 [0.6786, 0.7395]
0.6284 [0.5972, 0.6586]
0.5927 [0.5503, 0.6425]
0.6439 [0.6134, 0.6736]
NACC
Soft
0.9287 [0.8977, 0.9532]
0.8588 [0.8239, 0.8913]
0.7850 [0.7421, 0.8231]
0.8575 [0.8301, 0.8842]
NACC
Rigid
0.9157 [0.8853, 0.9419]
0.7870 [0.7538, 0.8140]
0.7085 [0.6719, 0.7385]
0.8037 [0.7696, 0.8323]
NACC
Multi-class
0.8215 [0.7913, 0.8563]
0.6824 [0.6499, 0.7102]
0.6596 [0.6172, 0.7014]
0.7212 [0.6941, 0.7541]
Appendix
Table 8: Comparison of label encoding methods with a linear model using brain anatomical features.
Model Type
Input size
AD vs CN
AD vs MCI
MCI vs CN
MACRO
ResNet 18
(128, 128, 128)
0.8674 [0.8328, 0.9011]
0.7268 [0.6885, 0.7734]
0.6852 [0.6449, 0.7286]
0.7598 [0.7277, 0.7889]
ResNet 18
(160, 160, 160)
0.8106 [0.7792, 0.8421]
0.6737 [0.6352, 0.7178]
0.6550 [0.6082, 0.7002]
0.7141 [0.6839, 0.7466]
ResNet 50
(128, 128, 128)
0.8382 [0.8014, 0.8671]
0.7029 [0.6711, 0.7401]
0.6638 [0.6229, 0.7073]
0.7350 [0.7012, 0.7653]
Appendix
Table 9: Comparison of model performance with different input sizes and model complexity. We adopt the best performing model as the baseline supervised model.
Figure 5: Lightweight region-conditioned 3D mask decoder used for ASP.
Task
Evaluation Setting
ASP (FF)
ASP (LoRA)
AD (Train: ADNI)
ADNI
0.7800
0.7815
NACC
0.8260
0.8264
AIBL
0.8318
0.8232
OASIS1
0.8733
0.8083
OASIS2
0.8092
0.8439
OASIS3
0.7454
0.7376
Appendix
Table 10: Comparison of ASP (FF/Full-Finetuning) and ASP (LoRA) AUROC performance across classification tasks.
Model
Internal: UK Biobank
External: NACC
Pearson r↑
MSE ↓
Abs. error ↓
Pearson r↑
MSE ↓
Abs. error ↓
ASP (LoRA)
0.9361
10.38
2.54
0.8589
29.26
4.19
ASP (Full-Finetuning)
0.9333
10.83
2.60
0.8464
32.06
4.32
No masked-autoencoder alternation
0.9291
11.49
2.67
0.8324
34.51
4.46
No alternation and no freezing
0.9220
12.59
2.78
0.7987
40.50
4.71
Masked autoencoder only
0.9159
13.61
2.89
0.8043
45.35
5.20
Appendix
Table 11: ASP training ablation evaluated on age estimation. Absolute error denotes mean absolute error in years.
Deep learning models have become essential tools in medical imaging, yet training them end-to-end for diverse tasks remains challenging due to limited labeled data and computational constraints. To address this, we introduce Brain2Vec, a foundation model pre-trained on 74,425 T1-weighted Magnetic Resonance Imaging (MRI) scans pooled from multiple studies. Brain2Vec can serve both as a weight initialization backbone for fine-tuning downstream tasks and as a standalone feature extractor. We evaluated Brain2Vec in three tasks: downstream predictive performance, convergence speed, and embedding quality. Using Brain2Vec's pretrained weights as initialization and fine-tuning on labeled data for a brain age prediction task, results demonstrate improved final accuracy and faster convergence compared to random initialization. Additionally, its frozen embeddings outperform anatomically defined volumetric features on four of five downstream classification tasks and perform competitively with a state-of-the-art foundation model pre-trained on substantially more data. Lastly, we show that Brain2Vec encodes information primarily from a spatially localized and neuroanatomically meaningful set of brain regions, and that manipulating a subset of latent features produces monotonic, region-specific effects on the reconstructed anatomy. Brain2Vec code and trained models are available as an open-source package on https://github.com/CBICA/Brain2Vec. Users can apply Brain2Vec locally by installing the package, or directly through the NiChart cloud platform https://cloud.neuroimagingchart.com, where Brain2Vec features can be derived without requiring local installation or specialized infrastructure.
Foundation models pretrained using self-supervised learning have transformed computer vision by learning transferable representations from large-scale unlabeled data. However, existing foundation models for neuroimaging remain limited by task-specific training, slice-based learning strategies, or relatively small pretraining datasets, restricting their generalizability across diverse brain MRI applications. In this work, we present BrainNext, a general-purpose self-supervised foundation model for volumetric brain MRI analysis. BrainNext combines masked autoencoder (MAE) pretraining with a native three-dimensional Bi-Directional xLSTM-UNet architecture to learn rich anatomical representations from 60,551 unlabeled brain MRI examinations spanning multiple MRI modalities. The pretrained model is subsequently adapted to downstream tasks through lightweight task-specific fine-tuning. We evaluate BrainNext on the Foundation Models for Medical Imaging (FOMO) 2025 Method Track, encompassing classification, segmentation, and brain-age estimation, where it achieved second place overall and ranked first in the meningioma segmentation task on the official FOMO 2025 challenge leaderboard, demonstrating strong transferability across heterogeneous neuroimaging tasks. These results highlight the potential of large-scale self-supervised pretraining to learn robust and transferable volumetric representations, establishing BrainNext as a scalable foundation model for diverse brain MRI applications.
Moona Mazher, Abdul Qayyum, Steven A. Niederer +1
UCL Hawkes Institute, Department of Computer Science, University College London, London, United Kingdom · National Heart & Lung Institute, Imperial College London, London, United Kingdom · Division of Cardiovascular Medicine, Stanford Cardiovascular Institute, Stanford University, California, USA
A key challenge in medical image analysis is the scarcity of large annotated datasets for specific populations and diseases. As deep learning models rely heavily on labeled data, effective transfer learning strategies are needed to reduce the dependence on manual annotations. Self-supervised learning has emerged as a promising approach for developing foundation models by enabling the learning of transferable feature representations from large-scale unlabeled medical imaging datasets. In this study, we investigate voxel-level brain age prediction as a domain-specific self-supervised pretext task and compare it with image inpainting, a widely used non-domain-specific alternative. We further propose a multitask self-supervised pretraining framework that jointly optimizes both objectives to learn complementary neuroimaging representations. The pretrained models are evaluated on three downstream magnetic resonance image segmentation tasks: multiple sclerosis lesion segmentation, ischemic stroke lesion segmentation, and cortical brain structure segmentation. Overall, the proposed multitask pretraining framework consistently outperformed the single-task pretrained models and training from scratch across most experimental settings, demonstrating the benefit of combining domain-specific and general self-supervised learning pretext tasks for the development of generalizable neuroimaging foundation models.\ Code Availability: The source code used in this study is publicly available at https://github.com/TasneemN/Combining-General-and-Domain-Specific-Pretext-Tasks-for-Brain-MR-Image-Segmentation/
Tasneem Nasser, Susanne Schmid, Roberto Souza +1
Biomedical Engineering University of Calgary Calgary AB Canada · Hotchkiss Brain Institute University of Calgary Calgary AB Canada · Electrical and Software Engineering University of Calgary Calgary AB Canada +1