Comparative Study of Anatomical and Learned Features in AI Models for Structural Brain MRI

Boyang Yu; Miquel Lopez Escoriza; Long Chen; Arjun V. Masurkar; Narges Razavian; Carlos Fernandez-Granda

cs.CVSep 6, 2026

Comparative Study of Anatomical and Learned Features in AI Models for Structural Brain MRI

Authors: Boyang Yu, Miquel Lopez Escoriza, Long Chen, Arjun V. Masurkar, Narges Razavian, Carlos Fernandez-Granda

Organizations: Center for Data Science, New York University, NYU, 60 Fifth Ave, New York, 10011, New York, United States. · Department of Neurology, NYU Grossman School of Medicine, 550 1st Ave, New York, 10016, New York, United States. · Courant Institute of Mathematical Sciences, NYU, 251 Mercer St, New York, 10012, New York, United States.

Abstract

In this work, we comprehensively evaluate three popular feature-extraction paradigms in AI-based neuroimaging modeling: (1) computation of anatomical surfaces and volumes, (2) supervised learning with convolutional neural networks (CNNs), and (3) unsupervised pretraining of vision transformer (ViT) foundation models, followed by supervised finetuning. Our study is based on 18 publicly available datasets containing 3D structural T1-weighted MRI scans from approximately 80,000 participants across seven distinct clinical tasks. We observe that a linear model based on anatomical features matches the diagnostic performance of complex nonlinear features learned by sophisticated AI frameworks, including foundation models trained on thousands of scans. Conversely, CNNs and pretrained ViTs learn features that implicitly capture relevant anatomical information, bypassing the need for explicit feature extraction. Building upon these insights, we propose Anatomy Segmentation Pretraining (ASP), a novel method to incorporate anatomical information during foundation-model pretraining, which outperforms existing models in biological age estimation.

Figures & tables

Figure 1: Overview of the study . (a) Cohorts used in the study. Cohorts used for pretraining, downstream training, internal testing, and external evaluation. Reported $n$ values denote MRI scans; all splits were performed at the participant level, such that all scans from the same participant were assigned to the same split. External cohorts were drawn from independent data sources. (b) AI models included in the study . These represent the main feature-extraction approaches for neuroimaging data: linear models based on anatomical features, convolutional neural networks (CNNs) and vision transformers (ViTs) pretrained on large general-population cohort. For the ViTs, in addition to standard pretraining based on masked autoencoding, we propose a novel scheme called Anatomical Segmentation Pretraining (ASP).

Dataset	Number of scans	Number of subjects	Age (years) (mean ± std)	Gender (female %)	Brain volume in mL (mean ± std)
UKBB	67294	62665	64.93 ± 7.69	52.81%	1156 ± 114
HCP	1350	1350	38.66 ± 25.76	54.97%	1179 ± 137
CamCAN	653	653	54.2 ± 18.5	49.35%	1156 ± 128
IXI	581	581	48.65 ± 16.47	53.87%	1137 ± 131
NACC	6956	4926	71.41 ± 11.25	53.34%	1034 ± 185
ADNI	6471	1817	74.68 ± 7.64	47.45%	1072 ± 116

Table 1: Datasets used to build the pretraining cohort, and the training and evaluation cohorts for each task in the study.

Figure 2: Performance comparison across diagnostic and segmentation tasks. Alzheimer’s disease diagnosis was evaluated across an internal ADNI test cohort and seven external cohorts (NACC, AIBL, MIRIAD, and OASIS 1-4), where ASP ViT achieved the highest macro-AUC in most cohorts, though statistical significance over the CNN and anatomical logistic regression was only reached for OASIS 2. In the diagnostics of Parkinson’s disease (PPMI) and Epilepsy (OpenNeuro), the Pretrained ViT reached the highest AUC, showing a statistically significant improvement only over BrainIAC. Estimated probabilities for Epilepsy from ASP ViT are also shown for EPISURG, an additional evaluation set with only epilepsy cases. For the identification of brain tumor IDH mutations (UCSF-PDGM) ASP ViT yielded the highest AUC, though these differences were not statistically significant. CNN could not be applied to the epilepsy task due to limited training data. In segmentation tasks, a U-Net achieved the highest Dice score for adult brain gliomas (BraTS 2023), while the Pretrained ViT performed best for stroke lesion segmentation (ATLAS R2.0); in both cases, the differences were only statistically significant relative to UNETR. Overall, no significant performance differences were observed between the pretrained ViTs, CNNs, and anatomical logistic regression models across the evaluated tasks.

Figure 3: Age estimation performance across internal and external cohorts. The ASP ViT model demonstrates superior performance, achieving a mean absolute error (MAE) of 2.54 years in the internal test cohort (a) and 4.19 years in both the NACC (b) and ADNI (c) external test cohorts. While the Pretrained ViT maintains competitive performance on the internal cohort with an MAE only 0.35 years greater than the ASP ViT, this performance gap widens to over 1 year in the external cohorts. Logistic regression underperforms relative to the ViT-based approaches by a margin of 1 to 2 years, though it consistently outperforms the CNN. These performance comparisons remain stable when the data is stratified by sex and age. Age-based stratification reveals more pronounced errors for individuals over 80, a demographic less represented in the training and pretraining datasets. When the best-performing model (ASP ViT) is applied to clinical populations, age estimates for dementia patients from the NACC (b) and ADNI (c) datasets are consistently higher than their biological age across most age intervals.

Figure 4: a. Prompt-based segmentation architecture . The input consists of an image and a brain region index, which is used to retrieve the corresponding brain region embedding. This embedding is added to the image representation, and the combined features are passed through a lightweight decoder to predict the segmentation mask for the specified region. b. Linear‑probe accuracy/ $R^{2}$ on six target variables—two global attributes (age, gender) and four region‑specific volumetric measures—tracked over pre‑training epochs. The differences in overall accuracy across regions arise because some regions are inherently easier to predict than others. For example, cortex volume tends to be more accurately segmented due to its larger size and higher contrast, compared to smaller and less distinct regions like the hippocampus. For demographic statistics the same happens. Gender is easier to infer than age since it is a binary attribute, while age requires finer-grained regression. c–d. Validation IoU by Brain Region. Regional performance varies based on anatomical complexity; for instance, the superior frontal cortex’s small, irregular geometry results in lower IoU, whereas the smoother, more consistent morphology of the pallidum allows for higher segmentation accuracy. e. ASP training ablation. Age-estimation performance for different ASP training variants on the external NACC cohort. The full ASP model achieves the lowest error, while removing alternating MAE updates, removing encoder freezing, using MAE alone, or training from scratch reduces performance. Detailed results are provided in Appendix Section 3.3 .

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	# images	# subjects	Age in yrs (mean ± std)	Gender (female %)	Brain volume in mL (mean ± std)
UKBB	67294	62665	64.93 ± 7.69	52.81%	1156 ± 114
HCP	1350	1350	38.66 ± 25.76	54.97%	1179 ± 137
CamCAN	653	653	54.2 ± 18.5	49.35%	1156 ± 128
IXI	581	581	48.65 ± 16.47	53.87%	1137 ± 131
NACC	6956	4926	71.41 ± 11.25	53.34%	1034 ± 185
ADNI	6471	1817	74.68 ± 7.64	47.45%	1072 ± 116

Appendix

Table 2: Dataset distribution in BrainATLAS. We include a total of 18 distinct neuroimaging dataset from diverse populations.

	Workflow	Explanation	Time (mean ± std, hours)
-autorecon1	steps 1-5	Initial processing of the MRI data	0.36 ± 0.06
-autorecon2	steps 6-23	Reconstruction of the brain’s cortical surface and refinement of subcortical structures	3.78 ± 0.76
-autorecon3	steps 24-31	Refinement of segmentation and cortical surfaces	1.97 ± 0.39

Appendix

Table 3: FreeSurfer main workflow explanation and processing time

Category	Feature Name
Volume	Right-VentralDC_Volume_mm3
Volume	CC_Posterior_Volume_mm3
Volume	3rd-Ventricle_Volume_mm3
Volume	inferiortemporal_GrayVol
Volume	lateraloccipital_GrayVol
Volume	lateralorbitofrontal_GrayVol

Appendix

Table 4: Brain anatomical features categorized by Volume, Surface Area, and Cortex Thickness.

Region	Number or MRIs	Percentage Brain Pixels	Label
2	66709	19.67	Left-Cerebral-White-Matter
4	66709	1.22	Left-Lateral-Ventricle
5	66706	0.04	Left-Inf-Lat-Vent
7	66709	1.29	Left-Cerebellum-White-Matter
8	66709	4.64	Left-Cerebellum-Cortex
10	66709	0.62	Left-Thalamus-Proper

Appendix

Table 5: Brain region segmentation statistics across all labeled regions before pre-processing.

Normalized Label	Percentage Brain Pixels	Region
Accumbens-area	0.08	[26, 58]
Amygdala	0.28	[18, 54]
Brain-Stem	1.88	[16]
CC	0.33	[251, 252, 253, 254, 255]
Caudate	0.59	[11, 50]
Cerebellum-Cortex	9.43	[8, 47]

Appendix

Table 6: Grouped brain region segmentation statistics after pre-processing.

Dataset	Linear 10%	Non-Linear 10%	Linear 30%	Non-Linear 30%	Linear 100%	Non-Linear 100%
ADNI	0.6882 [0.6566, 0.7159]	0.7055 [0.6719, 0.7395]	0.7470 [0.7165, 0.7798]	0.7398 [0.7144, 0.7721]	0.7586 [0.7279, 0.7874]	0.7529 [0.7203, 0.7834]
NACC	0.7221 [0.6723, 0.7693]	0.7510 [0.7187, 0.7901]	0.7931 [0.7527, 0.8463]	0.8061 [0.7546, 0.8516]	0.7954 [0.7566, 0.8531]	0.8216 [0.7741, 0.8661]
AIBL	0.7111 [0.6710, 0.7526]	0.7582 [0.7162, 0.8010]	0.8004 [0.7576, 0.8468]	0.7933 [0.7421, 0.8424]	0.8105 [0.7682, 0.8522]	0.8261 [0.7847, 0.8651]
OASIS1	0.8328 [0.7867, 0.8751]	0.7967 [0.7421, 0.8356]	0.8168 [0.7673, 0.8538]	0.8150 [0.7637, 0.8671]	0.8066 [0.7590, 0.8486]	0.8245 [0.7706, 0.8772]
OASIS2	0.6796 [0.6234, 0.7394]	0.6825 [0.6219, 0.7313]	0.7085 [0.6393, 0.7522]	0.7038 [0.6429, 0.7598]	0.7378 [0.6778, 0.7867]	0.7710 [0.7180, 0.8173]
OASIS3	0.6621 [0.5885, 0.7348]	0.6847 [0.6018, 0.7619]	0.7124 [0.6396, 0.7831]	0.7009 [0.6135, 0.7890]	0.7258 [0.6488, 0.7949]	0.7192 [0.6310, 0.7955]

Appendix

Table 7: Model comparison on dementia detection task with anatomical features. External cohort performance (95% CI) per dataset is also included.

Dataset	Label Encoding	AD vs CN	AD vs MCI	MCI vs CN	MACRO
ADNI	Soft	0.8873 [0.8551, 0.9201]	0.7445 [0.7099, 0.7742]	0.6926 [0.6532, 0.7389]	0.7586 [0.7279, 0.7874]
ADNI	Rigid	0.8442 [0.8165, 0.8762]	0.7184 [0.6887, 0.7412]	0.6565 [0.6092, 0.6997]	0.7397 [0.7088, 0.7703]
ADNI	Multi-class	0.7107 [0.6786, 0.7395]	0.6284 [0.5972, 0.6586]	0.5927 [0.5503, 0.6425]	0.6439 [0.6134, 0.6736]
NACC	Soft	0.9287 [0.8977, 0.9532]	0.8588 [0.8239, 0.8913]	0.7850 [0.7421, 0.8231]	0.8575 [0.8301, 0.8842]
NACC	Rigid	0.9157 [0.8853, 0.9419]	0.7870 [0.7538, 0.8140]	0.7085 [0.6719, 0.7385]	0.8037 [0.7696, 0.8323]
NACC	Multi-class	0.8215 [0.7913, 0.8563]	0.6824 [0.6499, 0.7102]	0.6596 [0.6172, 0.7014]	0.7212 [0.6941, 0.7541]

Appendix

Table 8: Comparison of label encoding methods with a linear model using brain anatomical features.

Model Type	Input size	AD vs CN	AD vs MCI	MCI vs CN	MACRO
ResNet 18	(128, 128, 128)	0.8674 [0.8328, 0.9011]	0.7268 [0.6885, 0.7734]	0.6852 [0.6449, 0.7286]	0.7598 [0.7277, 0.7889]
ResNet 18	(160, 160, 160)	0.8106 [0.7792, 0.8421]	0.6737 [0.6352, 0.7178]	0.6550 [0.6082, 0.7002]	0.7141 [0.6839, 0.7466]
ResNet 50	(128, 128, 128)	0.8382 [0.8014, 0.8671]	0.7029 [0.6711, 0.7401]	0.6638 [0.6229, 0.7073]	0.7350 [0.7012, 0.7653]

Appendix

Table 9: Comparison of model performance with different input sizes and model complexity. We adopt the best performing model as the baseline supervised model.

Figure 5: Lightweight region-conditioned 3D mask decoder used for ASP.

Task	Evaluation Setting	ASP (FF)	ASP (LoRA)
AD (Train: ADNI)	ADNI	0.7800	0.7815
	NACC	0.8260	0.8264
	AIBL	0.8318	0.8232
	OASIS1	0.8733	0.8083
	OASIS2	0.8092	0.8439
	OASIS3	0.7454	0.7376

Appendix

Table 10: Comparison of ASP (FF/Full-Finetuning) and ASP (LoRA) AUROC performance across classification tasks.

Model	Internal: UK Biobank			External: NACC
Model	Pearson $r\uparrow$	MSE $\downarrow$	Abs. error $\downarrow$	Pearson $r\uparrow$	MSE $\downarrow$	Abs. error $\downarrow$
ASP (LoRA)	0.9361	10.38	2.54	0.8589	29.26	4.19
ASP (Full-Finetuning)	0.9333	10.83	2.60	0.8464	32.06	4.32
No masked-autoencoder alternation	0.9291	11.49	2.67	0.8324	34.51	4.46
No alternation and no freezing	0.9220	12.59	2.78	0.7987	40.50	4.71
Masked autoencoder only	0.9159	13.61	2.89	0.8043	45.35	5.20

Appendix

Table 11: ASP training ablation evaluated on age estimation. Absolute error denotes mean absolute error in years.

Neuroimaging

Neuroimaging uses brain measurements and computational methods to map anatomy, activity, connectivity, and disease.

See all 235 papers

Brain2Vec: A Self-Supervised 3D Foundation Model for Structural Brain MRI
Oct 20, 2026Spiros Maggioros, Guray Erus, Gareth Harman +4Medical Imaging Foundation Models Medical Image Representation Learning
University of Pennsylvania
BrainNext: A General-Purpose Self-Supervised Foundation Model for Brain MRI Analysis
Jul 20, 2026Moona Mazher, Abdul Qayyum, Steven A. Niederer +1Medical Imaging Foundation Models Self-Supervised Pre-Training
UCL Hawkes Institute, Department of Computer Science, University College London, London, United Kingdom · National Heart & Lung Institute, Imperial College London, London, United Kingdom · Division of Cardiovascular Medicine, Stanford Cardiovascular Institute, Stanford University, California, USA
Combining General and Domain-Specific Pretext Tasks for Brain MR Image Segmentation
Sep 25, 2026Tasneem Nasser, Susanne Schmid, Roberto Souza +1Medical Image Representation Learning Self-Supervised Pre-Training
Biomedical Engineering University of Calgary Calgary AB Canada · Hotchkiss Brain Institute University of Calgary Calgary AB Canada · Electrical and Software Engineering University of Calgary Calgary AB Canada +1

Oct 20, 2026J/K move · Enter open · S save

Brain2Vec: A Self-Supervised 3D Foundation Model for Structural Brain MRI

Spiros Maggioros, Guray Erus, Gareth Harman, George Aidinis+3

University of Pennsylvania

Medical Imaging Foundation Models Medical Image Representation Learning Neuroimaging Self-Supervised Pre-Training

Open paper AskPDF arXiv

Deep learning models have become essential tools in medical imaging, yet training them end-to-end for diverse tasks remains challenging due to limited labeled data and computational constraints. To address this, we introduce Brain2Vec, a foundation model pre-trained on 74,425 T1-weighted Magnetic Resonance Imaging (MRI) scans pooled from multiple studies. Brain2Vec can serve both as a weight initialization backbone for fine-tuning downstream tasks and as a standalone feature extractor. We evaluated Brain2Vec in three tasks: downstream predictive performance, convergence speed, and embedding quality. Using Brain2Vec's pretrained weights as initialization and fine-tuning on labeled data for a brain age prediction task, results demonstrate improved final accuracy and faster convergence compared to random initialization. Additionally, its frozen embeddings outperform anatomically defined volumetric features on four of five downstream classification tasks and perform competitively with a state-of-the-art foundation model pre-trained on substantially more data. Lastly, we show that Brain2Vec encodes information primarily from a spatially localized and neuroanatomically meaningful set of brain regions, and that manipulating a subset of latent features produces monotonic, region-specific effects on the reconstructed anatomy. Brain2Vec code and trained models are available as an open-source package on https://github.com/CBICA/Brain2Vec. Users can apply Brain2Vec locally by installing the package, or directly through the NiChart cloud platform https://cloud.neuroimagingchart.com, where Brain2Vec features can be derived without requiring local installation or specialized infrastructure.

Abstract

Neuroimaging

Explore similar work

Brain2Vec: A Self-Supervised 3D Foundation Model for Structural Brain MRI

BrainNext: A General-Purpose Self-Supervised Foundation Model for Brain MRI Analysis

Combining General and Domain-Specific Pretext Tasks for Brain MR Image Segmentation