Brain2Vec: A Self-Supervised 3D Foundation Model for Structural Brain MRI

Spiros Maggioros; Guray Erus; Gareth Harman; George Aidinis; Alexander Getka; Aristeidis Sotiras; Christos Davatzikos

cs.CVOct 20, 2026

Brain2Vec: A Self-Supervised 3D Foundation Model for Structural Brain MRI

Authors: Spiros Maggioros, Guray Erus, Gareth Harman, George Aidinis, Alexander Getka, Aristeidis Sotiras, Christos Davatzikos

Organizations: University of Pennsylvania

Abstract

Deep learning models have become essential tools in medical imaging, yet training them end-to-end for diverse tasks remains challenging due to limited labeled data and computational constraints. To address this, we introduce Brain2Vec, a foundation model pre-trained on 74,425 T1-weighted Magnetic Resonance Imaging (MRI) scans pooled from multiple studies. Brain2Vec can serve both as a weight initialization backbone for fine-tuning downstream tasks and as a standalone feature extractor. We evaluated Brain2Vec in three tasks: downstream predictive performance, convergence speed, and embedding quality. Using Brain2Vec's pretrained weights as initialization and fine-tuning on labeled data for a brain age prediction task, results demonstrate improved final accuracy and faster convergence compared to random initialization. Additionally, its frozen embeddings outperform anatomically defined volumetric features on four of five downstream classification tasks and perform competitively with a state-of-the-art foundation model pre-trained on substantially more data. Lastly, we show that Brain2Vec encodes information primarily from a spatially localized and neuroanatomically meaningful set of brain regions, and that manipulating a subset of latent features produces monotonic, region-specific effects on the reconstructed anatomy. Brain2Vec code and trained models are available as an open-source package on https://github.com/CBICA/Brain2Vec. Users can apply Brain2Vec locally by installing the package, or directly through the NiChart cloud platform https://cloud.neuroimagingchart.com, where Brain2Vec features can be derived without requiring local installation or specialized infrastructure.

Figures & tables

Training Set				Validation Set
Study	n	Mean Age $\pm$ Std	(%) F	Study	n	Mean Age $\pm$ Std	(%) F
UKBIOBANK	32,719	$64.15\pm 7.53$	52.87	UKBIOBANK	4,104	$64.26\pm 7.53$	51.32
PENN	694	$73.10\pm 9.45$	57.49	PENN	95	$73.34\pm 9.73$	55.79
OASIS	1,925	$72.40\pm 10.57$	56.68	OASIS	250	$72.77\pm 10.65$	57.20
ADNI	8,672	$75.14\pm 7.38$	45.18	ADNI	1,074	$75.14\pm 7.41$	45.72
AIBL	1,499	$74.21\pm 6.83$	54.30	AIBL	210	$74.57\pm 7.00$	53.33

Table 1: Overview of datasets used for Brain2Vec pre-training, split by training and validation sets.

Figure 1: Overview of the Brain2Vec pre-training architecture and downstream applications. (Pre-training) Brain2Vec is a 3D Masked Autoencoder (MAE) pre-trained on 74,425 T1-weighted MRI scans from the iSTAGING consortium. Input volumes are divided into

8\times 8\times 8

patches, with 75% randomly masked. A 12-layer Vision Transformer (ViT) encoder processes only the visible patches to generate a 768-dimensional latent representation. Mask tokens are then introduced, and an 8-layer ViT decoder reconstructs the original scan. (Evaluation) The pre-trained encoder serves two complementary roles. First, it provides a robust weight initialization for end-to-end fine-tuning on a downstream task (brain age regression), accelerating convergence and improving accuracy relative to random initialization. Second, its frozen 768-dimensional embeddings serve as fixed features for lightweight downstream models, including a Support Vector Regressor (SVR) for brain age regression and SVMs for five binary classification tasks: CU vs. AD, elevated BMI, diabetes, smoking status, and hypertension. We additionally assess the quality and interpretability of these frozen embeddings via self-attention visualization, correlation with anatomically-defined Regions of Interest (ROIs), and latent-space manipulation, showing that Brain2Vec encodes spatially localized, neuroanatomically meaningful representations.

Panel of Figure 2 — Figure 2: Brain2Vec reconstruction results (75% random masking) for a representative subject from each cognitive group: (a) CU (Cognitively Unimpaired), (b) MCI (Mild Cognitive Impairment), and (c) AD (Alzheimer’s Disease). Within each panel, columns show the masked input, the Brain2Vec reconstruction (Recon.), and the ground truth (GT); rows show sagittal, coronal, and axial views.

Figure 3: ViT-BA validation metrics per epoch.

Dataset	Model	Pre-training	MAE	MSE	RMSE	$R^{2}$
Test	ViT-BA	No	4.0733	28.4987	5.3404	0.7275
Test	ViT-BA	Yes	3.5573	21.9879	4.6856	0.7902
IXI	ViT-BA	No	$6.1846$	$63.9272$	$7.9955$	$0.4306$
IXI	ViT-BA	Yes	$\bm{5.1667}$	$\bm{44.3969}$	$\bm{6.6631}$	$\bm{0.6045}$

Table 2: Brain age regression results with and without Brain2Vec pre-training on the held-out test set and the external IXI dataset.

Panel of Figure 4 — Figure 4: ViT-BA prediction plots on the held-out test set.

Panel of Figure 5 — Figure 5: ViT-BA and Brain2Vec-SVR prediction plots for IXI.

IXI Inference
Model	MAE	MSE	RMSE	$R^{2}$
SPARE-BA	$4.3695\pm 0.049$	$33.0857\pm 0.668$	$5.7517\pm 0.058$	$0.6798\pm 0.012$
NeuroJEPA-SVR	$3.4387\pm 0.070$	$\bm{19.6002\pm 0.787}$	$\bm{4.4263\pm 0.089}$	$0.8105\pm 0.008$
Brain2Vec-SVR	$\bm{3.4349\pm 0.017}$	$19.6184\pm 0.127$	$4.4292\pm 0.014$	$\bm{0.8126\pm 0.001}$
SPARE-BA	$7.8869$	$93.0798$	$9.6478$	$0.1709$
NeuroJEPA-SVR	$5.1895$	$40.9347$	$6.3980$	$0.6354$

Table 3: SVR brain age regression testing results.

Task	Method	Accuracy	AUC	F1	Recall	Precision
CU vs. AD	DLMUSE	$\bm{0.8325\pm 0.017}$	$\bm{0.9022\pm 0.014}$	$\bm{0.8323\pm 0.017}$	$\bm{0.8604\pm 0.029}$	$0.8154\pm 0.019$
	NeuroJEPA	$0.8040\pm 0.018$	$0.8778\pm 0.009$	$0.8036\pm 0.018$	$0.7702\pm 0.045$	$\bm{0.8269\pm 0.021}$
	Brain2Vec	$0.7845\pm 0.023$	$0.8586\pm 0.022$	$0.7844\pm 0.022$	$0.7725\pm 0.021$	$0.7928\pm 0.035$
CU vs. Elevated BMI	DLMUSE	$0.6065\pm 0.023$	$0.6453\pm 0.036$	$0.6058\pm 0.024$	$0.6317\pm 0.023$	$0.6041\pm 0.025$
	NeuroJEPA	$\bm{0.7884\pm 0.016}$	$\bm{0.8672\pm 0.013}$	$\bm{0.7884\pm 0.016}$	$\bm{0.7920\pm 0.017}$	$\bm{0.7865\pm 0.019}$
	Brain2Vec	$0.7756\pm 0.010$	$0.8505\pm 0.014$	$0.7756\pm 0.010$	$0.7636\pm 0.013$	$0.7825\pm 0.011$

Table 4: SVM disease classification testing results (mean

\pm

std across 5 folds).

Model	Accuracy	AUC	F1	Recall	Precision
DLMUSE	$\bm{0.9353\pm 0.012}$	$\bm{0.9776\pm 0.005}$	$\bm{0.9289\pm 0.013}$	$\bm{0.9333\pm 0.014}$	$0.9679\pm 0.011$
NeuroJEPA	$0.8305\pm 0.045$	$0.9708\pm 0.004$	$0.8243\pm 0.045$	$0.7550\pm 0.060$	$\bm{0.9929\pm 0.014}$
Brain2Vec	$0.8407\pm 0.017$	$0.9392\pm 0.011$	$0.8051\pm 0.028$	$0.9300\pm 0.019$	$0.8509\pm 0.030$

Table 5: CU vs AD inference on MIRIAD.

Figure 6: Self-attention map of a participant with Alzheimer’s Disease derived using the Vision Transformer (ViT) encoder of Brain2Vec. The attention map is computed using the [CLS] token on the heads of the encoder’s last ViT layer. Additional examples for all the heads are presented in supplementary Section A.6 .

Panel of Figure 7 — Figure 7: Correlation analysis of Brain2Vec latent embeddings with brain regions.

Model	FLOPs	#Params	GPU (A40)	CPU
DLMUSE	965.3B	16.6M	43.41 (min.) - $n=581$	6.39 (min.) - $n=1$
NeuroJEPA	172.9B	122.0M	3.62 (min.) - $n=581$	6.98 (sec.) - $n=1$
Brain2Vec	177.3B	88.6M	30.03 (sec.) - $n=581$	45.73 (sec.) - $n=1$

Table 6: Brain2Vec inference performance.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Training Set				Validation Set
Study	n	Mean Age $\pm$ Std	(%) F	Study	n	Mean Age $\pm$ Std	(%) F
UKBIOBANK	10,240	$66.23\pm 7.14$	45.39	UKBIOBANK	5,141	$66.25\pm 7.14$	45.75
HCP	465	$42.03\pm 18.89$	54.19	HCP	204	$41.83\pm 19.46$	58.33
AIBL	183	$72.92\pm 5.92$	55.74	AIBL	90	$73.03\pm 5.62$	60.00
BLSA	158	$56.71\pm 17.72$	65.19	BLSA	76	$54.95\pm 17.63$	63.16
SHIP	149	$40.24\pm 13.37$	42.95	SHIP	88	$38.95\pm 12.38$	34.09

Appendix

Table 7: Characteristics of the training and validation datasets for brain age prediction.

CU				AD
Study	n	Mean Age $\pm$ Std	(%) F	Study	n	Mean Age $\pm$ Std	(%) F
ADNI	419	$74.68\pm 7.12$	45.11	ADNI	418	$74.83\pm 7.72$	44.26
OASIS	313	$75.04\pm 7.33$	52.40	OASIS	313	$75.49\pm 7.49$	49.52
AIBL	92	$74.16\pm 7.72$	59.78	AIBL	92	$74.23\pm 7.92$	59.78
BLSA	9	$81.33\pm 8.76$	33.33	BLSA	9	$81.44\pm 8.91$	33.33
PENN	7	$70.01\pm 7.45$	42.86	PENN	7	$70.81\pm 6.96$	71.43

Appendix

Table 8: Characteristics of the training and validation datasets for CU vs. AD classification.

CU				Diabetes
Study	n	Mean Age $\pm$ Std	(%) F	Study	n	Mean Age $\pm$ Std	(%) F
UKBIOBANK	2,051	$66.12\pm 7.16$	36.03	UKBIOBANK	2,051	$66.14\pm 7.19$	36.03
SHIP	371	$59.84\pm 11.13$	47.44	SHIP	371	$60.13\pm 11.27$	42.05
MESA	169	$74.59\pm 7.45$	46.15	MESA	169	$74.72\pm 7.52$	46.15
BLSA	107	$68.47\pm 9.96$	55.14	BLSA	107	$68.41\pm 9.93$	55.14
CARDIA	108	$51.66\pm 3.75$	56.48	CARDIA	108	$51.74\pm 3.88$	56.48

Appendix

Table 9: Characteristics of the training and validation datasets for CU vs. Diabetes classification.

CU				BMI
Study	n	Mean Age $\pm$ Std	(%) F	Study	n	Mean Age $\pm$ Std	(%) F
UKBIOBANK	1,702	$62.17\pm 7.17$	61.69	UKBIOBANK	1,702	$62.16\pm 7.18$	61.69
CARDIA	132	$50.22\pm 3.94$	72.73	CARDIA	132	$50.16\pm 3.86$	72.73
HCP	122	$37.25\pm 13.59$	68.03	HCP	122	$37.27\pm 13.59$	68.03
SHIP	127	$56.00\pm 12.29$	63.78	SHIP	127	$56.00\pm 12.29$	63.78
MESA	106	$70.70\pm 5.76$	68.87	MESA	106	$70.61\pm 5.94$	68.87

Appendix

Table 10: Characteristics of the training and validation datasets for CU vs. Elevated BMI classification.

Non-smoker				Smoker
Study	n	Mean Age $\pm$ Std	(%) F	Study	n	Mean Age $\pm$ Std	(%) F
UKBIOBANK	1,051	$61.52\pm 7.57$	45.58	UKBIOBANK	1,051	$61.55\pm 7.56$	45.58
SHIP	680	$52.71\pm 13.72$	42.35	SHIP	680	$52.46\pm 13.67$	61.62
HCP	180	$29.41\pm 3.34$	43.89	HCP	180	$29.27\pm 3.22$	43.89
CARDIA	144	$50.54\pm 4.16$	48.61	CARDIA	144	$50.31\pm 3.80$	48.61
ADNI	95	$74.97\pm 4.69$	43.16	ADNI	95	$75.00\pm 4.59$	43.16

Appendix

Table 11: Characteristics of the training and validation datasets for smoker vs. non-smoker classification.

CU				Hypertension
Study	n	Mean Age $\pm$ Std	(%) F	Study	n	Mean Age $\pm$ Std	(%) F
UKBIOBANK	1,484	$61.85\pm 7.13$	58.22	UKBIOBANK	1,484	$61.95\pm 7.33$	64.35
SHIP	82	$51.96\pm 11.53$	53.66	SHIP	82	$50.00\pm 11.45$	71.95
PreventAD	9	$66.87\pm 6.47$	66.67	PreventAD	9	$63.98\pm 4.29$	100.00
MESA	46	$70.30\pm 6.40$	47.83	MESA	46	$71.35\pm 8.35$	52.17
HANDLS	12	$50.52\pm 9.33$	50.00	HANDLS	12	$51.02\pm 8.37$	83.33

Appendix

Table 12: Characteristics of the training and validation datasets for CU vs. Hypertension classification.

Panel of Figure 8 — Figure 8: Brain age gap differences across disease and risk-factor groups.

CU vs AD
Study	n	Accuracy	AUC	F1	Recall	Precision
ADNI	837	$0.8485\pm 0.005$	$0.9266\pm 0.008$	$0.8463\pm 0.005$	$0.8349\pm 0.006$	$0.8580\pm 0.007$
OASIS	626	$0.7843\pm 0.015$	$0.8589\pm 0.015$	$0.7833\pm 0.017$	$0.7802\pm 0.023$	$0.7867\pm 0.014$
AIBL	184	$0.8576\pm 0.004$	$0.9260\pm 0.009$	$0.8492\pm 0.006$	$0.8022\pm 0.016$	$0.9025\pm 0.011$
CU vs Elevated BMI
UKBIOBANK	3404	$0.8383\pm 0.005$	$0.9130\pm 0.005$	$0.8372\pm 0.006$	$0.8316\pm 0.009$	$0.8429\pm 0.003$

Appendix

Table 13: Disease classification performance by study using Brain2Vec embeddings as input features.

Panel of Figure 9 — Figure 9: Principal component analysis of Brain2Vec embeddings for samples in different disease and risk-factor groups, colored by acquisition site.

Brain2Vec dim.	Mean difference	Cohen’s $d$	DLMUSE ROI	Pearson $r$
497	$2.90\times 10^{-5}$	$0.8365$	509 (Ventricles)	$0.6624$
110	$-2.87\times 10^{-2}$	$-0.7359$	509 (Ventricles)	$-0.5890$
327	$2.74\times 10^{-2}$	$0.7299$	509 (Ventricles)	$0.4885$
175	$-1.63\times 10^{-2}$	$-0.7027$	509 (Ventricles)	$-0.5720$
693	$-9.09\times 10^{-3}$	$-0.6911$	509 (Ventricles)	$-0.5170$
691	$-1.95\times 10^{-2}$	$-0.6884$	509 (Ventricles)	$-0.4771$

Appendix

Table 14: Brain2Vec dimensions selected for latent manipulation and their best-matching DLMUSE ROIs

Figure 10: Self-attention map of a cognitively unimpaired patient

Figure 11: Self attention map of a participant with mild cognitive impairment

Figure 12: Self-attention map of a participant with Alzheimer’s disease

Figure 13: Effect on ventricles: we observe a monotonic increase of ventricle regions which is something clinically expected, as ventricular enlargement is a well-established radiological marker of neurodegeneration and disease progression in Alzheimer’s disease [ 30 ] .

Figure 14: Effect on frontal regions: we observe a monotonic decrease of frontal regions which is consistent with well-documented patterns of medial and lateral frontal lobe atrophy in AD patients [ 38 , 32 ] .

Figure 15: Effects on GM: we observe a similar trend on frontal regions as well as decreased GM and WM volumes, accompanied by an increase in cortical CSF. This pattern is consistent with enlargement of CSF spaces as brain tissue is lost, and is validated by prior volumetric findings in Alzheimer’s disease [ 18 ] .

Panel of Figure 16 — Figure 16: Example of ventricles region masking and reconstruction with Brain2Vec.

Panel of Figure 17 — Figure 17: Example of frontal region masking and reconstruction with Brain2Vec.

Panel of Figure 18 — Figure 18: Example of gray matter region masking and reconstruction with Brain2Vec.

Medical Imaging Foundation Models

Medical imaging foundation models learn reusable representations for tasks across clinical images, modalities, and institutions.

See all 234 papers

Jul 20, 2026

BrainNext: A General-Purpose Self-Supervised Foundation Model for Brain MRI Analysis

Foundation models pretrained using self-supervised learning have transformed computer vision by learning transferable representations from large-scale unlabeled data. However, existing foundation models for neuroimaging remain limited by task-specific training, slice-based learning strategies, or relatively small pretraining datasets, restricting their generalizability across diverse brain MRI applications. In this work, we present BrainNext, a general-purpose self-supervised foundation model for volumetric brain MRI analysis. BrainNext combines masked autoencoder (MAE) pretraining with a native three-dimensional Bi-Directional xLSTM-UNet architecture to learn rich anatomical representations from 60,551 unlabeled brain MRI examinations spanning multiple MRI modalities. The pretrained model is subsequently adapted to downstream tasks through lightweight task-specific fine-tuning. We evaluate BrainNext on the Foundation Models for Medical Imaging (FOMO) 2025 Method Track, encompassing classification, segmentation, and brain-age estimation, where it achieved second place overall and ranked first in the meningioma segmentation task on the official FOMO 2025 challenge leaderboard, demonstrating strong transferability across heterogeneous neuroimaging tasks. These results highlight the potential of large-scale self-supervised pretraining to learn robust and transferable volumetric representations, establishing BrainNext as a scalable foundation model for diverse brain MRI applications.

Moona Mazher, Abdul Qayyum, Steven A. Niederer +1

UCL Hawkes Institute, Department of Computer Science, University College London, London, United Kingdom · National Heart & Lung Institute, Imperial College London, London, United Kingdom · Division of Cardiovascular Medicine, Stanford Cardiovascular Institute, Stanford University, California, USA

Medical Imaging Foundation Models Self-Supervised Pre-Training Medical Image Representation Learning

Ask PDF

Sep 6, 2026

Comparative Study of Anatomical and Learned Features in AI Models for Structural Brain MRI

In this work, we comprehensively evaluate three popular feature-extraction paradigms in AI-based neuroimaging modeling: (1) computation of anatomical surfaces and volumes, (2) supervised learning with convolutional neural networks (CNNs), and (3) unsupervised pretraining of vision transformer (ViT) foundation models, followed by supervised finetuning. Our study is based on 18 publicly available datasets containing 3D structural T1-weighted MRI scans from approximately 80,000 participants across seven distinct clinical tasks. We observe that a linear model based on anatomical features matches the diagnostic performance of complex nonlinear features learned by sophisticated AI frameworks, including foundation models trained on thousands of scans. Conversely, CNNs and pretrained ViTs learn features that implicitly capture relevant anatomical information, bypassing the need for explicit feature extraction. Building upon these insights, we propose Anatomy Segmentation Pretraining (ASP), a novel method to incorporate anatomical information during foundation-model pretraining, which outperforms existing models in biological age estimation.

Boyang Yu, Miquel Lopez Escoriza, Long Chen +3

Center for Data Science, New York University, NYU, 60 Fifth Ave, New York, 10011, New York, United States. · Department of Neurology, NYU Grossman School of Medicine, 550 1st Ave, New York, 10016, New York, United States. · Courant Institute of Mathematical Sciences, NYU, 251 Mercer St, New York, 10012, New York, United States.

Neuroimaging Medical Imaging Foundation Models Brain MRI

Ask PDF

Apr 30, 2026

BrainDINO: A Brain MRI Foundation Model for Generalizable Clinical Representation Learning

Brain MRI underpins a wide range of neuroscientific and clinical applications, yet most learning-based methods remain task-specific and require substantial labeled data. Here we show that a single self-supervised representation can generalize across heterogeneous brain MRI endpoints. We trained BrainDINO, a self-distilled foundation model, on approximately 6.6 million unlabeled axial slices from 20 datasets encompassing broad variation in population, disease, and acquisition setting. Using a frozen encoder with lightweight task heads, BrainDINO supported transfer across tumor segmentation, neurodegenerative and neurodevelopmental conditions classification, brain age estimation, post-stroke temporal prediction, molecular status prediction, MRI sequence classification, and survival modeling. Across tasks and supervision regimes, BrainDINO consistently equaled or exceeded natural-image and MRI-specific self-supervised baselines, with particularly strong advantages under label scarcity. Representation analyses further showed anatomically organized and pathology-sensitive feature structure in the absence of task-specific supervision. Our findings indicate that large-scale slice-wise self-supervised learning can yield a unified brain MRI representation that supports diverse neuroimaging tasks without volumetric pretraining or full-network fine-tuning, establishing a scalable foundation for robust and data-efficient brain imaging analysis. Code is available at https://github.com/mclwu22/BrainDINO

Yizhou Wu, Shansong Wang, Yuheng Li +5

Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA, USA. · Department of Radiation and Cellular Oncology, The University of Chicago, Chicago, IL, USA. · Department of Biomedical Engineering, Georgia Institute of Technology, Atlanta, GA, USA. +2

Medical Imaging Foundation Models Self-Supervised Pre-Training Medical Image Representation Learning Brain MRI

Ask PDF

Abstract

Medical Imaging Foundation Models

Explore similar work

BrainNext: A General-Purpose Self-Supervised Foundation Model for Brain MRI Analysis

Comparative Study of Anatomical and Learned Features in AI Models for Structural Brain MRI

BrainDINO: A Brain MRI Foundation Model for Generalizable Clinical Representation Learning