Combining General and Domain-Specific Pretext Tasks for Brain MR Image Segmentation

Tasneem Nasser; Susanne Schmid; Roberto Souza; Naser El-Sheimy

cs.CVSep 25, 2026

Combining General and Domain-Specific Pretext Tasks for Brain MR Image Segmentation

Authors: Tasneem Nasser, Susanne Schmid, Roberto Souza, Naser El-Sheimy

Organizations: Biomedical Engineering University of Calgary Calgary AB Canada · Hotchkiss Brain Institute University of Calgary Calgary AB Canada · Electrical and Software Engineering University of Calgary Calgary AB Canada · Geomatics Engineering University of Calgary Calgary AB Canada

Abstract

A key challenge in medical image analysis is the scarcity of large annotated datasets for specific populations and diseases. As deep learning models rely heavily on labeled data, effective transfer learning strategies are needed to reduce the dependence on manual annotations. Self-supervised learning has emerged as a promising approach for developing foundation models by enabling the learning of transferable feature representations from large-scale unlabeled medical imaging datasets. In this study, we investigate voxel-level brain age prediction as a domain-specific self-supervised pretext task and compare it with image inpainting, a widely used non-domain-specific alternative. We further propose a multitask self-supervised pretraining framework that jointly optimizes both objectives to learn complementary neuroimaging representations. The pretrained models are evaluated on three downstream magnetic resonance image segmentation tasks: multiple sclerosis lesion segmentation, ischemic stroke lesion segmentation, and cortical brain structure segmentation. Overall, the proposed multitask pretraining framework consistently outperformed the single-task pretrained models and training from scratch across most experimental settings, demonstrating the benefit of combining domain-specific and general self-supervised learning pretext tasks for the development of generalizable neuroimaging foundation models.\ Code Availability: The source code used in this study is publicly available at https://github.com/TasneemN/Combining-General-and-Domain-Specific-Pretext-Tasks-for-Brain-MR-Image-Segmentation/

Figures & tables

Figure 1: Overview of the proposed framework. The figure illustrates the self-supervised pretraining strategies and their transfer to downstream MR segmentation tasks. Three pretraining strategies are investigated: (1) single-task image inpainting, in which masked MR volumes are used as input and the model reconstructs the original image; (2) single-task voxel-level brain age prediction, in which the full MR volume is used to predict a 3D voxel-wise brain age map; and (3) multitask pretraining, which jointly learns both image inpainting and voxel-level brain age prediction, producing both reconstructed MR and voxel-wise age map outputs. The pretrained SwinUNETR model is subsequently fine-tuned on three downstream segmentation tasks: multiple sclerosis lesion segmentation, and stroke lesion segmentation (non age-related), cortical brain structure segmentation (age-related)

Panel of Figure 2 — Figure 2: Age and sex distributions of the pretraining datasets. (a) Multi-source T1w dataset used for pretraining prior to ischemic stroke lesion and cortical brain structure segmentation. (b) Calgary Normative Study (CNS) dataset, containing paired T1w and FLAIR images, used for pretraining prior to MS lesion segmentation.

Architecture	Multi-source T1w	CNS (T1w+FLAIR)
	MAE (years)	MAE (years)
U-Net	$6.4\pm 4.3$	$6.9\pm 4.7$
UNETR	$7.1\pm 4.4$	$8.0\pm 5.0$
SwinUNETR	6.1 $\pm$ 4.1	5.3 $\pm$ 3.1

Table 1: Backbone architecture comparison for the voxel-level brain age prediction pretext task. Lower MAE indicates better performance.

Pretraining Strategy		Perceptual Loss	PSNR (dB)	Voxel MAE (years)
Image Inpainting	Multi-source T1w	0.0240	34.90	–
Image Inpainting	CNS (T1w+FLAIR)	0.0033	43.52	–
Voxel-Level Brain Age	Multi-source T1w	–	–	$6.1\pm 4.1$
Voxel-Level Brain Age	CNS (T1w+FLAIR)	–	–	$5.3\pm 3.1$
Multitask	Multi-source T1w	0.0020	46.24	5.46 $\pm$ 3.7
Multitask	CNS (T1w+FLAIR)	0.0026	45.78	5.25 $\pm$ 3.1

Table 2: Performance of the three self-supervised pretraining strategies under the two pretraining settings. Lower perceptual loss and voxel MAE indicate better performance, whereas higher PSNR indicates better reconstruction quality.

Training	Scratch	Voxel-Level Brain Age		Image Inpainting		Multitask
Size	End-to-End	All Layers	Decoder Only	All Layers	Decoder Only	All Layers	Decoder Only
11	0.662 $\pm$ 0.016	0.631 $\pm$ 0.013	0.593 $\pm$ 0.006	0.654 $\pm$ 0.022	0.623 $\pm$ 0.008	0.671 $\pm$ 0.016	0.654 $\pm$ 0.014
22	0.699 $\pm$ 0.015	0.692 $\pm$ 0.012	0.685 $\pm$ 0.017	0.696 $\pm$ 0.019	0.661 $\pm$ 0.009	0.695 $\pm$ 0.025	0.699 $\pm$ 0.010
30	0.751 $\pm$ 0.013	0.715 $\pm$ 0.015	0.722 $\pm$ 0.010	0.727 $\pm$ 0.012	0.700 $\pm$ 0.009	0.737 $\pm$ 0.006	0.706 $\pm$ 0.002
42	0.757 $\pm$ 0.017	0.733 $\pm$ 0.014	0.724 $\pm$ 0.014	0.743 $\pm$ 0.010	0.725 $\pm$ 0.013	0.752 $\pm$ 0.012	0.727 $\pm$ 0.005
50	0.757 $\pm$ 0.017	0.749 $\pm$ 0.011	0.736 $\pm$ 0.014	0.747 $\pm$ 0.012	0.724 $\pm$ 0.010	0.764 $\pm$ 0.010	0.741 $\pm$ 0.009

Table 3: Comparison of different fine-tuning strategies for MS lesion segmentation. For all pretrained models, the segmentation decoder was reinitialized prior to fine-tuning. Each experiment was independently repeated five times, and the mean Dice score

\pm

standard deviation is reported. The best result for each training set is highlighted in bold italic , while bold indicates results within a Dice difference of 0.005 from the best-performing method.

Figure 3: Comparison of the best-performing fine-tuning strategy for the MS lesion segmentation task, in which the entire network was fine-tuned after reinitializing the segmentation decoder.

Figure 4: MS lesion segmentation results for the same representative test image across the evaluated pretraining strategies with 11 images used for fine-tuning. Although both T1w and FLAIR images were used as model inputs, the segmentation results are overlaid on the FLAIR image for clearer visualization of the lesions. For all pretrained models, the decoder was reinitialized before fine-tuning; all layers were subsequently fine-tuned except for the Multitask decoder-only setting, in which only the reinitialized decoder was trained.

Figure 5: Per-image Dice scores for the five training strategies on the ischemic stroke lesion segmentation task. Results are shown for the fourth independent run using 50 labeled training cases. Each point represents one of the 100 test images, displayed in the same order across all subplots to enable direct comparison between methods. Colors indicate the acquisition center of each test image, and the dashed horizontal line denotes the mean Dice score. The evaluated strategies are training from scratch, image inpainting pretraining, voxel-level brain age prediction pretraining, multitask pretraining with fine-tuning of all pretrained layers (Multitask–FT All), and multitask pretraining with decoder reinitialization followed by end-to-end fine-tuning (Multitask–Decoder Reinitialized).

Figure 6: Stroke lesion segmentation results for the same representative test image across the evaluated pretraining strategies with 50 images used for fine-tuning in a single run. The segmentation results are overlaid on the T1w image, with the ground-truth lesion shown in green and the predicted lesions shown in violet.

Training Size	Scratch	Brain Age	Inpainting	Multitask–Decoder Reinitialized	Multitask–FT All
10	0.216 $\pm$ 0.033	0.203 $\pm$ 0.010	0.206 $\pm$ 0.028	0.276 $\pm$ 0.027	0.286 $\pm$ 0.011
20	0.229 $\pm$ 0.010	0.249 $\pm$ 0.010	0.231 $\pm$ 0.024	0.283 $\pm$ 0.030	0.300 $\pm$ 0.030
50	0.332 $\pm$ 0.025	0.351 $\pm$ 0.016	0.329 $\pm$ 0.023	0.399 $\pm$ 0.016	0.391 $\pm$ 0.009
100	0.459 $\pm$ 0.017	0.459 $\pm$ 0.009	0.464 $\pm$ 0.016	0.480 $\pm$ 0.014	0.470 $\pm$ 0.012
150	0.488 $\pm$ 0.009	0.485 $\pm$ 0.009	0.488 $\pm$ 0.015	0.481 $\pm$ 0.024	0.492 $\pm$ 0.011
200	0.506 $\pm$ 0.010	0.506 $\pm$ 0.008	0.507 $\pm$ 0.022	0.525 $\pm$ 0.008	0.502 $\pm$ 0.031

Table 4: Mean Dice score

\pm

standard deviation over five independent runs for ischemic stroke lesion segmentation across different training-set sizes and training strategies.

Training Size	Scratch	Brain Age	Inpainting	Multitask–FT All	Multitask–Decoder Reinitialized
10	0.186 $\pm$ 0.023	0.063 $\pm$ 0.010	0.221 $\pm$ 0.024	0.248 $\pm$ 0.033	0.251 $\pm$ 0.049
25	0.493 $\pm$ 0.063	0.313 $\pm$ 0.047	0.545 $\pm$ 0.080	0.633 $\pm$ 0.044	0.611 $\pm$ 0.033
45	0.726 $\pm$ 0.035	0.723 $\pm$ 0.019	0.762 $\pm$ 0.005	0.732 $\pm$ 0.041	0.674 $\pm$ 0.060
65	0.741 $\pm$ 0.016	0.737 $\pm$ 0.036	0.772 $\pm$ 0.015	0.762 $\pm$ 0.016	0.742 $\pm$ 0.033

Table 5: Mean Dice score

\pm

standard deviation over five independent runs for cortical brain structure segmentation across different training-set sizes and training strategies.

Figure 7: Cortical parcellation results for the same representative test image across the evaluated pretraining strategies with 25 images used for fine-tuning.

Explore similar work

BrainNext: A General-Purpose Self-Supervised Foundation Model for Brain MRI Analysis
Jul 20, 2026Moona Mazher, Abdul Qayyum, Steven A. Niederer +1Medical Imaging Foundation Models Self-Supervised Pre-Training
UCL Hawkes Institute, Department of Computer Science, University College London, London, United Kingdom · National Heart & Lung Institute, Imperial College London, London, United Kingdom · Division of Cardiovascular Medicine, Stanford Cardiovascular Institute, Stanford University, California, USA
BrainDINO: A Brain MRI Foundation Model for Generalizable Clinical Representation Learning
Apr 30, 2026Yizhou Wu, Shansong Wang, Yuheng Li +5Medical Imaging Foundation Models Self-Supervised Pre-Training
Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA, USA. · Department of Radiation and Cellular Oncology, The University of Chicago, Chicago, IL, USA. · Department of Biomedical Engineering, Georgia Institute of Technology, Atlanta, GA, USA. +2
Masked and Predictive Self-Supervised Foundation Models for 3D Brain MRI
Jun 11, 2026Esra Ergün, Hersh Chandarana, Dan Sodickson +1Medical Imaging Foundation Models Masked Autoencoders
Department of Computer Engineering, Istanbul Technical University, Istanbul, 34469, Türkiye · Department of Radiology, NYU Grossman School of Medicine, New York, NY, 10016, United States · Department of AI and Data Engineering, Istanbul Technical University, Istanbul, 34469, Türkiye

Jul 20, 2026J/K move · Enter open · S save

BrainNext: A General-Purpose Self-Supervised Foundation Model for Brain MRI Analysis

Moona Mazher, Abdul Qayyum, Steven A. Niederer, Daniel C. Alexander

UCL Hawkes Institute, Department of Computer Science, University College London, London, United Kingdom · National Heart & Lung Institute, Imperial College London, London, United Kingdom · Division of Cardiovascular Medicine, Stanford Cardiovascular Institute, Stanford University, California, USA

Medical Imaging Foundation Models Self-Supervised Pre-Training Medical Image Representation Learning

Open paper AskPDF arXiv

Foundation models pretrained using self-supervised learning have transformed computer vision by learning transferable representations from large-scale unlabeled data. However, existing foundation models for neuroimaging remain limited by task-specific training, slice-based learning strategies, or relatively small pretraining datasets, restricting their generalizability across diverse brain MRI applications. In this work, we present BrainNext, a general-purpose self-supervised foundation model for volumetric brain MRI analysis. BrainNext combines masked autoencoder (MAE) pretraining with a native three-dimensional Bi-Directional xLSTM-UNet architecture to learn rich anatomical representations from 60,551 unlabeled brain MRI examinations spanning multiple MRI modalities. The pretrained model is subsequently adapted to downstream tasks through lightweight task-specific fine-tuning. We evaluate BrainNext on the Foundation Models for Medical Imaging (FOMO) 2025 Method Track, encompassing classification, segmentation, and brain-age estimation, where it achieved second place overall and ranked first in the meningioma segmentation task on the official FOMO 2025 challenge leaderboard, demonstrating strong transferability across heterogeneous neuroimaging tasks. These results highlight the potential of large-scale self-supervised pretraining to learn robust and transferable volumetric representations, establishing BrainNext as a scalable foundation model for diverse brain MRI applications.

Abstract

Explore similar work

BrainNext: A General-Purpose Self-Supervised Foundation Model for Brain MRI Analysis

BrainDINO: A Brain MRI Foundation Model for Generalizable Clinical Representation Learning

Masked and Predictive Self-Supervised Foundation Models for 3D Brain MRI