Foundation models have transformed vision and language by learning general-purpose representations from large-scale unlabeled data, yet 3D medical imaging lacks analogous approaches. Existing self-supervised methods rely on low-level reconstruction or contrastive objectives that fail to capture the anatomical semantics critical for medical image analysis, limiting transfer to downstream tasks. We present MASS (MAsk-guided Self-Supervised learning), which treats in-context segmentation as the pretext task for learning general-purpose medical imaging representations. MASS's key insight is that automatically generated class-agnostic masks provide sufficient structural supervision for learning semantically rich representations. By training on thousands of diverse mask proposals spanning anatomical structures and pathological findings, MASS learns what semantically defines medical structures: the holistic combination of appearance, shape, spatial context, and anatomical relationships. We demonstrate effectiveness across data regimes: from small-scale pretraining on individual datasets (20-200 scans) to large-scale multi-modal pretraining on 5K CT, MRI, and PET volumes, all without annotations. MASS demonstrates: (i) few-shot segmentation on novel structures, (ii) matching full supervision with only 20-40% labeled data while outperforming self-supervised baselines by over 20 in Dice score in low-data regimes, and (iii) frozen-encoder classification on unseen pathologies that matches full supervised training with thousands of samples. Mask-guided self-supervised pretraining captures broadly generalizable knowledge, opening a path toward 3D medical imaging foundation models without expert annotations. Code is available: https://github.com/Stanford-AIMI/MASS.
Figures & tables
Figure 1: MASS framework overview. (A) Annotation-free mask generation: SAM2 generates class-agnostic masks by sampling 2D slices from unlabeled 3D images, applying automatic segmentation, and propagating masks through volumes. B) Mask-guided self-supervised learning: For each training step, we sample an image x and its auto-generated masks m . We then create two augmented views: a reference (xs,ys) and a query (xq,yq) . The model extracts a task embedding from the reference mask and uses it to predict the corresponding region in the query view. By solving many such in-context segmentation tasks, the model learns generalizable, semantically rich representations.
BCV
AMOS MR
SS H&N
KiTS Tumor
Method
1-shot
5-shot
10-shot
1-shot
5-shot
10-shot
1-shot
5-shot
10-shot
5-shot
30-shot
60-shot
Scratch
27.3(4.2)
72.8(1.6)
75.2(1.1)
32.8(3.8)
70.3(1.8)
75.9(1.2)
51.8(3.2)
58.5(2.4)
65.1(1.6)
5.7(2.8)
35.7(3.2)
45.9(2.4)
MG
47.1(2.8)
74.8(1.3)
77.7(0.9)
37.7(3.4)
72.2(1.5)
78.5(1.0)
55.2(2.6)
62.1(2.0)
68.8(1.3)
6.5(2.5)
38.2(2.8)
46.5(2.1)
MAE
46.3(3.0)
74.2(1.4)
78.0(1.0)
36.2(3.5)
73.8(1.4)
77.4(1.1)
54.2(2.7)
59.5(2.2)
72.6(1.2)
6.8(2.4)
42.2(2.5)
47.8(2.0)
S3D
46.7(2.9)
75.0(1.3)
77.9(1.0)
36.8(3.4)
73.9(1.4)
77.2(1.1)
55.2(2.6)
60.2(2.1)
71.5(1.2)
7.6(2.3)
40.7(2.6)
49.4(1.9)
SimCLR
44.9(3.1)
75.5(1.2)
78.4(0.9)
35.9(3.6)
71.9(1.6)
78.0(1.0)
53.6(2.8)
59.4(2.2)
65.8(1.5)
9.3(2.1)
41.5(2.4)
53.8(1.7)
Table 1: Single-dataset segmentation performance (Dice, %). All models are pretrained on each dataset’s full training set. Results shown as mean (std) over 3 runs. Baseline methods require finetuning on few-shot samples, while MASS can perform in-context inference without finetuning, already surpassing baselines. Finetuning MASS further improves performance. The number in the brackets of “Full supervision” indicates the number of labeled samples for full supervised training. Bold: best performance; underlined: second best.
BCV
AMOS MR
SS H&N
KiTS Tumor
LiTS Tumor
AutoPET
BraTS T1CE
ACDC
Pelvic
# shot
1
10
1
10
1
10
30
60
10
30
30
100
30
60
1
10
1
Full supervision
83.6 [23]
85.5 [38]
78.2 [39]
81.7 [159]
63.2 [99]
67.8 [983]
72.8 [241]
90.8 [70]
94.7 [80]
Scratch
27.3
75.2
32.9
75.9
51.8
65.1
35.7
45.9
42.5
50.4
40.1
53.4
54.0
62.8
38.7
69.8
57.8
Supervised pretrain
SuPreM
63.9
83.6
55.1
82.1
66.1
75.6
64.1
78.1
53.9
62.7
48.8
64.8
60.3
70.8
55.9
82.3
85.4
Iris (IC)
83.2
85.4
83.5
86.4
78.4
80.1
78.2
80.2
59.2
63.3
65.2
69.5
48.6
79.8
86.5
88.2
69.0
Table 2: Large-scale multi-modal pretraining segmentation performance (Dice, %). Results shown as mean over 3 runs (standard deviations in supplementary). Numbers in brackets under “Full supervision” indicate the number of labeled samples for full supervised training. Bold: best performance; underlined: second best.
RSNA ICH
Liver Trauma
Kidney Trauma
Spleen Trauma
5%
30%
100%
5%
30%
100%
5%
30%
100%
5%
30%
100%
Method
(761)
(4566)
(15220)
(153)
(918)
(3061)
(152)
(913)
(3046)
(152)
(917)
(3057)
Full training
Scratch
72.8
79.8
89.5
57.5
74.4
90.5
54.14
75.0
90.0
62.5
82.08
89.2
Frozen encoder
OM-MG
57.1
61.7
70.0
52.1
57.8
61.6
53.2
57.9
58.8
50.6
52.2
60.5
Table 3: Classification performance (AUC, %) across different downstream datasets and training data percentages. Training sample sizes are shown in parentheses. Bold: best performance; underlined: second best.
Table 4: Analysis experiments. (a) MASS learns strong representations from imperfect masks. (b) Impact of mask generation methods (c) MASS generalizes across different backbone architectures.
Figure 2: Progressive scaling analysis. Diversity along anatomical and modality dimensions drives generalization.
Preprocessing Method
Dice (%)
Linear mapping
54.7
Single window replication
64.5
Three-channel windows
65.5
Three-channel + multi-axis
65.4
Table 5: Ablation on SAM2 preprocessing strategies. Evaluated on BCV 1-shot segmentation.
Figure 3: SAM2 generated 2D masks on initial seed slices. We show examples from abdomen CT, head & neck CT, and abdomen MR. Images appear in color because we create 3-channel inputs for SAM2 using three complementary intensity windows. Each tissue’s color reflects its relative intensity across the three channels. SAM2 generates meaningful region proposals with good coverage of diverse anatomical structures including organs, bones, muscles, sub-organ regions, and pathologies (cysts). However, the masks also contain substantial noise from over-segmentation, missing objects and struggle with small subtle structures. Despite these imperfections, the diverse mask proposals spanning multiple granularities provide sufficient supervision for MASS to learn generalizable representations.
Figure 4: SAM2 3D mask propagation results. We show 3D masks generated by propagating the 2D seed masks from Figure 3 through the volume using SAM2’s video prediction capability. SAM2 successfully converts 2D masks into volumetric segmentations by tracking boundaries across slices. While propagation maintains anatomical coherence for most structures, the results still contain noise from inconsistent boundaries and occasional tracking failures. These imperfect but volumetrically consistent masks provide the structural supervision needed for MASS pretraining.
Figure 5: Task encoding module architecture. The module extracts compact task embeddings from reference image-mask pairs through two parallel streams: foreground feature encoding (top) captures fine anatomical details via high-resolution mask application, while contextual feature encoding (bottom) uses pixel shuffle operations and learnable query tokens with cross/self-attention to extract global context. The combined task embedding guides query image segmentation through the mask decoder. We follow [ 20 ] and refer the readers for more details in [ 20 ]
Figure 6: Probing learned knowledge in pretraining through one-shot in-context inference. We visualize the medical imaging understanding acquired by MASS during self-supervised pretraining by probing the model with one-shot in-context segmentation (no finetuning). Given a single reference image-mask pair, the pretrained model segments novel query images across diverse anatomical regions and modalities: BCV (abdominal CT), SS H&N (head & neck CT), AMOS MR (abdominal MRI), M&Ms (cardiac MRI), SS Thoracic (thoracic CT), and Pelvic (pelvic CT). The results demonstrate that MASS learns rich medical imaging knowledge including organ morphology, spatial relationships, and anatomical consistency entirely from mask-guided self-supervision without expert annotations.
Figure 7: PCA visualization of MASS learned features. We extract decoder output features from the pretrained MASS model (without finetuning) and reduce to 3 channels via PCA for visualization. Examples shown include abdomen CT, head & neck CT, and abdomen MR images. The visualizations reveal that MASS learns semantically meaningful representations with clear anatomical boundaries at full image resolution. Large structures (kidneys, liver, spleen, heart, lungs, bones, muscles) are clearly delineated, while semantically similar structures share consistent feature patterns (similar colors). Notably, MASS also captures fine-grained sub-anatomical details including pulmonary vasculature, hepatic vessels, and cardiac chambers, demonstrating multi-granular anatomical understanding learned entirely through mask-guided self-supervision without expert annotations.
Figure 8: Failure case analysis on pathology segmentation. We visualize MASS’s in-context segmentation on BraTS T1CE brain tumor data. Top row: reference image and mask defining the target tumor. Middle row: query image with ground truth tumor annotation. Bottom row: MASS’s prediction, which incorrectly segments a region at a similar spatial location as the reference rather than the actual tumor. This failure stems from the large cross-patient variance in tumor location, size, and shape. MASS’s pretraining objective learns within-image invariance but not cross-patient correspondence for highly variable pathologies. Notably, when using the query image as its own reference (with augmentation), MASS achieves > 70% Dice, confirming that the model learns meaningful tumor representations but lacks cross-patient semantic alignment. This gap is efficiently bridged through minimal finetuning with expert annotated samples.
BCV
AMOS MR
SS H&N
KiTS Tumor
LiTS Tumor
AutoPET
BraTS T1CE
ACDC
Pelvic
# shot
1
10
1
10
1
10
30
60
10
30
30
100
30
60
1
10
1
Full supervision
83.6 [23]
85.5 [38]
78.2 [39]
81.7 [159]
63.2 [99]
67.8 [983]
72.8 [241]
90.8 [70]
94.7 [80]
Scratch
27.3(3.8)
75.2(1.3)
32.9(4.0)
75.9(1.3)
51.8(3.2)
65.1(1.7)
35.7(3.6)
45.9(2.9)
42.5(2.6)
50.4(2.1)
40.1(2.4)
53.4(1.6)
54.0(1.9)
62.8(1.4)
38.7(3.4)
69.8(1.4)
57.8(3.1)
Supervised pretrain
SuPreM
63.9(1.9)
83.6(0.8)
55.1(2.1)
82.1(0.9)
66.1(1.7)
75.6(1.0)
64.1(1.2)
78.1(0.9)
53.9(1.6)
62.7(1.2)
48.8(1.5)
64.8(0.9)
60.3(1.2)
70.8(0.9)
55.9(2.0)
82.3(0.8)
85.4(1.3)
Iris (IC)
83.2(1.2)
85.4(0.7)
83.5(1.1)
86.4(0.7)
78.4(1.0)
80.1(0.7)
78.2(0.9)
80.2(0.7)
59.2(1.3)
63.3(1.0)
65.2(1.1)
69.5(0.8)
48.6(1.6)
79.8(0.8)
86.5(1.1)
88.2(0.7)
69.0(1.9)
Table 6: Large-scale multi-modal pretraining segmentation performance (Dice %). Results shown as mean (standard deviation) over 3 runs. Numbers in brackets for ”Full supervision” indicate the number of labeled samples for full supervised training. Bold: best performance; underlined: second best.
Method
# Samples
Modalities
Supervision
OpenMind
114K
24 MR
Self-supervised
Merlin
15K
CT
Report & EHR
AnatoMix
120K
Synthetic
Synthetic labels
SuPreM
2.1K
CT
32 organ/tumor masks
Iris
2K
CT, MR, PET
Expert annotated masks
MASS
5K
CT, MR, PET
Self-supervised
Table 7: Comparison of training data of large-scale pretraining baseline methods.
Metric
Value
Inference Time
Full in-context inference
88.84 ± 0.13 ms
Reference encoding
39.72 ± 0.07 ms
Query inference
49.00 ± 0.02 ms
Query inference FPS
20.41
Memory
Table 8: Model specifications and inference benchmark on NVIDIA H100 GPU with BF16 precision. Input size: 1283 .
Aug. Strategy
Dice (%)
Aug. Magnitude
Dice (%)
Spatial only
60.7
Small
61.3
Appearance only
54.6
Medium
65.5
Both
65.5
Large
62.9
Table 9: Ablation study on augmentation strategies. All experiments evaluated on BCV 1-shot segmentation.
Dataset
Body Region
Modality
Clinical Target
#Cls
Size
AMOS CT [ 29 ]
Abdomen
CT
Organs
15
300
AMOS MR [ 29 ]
Abdomen
MRI
Organs
13
60
AutoPET [ 22 ]
Whole body
CT + PET
Lesions
1
1014
BCV [ 23 ]
Abdomen
CT
Organs
13
30
BraTS [ 38 ]
Brain
T1/T2/FLAIR
Tumors
3
213
KiTS [ 26 ]
Abdomen
CT
Kidney & Tumor
2
210
Table 10: Dataset statistics. Upper section: upstream training datasets. Lower section: downstream evaluation datasets held out entirely from pretraining or reserved modalities.
Training deep learning models for three-dimensional (3D) medical imaging, such as Computed Tomography (CT), is fundamentally challenged by the scarcity of labeled data. While pre-training on natural images is common, it results in a significant domain shift, limiting performance. Self-Supervised Learning (SSL) on unlabeled medical data has emerged as a powerful solution, but prominent frameworks often fail to exploit the inherent 3D nature of CT scans. These methods typically process 3D scans as a collection of independent 2D slices, an approach that fundamentally discards critical axial coherence and the 3D structural context. To address this limitation, we propose the autoencoder for enhanced self-supervised medical image learning(MAESIL), a novel self-supervised learning framework designed to capture 3D structural information efficiently. The core innovation is the 'superpatch', a 3D chunk-based input unit that balances 3D context preservation with computational efficiency. Our framework partitions the volume into superpatches and employs a 3D masked autoencoder strategy with a dual-masking strategy to learn comprehensive spatial representations. We validated our approach on three diverse large-scale public CT datasets. Our experimental results show that MAESIL demonstrates significant improvements over existing methods such as AE, VAE and VQ-VAE in key reconstruction metrics such as PSNR and SSIM. This establishes MAESIL as a robust and practical pre-training solution for 3D medical imaging tasks.
Transformer architectures, including nnFormer,have demonstrated promising results in volumetric medical image segmentation by being able to capture long-range spatial interactions. Although they have high performance, these models need large quantities of labeled training data and are also likely to overfit and become training unstable. This is a serious practical problem because it is not only time-consuming but also expensive to obtain medical images that are annotated by experts. Moreover, fully supervised traditional training pipelines do not take advantage of the available large amounts of unlabeled medical imaging data that can be easily obtained in the clinics. We have solved these drawbacks by advancing the efficiency of the nnFormer with a self-supervised pretraining framework, which is based on the Masked Autoencoders (MAE). In this method, the model is pretrained on unlabeled volumetric medical images to reconstruct randomly masked parts of the input. This allows the encoder to learn meaningful anatomical and structural representations . The encoder is then further fine-tuned on a labeled dataset on the downstream segmentation task. Conducted Experiment shows that the offered method leads to a higher segmentation performance on the count of Dice score, a quicker convergence rate on the course of the fine-tuning procedure, and a superior generalization on the basis of limited labeled data . These findings validate that self-supervised learning combined with transformer-based segmentation models is an appropriate approach to the problem of data shortage in medical image analysis.
R. M. Krishna Sureddi, T. Satyanarayana Murthy, Nomula Varsha Reddy +2
Information Technology, Chaitanya Bharathi Institute of Technology, Hyderabad, India
Self-supervised pre-training paradigm has gained increasing prominence for learning transferable representations in medical imaging, yet existing methods for ultrasound (US) images operate at the image or frame level, overlooking the anatomical context for clinical-aligned representation learning. In this work, we propose an anatomy-anchored ultrasound self-supervision framework ANAUS that shifts representation learning from generic visual regions to clinically meaningful anatomical structures. Utilizing a learnable latent prompt engine alongside a one-time domain adaptation on existing public image-mask pairs, we empower the LP-SAM module to achieve annotation-free anatomy delineation at scale. Building upon this anatomical grounding, we propose a dual-policy self-supervised learning paradigm consisting of inter-view semantics-aware anatomy-separating alignment and contextual core-region prediction to enhance representation learning. Specifically, the former enforces feature invariance within identical anatomical regions while promoting discriminability across distinct structures; the latter compels the model to reconstruct corrupted regions, thereby capturing fine-grained structural details. Extensive evaluations on six public datasets demonstrate that ANAUS consistently outstrips current state-of-the-art methods while maintaining the computational efficiency essential for clinical deployment. Code is available at https://github.com/zhcz328/ANAUS.
Chunzheng Zhu, Yijun Wang, Jianxin Lin +5
Hunan University, Changsha, China · Shenzhen Maternity and Child Healthcare Hospital, Shenzhen, China