Where to Look Matters: Rethinking Sub-Volume Sampling in 3D Medical Self-supervised Learning

Junkai Liu; Le Zhang

cs.CVOct 9, 2026

Where to Look Matters: Rethinking Sub-Volume Sampling in 3D Medical Self-supervised Learning

Authors: Junkai Liu, Le Zhang

Organizations: School of Engineering, University of Birmingham

Abstract

Self-supervised learning on 3D medical images commonly relies on random sub-volume sampling during pretraining. However, random cropping implicitly treats all spatial regions as equally worth observing, despite anatomical priors and spatial redundancy that make regions inherently unequal in their value for representation learning. To this end, we first show through controlled crop-quality manipulations that better sub-volumes lead to better downstream transfer, revealing the observation policy as an overlooked bottleneck in 3D medical SSL. Motivated by this finding, we revisit sub-volume sampling as a where-to-look problem. We propose VolumeProbe, a plug-and-play framework that replaces random cropping with an offline coarse-to-fine probe bank guided by anatomy-awareness, informativeness, and diversity. We instantiate VolumeProbe as a lightweight Lite variant using geometric and image-statistical cues, and a diagnostic Feat variant using frozen visual features, to examine whether lightweight volume-intrinsic priors suffice or richer semantic cues reliably provide additional value. Extensive experiments across multiple 3D medical SSL backbones and downstream tasks demonstrate that VolumeProbe-Lite consistently improves transfer performance over random sampling. Mechanistic analyses further link these gains to more anatomically plausible and informative observations, reduced spatial redundancy, more efficient learning dynamics, and more structured representations.

Figures & tables

Figure 1: VolumeProbe reallocates the fixed crop budget toward better observations. (a) Random cropping in 3D medical SSL often samples low-yield, anatomy-agnostic, and redundant sub-volumes. (b) VolumeProbe builds a cached probe bank guided by anatomy-awareness, informativeness, and diversity, and samples higher-quality sub-volumes across epochs.

Figure 2: Crop quality matters for 3D medical representation learning. (a) Random cropping frequently spends the fixed crop budget on structure-sparse regions. (b) Policies defined by a semantic quality axis $Q_{1}$ yield a monotonic change in downstream Dice. (c) This trend remains under an alternative feature-level proxy $Q_{2}$ . (d) Better crop quality further improves learning efficiency.

Figure 3: Overview of VolumeProbe . VolumeProbe constructs an offline probe bank of normalized 3D crop centers guided by anatomy-awareness, informativeness, and diversity. Pretraining then samples sub-volumes from this bank without changing the SSL backbone or objective.

Method	Dice Score (%) $\uparrow$												Acc. (%) $\uparrow$
	BTCV			AMOS			WORD			Atlas-MRI			CC-CCII
	1%	10%	100%	1%	10%	100%	1%	10%	100%	1%	10%	100%	CC-CCII
From Scratch
UNETR Hatamizadeh et al. (2022)	28.05	42.85	79.82	23.67	60.06	77.02	30.89	73.76	78.73	18.65	40.83	49.97	88.65
Swin-UNETR Hatamizadeh et al. (2021)	27.71	51.33	80.53	28.94	63.45	82.51	43.68	74.03	80.39	17.19	39.46	48.50	88.32
General SSL

Table 1: Results on segmentation and classification tasks. For segmentation, we report Dice Score (%) under different training data proportions. For classification, we report Accuracy (%). The best results are bolded , and the second best results are underlined .

Table 5

Sampling Method	Dice Score (%) $\uparrow$
Sampling Method	BTCV	AMOS	WORD
VoCo Wu et al. (2024)	83.85	84.44	84.06
+ Center-Biased	83.79 ↓0.06	85.32 ↑0.88	83.95 ↓0.11
+ Uniform-Grid	83.88 ↑0.03	85.79 ↑1.35	85.26 ↑1.20
+ Grad-Only	83.74 ↓0.11	84.53 ↑0.09	83.87 ↓0.19
+ Entropy-Only	83.80 ↓0.05	84.31 ↓0.13	84.00 ↓0.06

Table 4: Comparison of different non-random sampling strategies for VoCo .

Figure 4: Anatomy-aware analysis. We aggregate organ masks to construct an anatomical reference density (purple), and overlay crop-center densities. VolumeProbe better aligns crop centers with organ-dense regions and achieves higher foreground and organ occupancy.

Figure 5: Representative cases for anatomy-aware sampling. We plot 3D crop boxes overlaid on organ masks, the coronal projection with sampled centers, 2D crop boxes, and example cropped views. Compared with Random , VolumeProbe places crops more consistently inside the body and around anatomically meaningful regions, raising foreground/organ occupancy ratios.

Figure 6: Informativeness analysis . Left : cumulative internal utility, measured by gradient- and entropy-based metrics. Right : cumulative anatomy-aware utility measured by the number of distinct organs observed and any-organ hits.

Figure 7: Diversity analysis. Left: within-bank coverage as candidate budget increases. VolumeProbe consistently achieves broader support. Right: same-bin candidate ratio measuring local redundancy. Our approach yields the lowest redundancy.

Figure 8: Learning relevance of VolumeProbe . (a) Early-stage learning curves, where VolumeProbe shows faster and more stable loss reduction. (b) Early representation diversity measured by token-level cosine diversity and effective rank. (c) Feature-map visualization, where VolumeProbe produces more structure-aligned representations.

Figure 9: Cross-strategy association between mechanistic diagnostics and downstream transfer.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Modality	Region of Interest	#Volumes
BTCV [ 32 ]	CT	Abdomen	24
TCIA Covid19 [ 10 ]	CT	Chest	722
LUNA16 [ 53 ]	CT	Chest	843
STOIC 2021 [ 51 ]	CT	Chest	2,000
FLARE23 [ 44 ]	CT	Abdomen	4,000
HNSCC [ 15 ]	CT	Head/Neck	1,071

Appendix

Table 5: Overview of pretraining datasets.

Task	Dataset	Modality	#Train				#Valid
Task	Dataset	Modality	Full	1%	10%	100%	#Valid
Segmentation	BTCV [ 32 ]	CT	24	1	2	24	6
	AMOS [ 28 ]	CT	200	2	20	200	100
	WORD [ 42 ]	CT	100	1	10	100	20
	Atlas-MRI [ 50 ]	MRI	45	1	5	45	15
Classification	CC-CCII [ 74 ]	CT	2,514	–	–	–	1,664

Appendix

Table 6: Overview of downstream datasets.

Setting	VoCo [ 68 ]	S2DC [ 48 ]	MedGMAE [ 13 ]
Preprocessing Settings
Spacing	$1.5\times 1.5\times 1.5$	$1.5\times 1.5\times 1.5$	$1.5\times 1.5\times 1.5$
Intensity	$[-175,250]$	$[-175,250]$	$[-175,250]$
Norm	$[0,1]$	$[0,1]$	$[0,1]$
Sub-volume size	$64\times 64\times 64$	$96\times 96\times 96$	$96\times 96\times 96$
Sub-volume number	1	1	3

Appendix

Table 7: Overview of preprocessing and pretraining settings used for pretraining.

Preprocessing Settings
Spacing	$1.5\times 1.5\times 1.5$
Intensity	$[-175,250]$
Norm	$[0,1]$
Sub-volume size	$96\times 96\times 96$
Sub-volume number	4
Augmentation	Random Rotate, Flip, Scale, Shift

Appendix

Table 8: Overview of preprocessing and fine-tuning settings for downstream evaluation.

Figure 10: Qualitative visualization of segmentation results under the 10% label setting.

Method

Dice Score (%)

\uparrow

BTCV

AMOS

WORD

Atlas-MRI

10%

100%

10%

100%

10%

100%

10%

100%

VoCo [ 68 ]

63.33 ±1.5

77.85 ±0.7

83.85 ±0.2

55.81 ±2.3

73.34 ±0.6

84.44 ±0.1

64.24 ±1.2

77.28 ±0.4

84.06 ±0.2

34.85 ±1.9

47.56 ±0.9

58.83 ±0.3

+ VolumeProbe - Lite

67.21 ±1.3

79.39 ±0.5

83.97 ±0.1

58.64 ±1.5

74.95 ±0.6

87.52 ±0.1

68.93 ±1.6

80.12 ±0.5

85.56 ±0.2

38.68 ±2.2

49.27 ±1.0

59.98 ±0.2

+ VolumeProbe - Feat

68.07 ±1.5

79.51 ±0.9

83.89 ±0.1

57.86 ±1.9

74.70 ±0.4

87.13 ±0.1

68.42 ±1.7

79.74 ±0.4

84.99 ±0.2

37.93 ±2.4

49.06 ±1.2

59.42 ±0.2

S2DC [ 48 ]

66.77 ±1.8

77.93 ±0.8

84.09 ±0.1

55.19 ±1.6

74.28 ±0.9

84.68 ±0.2

69.14 ±2.0

78.96 ±0.6

84.46 ±0.1

33.79 ±2.1

46.18 ±0.8

58.53 ±0.2

Appendix

Table 9: Multi-run robustness on segmentation tasks. Dice Score (%) is reported as mean ±std over three independent runs.

Table 19

Method

Dice Score (%)

\uparrow

BTCV

AMOS

WORD

Atlas-MRI

10%

100%

10%

100%

10%

100%

10%

100%

VoCo [ 68 ]

63.33

\pm

5.9

77.85

\pm

7.2

83.85

\pm

1.8

55.81

\pm

3.9

73.34

\pm

4.5

84.44

\pm

1.6

64.24

\pm

3.2

77.28

\pm

2.9

84.06

\pm

1.2

34.85

\pm

5.5

47.56

\pm

4.3

58.83

\pm

1.2

+ VolumeProbe

67.21 *

\pm

6.8

79.39 *

\pm

8.6

83.97 n.s.

\pm

1.3

58.64 ***

\pm

4.0

74.95 ****

\pm

4.2

87.52 ****

\pm

1.4

68.93 ****

\pm

3.7

80.12 ****

\pm

3.5

85.56 ****

\pm

1.5

38.68 **

\pm

4.2

49.27 n.s.

\pm

3.5

59.98 **

\pm

1.0

S2DC [ 48 ]

66.77

\pm

8.2

77.93

\pm

5.7

84.09

\pm

2.3

55.19

\pm

4.2

74.28

\pm

3.9

84.68

\pm

1.5

69.14

\pm

4.0

78.96

\pm

4.3

84.46

\pm

1.3

33.79

\pm

4.6

46.18

\pm

5.7

58.53

\pm

1.0

+ VolumeProbe

69.47 *

\pm

7.4

78.74 *

\pm

5.6

84.43 *

\pm

1.5

58.83 ****

\pm

4.0

75.31 ****

\pm

4.1

87.17 ****

\pm

1.9

69.73 ***

\pm

3.5

80.04 ***

\pm

3.9

85.22 ****

\pm

1.0

37.16 **

\pm

5.0

48.08 **

\pm

3.8

59.44 **

\pm

0.9

Appendix

Table 12: Subject-level variability and paired significance on segmentation tasks. Dice Score (%) is reported with subject-level SD in subscript and significance in superscript. Superscripts denote two-sided paired permutation tests against the corresponding random-cropping baseline: n.s.,

p>0.05

; *,

p\leq 0.05

; **,

p\leq 0.01

; ***,

p\leq 0.001

; ****,

p\leq 0.0001

. For BTCV (

N=6

), the minimum exact two-sided

p

-value is

0.03125

, so significance is capped at *.

Method	Dice Score (%) $\uparrow$
Method	AMOS		WORD
	1%	100%	1%	100%
nnU-Net (from scratch)	29.53 ±4.7	84.89 ±1.9	49.82 ±3.3	84.66 ±1.5
VoCo (nnU-Net) [ 68 ]	52.96 ±3.4	85.22 ±2.0	65.47 ±3.9	84.89 ±1.2
+ VolumeProbe	55.43 ±3.5	86.90 ±1.8	67.53 ±4.2	85.37 ±1.5
S2DC (nnU-Net) [ 48 ]	49.86 ±2.8	85.38 ±2.2	67.21 ±3.1	84.83 ±1.4

Appendix

Table 13: Evaluation under nnU-Net pipelines. Subject-level Dice Score (%) is reported as mean

\pm

SD. All variants within each dataset share the same auto-configured nnU-Net 3D full-resolution plan and training protocol.

Method	Foreground Occupancy (%) $\uparrow$	Organ Occupancy (%) $\uparrow$
Random	55.83	19.79
Center-Biased	69.12 ↑13.29	34.61 ↑14.82
Uniform-Grid	62.74 ↑6.91	28.57 ↑8.78
Grad-Only	71.34 ↑15.51	32.90 ↑13.11
Entropy-Only	68.76 ↑12.93	36.08 ↑16.29
VolumeProbe - Lite	75.63 ↑19.80	45.71 ↑25.92

Appendix

Table 14: Occupancy ratios of sampled sub-volumes. Foreground occupancy measures the average foreground voxel ratio within sampled crops, while organ occupancy measures the average organ-mask voxel ratio on the mask-available subset.

Table 23

Variant	Anatomy-aware		Informative		Dice Score (%) $\uparrow$
Variant	LocPrior	Margin	Grad	Entropy	BTCV	AMOS	WORD
VoCo [ 68 ]	–	–	–	–	83.85	84.44	84.06
+ VolumeProbe	✓	✓	✓	✓	83.97 ↑0.12	87.52 ↑3.08	85.56 ↑1.50
w/o LocPrior	$\times$	✓	✓	✓	83.90 ↑0.05	85.60 ↑1.16	84.21 ↑0.15
w/o Margin	✓	$\times$	✓	✓	83.87 ↑0.02	86.73 ↑2.29	85.40 ↑1.34
w/o Grad	✓	✓	$\times$	✓	83.91 ↑0.06	87.13 ↑2.69	85.36 ↑1.30

Appendix

Table 17: Fine-grained ablation of principle terms for VoCo [ 68 ] . We remove each term from the anatomy-aware or informativeness score while keeping the remaining pipeline unchanged.

Coarse Bins Partition	Dice Score (%) $\uparrow$
Coarse Bins Partition	BTCV	AMOS	WORD
$(G_{x},G_{y},G_{z})$
$(3,3,1)$	81.90	84.46	83.80
$(3,3,2)$	82.57	84.51	83.71
$(4,4,1)$	83.40	86.27	85.09
$(4,4,2)$	83.97	87.52	85.56

Appendix

Table 18: Ablation study of coarse-bin partition strategy for VoCo [ 68 ] .

Table 26

Table 27

Figure 11: Convergence under extended pretraining. Downstream Dice of VoCo with Random sampling and VolumeProbe from 100k to 160k steps. VolumeProbe consistently outperforms Random ; the gap narrows on WORD 1% but remains stable elsewhere, and VolumeProbe @100k still exceeds Random @160k in all settings.

Pretraining FOV	Sampling Policy	Foreground Occ. (%) $\uparrow$	Organ Occ. (%) $\uparrow$	AMOS 10% Dice (%) $\uparrow$
Canonical	Random	55.83	19.79	73.34
Canonical	VolumeProbe - Lite	75.63	45.71	74.95
Shifted/local FOV	Random	47.62	15.87	72.08
	VolumeProbe - Lite	54.12	25.13	72.40
	VolumeProbe -I+D	60.94	31.13	73.71

Appendix

Table 23: Robustness to non-canonical fields of view. Foreground occupancy, organ occupancy, and AMOS 10% Dice Score (%) are reported.

Method	Offline Construction	Cache Footprint	Per-epoch Time	Total Pretraining Time	Overhead
	Time (h)	(MB)	(min)	(GPU h)	vs. Random
VoCo [ 68 ]	–	–	392.6	269.3	–
+ VolumeProbe - Lite	3.79	41.5	406.8	281.1	+4.4%
+ VolumeProbe - Feat	9.26	41.5	406.8	281.1	+4.4%
S2DC [ 48 ]	–	–	1089.9	753.3	–
+ VolumeProbe - Lite	3.79	41.5	1137.4	768.9	+2.1%

Appendix

Table 24: Efficiency comparison between random sampling and VolumeProbe . For each SSL backbone, we report the probe-bank construction time, cache footprint, per-epoch pretraining time, total pretraining time, and relative overhead compared with Random under the same training settings.

Figure 12: Additional cases for anatomy-aware sampling. We compare Random , VolumeProbe - Lite , and VolumeProbe - Feat across three volumes. For each case, we show 3D crop boxes overlaid with organ masks, coronal crop-center density, sampled crop centers, 2D crop boxes, and example cropped views. Both VolumeProbe variants sample more anatomically valid and organ-rich regions than Random , leading to higher foreground and organ occupancy ratios.

Figure 13: Cross-epoch trajectory analysis. We visualize sampled crop centers across epochs for two representative cases under Random , VolumeProbe - Lite , and VolumeProbe - Feat on sagittal, coronal, and axial views. The light-to-dark trajectory denotes epoch progression, and the inset reports path length, unique bins, average revisit lag, and same-bin ratio.

Figure 14: Quantitative analysis of structure-aligned feature responses. We evaluate Random and VolumeProbe at 5k, 10k, and 20k early pretraining iterations using feature responses extracted from the same encoder stage. We report feature-body alignment, feature-organ alignment, and boundary alignment, with shaded regions indicating standard deviation across sampled sub-volumes. VolumeProbe consistently achieves higher alignment scores than Random , supporting the qualitative feature-map observations in Fig. 8 (c).

Medical Image Representation Learning

Medical image representation learning builds reusable visual features for diagnosis, segmentation, retrieval, and other clinical imaging tasks.

See all 260 papers

Learning Generalizable 3D Medical Image Representations from Mask-Guided Self-Supervision
Mar 14, 2026Yunhe Gao, Yabin Zhang, Chong Wang +5Medical Image Representation Learning Self-Supervised Pre-Training
Stanford University
Task-Aligned Self-Supervised Learning for Medical Image Analysis: A Task-Oriented Review with Practical Design Guidelines
May 18, 2026Chathura Wimalasiri, Kishor Nandakishor, Marimuthu PalaniswamiContrastive Learning Self-Supervised Learning
Department of Electrical and Electronic Engineering, University of Melbourne, Parkville, VIC 3052, Australia
VolTA-3D: Self-Supervised Learning for Brain MRI using 3D Volumetric Token Alignment
May 16, 2026Amy Makawana, Abhijeet Parida, Marius George Linguraru +23D ViTs Self-Supervised Pre-Training
Institute of Health Informatics, University College London, London, UK · Sheikh Zayed Institute for Pediatric Surgical Innovation, Children’s National Hospital, Washington, DC USA · School of Medicine and Health Sciences, George Washington University, Washington, DC, USA

Mar 14, 2026J/K move · Enter open · S save

Learning Generalizable 3D Medical Image Representations from Mask-Guided Self-Supervision

Yunhe Gao, Yabin Zhang, Chong Wang, Jiaming Liu+4

Stanford University

Medical Image Representation Learning Self-Supervised Pre-Training 3D Medical Imaging Medical Imaging Foundation Models Few-Shot Medical Image Segmentation

Open paper AskPDF arXiv

Foundation models have transformed vision and language by learning general-purpose representations from large-scale unlabeled data, yet 3D medical imaging lacks analogous approaches. Existing self-supervised methods rely on low-level reconstruction or contrastive objectives that fail to capture the anatomical semantics critical for medical image analysis, limiting transfer to downstream tasks. We present MASS (MAsk-guided Self-Supervised learning), which treats in-context segmentation as the pretext task for learning general-purpose medical imaging representations. MASS's key insight is that automatically generated class-agnostic masks provide sufficient structural supervision for learning semantically rich representations. By training on thousands of diverse mask proposals spanning anatomical structures and pathological findings, MASS learns what semantically defines medical structures: the holistic combination of appearance, shape, spatial context, and anatomical relationships. We demonstrate effectiveness across data regimes: from small-scale pretraining on individual datasets (20-200 scans) to large-scale multi-modal pretraining on 5K CT, MRI, and PET volumes, all without annotations. MASS demonstrates: (i) few-shot segmentation on novel structures, (ii) matching full supervision with only 20-40% labeled data while outperforming self-supervised baselines by over 20 in Dice score in low-data regimes, and (iii) frozen-encoder classification on unseen pathologies that matches full supervised training with thousands of samples. Mask-guided self-supervised pretraining captures broadly generalizable knowledge, opening a path toward 3D medical imaging foundation models without expert annotations. Code is available: https://github.com/Stanford-AIMI/MASS.

Abstract

Medical Image Representation Learning

Explore similar work

Learning Generalizable 3D Medical Image Representations from Mask-Guided Self-Supervision

Task-Aligned Self-Supervised Learning for Medical Image Analysis: A Task-Oriented Review with Practical Design Guidelines

VolTA-3D: Self-Supervised Learning for Brain MRI using 3D Volumetric Token Alignment