Structure is Supervision: Multiview Masked Autoencoders for Radiology

Sonia Laguna; Andrea Agostini; Alain Ryser; Samuel Ruiperez-Campillo; Irene Cannistraci; Moritz Vandenhirtz; Stephan Mandt; Nicolas Deperrois

cs.CVNov 27, 2025

Structure is Supervision: Multiview Masked Autoencoders for Radiology

Authors: Sonia Laguna, Andrea Agostini, Alain Ryser, Samuel Ruiperez-Campillo, Irene Cannistraci, Moritz Vandenhirtz, Stephan Mandt, Nicolas Deperrois,

Organizations: Department of Computer Science, ETH Zurich · Department of Computer Science, UC Irvine · Department of Quantitative Biomedicine, University of Zurich

Abstract

Building robust medical machine learning systems requires pretraining strategies that exploit the intrinsic structure present in clinical data. We introduce Multiview Masked Autoencoder (MVMAE), a self-supervised framework that leverages the natural multi-view organization of radiology studies to learn view-invariant and disease-relevant representations. MVMAE combines masked image reconstruction with cross-view alignment, transforming clinical redundancy across projections into a powerful self-supervisory signal. We further extend this approach with MVMAE-V2T, which incorporates radiology reports as an auxiliary text-based learning signal to enhance semantic grounding while preserving fully vision-based inference. Evaluated on a downstream disease classification task on three large-scale public datasets, MIMIC-CXR, CheXpert, and PadChest, MVMAE consistently outperforms supervised and vision-language baselines. Furthermore, MVMAE-V2T provides additional gains, particularly in low-label regimes where structured textual supervision is most beneficial. Together, these results establish the importance of structural and textual supervision as complementary paths toward scalable, clinically grounded medical foundation models.

Figures & tables

Figure 1: Framework overview . (a) Pretraining and (b) Finetuning stages of the proposed MVMAE and MVMAE-V2T frameworks. Each study includes multiple views processed jointly through masked reconstruction and cross-view alignment losses. MVMAE-V2T additionally incorporates a vision-to-text objective.

Figure 2: Study-level instance in MIMIC-CXR . The first row shows three different views in the same study, with their corresponding report and final labels.

Panel of Figure 3 — Figure 3: Per-label AUROC comparison across datasets (MIMIC-CXR, CheXpert Plus, PadChest). Left: overall performance with model selection done based on the best joint dataset. Right: model selection done to optimize per label results.

Panel of Figure 4 — Figure 4: Label efficiency under finetuning . Performance curves of the Combined dataset macro-average AUROC over 14 pathology labels, as a function of the number of labeled studies used for finetuning.

Model	Backbone	MIMIC-CXR		CheXpert		PadChest		Combined
Model	Backbone	AUROC	F1	AUROC	F1	AUROC	F1	AUROC	F1
Supervised	ViT-B/16	79.25 $\pm$ 0.42	25.30 $\pm$ 0.36	76.39 $\pm$ 0.95	23.77 $\pm$ 0.66	85.72 $\pm$ 0.28	27.16 $\pm$ 1.04	84.83 $\pm$ 0.25	29.33 $\pm$ 0.66
Independent	ViT-B/16	81.28 $\pm$ 0.15	29.16 $\pm$ 1.19	79.65 $\pm$ 1.67	30.58 $\pm$ 3.05	88.17 $\pm$ 0.29	33.22 $\pm$ 0.22	86.60 $\pm$ 0.04	35.48 $\pm$ 0.18
BiomedCLIP	ViT-B/16	78.88 $\pm$ 0.88	23.93 $\pm$ 2.29	76.72 $\pm$ 4.11	24.39 $\pm$ 1.87	84.87 $\pm$ 0.89	27.53 $\pm$ 3.51	83.97 $\pm$ 0.78	29.02 $\pm$ 3.36
CheXagent	ViT-L/14	80.65 $\pm$ 0.13	28.88 $\pm$ 3.60	83.02 $\pm$ 0.22	33.54 $\pm$ 4.81	88.31 $\pm$ 0.05	28.39 $\pm$ 3.69	86.25 $\pm$ 0.04	33.56 $\pm$ 0.70
MVMAE-V2T (ours)	ViT-B/16	81.20 $\pm$ 0.16	31.81 $\pm$ 6.12	80.87 $\pm$ 0.35	35.96 $\pm$ 9.35	88.27 $\pm$ 0.29	37.06 $\pm$ 6.74	86.74 $\pm$ 0.29	38.96 $\pm$ 6.59

Table 1: Classification performance across datasets . Comparison of MVMAE and baseline methods on the full labeled datasets. The table reports AUROC and F1 (mean

\pm

std in three seeds) averaged over the fourteen labels per dataset, along with a combined metric averaging performance across all three. ViT-B/16 stands for ViT-Base architecture with 16 patch size, and ViT-L/14 for ViT-Large with 14 patch size.

Model	Backbone	MIMIC-CXR		CheXpert		PadChest		Combined
Model	Backbone	AUROC	F1	AUROC	F1	AUROC	F1	AUROC	F1
Supervised	ViT-B/16	82.20 $\pm$ 3.18	38.84 $\pm$ 5.87	78.32 $\pm$ 6.03	45.47 $\pm$ 1.05	84.10 $\pm$ 5.47	29.45 $\pm$ 5.21	86.16 $\pm$ 3.14	38.09 $\pm$ 6.26
Independent	ViT-B/16	86.27 $\pm$ 1.66	50.18 $\pm$ 3.68	83.49 $\pm$ 2.27	46.17 $\pm$ 4.82	89.55 $\pm$ 3.13	42.32 $\pm$ 3.29	88.99 $\pm$ 1.90	50.80 $\pm$ 3.61
Chexagent	ViT-L/14	87.39 $\pm$ 0.98	52.28 $\pm$ 1.28	88.14 $\pm$ 0.52	55.30 $\pm$ 3.72	92.10 $\pm$ 0.50	40.74 $\pm$ 2.14	90.64 $\pm$ 0.48	53.37 $\pm$ 0.92
MVMAE-V2T (ours)	ViT-B/16	89.10 $\pm$ 0.15	57.83 $\pm$ 1.61	86.76 $\pm$ 0.41	54.14 $\pm$ 1.63	91.53 $\pm$ 0.56	43.38 $\pm$ 0.34	91.65 $\pm$ 0.05	57.47 $\pm$ 0.94
MVMAE (ours)	ViT-B/16	89.10 $\pm$ 0.08	55.01 $\pm$ 8.84	86.87 $\pm$ 1.48	52.34 $\pm$ 3.30	92.47 $\pm$ 0.95	40.41 $\pm$ 6.86	91.78 $\pm$ 0.10	55.32 $\pm$ 7.53

Table 2: Top-5 classification performance across datasets . Comparison of MVMAE and baseline methods on the full labeled datasets. The table reports Top-5 (labels) average AUROC and F1 (mean

\pm

std over three seeds) per dataset, along with a combined metric averaging performance across all three.

Model	MIMIC-CXR	CheXpert	PadChest	Combined
Supervised	0.0885 $\pm$ 0.0005	0.0955 $\pm$ 0.0010	0.0718 $\pm$ 0.0003	0.0815 $\pm$ 0.0004
Independent	0.0915 $\pm$ 0.0005	0.0963 $\pm$ 0.0016	0.0750 $\pm$ 0.0005	0.0845 $\pm$ 0.0003
BiomedCLIP	0.0924 $\pm$ 0.0052	0.0980 $\pm$ 0.0061	0.0769 $\pm$ 0.0053	0.0859 $\pm$ 0.0053
MVMAE-V2T (ours)	0.0964 $\pm$ 0.0015	0.1017 $\pm$ 0.0014	0.0825 $\pm$ 0.0022	0.0905 $\pm$ 0.0018
MVMAE (ours)	0.0903 $\pm$ 0.0006	0.0974 $\pm$ 0.0008	0.0743 $\pm$ 0.0008	0.0835 $\pm$ 0.0007

Table 3: Brier score under linear probing on 20K samples on the same encoder-type models, lower is better.

Figure 5: Average AUROC performance gain of MVMAE on Independent over three seeds.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Study-level structure representation . Each row illustrates a representative study with three distinct views (frontal, lateral, and unknown), accompanied by corresponding report excerpts (text) and the final diagnostic labels (e.g., Edema, Lung Opacity, Fracture, Pneumonia) for the different datasets (MIMIC CXR, CheXPert, Chest X-Ray and PadChest). We treat each study as the fundamental unit of analysis, and various combinations of views are possible (e.g., including various frontal images (rows $p$ and $q$ ), or missingness (row $p$ )). For Chest X-Ray dataset, no lateral or unknown views were available, nor text reports or labels.

Figure 7: Illustration of study acquisition . For a single study, various views could be acquired at different frequencies. This figure illustrates an example of a study composed of three lateral view images and two frontal ones. With these, the radiologist writes a report from which labels are extracted.

Dataset	Frontal (F)	Lateral (L)	Unknown (U)
MIMIC-CXR	239,931	116,555	15,465
CheXpert+	191,071	32,391	–
PadChest	62,901	27,526	68,095
ChestX-ray8	86,524	–	–

Appendix

Table 4: Distribution of radiographic views across datasets.

	# Studies			#	#	# Original	Original
	Train	Val	Test	Patients	Images	Labels	Image Size
MIMIC-CXR	222,758	1,808	3,269	65,086	371,951	14	$2500\times 3056$
CheXpert	187,511	200	500	64,725	223,462	14	$320\times 320$
PadChest	106,677	1,653	1,601	66,610	158,522	193	$256\times 256$
Chest X-ray	84,774	1,750	25,596	30,805	86,524	15	$1024\times 1024$

Appendix

Table 5: Dataset Specifications . A detailed breakdown of each dataset, including study counts for training, validation, and test splits, along with patient and image totals, label counts, and original image dimensions.

Figure 8: Label Distribution. Statistics for the MIMIC-CXR, CheXpert, and PadChest datasets, as well as for the Combined dataset in the training and validation splits. This figure illustrates how the frequency of each diagnostic label varies across individual datasets and the overall collection.

Figure 9: Binary Label Distribution. Statistics for the MIMIC-CXR, CheXpert, and PadChest datasets, as well as for the Combined dataset in the training and validation splits, after label binarization. This figure illustrates how the frequency of each diagnostic label varies across individual datasets and the overall collection.

Label	Supervised	Independent	BiomedCLIP	CheXagent	MVMAE (ours)
Backbone	ViT-B/16	ViT-B/16	ViT-B/16	ViT-L/14	ViT-B/16
Atelectasis	0.81	0.82	0.81	0.81	0.82
Cardiomegaly	0.80	0.81	0.80	0.81	0.82
Consolidation	0.81	0.82	0.80	0.81	0.83
Edema	0.88	0.89	0.88	0.89	0.89
Enlarged C.	0.68	0.73	0.74	0.71	0.74

Appendix

Table 6: Per-label classification AUROC on MIMIC-CXR . Results are shown for all compared models.

Label	Supervised	Independent	BiomedCLIP	CheXagent	MVMAE (ours)
Backbone	ViT-B/16	ViT-B/16	ViT-B/16	ViT-L/14	ViT-B/16
Atelectasis	0.76	0.81	0.74	0.73	0.78
Cardiomegaly	0.91	0.92	0.94	0.89	0.90
Consolidation	0.76	0.75	0.82	0.79	0.77
Edema	0.86	0.86	0.83	0.88	0.87
Enlarged C.	0.88	0.86	0.87	0.85	0.75

Appendix

Table 7: Per-label classification AUROC on CheXpert . Results are shown for all compared models.

Label	Supervised	Independent	BiomedCLIP	CheXagent	MVMAE (ours)
Backbone	ViT-B/16	ViT-B/16	ViT-B/16	ViT-L/14	ViT-B/16
Atelectasis	0.80	0.85	0.83	0.88	0.89
Cardiomegaly	0.91	0.92	0.91	0.92	0.92
Consolidation	0.88	0.91	0.88	0.91	0.91
Edema	0.91	0.94	0.93	0.93	0.95
Enlarged C.	0.81	0.84	0.81	0.80	0.84

Appendix

Table 8: Per-label classification AUROC on PadChest . Results are shown for all compared models.

Panel of Figure 10 — Figure 10: Per-label AUROC comparison across datasets for all models not shown in the main text.

Panel of Figure 11 — Figure 11: Label efficiency curves on Mimic-CXR, Chexpert Plus, and PadChest showing macro-AUROC across varying labeled data sizes.

Model	Setting	AUROC
Model	Setting	MIMIC-CXR	CheXpert	PadChest	Combined
MVMAE	One view	72.43	53.74	71.49	76.63
MVMAE	Two views (all)	75.22	57.33	77.38	79.50
Independent	One view	72.43	52.28	70.92	76.32
Independent	Two views (all)	74.46	57.21	76.96	79.23

Appendix

Table 9: Comparison of MVMAE and Independent models under single- and dual-view evaluation using 5k-sample linear probing. Both models were evaluated only on studies containing two available views to ensure a consistent comparison. Results show the mean macro-average AUROC across three random seeds.

Medical Image Representation Learning

Medical image representation learning builds reusable visual features for diagnosis, segmentation, retrieval, and other clinical imaging tasks.

See all 260 papers

Learning Generalizable 3D Medical Image Representations from Mask-Guided Self-Supervision
Mar 14, 2026Yunhe Gao, Yabin Zhang, Chong Wang +5Medical Image Representation Learning Self-Supervised Pre-Training
Stanford University
GreenRFM: Learning a resource-efficient radiology vision-language foundation model via supervision-centric pre-training
Mar 6, 2026Yingtai Li, Shuai Ming, Qiuli Wang +13Medical Imaging Foundation Models Radiology VLMs
School of Biomedical Engineering, Division of Life Sciences and Medicine, University of Science and Technology of China (USTC), Hefei, Anhui, 230026, China · Center for Medical Imaging, Robotics, Analytic Computing & Learning (MIRACLE), Suzhou Institute for Advanced Research, USTC, Suzhou, Jiangsu, 215123, China · Department of Radiology, The First Affiliated Hospital of USTC, Division of Life Sciences and Medicine, USTC, Hefei, Anhui, 230001, China +7
Masked and Predictive Self-Supervised Pretraining for Brain MRI: Effects of Objective Design on Disease Detection
Jun 11, 2026Esra Ergün, Hersh Chandarana, Dan Sodickson +1Medical Imaging Foundation Models Masked Autoencoders
Department of Computer Engineering, Istanbul Technical University, Istanbul, 34469, Türkiye · Department of Radiology, NYU Grossman School of Medicine, New York, NY, 10016, United States · Department of AI and Data Engineering, Istanbul Technical University, Istanbul, 34469, Türkiye

Mar 14, 2026J/K move · Enter open · S save

Learning Generalizable 3D Medical Image Representations from Mask-Guided Self-Supervision

Yunhe Gao, Yabin Zhang, Chong Wang, Jiaming Liu+4

Stanford University

Medical Image Representation Learning Self-Supervised Pre-Training 3D Medical Imaging Medical Imaging Foundation Models Few-Shot Medical Image Segmentation

Open paper AskPDF arXiv

Foundation models have transformed vision and language by learning general-purpose representations from large-scale unlabeled data, yet 3D medical imaging lacks analogous approaches. Existing self-supervised methods rely on low-level reconstruction or contrastive objectives that fail to capture the anatomical semantics critical for medical image analysis, limiting transfer to downstream tasks. We present MASS (MAsk-guided Self-Supervised learning), which treats in-context segmentation as the pretext task for learning general-purpose medical imaging representations. MASS's key insight is that automatically generated class-agnostic masks provide sufficient structural supervision for learning semantically rich representations. By training on thousands of diverse mask proposals spanning anatomical structures and pathological findings, MASS learns what semantically defines medical structures: the holistic combination of appearance, shape, spatial context, and anatomical relationships. We demonstrate effectiveness across data regimes: from small-scale pretraining on individual datasets (20-200 scans) to large-scale multi-modal pretraining on 5K CT, MRI, and PET volumes, all without annotations. MASS demonstrates: (i) few-shot segmentation on novel structures, (ii) matching full supervision with only 20-40% labeled data while outperforming self-supervised baselines by over 20 in Dice score in low-data regimes, and (iii) frozen-encoder classification on unseen pathologies that matches full supervised training with thousands of samples. Mask-guided self-supervised pretraining captures broadly generalizable knowledge, opening a path toward 3D medical imaging foundation models without expert annotations. Code is available: https://github.com/Stanford-AIMI/MASS.

Abstract

Medical Image Representation Learning

Explore similar work

Learning Generalizable 3D Medical Image Representations from Mask-Guided Self-Supervision

GreenRFM: Learning a resource-efficient radiology vision-language foundation model via supervision-centric pre-training

Masked and Predictive Self-Supervised Pretraining for Brain MRI: Effects of Objective Design on Disease Detection