Dataset-Specific Encoding of Image Content Underlies Cross-Dataset Failure in Brain MRI Foundation Models

Minh Sao Khue Luu; Bair N. Tuchinov

cs.CVOct 9, 2026

Dataset-Specific Encoding of Image Content Underlies Cross-Dataset Failure in Brain MRI Foundation Models

Authors: Minh Sao Khue Luu, Bair N. Tuchinov

Organizations: The Artificial Intelligence Research Center of Novosibirsk State University, Novosibirsk 630090, Russia

Abstract

Frozen features of brain MRI foundation models lose accuracy on new datasets, a failure usually blamed on the features encoding where each scan came from. We show that the failure lies elsewhere. Across 17 public datasets and eight frozen representations, every representation recognizes the MRI sequence within each dataset, yet most fail on an unseen one. The cause is neither dataset encoding nor a shift of each dataset: the content itself is encoded along different directions in each dataset. This explains which datasets fail, for the sequence and for sex, and why removing the dataset signal with INLP, LEACE, CORAL or adversarial training never improves sequence transfer. Only the broadly pretrained 3DINO and SAM-Med3D encode the sequence consistently enough to transfer; the brain-specific BrainIAC and BrainFM do no better than an untrained network. Code is available at https://github.com/BrainFM/brainfm-profiler/tree/main/representation_utility.

Figures & tables

Dataset	Sequences	Sampled
Brain tumor
BraTS25-MEN [ 40 , 41 , 42 ]	T1, T1c, T2, FLAIR	300
BraTS25-MET [ 43 ]	T1, T1c, T2, FLAIR	300
BraTS25-SSA [ 44 , 45 ]	T1, T1c, T2, FLAIR	300
BrainMetShare [ 46 ]	T1, T1c, FLAIR	300
NSK-GBM [ 47 ]	T1, T1c, T2, FLAIR	168

Table 1: The 17 public brain MRI datasets, grouped by domain. Sampled : number of volumes used in this study.

Representation	Architecture	Pretraining	Dim.
Baseline encoders
Handcrafted	—	none (image statistics)	26
Untrained CNN	DenseNet-121 [ 62 ]	none (random weights)	1024
MedicalNet [ 63 ]	ResNet-34 [ 64 ]	supervised segmentation, MRI and CT	512
SwinUNETR [ 65 ]	Swin	self-supervised, CT [ 66 ]	768
Brain-specific foundation models

Table 2: Frozen representations compared in this study. Dim.: feature dimension.

Encoder	Input size	Intensity scaling
Untrained CNN, MedicalNet, SwinUNETR, SAM-Med3D	$128^{3}$	$z$ -score, non-zero voxels
BrainIAC	$96^{3}$	$z$ -score, non-zero voxels
3DINO	$112^{3}$	0.05–99.95th percentile to $[-1,1]$
BrainFM	$128^{3}$	min–max to $[0,1]$
Handcrafted	$128^{3}$	none

Table 3: Encoder-specific input size and intensity scaling, applied after the shared geometry pipeline.

		Decodable			Across datasets
Group	Representation	Dataset	Sequence	Seq. within	10 multi-seq.	7 single-seq.
Baseline encoders	Handcrafted	0.92	0.72	0.85	0.44	0.46
	Untrained CNN	0.94	0.88	0.93	0.49	0.73
	MedicalNet	0.94	0.87	0.89	0.44	0.58
	SwinUNETR	0.91	0.83	0.91	0.47	0.67
Brain-specific FMs	BrainIAC	0.84	0.76	0.84	0.50	0.53

Table 4: Properties encoded by each frozen representation. Decodable : balanced accuracy of patient-level cross-validation for the source dataset and the sequence on the pooled corpus (chance 0.06 and 0.25; all outside the permutation null,

p<0.01

), and for the sequence within each multi-sequence dataset (mean over ten datasets). Across datasets : leave-one-dataset-out sequence prediction, as balanced accuracy averaged over the ten multi-sequence held-out datasets and as accuracy averaged over the seven single-sequence datasets.

Figure 1: Leave-one-dataset-out transfer of the imaging sequence. Balanced accuracy on each of the ten held-out datasets with more than one sequence (dots) and their mean (bar and number). Grey: baseline encoders; orange: brain-specific foundation models; blue: broadly pretrained medical foundation models; hatched: controls.

Figure 2: Source-dataset decodability (balanced accuracy, 17 classes) against cross-dataset sequence transfer (leave-one-dataset-out balanced accuracy on the ten multi-sequence held-out datasets). Grey: baseline encoders; orange: brain-specific foundation models; blue: broadly pretrained medical foundation models; open circle: 3DINO with random weights.

Figure 3: Sequence-direction consistency and transfer. (a) Consistency of each held-out multi-sequence dataset with the training datasets against transfer to that dataset, for every representation (filled) and control (open). (b) Mean transfer before and after centring each dataset’s features, label-free (squares) or class-balanced using the held-out labels (oracle, diamonds). Grey: baseline encoders; orange: brain-specific foundation models; blue: broadly pretrained medical foundation models.

Representation	Consistency	Before	Centring	LEACE	CORAL
Handcrafted	0.34	0.51	0.61 ∗	0.52 (0.51)	0.65 ∗ (0.52)
Untrained CNN	0.65	0.64	0.64	0.63 (0.65)	0.70 (0.59)
MedicalNet	0.49	0.65	0.70	0.62 ∗ (0.62)	0.67 (0.58)
SwinUNETR	$-$ 0.05	0.60	0.60	0.60 (0.60)	0.63 (0.60)
BrainIAC	0.13	0.54	0.54	0.54 (0.55)	0.53 (0.53)
BrainFM	0.40	0.65	0.68	0.65 (0.64)	0.69 (0.57)

Table 5: Sex as a second content label. Leave-one-dataset-out balanced accuracy averaged over the ten held-out datasets with sex labels; consistency of the held-out male–female direction with the training datasets. Control in brackets: a random projection of the same rank (LEACE) or alignment to a wrong dataset (CORAL). ∗ : differs from before,

p<0.05

, paired Wilcoxon test.

Representation	Before	INLP	LEACE	CORAL	Adversarial
Handcrafted	0.44	0.41 (0.44)	0.42 (0.40)	0.37 (0.27)	0.42 (0.43)
Untrained CNN	0.49	0.48 (0.47)	0.41 ∗ (0.44)	0.42 (0.28)	0.45 (0.44)
MedicalNet	0.44	0.44 (0.44)	0.43 (0.42)	0.42 (0.27)	0.40 (0.45)
SwinUNETR	0.47	0.41 ∗ (0.45)	0.44 (0.40)	0.45 (0.30)	0.36 (0.41)
BrainIAC	0.50	0.48 (0.50)	0.44 ∗ (0.48)	0.43 (0.25)	0.40 ∗ (0.45)
BrainFM	0.55	0.57 (0.55)	0.54 (0.53)	0.48 (0.33)	0.50 (0.51)

Table 6: Sequence transfer before and after removing the dataset signal. Leave-one-dataset-out balanced accuracy averaged over the ten multi-sequence held-out datasets; methods fitted on the training datasets of each split. Control in brackets: a random projection of the same rank (INLP, LEACE), alignment to a wrong dataset (CORAL), an untrained network of the same shape (adversarial). ∗ : after differs from before,

p<0.05

, paired Wilcoxon test over the ten datasets.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

	Before		Dataset after				Sequence after
Representation	Dataset	Seq.	INLP	LEACE	CORAL	Adv.	INLP	LEACE	Rand.	CORAL	Adv.
Handcrafted	0.92	0.73	0.88	0.84	0.14	0.88	0.68	0.56	0.67	0.54	0.71
Untrained CNN	0.94	0.88	0.89	0.26	0.11	0.90	0.83	0.71	0.87	0.62	0.78
MedicalNet	0.94	0.86	0.91	0.10	0.02	0.87	0.80	0.67	0.81	0.58	0.70
SwinUNETR	0.91	0.83	0.87	0.10	0.03	0.85	0.78	0.59	0.79	0.56	0.69
BrainIAC	0.84	0.76	0.75	0.03	0.01	0.72	0.71	0.57	0.72	0.49	0.59

Appendix

Table 7: Dataset and sequence decodability inside the training datasets, before and after each removal method, averaged over the leave-one-dataset-out splits (balanced accuracy). Chance: 0.06 for the dataset and 0.25 for the sequence. Adv.: adversarial. Rand.: removing 15 random directions, the control for LEACE.

Score	Within representation	Pooled	Across representations
Average confidence [ 68 ]	0.03	0.26	0.91
ATC [ 69 ]	0.01	0.11	0.68
Cluster consistency (ours)	0.37	0.27	0.11

Appendix

Table 8: Label-free prediction of sequence transfer to a held-out dataset. Spearman correlation between each score and the measured balanced accuracy, across the ten held-out datasets within each representation (mean over 11 representations and controls), across all 110 representation–dataset pairs, and across the 11 representation means. ATC also misestimates the balanced accuracy by 0.25 on average.

Explore similar work

BrainDINO: A Brain MRI Foundation Model for Generalizable Clinical Representation Learning
Apr 30, 2026Yizhou Wu, Shansong Wang, Yuheng Li +5Medical Imaging Foundation Models Self-Supervised Pre-Training
Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA, USA. · Department of Radiation and Cellular Oncology, The University of Chicago, Chicago, IL, USA. · Department of Biomedical Engineering, Georgia Institute of Technology, Atlanta, GA, USA. +2
A continually expandable foundation model for brain MRI
Aug 8, 2026Michail Mamalakis, Carmen Jimenez-Mesa, Yonghao Li +8Medical Imaging Foundation Models Continual Learning
1*Cancer Research UK Cambridge Institute, University of Cambridge Li Ka Shing Centre, Robinson Way, Cambridge, CB2 0RE, Cambridgeshire, United Kingdom. · Department of Computer Science and Technology, University of Cambridge, 15 JJ Thomson Ave, Cambridge, CB3 0FD, Cambridgeshire, United Kingdom. · Department of Psychiatry, University of Cambridge, Hills Road, Cambridge, CB2 2QQ, Cambridgeshire, United Kingdom. +7
Site Is Decodable Before Pretraining: Negative Controls for Probing Frozen Brain-MRI Foundation Models
Aug 10, 2026Saman RahbarMedical Imaging Medical Imaging Foundation Models
University of British Columbia, Vancouver, Canada

Apr 30, 2026J/K move · Enter open · S save

BrainDINO: A Brain MRI Foundation Model for Generalizable Clinical Representation Learning

Yizhou Wu, Shansong Wang, Yuheng Li, Mojtaba Safari+4

Department of Radiation Oncology and Winship Cancer Institute, Emory University, Atlanta, GA, USA. · Department of Radiation and Cellular Oncology, The University of Chicago, Chicago, IL, USA. · Department of Biomedical Engineering, Georgia Institute of Technology, Atlanta, GA, USA.+2

Medical Imaging Foundation Models Self-Supervised Pre-Training Medical Image Representation Learning Brain MRI

Open paper AskPDF arXiv

Brain MRI underpins a wide range of neuroscientific and clinical applications, yet most learning-based methods remain task-specific and require substantial labeled data. Here we show that a single self-supervised representation can generalize across heterogeneous brain MRI endpoints. We trained BrainDINO, a self-distilled foundation model, on approximately 6.6 million unlabeled axial slices from 20 datasets encompassing broad variation in population, disease, and acquisition setting. Using a frozen encoder with lightweight task heads, BrainDINO supported transfer across tumor segmentation, neurodegenerative and neurodevelopmental conditions classification, brain age estimation, post-stroke temporal prediction, molecular status prediction, MRI sequence classification, and survival modeling. Across tasks and supervision regimes, BrainDINO consistently equaled or exceeded natural-image and MRI-specific self-supervised baselines, with particularly strong advantages under label scarcity. Representation analyses further showed anatomically organized and pathology-sensitive feature structure in the absence of task-specific supervision. Our findings indicate that large-scale slice-wise self-supervised learning can yield a unified brain MRI representation that supports diverse neuroimaging tasks without volumetric pretraining or full-network fine-tuning, establishing a scalable foundation for robust and data-efficient brain imaging analysis. Code is available at https://github.com/mclwu22/BrainDINO

Abstract

Explore similar work

BrainDINO: A Brain MRI Foundation Model for Generalizable Clinical Representation Learning

A continually expandable foundation model for brain MRI

Site Is Decodable Before Pretraining: Negative Controls for Probing Frozen Brain-MRI Foundation Models