Natural Image Autoencoder-Based fMRI Representations for Trait and State Prediction

Juhyeon Park; Yeonwoo Kim; Peter Yongho Kim; Yansen Wang; Mingqing Xiao; Dongqi Han; Dongsheng Li; Taesup Moon

cs.CVSep 28, 2026

Natural Image Autoencoder-Based fMRI Representations for Trait and State Prediction

Authors: Juhyeon Park, Yeonwoo Kim, Peter Yongho Kim, Yansen Wang, Mingqing Xiao, Dongqi Han, Dongsheng Li, Taesup Moon

Organizations: IPAI, Seoul National University · Microsoft Research · ECE, Seoul National University · ASRI / INMC / AIIS, Seoul National University

Abstract

Foundation models pre-trained on large-scale fMRI datasets have shown strong downstream performance, but at substantial data and computation cost. To investigate how much fMRI-specific pre-training is actually needed for such performance, we introduce FReD, which derives fMRI representations from a frozen Deep Compression AutoEncoder (DCAE) pre-trained exclusively on natural images and pairs them with a task specific readout. For trait prediction, FReD summarizes frame-wise representations by their temporal mean and log-standard deviation and applies linear probing, with late fusion across two normalization schemes. For state prediction, it represents each frame as a single token and models temporal dependencies with a shallow Transformer. Across four resting-state datasets spanning six trait-prediction targets, linear probes on frozen DCAE features generally outperform those on fMRI foundation model representations and remain competitive with fully fine-tuned fMRI foundation models. On three task-fMRI state-prediction tasks, a temporal readout on DCAE features performs comparably to the strongest foundation models evaluated. A Gaussian injection analysis further shows that localized signal changes are recovered more accurately from the frozen DCAE features than from the evaluated foundation-model representations. Together, these results show that strong performance on current fMRI benchmarks is possible without fMRI-specific representation pre-training, making frozen natural-image features as a useful baseline for assessing its added value.

Figures & tables

Figure 1: Overview of FReD. A frozen DCAE pre-trained on natural images constructs a frame representation for each fMRI frame, for both $\mathbf{X}^{V}$ and $\mathbf{X}^{F}$ . For trait prediction, the frame representations are summarized by their temporal mean and log-standard deviation and read out by a linear predictor, and the predictions obtained under the two normalization schemes $n\in\{g,cf\}$ are combined by late fusion. For state prediction, the frame representations obtained under coordinate–frame normalization are projected into a common embedding space and processed by a Transformer.

Figure 2: Linear-probing results on trait prediction tasks over 500 random splits, reported as AUC/

\rho

Method	Input	ADHD-200		ABIDE-II		ADNI		HCP-A
		Diagnosis		Diagnosis		Diagnosis		Sex		Age		Intelligence
		AUC $\uparrow$	F1 $\uparrow$	AUC $\uparrow$	F1 $\uparrow$	AUC $\uparrow$	F1 $\uparrow$	AUC $\uparrow$	F1 $\uparrow$	MAE $\downarrow$	$\rho$ $\uparrow$	MAE $\downarrow$	$\rho$ $\uparrow$
Brain-JEPA	$\mathbf{X}^{P}$	$.660_{\pm.00}$	$.606_{\pm.01}$	$.720_{\pm.05}$	$.622_{\pm.04}$	$\underline{.693}_{\pm.07}$	$.597_{\pm.05}$	$.917_{\pm.00}$	$.829_{\pm.00}$	$.418_{\pm.00}$	$.849_{\pm.00}$	$.681_{\pm.01}$	$.487_{\pm.01}$
BH-F	$\mathbf{X}^{P}$	$.631_{\pm.05}$	$.573_{\pm.06}$	$.697_{\pm.03}$	$.592_{\pm.05}$	$.601_{\pm.06}$	$.452_{\pm.12}$	$.867_{\pm.01}$	$.729_{\pm.10}$	$.435_{\pm.02}$	$.838_{\pm.02}$	$.679_{\pm.04}$	$.498_{\pm.06}$
Brain-DiT	$\mathbf{X}^{P}$	$.648_{\pm.05}$	$.568_{\pm.04}$	$.712_{\pm.04}$	$.535_{\pm.08}$	$.659_{\pm.05}$	$.484_{\pm.14}$	$.923_{\pm.02}$	$.833_{\pm.03}$	$.386_{\pm.01}$	$.877_{\pm.00}$	$.674_{\pm.01}$	$.546_{\pm.01}$
SwiFT	$\mathbf{X}_{g}^{V}$	$.664_{\pm.05}$	$\textbf{.633}_{\pm.05}$	$.711_{\pm.03}$	$.606_{\pm.04}$	$.688_{\pm.07}$	$.582_{\pm.06}$	$.946_{\pm.01}$	$.853_{\pm.02}$	$.350_{\pm.03}$	$.897_{\pm.02}$	$.678_{\pm.03}$	$.507_{\pm.02}$

Table 1: Comparison of trait prediction performance. All foundation models are fully fine-tuned, TABLeT is trained from scratch, and FReD-Trait uses linear probing. Results are reported as the mean and standard deviation across data splits. Bold and underlined values indicate the best and second-best performance for each metric, respectively.

Panel of Figure 4 — Figure 4: (a) Fraction of variance explained by the leading ImageNet principal directions, as a function of the fraction of components retained, shown separately for $\mathbf{X}^{V}$ (left) and $\mathbf{X}^{F}$ (right). Each panel includes an ImageNet reference computed on held-out ImageNet latents at the corresponding DCAE input resolution; a curve close to the ImageNet reference indicates a similar concentration of variance along the leading ImageNet principal directions. For $\mathbf{X}^{V}$ , the curves are medians over the three anatomical slicing axes and the bands span their minimum and maximum. (b) Four-way localization accuracy as a function of the injected percent signal change, averaged over the three spatial scales $\sigma$ ; chance accuracy is 25%.

Method	Input	HBN-Movie		HBN-Sex
Method	Input	AUC $\uparrow$	F1 $\uparrow$	AUC $\uparrow$	F1 $\uparrow$
FReD-Trait	$\mathbf{X}^{V}_{g}$	$.667_{\pm.03}$	$.614_{\pm.04}$	$.882_{\pm.04}$	$.741_{\pm.06}$
FReD-Trait	$\mathbf{X}^{V}_{cf}$	$.863_{\pm.03}$	$.787_{\pm.03}$	$.822_{\pm.04}$	$.657_{\pm.06}$

Table 3: Linear probing results over 500 random splits on HBN-Movie/Sex.

Method	Input	Eval	ADHD-200		HCP-A
			Diagnosis		Intelligence
			AUC $\uparrow$	F1 $\uparrow$	MAE $\downarrow$	$\rho$ $\uparrow$
FReD-Trait	$\mathbf{X}_{g}^{V},\mathbf{X}_{cf}^{V}$	LP	$.686_{\pm.05}$	$.604_{\pm.05}$	$\textbf{.633}_{\pm.06}$	$\textbf{.607}_{\pm.05}$
FReD-Trait	$\mathbf{X}_{g}^{F},\mathbf{X}_{cf}^{F}$	LP	$\textbf{.717}_{\pm.03}$	$.607_{\pm.05}$	$.655_{\pm.03}$	$.576_{\pm.04}$
FReD-State	$\mathbf{X}_{g}^{V}$	TFS	$.663_{\pm.05}$	$.584_{\pm.07}$	$.652_{\pm.03}$	$.564_{\pm.03}$
FReD-State	$\mathbf{X}_{cf}^{V}$	TFS	$.637_{\pm.05}$	$.570_{\pm.06}$	$.664_{\pm.06}$	$.556_{\pm.05}$

Table 4: FReD-State performance on trait prediction tasks. Bold values indicate the best performance. LP: Linear probing, TFS: Train from scratch.

Panel of Figure 5 — Figure 5: (a) Effect of the log-standard deviation feature on trait prediction, measured as the AUC or $\rho$ difference against replacing it with a zero vector. (b) Trait prediction performance as a function of the number of groups $G$ . (c) State prediction accuracy as a function of $G$ .

Method	Input	Param.	HCP-Task	NSD
Method	Input	Param.	ACC	ACC
TABLeT	$\mathbf{X}^{V}_{cf}$	129.5M	$94.2_{\pm 0.2}$	$23.9_{\pm 1.0}$
TABLeT with $\mathbf{\bar{Z}}^{V}_{a,t}$	$\mathbf{X}^{V}_{cf}$	145.4M	$94.7_{\pm 0.3}$	$27.5_{\pm 0.4}$
FReD-State	$\mathbf{X}^{V}_{cf}$	78.2M	$\textbf{97.4}_{\pm 0.1}$	$\textbf{29.5}_{\pm 0.2}$

Table 5: Comparison between TABLeT and FReD-State, trained from scratch.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Target	Subjects	Samples	Duration	# Classes	Majority (%)
ADHD-200	ADHD Dx	430	430	300 s	2	56.3
ABIDE-II	ASD Dx	616	616	300 s	2	56.3
ADNI	MCI Dx	464	464	300 s	2	59.5
HCP-A	Sex	1,069	1,069	382.4 s	2	57.1
HCP-A	Age, Intelligence	1,069	1,069	382.4 s	–	–
HBN-Movie	Movie	428	856	200 s	2	50.0

Appendix

Table 6: Summary of the datasets used for trait prediction (top) and state prediction (bottom). The numbers of subjects and samples represent the totals after preprocessing. Duration indicates the temporal duration of each sample. Trait prediction datasets contain one sample per subject, whereas state prediction datasets contain multiple samples per subject. Dx denotes diagnosis classification; Task21 denotes 21-class cognitive task-state decoding; and COCO24 denotes 24-class object-category decoding. The majority-class proportion is omitted for regression tasks.

Figure 6: Linear-probing results over 500 random train-test splits after reducing the FReD-Trait features to 768 dimensions, reported as AUC for classification and

\rho

for regression. RP768 and PCA768 denote Gaussian random projection and PCA, respectively.

Figure 7: Distribution of the late-fusion weight

\alpha

over the 500 train-test splits, shown separately for FReD-Trait with

\mathbf{X}^{V}

and

\mathbf{X}^{F}

. Larger

\alpha

places more weight on the globally normalized branch, and the dashed line marks equal weighting. White markers indicate the mean.

Method	Input	ADHD-200		ABIDE-II		ADNI		HCP-A
		Diagnosis		Diagnosis		Diagnosis		Sex		Age		Intelligence
		AUC $\uparrow$	F1 $\uparrow$	AUC $\uparrow$	F1 $\uparrow$	AUC $\uparrow$	F1 $\uparrow$	AUC $\uparrow$	F1 $\uparrow$	MAE $\downarrow$	$\rho$ $\uparrow$	MAE $\downarrow$	$\rho$ $\uparrow$
Brain-JEPA	$\mathbf{X}^{P}$	$.662_{\pm.04}$	$.595_{\pm.03}$	$.721_{\pm.03}$	$.572_{\pm.06}$	$.674_{\pm.04}$	$.527_{\pm.07}$	$.905_{\pm.02}$	$.817_{\pm.01}$	$.454_{\pm.01}$	$.830_{\pm.01}$	$.704_{\pm.04}$	$.429_{\pm.05}$
BH-F	$\mathbf{X}^{P}$	$.662_{\pm.05}$	$\underline{.617}_{\pm.03}$	$.710_{\pm.03}$	$.593_{\pm.05}$	$.614_{\pm.06}$	$.421_{\pm.10}$	$.896_{\pm.01}$	$.781_{\pm.01}$	$.430_{\pm.02}$	$.839_{\pm.01}$	$.677_{\pm.04}$	$.502_{\pm.05}$
Brain-DiT	$\mathbf{X}^{P}$	$.650_{\pm.03}$	$.604_{\pm.04}$	$.704_{\pm.04}$	$.558_{\pm.09}$	$.668_{\pm.05}$	$.562_{\pm.04}$	$.936_{\pm.03}$	$.856_{\pm.01}$	$.408_{\pm.02}$	$.863_{\pm.01}$	$.674_{\pm.02}$	$.517_{\pm.02}$
SwiFT	$\mathbf{X}^{V}_{g}$	$.654_{\pm.06}$	$\textbf{.618}_{\pm.04}$	$.705_{\pm.03}$	$.588_{\pm.04}$	$\textbf{.699}_{\pm.07}$	$\textbf{.613}_{\pm.08}$	$.945_{\pm.02}$	$.839_{\pm.02}$	$\underline{.391}_{\pm.02}$	$\underline{.866}_{\pm.01}$	$.683_{\pm.03}$	$.500_{\pm.03}$

Appendix

Table 7: Partial fine-tuning results on trait prediction tasks. Each foundation model is partially fine-tuned, while FReD-Trait uses linear probing on frozen features. Results are reported as the mean and standard deviation across 12 data splits for the clinical diagnosis tasks and four splits for HCP-A. Bold and underlined values indicate the best and second-best performance for each metric.

Method

Input

Eval

ADHD-200

ABIDE-II

ADNI

HCP-A

Diagnosis

Sex

Age

Intelligence

AUC

\uparrow

\uparrow

AUC

\uparrow

\uparrow

AUC

\uparrow

\uparrow

AUC

\uparrow

\uparrow

MAE

\downarrow

\rho\uparrow

MAE

\downarrow

\rho\uparrow

Brain-JEPA

\mathbf{X}^{P}

.660_{\pm.04}

.589_{\pm.04}

.728_{\pm.03}

.623_{\pm.04}

.693_{\pm.03}

.564_{\pm.05}

.892_{\pm.03}

.785_{\pm.04}

.516_{\pm.01}

.773_{\pm.01}

.715_{\pm.05}

.407_{\pm.05}

Brain-JEPA

\mathbf{X}^{P}

.660_{\pm.00}

.606_{\pm.01}

.720_{\pm.05}

.622_{\pm.04}

.693_{\pm.07}

.597_{\pm.05}

.917_{\pm.00}

.829_{\pm.00}

.418_{\pm.00}

.849_{\pm.00}

.681_{\pm.01}

.487_{\pm.01}

BH-F

\mathbf{X}^{P}

.669_{\pm.03}

\underline{.616}_{\pm.02}

.731_{\pm.03}

.608_{\pm.03}

.601_{\pm.05}

.432_{\pm.07}

.872_{\pm.01}

.759_{\pm.02}

.434_{\pm.00}

.840_{\pm.00}

.677_{\pm.04}

.520_{\pm.05}

BH-F

\mathbf{X}^{P}

.631_{\pm.05}

.573_{\pm.06}

.697_{\pm.03}

.592_{\pm.05}

.601_{\pm.06}

.452_{\pm.12}

.867_{\pm.01}

.729_{\pm.10}

.435_{\pm.02}

.838_{\pm.02}

.679_{\pm.04}

.498_{\pm.06}

Appendix

Table 8: Comparison of trait prediction performance. Each foundation model is evaluated both by linear probing on its frozen features and by full fine-tuning, while FReD-Trait uses linear probing. Results are reported as the mean and standard deviation across data splits. Bold and underlined values indicate the best and second-best performance for each metric, respectively. LP and FT denote linear probing and full fine-tuning, respectively.

Method	Input	ADHD-200		ABIDE-II		ADNI		HCP-A
		Diagnosis		Diagnosis		Diagnosis		Sex		Age		Intelligence
		AUC $\uparrow$	F1 $\uparrow$	AUC $\uparrow$	F1 $\uparrow$	AUC $\uparrow$	F1 $\uparrow$	AUC $\uparrow$	F1 $\uparrow$	MAE $\downarrow$	$\rho\uparrow$	MAE $\downarrow$	$\rho\uparrow$
BrainMASS	$\mathbf{X}^{P}$	$.638_{\pm.06}$	$.552_{\pm.07}$	$.695_{\pm.04}$	$.596_{\pm.05}$	$.637_{\pm.07}$	$.533_{\pm.07}$	$.823_{\pm.03}$	$.699_{\pm.04}$	$.635_{\pm.03}$	$.628_{\pm.04}$	$.679_{\pm.04}$	$.493_{\pm.06}$
Brain-JEPA	$\mathbf{X}^{P}$	$.655_{\pm.04}$	$.589_{\pm.04}$	$.720_{\pm.03}$	$.619_{\pm.04}$	$\textbf{.684}_{\pm.06}$	$.561_{\pm.07}$	$.874_{\pm.03}$	$.763_{\pm.04}$	$.502_{\pm.03}$	$.784_{\pm.02}$	$.712_{\pm.04}$	$.418_{\pm.06}$
BH-F	$\mathbf{X}^{P}$	$.662_{\pm.03}$	$\textbf{.631}_{\pm.02}$	$.740_{\pm.03}$	$.621_{\pm.04}$	$.619_{\pm.07}$	$.453_{\pm.09}$	$.883_{\pm.03}$	$.770_{\pm.04}$	$.420_{\pm.02}$	$.854_{\pm.02}$	$.682_{\pm.04}$	$.507_{\pm.05}$
Brain-DiT	$\mathbf{X}^{P}$	$.633_{\pm.04}$	$.586_{\pm.03}$	$.698_{\pm.03}$	$.577_{\pm.04}$	$.651_{\pm.06}$	$.542_{\pm.07}$	$.925_{\pm.02}$	$.823_{\pm.03}$	$.419_{\pm.02}$	$.855_{\pm.02}$	$.680_{\pm.04}$	$.501_{\pm.06}$

Appendix

Table 9: Full results of Fig. 2 . Bold and underlined values indicate the best and second-best performance for each metric, respectively.

Method	Eval.	Input	HBN-Movie
Method	Eval.	Input	AUC $\uparrow$	F1 $\uparrow$
BrainMASS	LP	$\mathbf{X}^{P}$	$.895_{\pm.02}$	$.793_{\pm.02}$
Brain-JEPA	LP	$\mathbf{X}^{P}$	$.631_{\pm.02}$	$.610_{\pm.04}$
Brain-JEPA	FT	$\mathbf{X}^{P}$	$.716_{\pm.10}$	$.656_{\pm.08}$
BH-F	LP	$\mathbf{X}^{P}$	$.724_{\pm.04}$	$.647_{\pm.05}$
BH-F	FT	$\mathbf{X}^{P}$	$.707_{\pm.18}$	$.673_{\pm.16}$

Appendix

Table 10: Full results of Fig. 3 . LP rows are highlighted in yellow; FT and TFS denote fine-tuning and training from scratch, respectively. Bold and underlined values indicate the best and second-best distinct performance for each metric, respectively.

Component	TABLeT	FReD-State with $\mathbf{X}^{V}$
Input representation and tokenization
DCAE latent input	$\mathbf{x}\in\mathbb{R}^{T\times 3\times 96\times 32\times 3\times 3}$	Same input tensor
Intra-frame aggregation	No slice averaging. Three contiguous 32-slice blocks are formed per axis, and corresponding features from the three anatomical axes are concatenated at each latent-grid position.	Contiguous slice-group averaging with $G=24$ groups per axis ( $96\rightarrow 24$ ), followed by concatenation of all axes, groups, channels, and spatial locations into one frame vector.
Tokens per frame	$3$ slice blocks $\times\,3\times 3$ spatial positions $=27$	$1$ whole-frame token
Pre-projection feature width	$3{,}072$	$20{,}736$
Frame projection	$\operatorname{LN}(3072)\rightarrow\operatorname{Linear}(3072,896)\rightarrow\operatorname{LN}(896)$	$\operatorname{LN}(20736)\rightarrow\operatorname{Linear}(20736,896)\rightarrow\operatorname{LN}(896)$

Appendix

Table 11: Architectural comparison between TABLeT and FReD-State with

\mathbf{X}^{V}

. Both models receive the same DCAE latent tensor and use the same downstream loss;

T

denotes the number of input fMRI frames.

Explore similar work

Oct 15, 2025cs.CV

Scaling Vision Transformers for Functional MRI with Flat Maps

We study the problem of training self-supervised foundation models for functional MRI. Our main contributions are: (1) we introduce a new model family (CortexMAE) trained using the masked autoencoder framework on 2.1K hours of open fMRI data, and (2) we release the first open evaluation suite (Brainmarks) for fMRI foundation models. Our core innovation is simple: we adapt the Vision Transformer to fMRI by first converting each 3D fMRI volume to a 2D map using a cortical flat map projection. We directly compare flat maps to both parcellation and volume-based representations. While each has its advantages, flat maps generally perform best. We perform the first systematic scaling analysis for fMRI and observe strict power law scaling, albeit with limits. Finally, we use Brainmarks to do controlled benchmark comparisons. On subject-level trait prediction, we report a challenging null result: no single model achieves clear state-of-the-art performance. Moreover, all models struggle to outperform a simple functional connectivity baseline. On cognitive state decoding, we observe more robust performance, and in this setting our CortexMAE family outperforms prior models by a large margin. Code, models, and datasets are available at https://github.com/MedARC-AI/CortexMAE and https://github.com/MedARC-AI/Brainmarks.

Connor Lane, Mihir Tripathy, Leema Krishna Murali +15

MedARC · Sophont · Baylor College of Medicine +3

Functional Magnetic Resonance Imaging Vision Transformer Transformer Architectures

Ask PDF

Apr 14, 2026cs.CV

Brain-DiT: A Universal Multi-state fMRI Foundation Model with Metadata-Conditioned Pretraining

Current fMRI foundation models primarily rely on a limited range of brain states and mismatched pretraining tasks, restricting their ability to learn generalized representations across diverse brain states. We present Brain-DiT, a universal multi-state fMRI foundation model pretrained on 349,898 sessions from 24 datasets spanning resting, task, naturalistic, disease, and sleep states. Unlike prior fMRI foundation models that rely on masked reconstruction in the raw-signal space or a latent space, Brain-DiT adopts metadata-conditioned diffusion pretraining with a Diffusion Transformer (DiT), enabling the model to learn multi-scale representations that capture both fine-grained functional structure and global semantics. Across extensive evaluations and ablations on 7 downstream tasks, we find consistent evidence that diffusion-based generative pretraining is a stronger proxy than reconstruction or alignment, with metadata-conditioned pretraining further improving downstream performance by disentangling intrinsic neural dynamics from population-level variability. We also observe that downstream tasks exhibit distinct preferences for representational scale: ADNI classification benefits more from global semantic representations, whereas age/sex prediction comparatively relies more on fine-grained local structure.

Junfeng Xia, Wenhao Ye, Xuanye Pan +3

Department of Biomedical Engineering, Southern University of Science and Technology, China · School of Biomedical Engineering, Shenzhen University, China

Pretraining Foundation Model Representational Capacity

Ask PDF

Aug 10, 2026cs.CV

Frozen Brain-MRI Foundation Models Are Site Fingerprints

Frozen foundation-model (FM) embeddings are increasingly used as off-the-shelf brain-MRI representations, on the assumption that they capture anatomy. We audit what they actually encode and find that acquisition site is a large, intrinsic component of the representation. Across two independent cohorts (ABIDE-I, ABIDE-II), three frozen 3-D encoders (brain-pretrained, CT-pretrained, and randomly initialized), and every network depth, site is linearly decodable at roughly 0.9 balanced accuracy at deep layers, exceeding the decodability of every clinical or demographic variable (sex, age, autism diagnosis) at every layer. The effect is intrinsic rather than learned: a randomly initialized encoder is already a ~0.9 site classifier on both cohorts and across three architecture families (Swin, ViT, ResNet), and site is decodable at ~0.95 directly from the raw downsampled image with no encoder, so the fingerprint reflects low-level image statistics that any encoder preserves rather than a product of pretraining. Residualizing measured population covariates leaves site decodability essentially unchanged, indicating an acquisition- rather than population-driven effect. A nonlinear probe matches the linear one, so the fingerprint is fully linearly accessible. The site subspace is removable post hoc by iterative null-space projection or ComBat (site decodability 0.94 -> 0.07/0.00), and is a site-attribution concern for shared or federated embeddings; but for dense segmentation this removal is not free, because site and anatomy occupy an entangled linear subspace (a matched-rank random-direction projection is Dice-neutral, whereas removing the site subspace is destructive). We recommend site-audited use of frozen brain-MRI FMs and release an open audit toolkit.

Saman Rahbar

University of British Columbia, Vancouver, Canada

Functional Magnetic Resonance Imaging Fingerprint Foundation Model Decoding Covariates Web

Ask PDF