FlatClip: Reusing Image Foundation Models for fMRI Representation Learning via Cortical Flatmaps

Mo Wang; Wenhao Ye; Zihan Ning; Jiayu Zuo; Junfeng Xia; Hongkai Wen; Quanying Liu

cs.CESep 25, 2026

FlatClip: Reusing Image Foundation Models for fMRI Representation Learning via Cortical Flatmaps

Authors: Mo Wang, Wenhao Ye, Zihan Ning, Jiayu Zuo, Junfeng Xia, Hongkai Wen, Quanying Liu

Organizations: Department of Biomedical Engineering, Southern University of Science and Technology, China · Department of Computer Science, University of Warwick, The UK · School of Biomedical Engineering, Shenzhen University, Shenzhen, China

Abstract

Can the spatial organization of cortical activity help transfer frozen image representations to fMRI prediction? We introduce FlatClip, which renders cortical activity as anatomy-linked flatmaps and extracts features with a frozen SigLIP2 encoder. Only a lightweight downstream probe is trained, without fMRI-specific encoder pretraining or fine-tuning. Across four resting-state benchmarks, FlatClip outperforms the evaluated ROI-level baselines on HCP and ADNI, but is weaker on PPMI and below the strongest voxel-level models overall. On NSD visual-fMRI decoding, restricting inputs to visual or NSD-provided task-active cortex improves recognition over whole-cortex inputs. Spatial perturbations reduce performance with both retrained and fixed readouts, and anatomy-linked arrangements outperform cortical-pixel permutations across three colormaps. Together, these results demonstrate a practical route for reusing image-pretrained features in fMRI and highlight cortical arrangement and input-region selection as important design choices. Code is available at https://github.com/OneMore1/FlatClip.

Figures & tables

Figure 1: Motivation. fMRI representations vary in spatial granularity. FlatClip studies surface-level cortical flatmaps as a geometry-aware middle ground.

Figure 2: FlatClip pipeline. Preprocessed cortical signals are rendered in a fixed surface layout and encoded by frozen SigLIP2. Resting-state prediction averages 40 frame features into one subject feature; NSD prediction uses a stimulus-level GLM response map. Only the downstream MLP is trained.

Panel of Table 1 — Table 1: Performance on HCP, ADNI (MCI), ADNI (AD), and PPMI with Accuracy/weighted F1. Darker cell color indicates stronger performance after column-wise min–max normalization.

Figure 3: Visual-fMRI COCO80 multi-label recognition on NSD. (a) Illustration of the cortical input regions used by FlatClip: whole cortex, HCP-MMP visual cortex, and the NSD-provided subject-level task-active region (NSDgeneral). (b) Subject-wise mAP across four NSD subjects for Omni-fMRI, NeuroSTORM, SwiFT and FlatClip variants. (c,d) Mean mAP and weighted F1 across subjects. All FlatClip variants use one global representation per GLM response map and the same downstream MLP classifier. Dots denote individual subjects, and error bars denote SEM across subjects. Stimulus-level alignment statistics are reported in Appendix B.2 .

Control input	ACC $\uparrow$	wF1 $\uparrow$	Balanced ACC $\uparrow$	Macro F1 $\uparrow$
Real flatmap	82.06 ${\pm}$ 2.05	82.04 ${\pm}$ 2.05	81.81 ${\pm}$ 2.03	81.88 ${\pm}$ 2.06
Spatial block-shuffle	76.14 ${\pm}$ 1.83 ∗	76.18 ${\pm}$ 1.83 ∗	76.21 ${\pm}$ 1.85 ∗	76.06 ${\pm}$ 1.84 ∗
Left-right hemisphere swap	74.96 ${\pm}$ 1.17 ∗	74.89 ${\pm}$ 1.23 ∗	74.56 ${\pm}$ 1.31 ∗	74.64 ${\pm}$ 1.28 ∗
Within-hemisphere region shuffle	70.90 ${\pm}$ 1.63 ∗	70.87 ${\pm}$ 1.62 ∗	70.62 ${\pm}$ 1.61 ∗	70.64 ${\pm}$ 1.62 ∗
Cortical-pixel permutation	72.93 ${\pm}$ 1.92 ∗	72.87 ${\pm}$ 1.97 ∗	72.58 ${\pm}$ 2.05 ∗	72.62 ${\pm}$ 2.02 ∗
Real flatmap with random initialized	70.56 ${\pm}$ 1.34 ∗	70.56 ${\pm}$ 1.36 ∗	70.37 ${\pm}$ 1.41 ∗	70.35 ${\pm}$ 1.38 ∗

Table 2: Geometry-control ablation on HCP sex classification. All input-control results use the frozen pretrained SigLIP2 encoder with one global representation per frame, except the explicitly labeled random-initialization control. The MLP architecture and training protocol are fixed, and a separate probe is trained for each condition. Results are reported as mean

\pm

std over three seeds. Significance markers indicate paired

t

-tests over the three probe-seed results, comparing Real flatmap against each control after Holm correction. Red indicates the best performance.

Representation	Resolution	HCP Sex		AD Diagnosis
Representation	Resolution	ACC $\uparrow$	wF1 $\uparrow$	ACC $\uparrow$	wF1 $\uparrow$
ROI time series	450 ROIs	65.68 $\pm$ 4.68	65.51 $\pm$ 4.47	60.65 $\pm$ 5.61	60.75 $\pm$ 3.90
Voxel model	Native 4D volume	80.74 $\pm$ 2.41	80.32 $\pm$ 3.21	72.22 $\pm$ 10.2	71.66 $\pm$ 9.54
Voxel model	Adaptive patches	82.81 $\pm$ 0.57	82.88 $\pm$ 0.55	72.99 $\pm$ 1.03	70.76 $\pm$ 3.16

Table 3: Comparison of ROI time-series and voxel-based models. The ROI input contains 450 regional time series; voxel models use native volumetric signals with standard or adaptive patching. Results are Accuracy and weighted F1 (%).

Panel of Table 4 — Table 4: Implementation ablations for frozen image backbone, feature representation, and flatmap normalization. Results are reported as Accuracy / weighted F1. (%)

Task	FlatClip	CortexMAE-F	SiT-style parcel-token surface
HCP sex	$82.06\pm 2.05$ / $82.04\pm 2.05$	$\mathbf{82.40\pm 1.27}$ / $\mathbf{82.37\pm 1.26}$	$55.84\pm 5.19$ / $55.74\pm 5.15$
ADNI AD/CN	$\mathbf{75.46\pm 2.12}$ / $\mathbf{75.41\pm 2.11}$	$69.91\pm 6.26$ / $70.43\pm 6.31$	$74.54\pm 2.85$ / $73.07\pm 2.90$
ADNI MCI/CN	$\mathbf{61.11\pm 1.48}$ / $\mathbf{60.75\pm 1.16}$	$58.12\pm 0.74$ / $57.46\pm 1.46$	$55.98\pm 3.96$ / $55.59\pm 2.79$

Table 5: Comparison with surface models. Accuracy / weighted F1 (%), mean

\pm

std over three runs. CortexMAE-F pretraining includes HCP. The SiT-style column reports the parcel-token configuration; implementation details are given in Appendix D.2 .

Renderer	Real minus permuted wF1
Inferno	$+7.26$
Grayscale	$+8.24$
RdBu_r	$+8.74$

Table 6: Geometry effect across renderers. Weighted-F1 gain (percentage points) of real flatmaps over the corresponding fixed cortical-pixel permutation.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Backbone	Feature	HCP		PPMI		ADNI (MCI)		ADNI (AD)
		Sex Classif.		PD Diagnosis		Diagnosis		Diagnosis
		ACC $\uparrow$	wF1 $\uparrow$	ACC $\uparrow$	wF1 $\uparrow$	ACC $\uparrow$	wF1 $\uparrow$	ACC $\uparrow$	wF1 $\uparrow$
DINOv2	One global token	80.37 $\pm$ 2.55	80.33 $\pm$ 2.53	51.39 $\pm$ 3.18	51.58 $\pm$ 2.25	60.68 $\pm$ 2.67	60.44 $\pm$ 1.98	75.46 $\pm$ 4.01	75.20 $\pm$ 3.69
DINOv2	40 Global tokens	79.53 $\pm$ 1.17	79.50 $\pm$ 1.15	47.57 $\pm$ 5.14	48.82 $\pm$ 2.67	55.98 $\pm$ 2.67	55.58 $\pm$ 2.09	75.00 $\pm$ 2.41	74.61 $\pm$ 2.34
ViT-B	40 Global tokens	70.00 $\pm$ 1.13	70.02 $\pm$ 1.15	60.00 $\pm$ 1.36	55.45 $\pm$ 1.72	52.76 $\pm$ 4.57	52.19 $\pm$ 4.71	78.11 $\pm$ 2.56	76.84 $\pm$ 2.71
SigLIP2	One Global token	82.06 $\pm$ 2.05	82.04 $\pm$ 2.05	54.86 $\pm$ 1.96	55.72 $\pm$ 1.63	61.11 $\pm$ 1.48	60.75 $\pm$ 1.16	75.46 $\pm$ 2.12	75.41 $\pm$ 2.11

Appendix

Table 7: Ablation of frozen image backbone and feature extraction strategy. Results are reported as Accuracy / weighted F1.

Flatmap rendering
Configuration	Value
surface space	fsLR cortical surface
renderer	PyCortex / cached flatmap projection
Normalization	global z-score with bounded color scale
colormap	Inferno (HCP); RdBu_r (NSD)
background pixels	transparent outside cortex/ROI, composited on white RGB

Appendix

Table 8: Hyperparameter settings of FlatClip.

Panel of Table 9 — Table 9: Within-subject COCO80 multi-label classification on NSD. Red indicates the best performance. wF1 denotes weighted-F1. Unmarked FlatClip rows use the main setting: one 768-dimensional feature obtained by averaging a $16\times 16$ feature grid adaptively pooled from valid NaFlex output tokens. This grid is formed after encoding and is distinct from the processor’s variable-size patch grid. Rows marked with ∗ retain all 256 pooled feature tokens.

Panel of Table 10 — Table 10: Ablation on the number of resting-state frames used by FlatClip. All results use frozen SigLIP2 with one global representation per frame. Selected frame features are mean-pooled into a 768-dimensional subject-level token. Results are reported as Accuracy / weighted F1 over three seeds.

Figure 4: Stimulus-level image–fMRI feature alignment analysis on a randomly sampled subset of NSD shared1000 in SigLIP2 space. (a) Paired-random cosine margin, defined as the cosine similarity between the aligned fMRI-derived feature and the matched image feature minus a randomly mismatched image cosine. (b) Paired cosine similarity between the aligned fMRI-derived feature and its matched image feature. For each shared image, scores are first computed for each subject and then averaged across the four subjects. Dots denote individual shared stimuli, and large markers denote the mean. Significance brackets compare SigLIP2-NSDgeneral against each other method using paired Wilcoxon signed-rank tests over stimuli. Cosines are computed after separately fitted subject- and model-specific ridge alignment.

Metric	Comparison	Baseline	SigLIP2-NSDgeneral	$\Delta$	95% CI	$p_{t}$	$p_{w}$	Improved
Local-RDM	vs SwiFT	0.026	0.134	0.108	[0.086, 0.129]	$9.01{\times}10^{-16}$	$3.18{\times}10^{-12}$	83.0%
Local-RDM	vs Omni-fMRI	0.009	0.134	0.125	[0.102, 0.147]	$2.70{\times}10^{-17}$	$5.27{\times}10^{-13}$	86.0%
Local-RDM	vs NeuroSTORM	0.052	0.134	0.083	[0.064, 0.101]	$5.78{\times}10^{-13}$	$1.34{\times}10^{-10}$	79.0%
Local-RDM	vs SigLIP2-Cortex	0.032	0.134	0.102	[0.084, 0.121]	$2.09{\times}10^{-17}$	$2.87{\times}10^{-13}$	89.0%
Local-RDM	vs SigLIP2-Visual	0.091	0.134	0.043	[0.026, 0.061]	$5.41{\times}10^{-6}$	$5.73{\times}10^{-6}$	74.0%
Paired cosine	vs SwiFT	0.222	0.325	0.103	[0.087, 0.119]	$3.82{\times}10^{-21}$	$7.31{\times}10^{-15}$	90.0%

Appendix

Table 11: Stimulus-level image–fMRI feature alignment analysis on a randomly sampled subset of NSD shared1000 in SigLIP2 space. Values are averaged over four subjects, leaving one value per image.

\Delta

denotes SigLIP2-NSDgeneral minus the comparison method.

p_{t}

and

p_{w}

denote one-sided paired

t

-test and Wilcoxon signed-rank p-values, with Holm correction across the 15 comparisons within each test family. Confidence intervals are percentile bootstrap intervals over stimulus-level paired differences.

Control input	ACC $\uparrow$	wF1 $\uparrow$	Balanced ACC $\uparrow$	Macro F1 $\uparrow$
Real flatmap	80.37 ${\pm}$ 2.55	80.33 ${\pm}$ 2.53	80.08 ${\pm}$ 2.47	80.15 ${\pm}$ 2.53
Spatial block-shuffle	75.47 ${\pm}$ 0.78 ∗	75.49 ${\pm}$ 0.76 ∗	75.44 ${\pm}$ 0.70 ∗	75.34 ${\pm}$ 0.75 ∗
Cortical-pixel permutation	73.77 ${\pm}$ 2.80 ∗	73.80 ${\pm}$ 2.81 ∗	74.06 ${\pm}$ 2.61 ∗	73.73 ${\pm}$ 2.76 ∗
FC heatmap	54.99 ${\pm}$ 5.08 ∗∗	55.04 ${\pm}$ 5.07 ∗∗	54.80 ${\pm}$ 5.08 ∗∗	54.77 ${\pm}$ 5.07 ∗∗
Random image	49.24 ${\pm}$ 1.52 ∗∗	49.33 ${\pm}$ 1.52 ∗∗	49.24 ${\pm}$ 1.54 ∗∗	49.14 ${\pm}$ 1.53 ∗∗

Appendix

Table 12: Geometry-control ablation on HCP sex classification. All results use the frozen DINOv2 encoder with one global token per frame, followed by an independently trained MLP with the same architecture and training protocol for every condition. Results are reported as mean

\pm

std over three seeds. Significance markers indicate paired

t

-tests over the three probe-seed results, comparing Real flatmap against each control after Holm correction. Red indicates the best performance.

Input region	Width $\times$ height
Whole cortex	$943\times 384$
HCP-MMP visual	$273\times 264$
NSDgeneral, subject 1	$293\times 208$
NSDgeneral, subjects 2 and 5	$293\times 209$
NSDgeneral, subject 7	$292\times 209$

Appendix

Table 13: Native cropped flatmap dimensions before encoder processing.

Task	Adjustment	ACC	wF1	$\Delta$ wF1
HCP sex	Raw	$82.06\pm 2.05$	$82.04\pm 2.05$	Ref.
	Age	$80.88\pm 1.06$	$80.88\pm 1.07$	$-1.16$
ADNI MCI/CN	Raw	$61.11\pm 1.48$	$60.75\pm 1.16$	Ref.
	Age + sex	$59.40\pm 0.74$	$59.03\pm 0.38$	$-1.72$
	Age + sex + site	$57.69\pm 0.00$	$56.47\pm 0.60$	$-4.28$
	Age + sex + site + protocol	$58.12\pm 1.96$	$57.69\pm 1.57$	$-3.06$

Appendix

Table 14: Measured-confound sensitivity. Mean

\pm

std (%) over three probe seeds.

\Delta

wF1 is the percentage-point change from the raw row for the same task.

Task	Window	ACC	wF1
ADNI MCI/CN	TR0–39	62.39	62.69
	TR98–137	62.82	63.20
ADNI AD/CN	TR0–39	75.46	75.41
	TR98–137	$73.15\pm 2.89$	$73.35\pm 3.10$

Appendix

Table 15: Alternative 40-TR windows. Accuracy and weighted F1 (%). The first-window MCI/CN row is the paired re-evaluation for this experiment.

Split	Class 0	Class 1	Total
Training	321	281	602
Validation	107	94	201
Test	107	90	197

Appendix

Table 16: HCP class distribution for the supplementary frozen-feature evaluations. Class labels follow the saved split manifest.

Task	Main MLP	Linear probe
HCP sex	$82.06\pm 2.05$ / $82.04\pm 2.05$	$84.77\pm 1.34$ / $84.77\pm 1.34$
ADNI MCI/CN	$61.11\pm 1.48$ / $60.75\pm 1.16$	$50.43\pm 6.45$ / $50.61\pm 6.41$
ADNI AD/CN	$75.46\pm 2.12$ / $75.41\pm 2.11$	$70.37\pm 10.79$ / $71.10\pm 10.59$
PPMI three-way	$54.51\pm 5.35$ / $55.59\pm 4.20$	$44.10\pm 0.60$ / $47.75\pm 0.39$

Appendix

Table 17: Probe sensitivity on resting-state tasks. Accuracy / weighted F1 (%), mean

\pm

std over three runs. PPMI uses the raw re-evaluation paired with the linear experiment.

Input	Retrained MLP	Fixed MLP	Retrained linear	Fixed linear
Real flatmap	$82.04\pm 2.05$	$82.04\pm 2.05$	$84.77\pm 1.34$	$84.77\pm 1.34$
Spatial block shuffle	$76.18\pm 1.83$	$28.65\pm 0.00$	$77.53\pm 1.05$	$32.18\pm 3.48$
Cortical-pixel permutation	$72.87\pm 1.97$	$28.65\pm 0.00$	$76.43\pm 1.41$	$28.65\pm 0.00$
Phase randomization	$64.49\pm 2.14$	$29.74\pm 1.89$	$65.87\pm 3.09$	$28.65\pm 0.00$

Appendix

Table 18: Readout controls on HCP. Weighted F1 (%), mean

\pm

std over seeds 42, 43, and 44. Retrained probes are fitted separately for each condition; fixed probes are fitted only on real flatmaps.

	Visual		NSDgeneral
Method	mAP	wF1	mAP	wF1
ROI-linear	14.28	22.91	16.45	23.80
Small MLP	14.06	24.66	15.18	26.04
FlatClip	15.84	26.92	16.35	26.97

Appendix

Table 19: Same-mask baselines on NSD. Mean mAP and weighted F1 (%) across the four evaluated subjects.

Representation	ACC	wF1
Axial mean projection	$72.93\pm 1.63$	$72.81\pm 1.75$
FlatClip cortical flatmap	$82.06\pm 2.05$	$82.04\pm 2.05$
Independent slices, then temporal mean	$89.34\pm 1.34$	$89.32\pm 1.34$

Appendix

Table 20: HCP direct-volume 40-frame results. Accuracy and weighted F1 (%), mean

\pm

std over three probe seeds.

Input	mAP	wF1
FlatClip, whole cortex	11.100	24.576
FlatClip, Visual	14.050	25.829
FlatClip, NSDgeneral	15.311	26.922
Three-view mean projection	4.95	17.00
Independent axial slices	$4.342\pm 0.171$	$15.922\pm 0.579$

Appendix

Table 21: NSD subj01 direct-volume comparisons. mAP and weighted F1 (%). Original FlatClip and mean-projection values are reported alongside the three-seed slice-wise experiment.

Original beta-map results
Input	mAP	wF1
FlatClip–NSDgeneral	15.311	26.922
Omni-fMRI, repeated beta $\times 40$	3.949	13.818
Native temporal inputs: 7,000/1,000/2,000 split
Omni-fMRI, continuous BOLD40	$3.804\pm 0.052$	$16.531\pm 0.357$
Omni-fMRI, event stack $3\times 13+1$	$3.840\pm 0.035$	$16.188\pm 0.183$

Appendix

Table 22: Omni-fMRI temporal input adaptations on NSD subj01. mAP and weighted F1 (%). Original beta-map results and new temporal-input results are grouped separately. Temporal rows report mean

\pm

std over three seeds.

Training fraction	Subjects	Linear	MLP
10%	60	$70.05\pm 4.51$ / $70.01\pm 4.47$	$70.39\pm 2.55$ / $69.92\pm 3.04$
25%	150	$77.83\pm 2.55$ / $77.83\pm 2.59$	$75.97\pm 4.93$ / $75.89\pm 4.96$
50%	300	$82.91\pm 0.29$ / $82.89\pm 0.27$	$79.19\pm 1.76$ / $79.11\pm 1.71$
100%	602	$84.77\pm 1.34$ / $84.77\pm 1.34$	$82.06\pm 2.05$ / $82.04\pm 2.05$

Appendix

Table 23: HCP labeled-subject scaling. Accuracy / weighted F1 (%), mean

\pm

std over three seeds.

fMRI Representation Learning

See all 13 papers

Scaling Vision Transformers for Functional MRI with Flat Maps
Oct 15, 2025Connor Lane, Mihir Tripathy, Leema Krishna Murali +15fMRI Foundation Models Masked Autoencoders
MedARC · Sophont · Baylor College of Medicine +3
CoFlu-Brain: An fMRI Connectivity Foundation Model with Cofluctuation-Based Pretraining
Oct 11, 2026Ziteng Sui, Junfeng Xia, Wenhao Ye +2fMRI Foundation Models fMRI Representation Learning
Department of Biomedical Engineering, Southern University of Science and Technology, China · School of Biomedical Engineering, Shenzhen University, China · Department of Computer Science, University of Warwick, UK
Natural Image Autoencoder-Based fMRI Representations for Trait and State Prediction
Sep 28, 2026Juhyeon Park, Yeonwoo Kim, Peter Yongho Kim +5Medical Imaging Foundation Models fMRI Foundation Models
IPAI, Seoul National University · Microsoft Research · ECE, Seoul National University +1

Oct 15, 2025J/K move · Enter open · S save

Scaling Vision Transformers for Functional MRI with Flat Maps

Connor Lane, Mihir Tripathy, Leema Krishna Murali, Ratna Sagari Grandhi+14

MedARC · Sophont · Baylor College of Medicine+3

fMRI Foundation Models Masked Autoencoders Neuroimaging Medical Image Benchmarks Medical Imaging Foundation Models fMRI Representation Learning+1

Open paper AskPDF arXiv

We study the problem of training self-supervised foundation models for functional MRI. Our main contributions are: (1) we introduce a new model family (CortexMAE) trained using the masked autoencoder framework on 2.1K hours of open fMRI data, and (2) we release the first open evaluation suite (Brainmarks) for fMRI foundation models. Our core innovation is simple: we adapt the Vision Transformer to fMRI by first converting each 3D fMRI volume to a 2D map using a cortical flat map projection. We directly compare flat maps to both parcellation and volume-based representations. While each has its advantages, flat maps generally perform best. We perform the first systematic scaling analysis for fMRI and observe strict power law scaling, albeit with limits. Finally, we use Brainmarks to do controlled benchmark comparisons. On subject-level trait prediction, we report a challenging null result: no single model achieves clear state-of-the-art performance. Moreover, all models struggle to outperform a simple functional connectivity baseline. On cognitive state decoding, we observe more robust performance, and in this setting our CortexMAE family outperforms prior models by a large margin. Code, models, and datasets are available at https://github.com/MedARC-AI/CortexMAE and https://github.com/MedARC-AI/Brainmarks.

Abstract

fMRI Representation Learning

Explore similar work

Scaling Vision Transformers for Functional MRI with Flat Maps

CoFlu-Brain: An fMRI Connectivity Foundation Model with Cofluctuation-Based Pretraining

Natural Image Autoencoder-Based fMRI Representations for Trait and State Prediction