Can Natural Image Autoencoders Compactly Tokenize fMRI Volumes for Long-Range Dynamics Modeling?

Peter Yongho Kim; Juhyeon Park; Jungwoo Park; Jubin Choi; Jungwoo Seo; Jiook Cha; Taesup Moon

cs.CVApr 4, 2026

Can Natural Image Autoencoders Compactly Tokenize fMRI Volumes for Long-Range Dynamics Modeling?

Authors: Peter Yongho Kim, Juhyeon Park, Jungwoo Park, Jubin Choi, Jungwoo Seo, Jiook Cha, Taesup Moon

Organizations: ECE, Seoul National University · IPAI, Seoul National University · BCS, Seoul National University · Psychology, Seoul National University ·

Abstract

Modeling long-range spatiotemporal dynamics in functional Magnetic Resonance Imaging (fMRI) remains a key challenge due to the high dimensionality of the four-dimensional signals. Prior voxel-based models, although demonstrating excellent performance and interpretation capabilities, are constrained by prohibitive memory demands and thus can only capture limited temporal windows. To address this, we propose TABLeT (Two-dimensionally Autoencoded Brain Latent Transformer), a novel approach that tokenizes fMRI volumes using a pre-trained 2D natural image autoencoder. Each 3D fMRI volume is compressed into a compact set of continuous tokens, enabling long-sequence modeling with a simple Transformer encoder with limited VRAM. Across large-scale benchmarks including the UK-Biobank (UKB), Human Connectome Project (HCP), and ADHD-200 datasets, TABLeT outperforms existing models in multiple tasks, while demonstrating substantial gains in computational and memory efficiency over the state-of-the-art voxel-based method given the same input. Furthermore, we develop a self-supervised masked token modeling approach to pre-train TABLeT, which improves the model's performance for various downstream tasks. Our findings suggest a promising approach for scalable and interpretable spatiotemporal modeling of brain activity. Our code is available at https://github.com/beotborry/TABLeT.

Figures & tables

Figure 1: We show that a 2D natural image autoencoder can be transferred well to 4D fMRI data (left). Building on this, we propose to tokenize a single 3D volume of fMRI only into 27 tokens, enabling longer spatiotemporal modeling with a simple Transformer architecture (right).

Panel of Figure 2 — Figure 2: In TABLeT, each frame of the fMRI timeseries is tokenized by a 2D autoencoder, and the tokens are processed by a Transformer.

Method UKB ADHD-200 Sex Age Diagnosis ACC AUC F1 MSE MAE

\rho

ACC AUC F1 XGBoost 84.1 0.916 0.830 0.698 0.686 0.553 62.3 0.650 0.555 BrainNetCNN 91.7 0.969 0.912 0.597 0.618 0.647 59.2 0.640 0.545 BNT 92.4 0.980 0.919 0.540 0.588 0.685 63.6 0.677 0.624 meanMLP 87.7 0.949 0.919 0.672 0.662 0.586 56.8 0.617 0.532 Brain-JEPA 1 1 1 Since the ADHD-200 dataset contains fMRI data with varying repetition time (TR) values and fewer than 160 frames, the default frame number used in Brain-JEPA, we were unable to conduct experiments. 86.8 0.943 0.862 0.688 0.669 0.574 – – – TFF (

T=20

) 98.3 0.998 0.982 0.440 0.525 0.760 63.3 0.700 0.608 SwiFT (

T=20

) 97.4 0.998 0.972 0.366 0.480 0.800 63.3 0.693 0.623 SwiFT (

T=50

) 98.1 0.999 0.980 0.364 0.477 0.802 63.9 0.701 0.627 TABLeT (

T=256

) 97.7 0.998 0.976 0.340 0.466 0.814 65.8 0.729 0.630

Method HCP Sex Age Intelligence ACC AUC F1 MSE MAE

\rho

MSE MAE

\rho

XGBoost 82.2 0.890 0.837 0.859 0.769 0.296 0.908 0.779 0.292 BrainNetCNN 86.3 0.937 0.866 0.847 0.749 0.372 0.967 0.788 0.286 BNT 86.3 0.935 0.872 0.794 0.719 0.444 0.920 0.778 0.318 meanMLP 84.5 0.915 0.855 0.846 0.751 0.370 0.887 0.767 0.340 Brain-JEPA 73.9 0.809 0.761 0.814 0.746 0.369 0.959 0.799 0.171 TFF (

T=20

) 88.1 0.937 0.892 0.888 0.779 0.246 0.898 0.767 0.312 SwiFT (

T=20

) 93.1 0.978 0.937 0.776 0.719 0.450 0.940 0.782 0.297 SwiFT (

T=50

) 92.2 0.972 0.929 0.764 0.699 0.460 0.865 0.758 0.354 TABLeT (

T=256

) 93.8 0.987 0.943 0.773 0.705 0.473 0.835 0.741 0.392

Table 1: Performance comparison to baselines on classification and regression tasks. The best results are bolded and the second best results are underlined . Standard deviations are provided in Sec. D .

Figure 3: Visualization of reconstructions from 3D and 2D DCAE.

Figure 4: Information preservation of 3D and 2D DCAE.

Panel of Figure 5 — Figure 5: Comparison of (a) memory and (b) training time, between TABLeT and SwiFT.

Axis	UKB
	Sex			Age
	ACC	AUC	F1	MSE	MAE	$\rho$
Sagittal	$\underline{97.3}_{\pm 0.9}$	$0.996_{\pm 0.001}$	$\underline{0.971}_{\pm 0.009}$	$\underline{0.369}_{\pm 0.015}$	$\underline{0.486}_{\pm 0.010}$	$\underline{0.796}_{\pm 0.008}$
Coronal	$97.1_{\pm 0.4}$	$0.996_{\pm 0.002}$	$0.969_{\pm 0.004}$	$0.435_{\pm 0.012}$	$0.525_{\pm 0.008}$	$0.756_{\pm 0.009}$
Axial	$\underline{97.3}_{\pm 0.4}$	$\underline{0.997}_{\pm 0.000}$	$\underline{0.971}_{\pm 0.004}$	$0.410_{\pm 0.020}$	$0.509_{\pm 0.014}$	$0.771_{\pm 0.013}$
All	$\textbf{97.7}_{\pm 0.2}$	$\textbf{0.998}_{\pm 0.000}$	$\textbf{0.976}_{\pm 0.002}$	$\textbf{0.340}_{\pm 0.011}$	$\textbf{0.466}_{\pm 0.010}$	$\textbf{0.814}_{\pm 0.009}$

Table 4: Effect of the choice of slicing axis and aggregation of the three axes on classification and regression tasks. The best results are bolded and the second best results are underlined .

Panel of Figure 6 — Figure 6: Performance of TABLeT on HCP-Intelligence and ADHD-200 with varying $T$ .

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Validation Loss Curve for Training of 3D DCAE.

Model	HBN-Movie
Model	ACC	AUC	F1
SwiFT $(T=50)$	$71.7_{\pm 4.29}$	$0.810_{\pm 0.040}$	$0.717_{\pm 0.040}$
TABLeT $(T=50)$	$74.7_{\pm 2.38}$	$0.826_{\pm 0.033}$	$0.750_{\pm 0.024}$
TABLeT $(T=250)$	$\textbf{82.1}_{\pm 1.82}$	$\textbf{0.976}_{\pm 0.014}$	$\textbf{0.847}_{\pm 0.015}$

Appendix

Table A6 : Results on HBN-Movie.

Model	# Token	Dim.	HCP			ADHD-200
			Intelligence			Diagnosis
			MSE	MAE	$\rho$	ACC	AUC	F1
TABLeT	27	3072	$0.835_{\pm 0.053}$	$0.741_{\pm 0.028}$	$0.392_{\pm 0.062}$	$\textbf{65.8}_{\pm 3.50}$	$0.729_{\pm 0.029}$	$0.630_{\pm 0.038}$
TABLeT Alt.1	9	9216	$\textbf{0.814}_{\pm 0.044}$	$\textbf{0.733}_{\pm 0.026}$	$\textbf{0.416}_{\pm 0.061}$	$65.7_{\pm 2.74}$	$0.710_{\pm 0.031}$	$0.632_{\pm 0.043}$
TABLeT Alt.2	3	27648	$0.840_{\pm 0.093}$	$0.736_{\pm 0.044}$	$0.401_{\pm 0.035}$	$65.1_{\pm 3.59}$	$\textbf{0.690}_{\pm 0.032}$	$\textbf{0.636}_{\pm 0.042}$

Appendix

Table A7 : Results on HCP and ADHD-200 with varying axis aggregation schemes.

Category	UKB	HCP	ADHD -200
Number of subjects	8,178	1,061	533
Sex
Male, n (%)	4,295 (52.5%)	488 (46.0%)	207 (38.8%)
Female, n (%)	3,883 (47.5%)	573 (54.0%)	325 (61.0%)
N/A, n (%)	–	–	1 (0.2%)
Age (years)	$54.98_{\pm 7.53}$	$28.79_{\pm 3.70}$	$11.94_{\pm 3.40}$

Appendix

Table A11 : Demographic information of the datasets used in our study

fMRI Representation Learning

See all 13 papers

FlexiBrain: Resolution-Agnostic Voxel-Level Encoding for Native fMRI
Jun 9, 2026Mo Wang, Wenhao Ye, Junfeng Xia +3Medical Imaging Foundation Models fMRI Foundation Models
Southern University of Science and Technology · University of Warwick
Scaling Vision Transformers for Functional MRI with Flat Maps
Oct 15, 2025Connor Lane, Mihir Tripathy, Leema Krishna Murali +15fMRI Foundation Models Masked Autoencoders
MedARC · Sophont · Baylor College of Medicine +3
Natural Image Autoencoder-Based fMRI Representations for Trait and State Prediction
Sep 28, 2026Juhyeon Park, Yeonwoo Kim, Peter Yongho Kim +5Medical Imaging Foundation Models fMRI Foundation Models
IPAI, Seoul National University · Microsoft Research · ECE, Seoul National University +1

Jun 9, 2026J/K move · Enter open · S save

FlexiBrain: Resolution-Agnostic Voxel-Level Encoding for Native fMRI

Mo Wang, Wenhao Ye, Junfeng Xia, Minghao Xu+2

Southern University of Science and Technology · University of Warwick

Medical Imaging Foundation Models fMRI Foundation Models fMRI Representation Learning Neuroimaging Medical Image Representation Learning

Open paper AskPDF arXiv

The success of large-scale deep learning models in neuroscience is fundamentally constrained by severe data heterogeneity. Native fMRI data aggregated from diverse sources exhibit substantial variation in both spatial and temporal resolutions. Consequently, most existing frameworks rely on lengthy, rigid preprocessing pipelines that enforce uniformity across datasets. This practice introduces two critical limitations: (1) potential degradation of subject-specific anatomical information; (2) significant computational overhead, often requiring hours of processing per subject. Here, we propose FlexiBrain, a resolution-agnostic voxel-level encoding framework for native fMRI based on Mamba-JEPA. FlexiBrain defines patch sizes in real-world physical units and employs a dynamic patch resizing, thereby bypassing destructive spatial standardization while enabling direct ingestion of data in native space. We instantiate the framework using an efficient Mamba-JEPA backbone to model high-dimensional 4D fMRI signals. Across five diverse downstream neuroscience tasks, FlexiBrain consistently outperforms recent state-of-the-art methods, achieving gains of up to 12 percentage points without external data augmentation. Importantly, FlexiBrain functions as a seamless plug-in module, substantially reducing preprocessing costs and accelerating the development of robust voxel-level fMRI foundation models. Code is available at https://github.com/OneMore1/FlexiBrain.

Abstract

fMRI Representation Learning

Explore similar work

FlexiBrain: Resolution-Agnostic Voxel-Level Encoding for Native fMRI

Scaling Vision Transformers for Functional MRI with Flat Maps

Natural Image Autoencoder-Based fMRI Representations for Trait and State Prediction