MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data

Paul S. Scotti; Mihir Tripathy; Cesar Kadir Torrico Villanueva; Reese Kneeland; Tong Chen; Ashutosh Narang; Charan Santhirasegaran; Jonathan Xu

cs.CVMar 17, 2024

MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data

Authors: Paul S. Scotti, Mihir Tripathy, Cesar Kadir Torrico Villanueva, Reese Kneeland, Tong Chen, Ashutosh Narang, Charan Santhirasegaran, Jonathan Xu,

Organizations: Stability AI · Medical AI Research Center (MedARC) · Princeton Neuroscience Institute · University of Minnesota ·

Abstract

Reconstructions of visual perception from brain activity have improved tremendously, but the practical utility of such methods has been limited. This is because such models are trained independently per subject where each subject requires dozens of hours of expensive fMRI training data to attain high-quality results. The present work showcases high-quality reconstructions using only 1 hour of fMRI training data. We pretrain our model across 7 subjects and then fine-tune on minimal data from a new subject. Our novel functional alignment procedure linearly maps all brain data to a shared-subject latent space, followed by a shared non-linear mapping to CLIP image space. We then map from CLIP space to pixel space by fine-tuning Stable Diffusion XL to accept CLIP latents as inputs instead of text. This approach improves out-of-subject generalization with limited training data and also attains state-of-the-art image retrieval and reconstruction metrics compared to single-subject approaches. MindEye2 demonstrates how accurate reconstructions of perception are possible from a single visit to the MRI facility. All code is available on GitHub.

Figures & tables

Figure 1: MindEye2 overall schematic. MindEye2 is trained using samples from 7 subjects in the Natural Scenes Dataset and then fine-tuned using a target held-out subject who may have scarce training data. Ridge regression maps fMRI activity to an initial shared-subject latent space. An MLP backbone and diffusion prior output OpenCLIP ViT-bigG/14 embeddings which SDXL unCLIP uses to reconstruct the seen image, which are then refined with base SDXL. The submodules help retain low-level information and support retrieval tasks. Snowflakes=frozen models used during inference, flames=actively trained.

Figure 2: SDXL unCLIP reconstructions + predicted image captions (left) are fed to base SDXL for refinement (right).

Figure 3: Reconstructions from different model approaches using 1 hour of training data from NSD.

Method	Low-Level				High-Level				Retrieval
	PixCorr $\uparrow$	SSIM $\uparrow$	Alex(2) $\uparrow$	Alex(5) $\uparrow$	Incep $\uparrow$	CLIP $\uparrow$	Eff $\downarrow$	SwAV $\downarrow$	Image $\uparrow$	Brain $\uparrow$
MindEye2	0.322	0.431	96.1%	$\underline{98.6\%}$	$\underline{95.4\%}$	$93.0\%$	0.619	$\underline{0.344}$	98.3%	98.8%
MindEye2 (unrefined)	$0.278$	$0.328$	$\underline{95.2\%}$	99.0%	96.4%	94.5%	$\underline{0.622}$	0.343	$-$	$-$
MindEye1	$\underline{0.319}$	$0.360$	$92.8\%$	$96.9\%$	$94.6\%$	$\underline{93.3\%}$	$0.648$	$0.377$	$\underline{84.1\%}$	$\underline{90.0\%}$
Ozcelik and VanRullen (2023)	$0.273$	$\underline{0.365}$	$94.4\%$	$96.6\%$	$91.3\%$	$90.9\%$	$0.728$	$0.421$	$18.8\%$	$26.3\%$
Takagi and Nishimoto (2023)	$0.246$	$0.410$	$78.9\%$	$85.6\%$	$83.8\%$	$82.1\%$	$0.811$	$0.504$	$-$	$-$

Table 1: Quantitative comparison of fMRI-to-image models. Results average across subjects 1, 2, 5, and 7 from the Natural Scenes Dataset. Results from all previous work were recalculated using their respective public codebases using the full 40 sessions of NSD data, which was not released until the recent completion of the 2023 Algonauts challenge . Image retrieval uses a brain query to identify its corresponding image among 300 image candidates; brain retrieval uses an image query to identify its corresponding brain sample among 300 brain candidates (chance=0.3%). MindEye2 and MindEye1 retrieval scores use the corrected directions described in Section 3.3 . PixCorr=pixelwise correlation between ground truth and reconstructions; SSIM=structural similarity index metric ( Wang et al., 2004 ) ; EfficientNet-B1 (“Eff”) ( Tan and Le, 2020 ) and SwAV-ResNet50 (“SwAV”) ( Caron et al., 2021 ) refer to average correlation distance; all other metrics refer to two-way identification (chance = 50%). Two-way identification refers to percent correct across comparisons gauging if the original image embedding is more similar to its paired brain embedding or a randomly selected brain embedding (see Appendix A.9 ). Missing values are from metrics being non-applicable. Bold indicates best performance, underline second-best performance.

Figure 4: Normalized reconstruction metrics for MindEye2 with (connected) or without (dotted) pretraining on other subjects, using varying amounts of training/fine-tuning data. Normalization was such that $0$ on the y-axis corresponds to metrics using random COCO images (not from NSD test set) as reconstructions and $1$ corresponds to metrics using 40-session pretrained MindEye2. Black lines indicate median. Test data is the same across all comparisons (see section 3 ).

	COCO captions		GIT captions
Metric	MindEye2	UniBrain	MindEye2	Ferrante et al.
METEOR $\uparrow$	0.248	0.170	0.344	0.305
ROUGE-L $\uparrow$	0.326	0.225	0.427	-
ROUGE-1 $\uparrow$	0.353	0.247	0.455	-
Sentence $\uparrow$	47.9%	-	52.3%	44.7%
CLIP-B $\uparrow$	73.7%	-	75.4%	70.5%

Table 2: FMRI-to-image caption evaluations. Previous works used different ground truth captions for comparison (COCO captions or captions generated from GIT), necessitating separate comparisons. Results were calculated exclusively on NSD subject 1. MindEye2 metrics come from the model trained on all 40 sessions of NSD data whereas previous work used 37 sessions.

Brain Region	MindEye2	MindEye2 (unrefined)	MindEye2 (1 hour)	Brain Diffuser	Takagi et al.
Visual cortex $\uparrow$	0.373	0.384	0.348	0.381	0.247
V1 $\uparrow$	0.364	0.385	0.309	0.362	0.181
V2 $\uparrow$	0.352	0.366	0.314	0.340	0.152
V3 $\uparrow$	0.342	0.353	0.315	0.332	0.152
V4 $\uparrow$	0.327	0.339	0.300	0.323	0.170
Higher vis. $\uparrow$	0.368	0.373	0.351	0.375	0.288

Table 3: Brain correlation scores calculated in different brain regions including visual cortex, early visual cortical regions V1, V2, V3, and V4, and higher visual areas (set complement of visual cortex and early visual cortex).

Metric		ME2	ME1	CLIP L
Low-Level	PixCorr $\uparrow$	0.292	0.225	0.243
	SSIM $\uparrow$	0.386	0.380	0.371
	Alex(2) $\uparrow$	92.7%	87.3%	84.8%
	Alex(5) $\uparrow$	97.6%	94.7%	93.7%
High-Level	Incep $\uparrow$	91.5%	88.9%	87.7%
	CLIP $\uparrow$	90.5%	86.2%	89.2%

Table 4: Ablations on how MindEye2 (ME2) improves upon MindEye1. "ME1" results replace the initial linear mapping of fMRI voxels with MindEye1’s MLP with dropout. "CLIP L" results map voxels to CLIP L (reconstructions via Versatile Diffusion) instead of OpenCLIP bigG (reconstructions via SDXL unCLIP).

Metric		Prior	Prior+Low	Prior+Ret.	All
Low-Level	PixCorr $\uparrow$	0.155	0.281	0.233	0.267
	SSIM $\uparrow$	0.309	0.385	0.319	0.380
	Alex(2) $\uparrow$	79.6%	89.4%	90.6%	89.7%
	Alex(5) $\uparrow$	88.6%	96.2%	96.8%	96.4%
High-Level	Incep $\uparrow$	85.3%	91.5%	91.9%	91.4%
	CLIP $\uparrow$	79.5%	88.4%	89.4%	87.9%

Table 5: Ablations compare reconstruction and retrieval metrics for MindEye2 trained with various combinations of model components. Retr.=Retrieval submodule, Low=Low-level submodule.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Method		MindEye2	MindEye1
Low-Level	PixCorr $\uparrow$	0.376	0.388
	SSIM $\uparrow$	0.440	0.355
	Alex(2) $\uparrow$	97.5%	96.1%
	Alex(5) $\uparrow$	99.1%	98.3%
High-Level	Incep $\uparrow$	95.4%	95.0%
	CLIP $\uparrow$	92.6%	93.7%

Appendix

Table 6: Performance comparison between MindEye2 (refined) and MindEye1 both trained from scratch across all 40 NSD sessions using only subject 1 data.

Figure 5: Low-level metric performance (y-axis) plotted against the number of fMRI scanning sessions used in the training data (x-axis) for subject 1. All values are normalized to the same y-axis. The bolded line represents the average performance across all metrics.

Figure 6: High-level metric performance (y-axis) plotted against the number of fMRI scanning sessions used in the training data (x-axis) for subject 1. All values are normalized to the same y-axis. The bolded line represents the average performance across all metrics. SwAV and EffNet-B scores are inverted in this plot so that higher is better for all metrics.

Figure 7: Brain correlation scores (y-axis) in different brain regions including visual cortex (defined by the nsdgeneral mask, bolded), V1, V2, V3, V4 (collectively called early visual cortex) and higher visual areas (the set complement of nsdgeneral and early visual cortex) plotted against the number of fMRI scanning sessions used in the training data (x-axis) for subject 1. All values are normalized to the same y-axis.

$40$ Session Subject Results		Subject 1	Subject 2	Subject 5	Subject 7
Low	PixCorr $\uparrow$	0.374	0.328	0.301	0.283
	SSIM $\uparrow$	0.439	0.430	0.432	0.423
	Alex(2) $\uparrow$	97.82%	97.01%	95.32%	94.25%
	Alex(5) $\uparrow$	99.10%	98.83%	98.54%	97.97%
High	Incep $\uparrow$	96.15%	94.90%	96.52%	94.09%
	CLIP $\uparrow$	93.56%	91.66%	94.32%	92.36%

Appendix

Table 7: Single subject quantitative results for 40 sessions of training data.

$1$ Session Subject Results		Subject 1	Subject 2	Subject 5	Subject 7
Low	PixCorr $\uparrow$	0.235	0.200	0.175	0.170
	SSIM $\uparrow$	0.428	0.433	0.405	0.408
	Alex(2) $\uparrow$	88.02%	85.00%	83.11%	80.70%
	Alex(5) $\uparrow$	93.33%	92.13%	91.00%	85.90%
High	Incep $\uparrow$	83.56%	81.86%	84.33%	74.90%
	CLIP $\uparrow$	80.75%	79.39%	82.53%	74.29%

Appendix

Table 8: Single subject quantitative results for 1 session of training data.

Figure 8: Generating images from their CLIP image embeddings. SDXL unCLIP (middle) outperforms Versatile Diffusion (right) in capturing perceptual details.

Metrics	SDXL unCLIP	VD
FID $\downarrow$	13.69	22.04
PixCorr $\uparrow$	0.676	0.266
SSIM $\uparrow$	0.232	0.055
Alex(2) $\uparrow$	0.998	0.972
Alex(5) $\uparrow$	0.998	0.966
Incep $\uparrow$	0.997	0.994

Appendix

Table 9: SDXL unCLIP reconstructions from ground truth OpenCLIP image latents consistently outperform Versatile Diffusion reconstructions from ground truth CLIP image latents.

Figure 9: UMAP plots depict CLIP image latents (blue), backbone latents (green), retrieval submodule latents (orange), and diffusion prior latents (red). UMAPs were estimated across the 1,000 test samples for subject 1, using the full 40-session model. CLIP image latents correspond to the $256\times 1664$ dimensionality of OpenCLIP ViT-bigG/14 image token embeddings. Euclidean distance between the given MindEye2 embedding space and CLIP image space is lowest for the diffusion prior, suggesting that the diffusion prior helps to align the two embedding spaces.

Metric		Sub 2-8	Sub 2,5,7	Sub 5
Low-Level	PixCorr $\uparrow$	0.235	0.234	0.232
	SSIM $\uparrow$	0.428	0.421	0.421
	Alex(2) $\uparrow$	88.0%	89.1%	88.9%
	Alex(5) $\uparrow$	93.3%	94.1%	93.6%
High-Level	Incep $\uparrow$	83.6%	83.8%	83.8%
	CLIP $\uparrow$	80.8%	83.5%	82.7%

Appendix

Table 10: Evaluation metrics for MindEye2 models both fine-tuned on 1 hour of training data from subject 1 but pretrained on different numbers of subjects.

Metrics	Brain captions	Empty prompt
PixCorr $\uparrow$	0.235	0.235
SSIM $\uparrow$	0.428	0.423
Alex(2) $\uparrow$	88.02%	87.46%
Alex(5) $\uparrow$	93.33%	93.22%
Incep $\uparrow$	83.56%	82.67%
CLIP $\uparrow$	80.75%	81.21%

Appendix

Table 11: Using image captions predicted from brain activity for refinement seems to modestly improve subsequent image reconstructions from fMRI activity compared to using empty text prompts.

Figure 10: Subject-specific UMAPs on the diffusion prior outputs from the 40-hour MindEye2 models help to visualize individual differences with regard to reconstuction quality for different categories of natural images.

Figure 11: Unrefined reconstructions and decoded captions from synthetic fMRI activity. Voxels in the desired target brain region were set to a one while all other voxels were set to zero, and this synthetic brain data was fed through each subject’s MindEye2 model.

Figure 12: Examples of trials where subjects preferred Brain Diffuser (1-hour) reconstructions over MindEye2 (1-hour) reconstructions. It seems possible human raters tended to select the more distorted reconstruction of the two when both reconstructions were of bad quality, but the distortions trend towards correct features.

Image Reconstruction

Image reconstruction recovers images, volumes, or scenes from incomplete, noisy, indirect, or compressed measurements using physical models and learned priors.

See all 304 papers

Align, Then Adapt: Few-Shot fMRI Visual Reconstruction For New Subjects
May 3, 2025Christos Zangos, Danish Ebadulla, Thomas Christopher Sprague +1fMRI-Based Visual Decoding fMRI Representation Learning
Department of Computer Science, University of California, Santa Barbara, USA · Department of Psychological and Brain Sciences, University of California, Santa Barbara, USA
StableMind: Source-Free Cross-Subject fMRI Decoding with Regularized Adaptation
May 4, 2026Jintao Guo, Lin Wang, Shumeng Li +5Source-Free Domain Adaptation Neural Decoding
National Key Laboratory for Novel Software Technology and the Institute of Brain-Machine Interface, Nanjing University, Nanjing 210023, China · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore 639798 · School of Intelligence Science and Technology, Nanjing University, Nanjing 215163, China +2
Seeing the imagined: latent functional alignment in visual imagery decoding from fMRI data
Apr 15, 2026Fabrizio Spera, Tommaso Boccato, Michal Olak +2Neural Decoding Visual Neuroscience
Department of Biomedicine and Prevention, University of Rome Tor Vergata, Rome, Italy · Tether Evo · A.A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Boston, USA

May 3, 2025J/K move · Enter open · S save

Align, Then Adapt: Few-Shot fMRI Visual Reconstruction For New Subjects

Christos Zangos, Danish Ebadulla, Thomas Christopher Sprague, Ambuj Singh

Department of Computer Science, University of California, Santa Barbara, USA · Department of Psychological and Brain Sciences, University of California, Santa Barbara, USA

fMRI-Based Visual Decoding fMRI Representation Learning Few-Shot Domain Adaptation Latent Space Alignment

Open paper AskPDF arXiv

Reconstructing visual images from fMRI data is a central but challenging problem in neuroscience. Despite recent progress, current methods fall short when data and computation are limited-precisely the conditions under which this task is most critical. We introduce Adapter Alignment, a staged, architecture-agnostic training procedure for adapting a pre-trained reconstruction model to a new subject with little data. Rather than letting a shared representation space emerge implicitly, lightweight subject-specific adapters are first explicitly aligned to a pre-trained reference subject on a small set of shared calibration images, and the full pipeline is then fine-tuned while preserving this alignment. The resulting representations not only place different subjects in a shared space but also align them semantically. Our results demonstrate significant gains in low-data regimes, improving reconstruction metrics by up to 8%, and pre-diffusion retrieval shows that these gains reflect additional image-specific information in the fMRI embeddings rather than the generative prior. We also propose a coverage-based algorithm to select representative calibration images. Using both techniques together, we surpass baseline models fine-tuned with one hour of data using only 24 minutes of subject-specific data. The procedure is effective across architectures, reference subjects, and datasets, including adapting 3T subjects to a 7T reference.

Abstract

Image Reconstruction

Explore similar work

Align, Then Adapt: Few-Shot fMRI Visual Reconstruction For New Subjects

StableMind: Source-Free Cross-Subject fMRI Decoding with Regularized Adaptation

Seeing the imagined: latent functional alignment in visual imagery decoding from fMRI data