Align, Then Adapt: Few-Shot fMRI Visual Reconstruction For New Subjects

Christos Zangos; Danish Ebadulla; Thomas Christopher Sprague; Ambuj Singh

eess.IVMay 3, 2025

Align, Then Adapt: Few-Shot fMRI Visual Reconstruction For New Subjects

Authors: Christos Zangos, Danish Ebadulla, Thomas Christopher Sprague, Ambuj Singh

Organizations: Department of Computer Science, University of California, Santa Barbara, USA · Department of Psychological and Brain Sciences, University of California, Santa Barbara, USA

Abstract

Reconstructing visual images from fMRI data is a central but challenging problem in neuroscience. Despite recent progress, current methods fall short when data and computation are limited-precisely the conditions under which this task is most critical. We introduce Adapter Alignment, a staged, architecture-agnostic training procedure for adapting a pre-trained reconstruction model to a new subject with little data. Rather than letting a shared representation space emerge implicitly, lightweight subject-specific adapters are first explicitly aligned to a pre-trained reference subject on a small set of shared calibration images, and the full pipeline is then fine-tuned while preserving this alignment. The resulting representations not only place different subjects in a shared space but also align them semantically. Our results demonstrate significant gains in low-data regimes, improving reconstruction metrics by up to 8%, and pre-diffusion retrieval shows that these gains reflect additional image-specific information in the fMRI embeddings rather than the generative prior. We also propose a coverage-based algorithm to select representative calibration images. Using both techniques together, we surpass baseline models fine-tuned with one hour of data using only 24 minutes of subject-specific data. The procedure is effective across architectures, reference subjects, and datasets, including adapting 3T subjects to a 7T reference.

Figures & tables

Panel of Figure 1 — Figure 1: Cross-subject cosine similarity of fMRI embeddings in the common space extracted from a pre-trained MindEye2 model. (a) Cross-subject alignment is weak before any transformation. (b) Applying an orthogonal transformation substantially improves cross-subject similarity. (c) Explicit alignment through lightweight adapter modules achieves near-perfect alignment. Only Subjects 1, 2, 5, and 7 are shown for consistency throughout the manuscript.

Figure 2: Training procedures for fMRI-Image Reconstruction. Top: Traditional Approach to reconstruction. All subjects are trained simultaneously and loss functions are applied at the MLP Mapper to align the output to pre-trained CLIP space. Bottom: Proposed AdapterAlign training pipeline. First, we train a single reference subject and obtain its embeddings in the shared space. Then we train subsequent subjects to minimize loss at both the shared space and the mapper. The same mapper weights are transferred over in step 2.

	Low-Level				High-Level
Method	PixCorr $\uparrow$	SSIM $\uparrow$	Alex(2) $\uparrow$	Alex(5) $\uparrow$	Incep $\uparrow$	CLIP $\uparrow$	Eff $\downarrow$	SwAV $\downarrow$
Subject 1 Pre-train	0.345	0.346	92.80%	96.88%	94.40%	90.02%	0.692	0.399
Baseline FT AVG	0.258	0.339	89.64%	95.05%	92.63%	89.64%	0.717	0.427
AAMax FT AVG	0.259	0.337	89.20%	95.16%	92.68%	89.49%	0.713	0.423

Table 1: Quantitative results on training the reconstruction models on 40 hours of data. The fine-tuning results are averaged over subjects 2,5 and 7. AAMax and baseline fine-tuning perform similarly across all metrics. Std. deviations are provided in Appendix U .

	Low-Level				High-Level
Method	PixCorr $\uparrow$	SSIM $\uparrow$	Alex(2) $\uparrow$	Alex(5) $\uparrow$	Incep $\uparrow$	CLIP $\uparrow$	Eff $\downarrow$	SwAV $\downarrow$
Subject 2 FT 1HR Baseline	0.078 $\pm$ 0.010	0.264 $\pm$ 0.007	75.33% $\pm$ 1.62%	84.18% $\pm$ 1.40%	74.15% $\pm$ 0.99%	73.66% $\pm$ 1.3%	0.862 $\pm$ 0.011	0.524 $\pm$ 0.009
Subject 2 FT Selected 1HR Baseline	0.097 $\pm$ 0.003	0.288 $\pm$ 0.004	78.78% $\pm$ 0.41%	85.78% $\pm$ 0.69%	77.13% $\pm$ 0.76%	76.17% $\pm$ 0.03%	0.842 $\pm$ 0.003	0.513 $\pm$ 0.005
Subject 2 FT 1HR AAMax	0.112 $\pm$ 0.005	0.259 $\pm$ 0.025	78.75% $\pm$ 1.29%	88.04% $\pm$ 1.19%	78.99% $\pm$ 0.97%	78.28% $\pm$ 0.39%	0.824 $\pm$ 0.003	0.513 $\pm$ 0.014
Subject 2 FT Selected 1HR AAMax	0.102 $\pm$ 0.003	0.267 $\pm$ 0.11	81.41% $\pm$ 0.27%	89.55% $\pm$ 0.26%	80.43% $\pm$ 0.13%	80.33% $\pm$ 0.01%	0.823 $\pm$ 0.002	0.496 $\pm$ 0.002
Subject 5 FT 1HR Baseline	0.082 $\pm$ 0.002	0.267 $\pm$ 0.002	75.05% $\pm$ 0.08%	83.28% $\pm$ 0.27%	76.04% $\pm$ 1.12%	76.40% $\pm$ 0.64%	0.855 $\pm$ 0.002	0.526 $\pm$ 0.006

Table 2: Limited data fine-tuning results (1 hour, 250 images) using Subject 1 as the reference and Subjects 2, 5, and 7 as fine-tuning subjects. Each subject is evaluated under four settings: Baseline with random sampling, AAMax with random sampling, Baseline with image selection (IS), and AAMax with IS. Results are averaged over three runs with fixed random splits. AAMax consistently outperforms the baseline, IS provides additional gains in both training regimes, and the best performance is achieved by combining AAMax with IS. Pixel-level metrics are shown in gray since they are not optimized in these runs but are still valid.

Panel of Figure 3 — Figure 3: Pushing the limits of data efficiency with AAMax+Image Selection. Left: Subject 2 fine-tuned from Subject 1. Right: Subject 5 fine-tuned from Subject 1. In both cases, AAMax and AAMax+Image Selection outperform the baseline trained with 250 images, even when using only 150 images (a 40% reduction) and match performance with 100 images (a 60% reduction). Image selection in these experiments uses the top 20 eigenvector dimensions (Appendix F ).

	Low-Level				High-Level
Method	PixCorr $\uparrow$	SSIM $\uparrow$	Alex(2) $\uparrow$	Alex(5) $\uparrow$	Incep $\uparrow$	CLIP $\uparrow$	Eff $\downarrow$	SwAV $\downarrow$
MindAligner (MA)	0.116	0.280	82.11%	88.68%	81.36%	79.69%	0.809	0.483
AAMax + IS	0.118	0.285	81.78%	89.61%	81.49%	79.55%	0.810	0.491
MA + AAMax + IS	0.124	0.284	82.24%	89.95%	81.49%	80.99%	0.795	0.480

Table 3: Comparison and combination of MindAligner and Adapter Alignment. All methods use MindEye1 with pretraining on Subject 1 and fine-tuning on Subject 2 for 1 hour of data. MindAligner follows the original paper, using categorical stimulus pairs. AA uses our 250-image Image Selection subset that spans the breadth of the principal eigenspace. The combined method applies MindAligner for coarse voxel-space alignment followed by AAMax + Image Selection for fine-grained common-space alignment. The combined approach achieves the strongest overall performance.

Figure 4: Examples of reconstructed images from Subject 1 of the THINGS dataset after fine-tuning. Each pair depicts the ground truth (left) and the reconstructed image (right).

	Low-Level		High-Level
Method	Alex(2) $\uparrow$	Alex(5) $\uparrow$	Incep $\uparrow$	CLIP $\uparrow$	Eff $\downarrow$	SwAV $\downarrow$
NSD Subj1 Pre-train	92.80%	96.88%	94.40%	90.02%	0.692	0.399
THINGS Subj1 Individual	68.02%	72.6%	58.8%	64.29%	0.947	0.649
THINGS Subj1 Baseline Fine-Tune	77.46%	81.07%	71.06%	73.16%	0.911	0.619
THINGS Subj1 AAMax Fine-Tune	78.27%	85.55%	74.10%	81.37%	0.902	0.595

Table 4: Quantitative results on fine-tuning THINGS-fMRI Subject 1 using NSD Subject 1 as the reference subject. Only the high level pipeline is trained and therefore, pixelwise metrics are excluded. AAMax outperforms the baseline even when an explicit data mapping is not available.

Appendix figures & tables38 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Where alignment acts	Alignment signal	How alignment is trained
Hyperalignment / SRM ( Haxby et al., 2011 ; Chen et al., 2015 )	Voxel / response space	Shared stimuli	Separate alignment model (no reconstruction)
MindEye2 ( Scotti et al., 2024 )	Shared space after linear adapters	Implicit, via reconstruction losses	Jointly end-to-end; no explicit alignment
Ferrante et al. (2024)	Adapter output (reference space)	Shared stimuli	Explicit alignment only; no subsequent end-to-end training
MindBridge ( Wang et al., 2024 )	Subject-invariant shared space	Cyclic reconstruction	Jointly with the reconstruction objective
MindAligner ( Dai et al., 2025 )	Voxel space	Category-level pairs	Explicit voxel-space alignment module
Adapter Alignment (ours)	Adapter output (reference space)	Shared or semantically matched calibration images	Explicit alignment, then end-to-end fine-tuning retaining the alignment loss

Appendix

Table 5: Where and how related methods enforce cross-subject alignment.

Method	Split	Alex(2) $\uparrow$	Alex(5) $\uparrow$	Incep $\uparrow$	CLIP $\uparrow$	Eff $\downarrow$
MindEye2 Baseline FT	Standard	82.94%	89.68%	80.36%	78.74%	0.814
MindEye2 Baseline FT	Modified	79.54%	86.46%	78.50%	75.13%	0.847
MindEye2 AAMax FT	Modified	81.23%	88.80%	82.79%	79.08%	0.819

Appendix

Table 6: Effect of the dataset split on MindEye2 fine-tuning (1 hour of data, Subject 1 as the reference). The standard-split baseline was rerun with the hyperparameters recommended by MindEye2; only the split differs between the standard and modified rows. All rows are averaged over Subjects 2, 5, and 7; per-subject modified-split results are given in Appendices W and C .

	Low-Level				High-Level
Method	PixCorr $\uparrow$	SSIM $\uparrow$	Alex(2) $\uparrow$	Alex(5) $\uparrow$	Incep $\uparrow$	CLIP $\uparrow$	Eff $\downarrow$	SwAV $\downarrow$
Subject 5 FT 1HR Baseline	0.102	0.358	78.40%	86.61%	79.85%	77.23%	0.839	0.494
Subject 5 FT 1HR AAMax	0.121	0.354	80.68%	88.86%	85.12%	81.89%	0.799	0.469
Subject 7 FT 1HR Baseline	0.103	0.356	78.50%	84.70%	76.81%	74.02%	0.858	0.506
Subject 7 FT 1HR AAMax	0.131	0.390	80.80%	87.86%	80.29%	76.55%	0.836	0.508

Appendix

Table 7: Fine-tuning on Subjects 5 and 7 using MindEye2’s architecture. AAMax outperforms the baseline on most metrics. The same reference subject checkpoint was used in all the runs and all models were trained with the exact same hyperparameter setup.

	Low-Level				High-Level
Method	PixCorr $\uparrow$	SSIM $\uparrow$	Alex(2) $\uparrow$	Alex(5) $\uparrow$	Incep $\uparrow$	CLIP $\uparrow$	Eff $\downarrow$	SwAV $\downarrow$
Subject 2 FT 1HR AAMAX linear	0.110	0.277	79.70%	88.61%	80.35%	79.88%	0.815	0.486
Subject 2 FT 1HR AAMAX GELU	0.112	0.259	78.75%	88.04%	78.99%	78.28%	0.824	0.513
Subject 2 FT 1HR AAMAX ReLU	0.101	0.273	80.44%	88.18%	78.72%	78.17%	0.827	0.508
Subject 2 FT 1HR AAMAX 2 linear	0.107	0.268	78.28%	87.12%	78.78%	78.62%	0.827	0.504

Appendix

Table 8: Ablation on adapter variants for Subject 2 fine-tuned with 1 hour of data under AAMax training. We compare a nonlinear adapter with GELU, a nonlinear adapter with ReLU, and a two-layer linear adapter. Results show that adapter choice has little impact on performance, with all variants yielding similar alignment and reconstruction quality.

Method	PixCorr $\uparrow$	SSIM $\uparrow$	Alex(2) $\uparrow$	Alex(5) $\uparrow$	Incep $\uparrow$	CLIP $\uparrow$	Eff $\downarrow$	SwAV $\downarrow$
	Low-Level				High-Level
Subject 1 FT From Subject 2 1HR Baseline	0.007	0.268	75.55%	84.00%	75.20%	74.60%	0.862	0.520
Subject 1 FT From Subject 2 1HR AAMAX	0.060	0.277	81.13%	88.99%	79.71%	78.89%	0.824	0.494
Subject 1 FT From Subject 5 1HR Baseline	0.094	0.239	71.54%	80.10%	73.11%	74.02%	0.875	0.537
Subject 1 FT From Subject 5 1HR AAMAX	0.105	0.259	76.77%	86.20%	80.08%	80.14%	0.828	0.496
Subject 1 FT From Subject 7 1HR Baseline	0.010	0.266	74.96%	81.55%	71.71%	73.72%	0.874	0.528

Appendix

Table 9: Effect of reference subject choice on limited-data fine-tuning. We compare AAMax and normal fine-tuning (160 epochs) when Subjects 2, 5, and 7 are used as reference subjects and Subject 1 is fine-tuned. AAMax consistently outperforms normal fine-tuning regardless of the reference, though overall performance reflects the quality of the chosen reference subject.

Method	PixCorr $\uparrow$	SSIM $\uparrow$	Alex(2) $\uparrow$	Alex(5) $\uparrow$	Incep $\uparrow$	CLIP $\uparrow$	Eff $\downarrow$	SwAV $\downarrow$
	Low-Level				High-Level
Subject 5 FT From Subject 2 1HR AAMAX	0.097	0.275	77.31%	86.98%	81.41%	80.06%	0.829	0.488
Subject 7 FT From Subject 2 1HR AAMAX	0.112	0.272	77.05%	84.75%	78.39%	76.95%	0.843	0.502
Subject 2 FT From Subject 5 1HR AAMAX	0.092	0.264	76.74%	86.27%	78.80%	78.24%	0.823	0.489
Subject 7 FT From Subject 5 1HR AAMAX	0.093	0.252	76.49%	85.31%	79.21%	77.99%	0.834	0.504
Subject 2 FT From Subject 7 1HR AAMAX	0.098	0.264	75.73%	84.33%	77.69%	77.36%	0.838	0.494

Appendix

Table 10: AAMax fine-tuning (1 hour of data) for all remaining reference–target combinations among Subjects 2, 5, and 7. Results are shown for reference; performance depends on the pairing of reference and target subject. Low-level metrics are shown in gray because they are not optimized in these runs.

	Low-Level				High-Level
Method	PixCorr $\uparrow$	SSIM $\uparrow$	Alex(2) $\uparrow$	Alex(5) $\uparrow$	Incep $\uparrow$	CLIP $\uparrow$	Eff $\downarrow$	SwAV $\downarrow$
Pre-train Subjects 1+2 $\rightarrow$ FT Subject 5 1HR AAMAX	0.102	0.263	78.79%	88.45%	81.08%	81.41%	0.804	0.495
Pre-train Subject 1 $\rightarrow$ FT Subject 5 1HR AAMAX	0.105	0.278	79.90%	88.68%	80.63%	82.20%	0.808	0.493

Appendix

Table 11: Single- versus multi-subject reference pre-training. Subject 5 is fine-tuned with AAMax on 1 hour of data. The single-reference row corresponds to Subject 5 FT 1HR AAMax in Table 2 .

Figure 5: Coverage of the image selection algorithm compared to random sampling. We project embeddings onto the top 20 eigenvectors and count empty bins across 250 selected images. The histogram shows empty-bin counts for 1,000 random subsets, while the dashed line marks our algorithm’s selection. Our method leaves only 15 bins empty, whereas random sampling leaves $\approx 200\pm 10$ on average, indicating significantly more uniform coverage of the representation space ( $p=0.0002$ ).

	Low-Level				High-Level
Method	PixCorr $\uparrow$	SSIM $\uparrow$	Alex(2) $\uparrow$	Alex(5) $\uparrow$	Incep $\uparrow$	CLIP $\uparrow$	Eff $\downarrow$	SwAV $\downarrow$
Random Selection	0.112	0.259	78.75%	88.04%	78.99%	78.28%	0.824	0.513
10 EV Dimensions	0.114	0.303	78.99%	86.53%	77.30%	75.23%	0.850	0.541
20 EV Dimensions	0.102	0.267	81.41%	89.55%	80.43%	80.33%	0.823	0.496
30 EV Dimensions	0.112	0.275	79.60%	88.03%	78.58%	76.89%	0.850	0.537
40 EV Dimensions	0.118	0.285	81.78%	89.61%	81.49%	79.55%	0.810	0.491

Appendix

Table 12: Ablation on the number of eigenvector dimensions used for image selection. We compare random image selection and Adapter Alignment with images selected using our algorithm across different eigenvector settings. Results are reported for Subject 1 as the reference and Subject 2 fine-tuned with 250 images under AAMax training, averaged over 3 runs. 20 and 40 eigenvectors are competitive, and 40 eigenvectors are used for the image-selection results in Tables 2 and 3 .

	Low-Level				High-Level
Method	PixCorr $\uparrow$	SSIM $\uparrow$	Alex(2) $\uparrow$	Alex(5) $\uparrow$	Incep $\uparrow$	CLIP $\uparrow$	Eff $\downarrow$	SwAV $\downarrow$
Ours	0.118	0.285	81.78%	89.61%	81.49%	79.55%	0.810	0.491
K-Means	0.110	0.281	82.40%	89.14%	80.51%	80.26%	0.815	0.490
K-Center Greedy	0.114	0.271	81.93%	89.54%	80.10%	79.35%	0.817	0.489
DPP Diversity	0.115	0.270	80.80%	89.22%	80.21%	78.66%	0.820	0.493
CLIP K-Center	0.112	0.290	81.15%	88.73%	78.56%	79.92%	0.831	0.499

Appendix

Table 13: Comparison of image-selection strategies. Subject 1 is the reference and Subject 2 is fine-tuned with AAMax on 250 selected images; only the selection rule differs between rows. Best results are in bold.

	Ours	K-Center	K-Means	DPP	CLIP K-Center
Ours	250 (100.0%)	73 (29.2%)	74 (29.6%)	65 (26.0%)	58 (23.2%)
K-Center	73 (29.2%)	250 (100.0%)	43 (17.2%)	131 (52.4%)	77 (30.8%)
K-Means	74 (29.6%)	43 (17.2%)	250 (100.0%)	43 (17.2%)	60 (24.0%)
DPP	65 (26.0%)	131 (52.4%)	43 (17.2%)	250 (100.0%)	64 (25.6%)
CLIP K-Center	58 (23.2%)	77 (30.8%)	60 (24.0%)	64 (25.6%)	250 (100.0%)

Appendix

Table 14: Pairwise overlap between the 250-image subsets selected by each method. Entries are the number of shared images, with percentages in parentheses.

	Low-Level				High-Level
Method	PixCorr $\uparrow$	SSIM $\uparrow$	Alex(2) $\uparrow$	Alex(5) $\uparrow$	Incep $\uparrow$	CLIP $\uparrow$	Eff $\downarrow$	SwAV $\downarrow$
AAMAX	0.112	0.259	78.75%	88.04%	78.99%	78.28%	0.824	0.513
AA Step 1 only	0.008	0.240	77.14%	87.04%	77.00%	75.43%	0.857	0.510
End-to-End with MSE+Prior+CLIP Loss (MLP Frozen)	0.007	0.238	76.40%	86.42%	77.72%	75.04%	0.852	0.507

Appendix

Table 15: Comparison of different training regimes for fine-tuning Subject 2 with 250 training images. We evaluate (1) AAMax (full Adapter Alignment with staged alignment + end-to-end fine-tuning), (2) AA Step 1 only (adapter overfitting without subsequent end-to-end training), and (3) End-to-End with MSE+Prior+CLIP Loss (MLP Frozen) (reset-tuning). AAMax achieves the best performance across both low-level and high-level metrics, demonstrating that explicit staged alignment followed by end-to-end fine-tuning is more effective than either alignment-only or combined end-to-end training when compared under identical architectural conditions.

	Low-Level				High-Level
Method	PixCorr $\uparrow$	SSIM $\uparrow$	Alex(2) $\uparrow$	Alex(5) $\uparrow$	Incep $\uparrow$	CLIP $\uparrow$	Eff $\downarrow$	SwAV $\downarrow$
Subject 1 FT 1HR Baseline E1	0.031	0.234	58.31%	62.56%	58.51%	58.71%	0.947	0.611
Subject 1 FT 1HR AAMAX E1	0.095	0.254	77.20%	85.98%	78.80%	79.39%	0.833	0.503
Subject 1 FT 1HR Baseline E160	0.094	0.239	71.54%	80.10%	73.11%	74.02%	0.875	0.537
Subject 1 FT 1HR AAMAX E160	0.105	0.259	76.77%	86.20%	80.08%	80.14%	0.828	0.496
Subject 1 FT 2HR Baseline E1	0.049	0.242	62.41%	68.06%	62.64%	64.80%	0.925	0.583

Appendix

Table 16: Quantitative results on fine-tuning a new subject with limited data. FT=Fine tuning. The first section shows the result of fine-tuning with 250 (1 hour) images after 1 epoch and after 160 epochs on the baseline and using AAMax. The second section compares performance on 500 (2 hours) images and the third section compares performance on 1000 (4 hours) images. In all cases AAMax, significantly outperforms the baseline. The greyed out low level metrics are not optimized for in these experiments.

Panel of Figure 6 — Figure 6: Comparing finetuning reconstruction performance over epochs with limited data. From left to right: (a) Finetuning Subject 1 with 250 images or 1 hour of data (b) Finetuning Subject 1 with 500 images or 2 hours of data (c) Finetuning Subject 1 with 1000 images or 4 hours of data. AAMax not only outperforms normal end-to-end training, but the performance at Epoch 1 is better than the baseline’s performance at Epoch 160.

Figure 7: Reconstruction performance across epochs for Subject 2 fine-tuned with 1 hour of data using Subject 1 as the reference. The shaded regions denote Step 1 (adapter alignment) and Step 2 (end-to-end fine-tuning). Performance steadily increases during adapter alignment but plateaus short of optimal results. Transitioning to end-to-end fine-tuning yields an immediate jump in accuracy and ultimately outperforms adapter-only training. The dashed lines indicate the transition from AA to E2E. Performance at Epoch 300 of AA is the same as Epoch 0 of E2E, since no further fine-tuning has been done yet.

Panel of Figure 8 — Figure 8: Comparing performance with scale for AAMax. Left: AAMax vs regular finetuning performance over increasing shared space size. AAMax scales with shared space dimensionality. Right: AAMax finetuning performance at 1024 shared space with Versatile Diffusion and Stable Diffusion XL. All metrics improve when moving to a larger diffusion model.

Panel of Figure 9 — Figure 9: Comparing eigenvector cosine similarity for the test data across subjects in both training techniques. We use the first 5 principal eigenvectors. Row 1 shows the results for normal end-to-end training. Except for the first eigenvector, cosine similarity is very low across subjects. Row 2 shows the results for AAMax training. Cosine similarity is high even at the 5th principal eigenvector.

Figure 10: Comparing eigenvector cosine similarity for the test data across datasets in both training techniques. 1. Reference Subject 1 NSD 2. Normal Finetune Subject 1 THINGS 3. AAMax Finetune Subject 1 THINGS. AAMax training generates better aligned embedding spaces.

Figure 11: Comparing Mean Squared Error for the shared images across subjects in both training techniques. The MSE is compared after training is completed in both techniques. MSE for AAMax is an order of magnitude lower than the baseline

Figure 12: Comparing kNN consistency @ 50 for the shared images across subjects in both training techniques. The kNN-C is compared after training is completed in both techniques. AAMax significantly outperforms the baseline.

Metric	Baseline	AAMax
Paired cosine $\uparrow$ (95% CI)	0.366 [0.360, 0.373]	0.576 [0.569, 0.582]
Paired MSE $\downarrow$	8.42	6.30
kNN consistency @50 $\uparrow$	0.339	0.427
Subject 1 $\rightarrow$ 2 identification, top-1 / top-5 $\uparrow$	50.80% / 76.27%	92.40% / 98.93%
Subject 2 $\rightarrow$ 1 identification, top-1 / top-5 $\uparrow$	49.47% / 78.13%	87.73% / 97.47%

Appendix

Table 17: Cross-subject alignment of the full adapter output on 750 held-out shared images (Subject 1

\rightarrow

Subject 2, MindEye2). Identification is 750-way.

	Region	Baseline	AAMax	$\Delta$
pRF areas	All pRF voxels	0.344	0.505	+0.160
	V1	0.286	0.394	+0.108
	V2	0.241	0.335	+0.094
	V3	0.186	0.266	+0.080
	hV4	0.171	0.250	+0.079
Eccentricity	$0.5^{\circ}$	0.102	0.180	+0.078

Appendix

Table 18: Region-resolved paired cosine similarity on 750 held-out shared images. Each entry compares only the contribution of the given pRF-defined area, eccentricity band, or fLOC-defined category system. 95% paired-bootstrap confidence intervals of the difference are shown for the fLOC systems.

Subregion	Baseline	AAMax	$\Delta$ [95% CI]
OFA	0.121	0.133	+0.011 [0.006, 0.016]
FFA-1	0.120	0.186	+0.066 [0.061, 0.070]
FFA-2	0.156	0.236	+0.079 [0.074, 0.086]
OPA	0.099	0.173	+0.074 [0.069, 0.080]
PPA	0.292	0.359	+0.068 [0.064, 0.072]
EBA	0.271	0.395	+0.124 [0.118, 0.130]

Appendix

Table 19: Paired cosine similarity for individual bilateral fLOC subregions on 750 held-out shared images, with 95% paired-bootstrap confidence intervals of the difference. Subregions absent in either subject (mTL-faces, aTL-faces, RSC, FBA-1, mTL-bodies, mTL-words) are excluded.

	Region removed	Baseline	AAMax
pRF	V1	0.020	0.026
	V2	0.011	0.018
	V3	0.008	0.019
	hV4	0.005	0.010
	All pRF voxels	0.029	0.056
fLOC	Faces	0.006	0.011

Appendix

Table 20: Leave-one-region-out analysis: decrease in the full paired cosine similarity when a region’s exact contribution is removed from the adapter output. Larger values indicate that the region contributes more to the cross-subject alignment.

		1,000-way (T1 / T5 / T10)		300-way (top-1)
Model	Training	Image retrieval $\uparrow$	Brain retrieval $\uparrow$	Image retrieval $\uparrow$	Brain retrieval $\uparrow$
MindEye1	Baseline	21.7 / 45.2 / 55.1	25.4 / 50.1 / 61.7	34.78 $\pm$ 0.76	39.24 $\pm$ 0.99
MindEye1	AAMax	29.0 / 55.9 / 66.6	36.1 / 64.8 / 74.2	44.20 $\pm$ 0.80	52.08 $\pm$ 0.92
MindEye2	Baseline	33.2 / 63.8 / 75.0	39.8 / 70.0 / 80.2	50.87 $\pm$ 0.88	56.18 $\pm$ 1.23
MindEye2	AAMax	39.6 / 69.6 / 78.6	45.1 / 75.2 / 85.5	56.60 $\pm$ 0.79	63.00 $\pm$ 0.74

Appendix

Table 21: Pre-diffusion retrieval for Subject 2 fine-tuned with 1 hour of data from Subject 1. Global retrieval reports top-1/5/10 accuracy (%) among all 1,000 test images. The 300-way protocol reports mean

\pm

standard deviation (%) of top-1 accuracy across 30 random candidate samplings of a single trained model.

Figure 13: Visualization of the Animate/Inanimate semantic category on the two principal singular vectors. Each point (embedding) is colored based on whether its corresponding image contains ‘objects’ that fall into the Animate category (red), Inanimate category (blue) or both categories (green).

Dimension 1	Animals (giraffes, zebras, and elephants), nature, and outdoor scenes
Dimension 2	Cats and food
Dimension 3	Kite flying, airplanes, and flying objects
Dimension 4	Surfing, water sports, beach, and ocean scenes
Dimension 5	Airplanes, flying objects, outdoor activities and events, transportation scenes

Appendix

Table 22: Concept analysis of images along principal dimensions.

Panel of Figure 14 — Figure 14: Comparison of images from Subject 2 with the highest and lowest projection values along the first principal left eigenvector. (a) Images whose corresponding 1024-dimensional embeddings yielded the highest projection values. (b) Images whose corresponding 1024-dimensional embeddings yielded the lowest projection values. The presence of the animate concept (e.g., giraffes, zebras, elephants) is prominent in the first set, while it fades in the second.

Panel of Figure 15 — Figure 15: Comparison of images from Subject 2 with the highest and lowest projection values along the 4 th principal left eigenvector. (a) Images whose corresponding 1024-dimensional embeddings yielded the highest projection values. (b) Images whose corresponding 1024-dimensional embeddings yielded the lowest projection values. Again the presence of common concepts (e.g. surfing, water sports, beach, and ocean scenes) is prominent in the first set, while it fades in the second.

Training Method	Adapter Alignment (Total Time)	End-to-End Fine-tuning		Total Time to Train
Training Method	Adapter Alignment (Total Time)	Time per Epoch	Total Time (160 epochs)	Total Time to Train
Baseline	-	221.40 seconds	590.40 minutes	590.40 minutes
AAMax	67 seconds	223.61 seconds	596.29 minutes	597.41 minutes

Appendix

Table 23: Training times for baseline and AAMax fine-tuning, averaged over the three fine-tuned subjects (Subjects 2, 5, and 7). The total time is the time to train for 160 epochs of end-to-end fine-tuning.

Architecture	Subject adapters	Shared voxel-to-CLIP	Diffusion prior	Total
MindEye1	122,912,768	883,162,112	56,056,564	1,062,131,444
MindEye2	122,896,384	1,903,020,028	259,866,596	2,285,783,008

Appendix

Table 24: Learned parameters of the brain-decoding checkpoints, which are identical for baseline fine-tuning and AAMax. Subject adapters are those for Subjects 1 and 2.

Figure 16: Visual comparison among MindEye 1 without AAMax (Normal ME1), MindEye1 with AAMax (AAMax ME1), MindEye 2 without AAMax (Normal ME2) and MindEye2 with AAMax (AAMax ME2).

	Low-Level				High-Level
Method	PixCorr $\uparrow$	SSIM $\uparrow$	Alex(2) $\uparrow$	Alex(5) $\uparrow$	Incep $\uparrow$	CLIP $\uparrow$	Eff $\downarrow$	SwAV $\downarrow$
B5k Subj1 Pre-train	0.041	0.211	65.71%	76.87%	66.24%	74.77%	0.928	0.612
FT B5k Subj2 on B5k Subj1	0.047	0.247	60.53%	66.23%	60.06%	64.63%	0.952	0.631
FT B5k Subj2 on NSD Subj1	0.053	0.252	63.32%	69.13%	59.75%	66.07%	0.948	0.634
FT NSD Subj2 on B5k Subj1	0.115	0.289	79.76%	86.73%	79.57%	79.88%	0.825	0.499
FT NSD Subj2 on NSD Subj1	0.114	0.270	83.70%	91.76%	83.71%	82.00%	0.789	0.469

Appendix

Table 25: Cross-dataset and within-dataset transfer performance between NSD and BOLD5000.

	Low-Level				High-Level
Method	PixCorr $\uparrow$	SSIM $\uparrow$	Alex(2) $\uparrow$	Alex(5) $\uparrow$	Incep $\uparrow$	CLIP $\uparrow$	Eff $\downarrow$	SwAV $\downarrow$
Baseline	0.078	0.264	75.33%	84.18%	74.15%	73.66%	0.862	0.524
AAMax, approximate pairs	0.086	0.275	78.11%	85.74%	74.79%	75.29%	0.843	0.518
AAMax, exact pairs	0.112	0.259	78.75%	88.04%	78.99%	78.28%	0.824	0.513

Appendix

Table 26: Effect of correspondence quality on AAMax. Subject 1 is the reference and Subject 2 is fine-tuned with 1 hour of data. Approximate pairs replace the identical shared images with CLIP-matched images. All rows are averaged over three independently generated splits. Low-level metrics are shown in gray because they are not optimized in these runs.

	Low-Level				High-Level
Method	PixCorr $\uparrow$	SSIM $\uparrow$	Alex(2) $\uparrow$	Alex(5) $\uparrow$	Incep $\uparrow$	CLIP $\uparrow$	Eff $\downarrow$	SwAV $\downarrow$
Subject 1 Pre-train	0.345	0.346	92.80%	96.88%	94.40%	90.02%	0.692	0.399
Baseline FT AVG	0.258 $\pm$ 0.03	0.339 $\pm$ 0.005	89.64% $\pm$ 1.8%	95.05% $\pm$ 0.87%	92.63% $\pm$ 1.62%	89.64% $\pm$ 1.55%	0.717 $\pm$ 0.017	0.427 $\pm$ 0.01
AAMax FT AVG	0.259 $\pm$ 0.03	0.337 $\pm$ 0.00	89.20% $\pm$ 2.27%	95.16% $\pm$ 1.03%	92.68% $\pm$ 1.69%	89.49% $\pm$ 1.25%	0.713 $\pm$ 0.02	0.423 $\pm$ 0.02

Appendix

Table 27: Summary statistics of the quantitative results on training the reconstruction models on all 40 hours of data. The fine-tuning results are averaged over subjects 2,5 and 7. AAMax and baseline fine-tuning perform similarly across all metrics.

Panel of Figure 17 — Figure 17: Performance of AAMax compared to baseline fine-tuning as a function of training data size. Left: AlexNet-5 2-way percent correct. Middle: Inception 2-way percent correct. Right: CLIP 2-way percent correct. Across all three metrics, AAMax achieves higher accuracy in limited-data regimes, with the advantage diminishing as more data becomes available.

MindEye1
	Low-Level				High-Level
Method	PixCorr $\uparrow$	SSIM $\uparrow$	Alex(2) $\uparrow$	Alex(5) $\uparrow$	Incep $\uparrow$	CLIP $\uparrow$	Eff $\downarrow$	SwAV $\downarrow$
Baseline	0.078	0.264	75.33%	84.18%	74.15%	73.66%	0.862	0.524
AAMax	0.112	0.259	78.75%	88.04%	78.99%	78.28%	0.824	0.513
MindEye2
Baseline	0.104	0.354	81.71%	88.06%	78.84%	74.15%	0.845	0.499

Appendix

Table 28: Evaluating Adapter Alignment across architectures. Subject 1 is trained as the reference, and Subject 2 is fine-tuned with 1 hour of training data. Results are shown for MindEye1 and MindEye2 under baseline and AAMax fine-tuning. Adapter Alignment consistently improves performance and scales with stronger architectures.

Figure 18: Reconstruction results after fine-tuning on 250 common images.

fMRI-Based Visual Decoding

See all 25 papers

May 4, 2026

StableMind: Source-Free Cross-Subject fMRI Decoding with Regularized Adaptation

Existing cross-subject fMRI decoding methods typically train a model on multiple scanned subjects and then adapt it to a new subject using substantial paired fMRI-image data. However, in realistic scenarios, new-subject fMRI data are often limited due to costly data acquisition, and raw data from previous subjects may be inaccessible, leading existing methods to suffer performance degradation during new-subject adaptation. In this paper, we identify that this degradation stems from two key issues: brain-side instability caused by large subject differences in fMRI responses, and image-side supervision unreliability caused by fine-grained visual details that are not reliably supported by limited fMRI signals. To address these challenges, we propose StableMind, a regularized adaptation framework designed to improve brain-side representation stability and image-side supervision reliability. (1) To stabilize brain representations, StableMind reuses ridge projections from the pretrained model as adaptation priors to constrain limited-data new-subject adaptation, and applies Fourier-based feature-level brain augmentation to improve robustness to individual variability. (2) To improve image supervision reliability, StableMind introduces difficulty-aware image blur for brain-image alignment, reducing the influence of fine-grained visual details that are weakly supported by limited fMRI signals while preserving stable visual structure. Experiments on the Natural Scenes Dataset under a unified 1-hour adaptation protocol demonstrate that StableMind achieves 84.02% image retrieval accuracy and 81.66% brain retrieval accuracy averaged over four subjects, surpassing the state-of-the-art method by 5.71% brain retrieval accuracy with fewer trainable adaptation parameters. Our code is available at https://github.com/lingeringlight/StableMind.

Jintao Guo, Lin Wang, Shumeng Li +5

National Key Laboratory for Novel Software Technology and the Institute of Brain-Machine Interface, Nanjing University, Nanjing 210023, China · School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore 639798 · School of Intelligence Science and Technology, Nanjing University, Nanjing 215163, China +2

Source-Free Domain Adaptation Neural Decoding Cross-Modal Alignment fMRI-Based Visual Decoding

Ask PDF

May 23, 2026

MindAdapter: Few-Shot Parameter-Efficient Residual Calibration of Cross-Subject Brain-to-Visual Decoding Models

Cross-subject brain-to-visual decoding remains a core challenge in brain-computer interfaces due to severe inter-individual variability that induces systematic subject-specific functional misalignment. To address this issue, we propose MindAdapter, a parameter-efficient few-shot calibration framework for pretrained brain-to-visual decoding models. MindAdapter adopts a decoupled linear-residual cascade alignment paradigm by freezing a pretrained explicit brain functional alignment backbone (coarse) and introducing a lightweight nonlinear residual adapter (fine), thereby disentangling global cross-subject correspondence from subject-specific residual corrections for fine-grained spatial and semantic calibration. To further preserve global representational stability, we design a topology-anchored dual-stream manifold constraint, where a small set of shared stimuli serves as topological pins with voxel-level paired supervision, while a semantic stream enforces consistency through a frozen vision-language decoder on unpaired brain data. Together, MindAdapter efficiently injects subject-specific corrections while maintaining the global representational geometry learned during pretraining. Experiments on the Natural Scenes Dataset (NSD) demonstrate that MindAdapter substantially improves cross-subject visual reconstruction and retrieval accuracy using only a few shared stimuli, offering a practical and data-efficient solution for personalized brain-to-visual decoding.

Jiaxiang Liu, Jiawei Du, Xupeng Chen +4

Guangdong Institute of Intelligence Science and Technology Hengqin, China · Agency for Science, Technology and Research Singapore · New York University Brooklyn, US +2

Neural Decoding Cross-Modal Alignment Few-Shot Learning Parameter-Efficient Fine-Tuning fMRI-Based Visual Decoding

Ask PDF

Mar 8, 2026

Duala: Dual-Level Alignment of Subjects and Stimuli for Cross-Subject fMRI Decoding

Cross-subject visual decoding aims to reconstruct visual experiences from brain activity across individuals, enabling more scalable and practical brain-computer interfaces. However, existing methods often suffer from degraded performance when adapting to new subjects with limited data, as they struggle to preserve both the semantic consistency of stimuli and the alignment of brain responses. To address these challenges, we propose Duala, a dual-level alignment framework designed to achieve stimulus-level consistency and subject-level alignment in fMRI-based cross-subject visual decoding. (1) At the stimulus level, Duala introduces a semantic alignment and relational consistency strategy that preserves intra-class similarity and inter-class separability, maintaining clear semantic boundaries during adaptation. (2) At the subject level, a distribution-based feature perturbation mechanism is developed to capture both global and subject-specific variations, enabling adaptation to individual neural representations without overfitting. Experiments on the Natural Scenes Dataset (NSD) demonstrate that Duala effectively improves alignment across subjects. Remarkably, even when fine-tuned with only about one hour of fMRI data, Duala achieves over 81.1% image-to-brain retrieval accuracy and consistently outperforms existing fine-tuning strategies in both retrieval and reconstruction. Our code is available at https://github.com/ShumengLI/Duala.

Shumeng Li, Jintao Guo, Jian Zhang +3

State Key Laboratory of Novel Software Technology, Nanjing University · Institute of Brain-Machine Interface, Nanjing University

fMRI-Based Visual Decoding Neural Decoding Latent Space Alignment Cross-Modal Alignment

Ask PDF

Abstract

fMRI-Based Visual Decoding

Explore similar work

StableMind: Source-Free Cross-Subject fMRI Decoding with Regularized Adaptation

MindAdapter: Few-Shot Parameter-Efficient Residual Calibration of Cross-Subject Brain-to-Visual Decoding Models

Duala: Dual-Level Alignment of Subjects and Stimuli for Cross-Subject fMRI Decoding