Authors: Mu Nan, Muquan Yu, Weijian Mai, Jacob S. Prince, Hossein Adeli, Rui Zhang, Jiahang Cao, Benjamin Becker, John A. Pyles, Margaret M. Henderson, Chunfeng Song, Nikolaus Kriegeskorte, Michael J. Tarr, Xiaoqing Hu, Andrew F. Luo
Organizations: University of Hong Kong · Shenzhen Loop Area Institute · Chinese University of Hong Kong · Shanghai Artificial Intelligence Laboratory · Harvard University · Columbia University · University of Washington · Carnegie Mellon University
Visual decoding from brain signals is a key challenge at the intersection of computer vision and neuroscience, requiring methods that bridge neural representations and computational models of vision. A field-wide goal is to achieve generalizable, cross-subject models. A major obstacle towards this goal is the substantial variability in neural representations across individuals, which has so far required training bespoke models or fine-tuning separately for each subject. To address this challenge, we introduce a meta-optimized approach for semantic visual decoding from fMRI that generalizes to novel subjects without any fine-tuning. By simply conditioning on a small set of image-brain activation examples from the new individual, our model rapidly infers their unique neural encoding patterns to facilitate robust and efficient visual decoding. Our approach is explicitly optimized for in-context learning of the new subject's encoding model and performs decoding by hierarchical inference, inverting the encoder. First, for multiple brain regions, we estimate the per-voxel visual response encoder parameters by constructing a context over multiple stimuli and responses. Second, we construct a context consisting of encoder parameters and response values over multiple voxels to perform aggregated functional inversion. We demonstrate strong cross-subject and cross-scanner generalization across diverse visual backbones without retraining or fine-tuning. Moreover, our approach requires neither anatomical alignment nor stimulus overlap. This work is a critical step towards a generalizable foundation model for non-invasive brain decoding.
Figures & tables
Figure 1: Overview of our hierarchical brain decoding framework. Encoders predict brain activity from stimulus, while decoders reconstruct stimulus from brain activity. (a) Our framework can generalize to novel subjects without any fine-tuning . In the first stage, we infer parameters of a forward model (image-computable encoder) by constructing a context using stimuli/activity pairs for a single voxel, repeated for every voxel. In the second stage, we construct a context across multiple voxels, fusing the encoder parameters with observed brain activations to decode the stimuli. Our approach requires neither anatomical alignment, nor stimulus overlap. (b) Decoding results on BOLD5000 after training on NSD, our method can generalize without fine-tuning across scanners, voxel size, and subjects . (c) Our model performance positively scales with both the number of images provided in Stage 1, and the number of voxels provided in Stage 2.
Figure 2: Model architecture of BrainCoDec. In stage one, the in-context encoder infers encoder parameters by in-context learning across stimuli/activation pairs for a single voxel. This is repeated across the voxels of interest. In stage two, we integrate across multiple voxels, taking as input the voxelwise parameters and activation corresponding to a novel image. Both stages can vary the context sizes.
Figure 3: Contextual scaling and ablation analysis of BrainCoDec. Top: Image-context scaling from stage 1. Decoder performance scales positively with more images collected for the novel subject. Middle: Voxel-context scaling from stage 2. Top-1 retrieval accuracy improves consistently as the number of in-context voxels increases with all visual backbones across all subjects. Bottom: Ablation comparison. Cosine similarities for four variants using CLIP backbone, synthetic data pretraining (PT Only), gradient inversion (Inversion), training with subject hold-out (FT HO; BrainCoDec), and training on seen subjects (FT no HO). These results show that models trained with real neural data outperform the models trained with only synthetic data, with only marginal gains from fine-tuning on a subject.
Figure 4: Image retrieval comparison on a subject unseen during training (S1) . For each method (BrainCoDec-200, MindEye2 + anatomical alignment, TGBD), columns list the Top- 1 – 3 retrieved images out of 907 test images from left to right, ranked by similarity in the evaluation embedding space. Red boxes mark correct hits. Our model can yield very high semantic retrieval consistency without any fine-tuning.
S1
S2
S5
S7
Mean
Models
Top-1 ↑
Top-5 ↑
Top-1 ↑
Top-5 ↑
Top-1 ↑
Top-5 ↑
Top-1 ↑
Top-5 ↑
Top-1 ↑
Top-5 ↑
MindEye2 [ 95 ]
4.11%
12.9%
3.82%
10.70%
2.87%
9.58%
2.51%
6.49%
3.90%
9.81%
TGBD [ 55 ]
1.27%
3.89%
0.56%
2.33%
0.84%
3.34%
0.39%
1.41%
0.82%
3.09%
BrainCoDec-200
25.5%
56.6%
22.9%
52.4%
23.2%
55.8%
19.2%
51.2%
22.7%
54.0%
Table 1: Quantitative comparison of unseen subject brain decoding performance. Top-1 and Top-5 retrieval accuracy (%) on unseen NSD subjects (S1, S2, S5, S7) for MindEye2 + anatomical alignment, TGBD, and BrainCoDec-200 (200 in-context images). Our method substantially outperforms prior methods while requiring neither subject-specific fine-tuning nor large-scale training data. Mean accuracies across subjects are reported in the rightmost column; additional metrics and standard deviations are provided in the Appendix .
Figure 5: Robustness of removing voxels from ROIs. Cosine similarity of masking out category-specific voxels (Food, Faces, Places, Words) across four unseen NSD subjects on top-activating images from the test set. For each category, we compare performance using full context voxels from higher visual cortex versus masking out category-selective ROIs. Across nearly all conditions, masking the corresponding functional region has minimal impact on decoding performance, indicating strong robustness and distributed representation learning in BrainCoDec. Masking scene-selective regions (PPA/OPA/RSC) leads to some performance drop.
Figure 6: Semantic attention patterns in BrainCoDec. Left: Example face and place stimuli used for category-specific analysis. Middle: Comparison of category t -values from an independent NSD functional localizer, with the corresponding attention-weight maps from the final self-attention layer when decoding these stimuli, showing closely matched spatial distributions. Right: UMAP projection of voxelwise attention weights across the full test set. Color-coded clusters separate body/face-selective regions in green (EBA, FFA/aTL-faces) and scene-selective regions in red (RSC, OPA, PPA).
Figure 7: Image-context scaling on BOLD5000. Retrieval Mean rank (lower is better) across three unseen subjects as the number of in-context image–brain pairs increases.
Figure 8: Top-4 image retrieval on BOLD5000. We visualize the retrieval result on a new-scanner unseen subject (CSI1) using 20 images as context. The right columns display the Top- 4 retrieved images. Red boxes indicate correct hits. Our model can generalize to novel datasets and scanning parameters without training.
Backbones
Top-1 Acc.
Top-5 Acc.
Mean Rank
Cosine Sim.
CLIP
31.45 ± 12.80%
81.67 ± 9.42%
3.49 ± 0.76
0.72 ± 0.02
DINOv2
13.99 ± 5.83%
53.33 ± 6.74%
6.78 ± 0.87
0.08 ± 0.01
SigLIP
23.67 ± 8.05%
73.41 ± 8.25%
4.47 ± 0.93
0.66 ± 0.01
Table 2: Quantitative results of BOLD5000. We directly test BrainCoDec using just 20 images as in-context and a different 20 images as the test set on three unseen subjects from BOLD5000. Chance Top-1 Acc is 5% . All metrics are averaged across all folds of all three unseen subjects.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Model
S1
S2
S5
S7
% Top-1 Accuracy ( ↑ )
MindEye2
4.11±1.41
3.82±1.10
2.87±1.19
2.51±1.64
TGBD
1.27±0.16
0.56±0.12
0.84±0.16
0.39±0.09
BrainCodec-200
25.5±3.02
22.9±2.98
23.2±2.63
19.2±2.42
% Top-5 Accuracy ( ↑ )
MindEye2
12.9±2.55
10.7±3.14
9.58±3.61
6.49±2.87
Appendix
Table S.1 : Quantitative comparison on NSD Subjects 1, 2, 5, and 7.
Model
S3
S4
S6
S8
% Top-1 Accuracy ( ↑ )
MindEye2
3.50±1.13
3.19±1.16
2.69±1.42
2.33±1.86
TGBD
0.65±0.13
0.75±0.15
0.61±0.12
0.17±0.05
BrainCodec-200
19.0±1.86
16.1±1.75
20.1±2.52
14.4±1.56
% Top-5 Accuracy ( ↑ )
MindEye2
10.33±3.30
9.95±3.45
8.04±3.24
4.95±2.50
Appendix
Table S.2 : Quantitative comparison on NSD Subjects 3, 4, 6, and 8.
Figure 13Figure 14Figure 15Figure 16
Figure S.9 : Image-context scaling of BrainCoDec on NSD subjects 3, 4, 6, and 8.
Figure S.10 : Voxel-context scaling of BrainCoDec on NSD subjects 3, 4, 6, and 8.
Figure S.11 : Image-context scaling of BrainCoDec on BOLD5000 subjects.
Figure S.12 : Voxel-context scaling of BrainCoDec on BOLD5000 subjects.
Figure S.13 : Semantic attention patterns in BrainCoDec.
Figure S.14 : Image retrieval results on BOLD5000 unseen subjects from fold 1 using 80 images as context. To note, since BOLD5000 provides only 20 test images, we visualize retrieval results from a pool of 500 images for rigorous evaluation.
Model
S1
S2
S5
S7
% Top-1 Accuracy ( ↑ )
PT only
3.67±0.69
3.24±0.71
2.96±0.64
2.68±0.63
Inversion
1.61±0.73
1.39±0.86
2.04±0.81
1.90±0.77
BrainCoDec-200
25.5±3.02
22.9±2.98
23.2±2.63
19.2±2.42
BrainCoDec-200 no HO
28.3±3.40
27.1±3.21
29.4±3.40
24.0±3.36
% Top-5 Accuracy ( ↑ )
Appendix
Table S.3 : Quantitative comparison on model variants and ablations.
Guangdong Institute of Intelligence Science and Technology Hengqin, China · Agency for Science, Technology and Research Singapore · New York University Brooklyn, US +2