Authors: Mu Nan, Muquan Yu, Weijian Mai, Jacob S. Prince, Hossein Adeli, Rui Zhang, Jiahang Cao, Benjamin Becker, John A. Pyles, Margaret M. Henderson, Chunfeng Song, Nikolaus Kriegeskorte, Michael J. Tarr, Xiaoqing Hu, Andrew F. Luo
Organizations: University of Hong Kong · Shenzhen Loop Area Institute · Chinese University of Hong Kong · Shanghai Artificial Intelligence Laboratory · Harvard University · Columbia University · University of Washington · Carnegie Mellon University
Visual decoding from brain signals is a key challenge at the intersection of computer vision and neuroscience, requiring methods that bridge neural representations and computational models of vision. A field-wide goal is to achieve generalizable, cross-subject models. A major obstacle towards this goal is the substantial variability in neural representations across individuals, which has so far required training bespoke models or fine-tuning separately for each subject. To address this challenge, we introduce a meta-optimized approach for semantic visual decoding from fMRI that generalizes to novel subjects without any fine-tuning. By simply conditioning on a small set of image-brain activation examples from the new individual, our model rapidly infers their unique neural encoding patterns to facilitate robust and efficient visual decoding. Our approach is explicitly optimized for in-context learning of the new subject's encoding model and performs decoding by hierarchical inference, inverting the encoder. First, for multiple brain regions, we estimate the per-voxel visual response encoder parameters by constructing a context over multiple stimuli and responses. Second, we construct a context consisting of encoder parameters and response values over multiple voxels to perform aggregated functional inversion. We demonstrate strong cross-subject and cross-scanner generalization across diverse visual backbones without retraining or fine-tuning. Moreover, our approach requires neither anatomical alignment nor stimulus overlap. This work is a critical step towards a generalizable foundation model for non-invasive brain decoding.
Figures & tables
Figure 1: Overview of our hierarchical brain decoding framework. Encoders predict brain activity from stimulus, while decoders reconstruct stimulus from brain activity. (a) Our framework can generalize to novel subjects without any fine-tuning . In the first stage, we infer parameters of a forward model (image-computable encoder) by constructing a context using stimuli/activity pairs for a single voxel, repeated for every voxel. In the second stage, we construct a context across multiple voxels, fusing the encoder parameters with observed brain activations to decode the stimuli. Our approach requires neither anatomical alignment, nor stimulus overlap. (b) Decoding results on BOLD5000 after training on NSD, our method can generalize without fine-tuning across scanners, voxel size, and subjects . (c) Our model performance positively scales with both the number of images provided in Stage 1, and the number of voxels provided in Stage 2.
Figure 2: Model architecture of BrainCoDec. In stage one, the in-context encoder infers encoder parameters by in-context learning across stimuli/activation pairs for a single voxel. This is repeated across the voxels of interest. In stage two, we integrate across multiple voxels, taking as input the voxelwise parameters and activation corresponding to a novel image. Both stages can vary the context sizes.
Figure 3: Contextual scaling and ablation analysis of BrainCoDec. Top: Image-context scaling from stage 1. Decoder performance scales positively with more images collected for the novel subject. Middle: Voxel-context scaling from stage 2. Top-1 retrieval accuracy improves consistently as the number of in-context voxels increases with all visual backbones across all subjects. Bottom: Ablation comparison. Cosine similarities for four variants using CLIP backbone, synthetic data pretraining (PT Only), gradient inversion (Inversion), training with subject hold-out (FT HO; BrainCoDec), and training on seen subjects (FT no HO). These results show that models trained with real neural data outperform the models trained with only synthetic data, with only marginal gains from fine-tuning on a subject.
Figure 4: Image retrieval comparison on a subject unseen during training (S1) . For each method (BrainCoDec-200, MindEye2 + anatomical alignment, TGBD), columns list the Top- 1 – 3 retrieved images out of 907 test images from left to right, ranked by similarity in the evaluation embedding space. Red boxes mark correct hits. Our model can yield very high semantic retrieval consistency without any fine-tuning.
S1
S2
S5
S7
Mean
Models
Top-1 ↑
Top-5 ↑
Top-1 ↑
Top-5 ↑
Top-1 ↑
Top-5 ↑
Top-1 ↑
Top-5 ↑
Top-1 ↑
Top-5 ↑
MindEye2 [ 95 ]
4.11%
12.9%
3.82%
10.70%
2.87%
9.58%
2.51%
6.49%
3.90%
9.81%
TGBD [ 55 ]
1.27%
3.89%
0.56%
2.33%
0.84%
3.34%
0.39%
1.41%
0.82%
3.09%
BrainCoDec-200
25.5%
56.6%
22.9%
52.4%
23.2%
55.8%
19.2%
51.2%
22.7%
54.0%
Table 1: Quantitative comparison of unseen subject brain decoding performance. Top-1 and Top-5 retrieval accuracy (%) on unseen NSD subjects (S1, S2, S5, S7) for MindEye2 + anatomical alignment, TGBD, and BrainCoDec-200 (200 in-context images). Our method substantially outperforms prior methods while requiring neither subject-specific fine-tuning nor large-scale training data. Mean accuracies across subjects are reported in the rightmost column; additional metrics and standard deviations are provided in the Appendix .
Figure 5: Robustness of removing voxels from ROIs. Cosine similarity of masking out category-specific voxels (Food, Faces, Places, Words) across four unseen NSD subjects on top-activating images from the test set. For each category, we compare performance using full context voxels from higher visual cortex versus masking out category-selective ROIs. Across nearly all conditions, masking the corresponding functional region has minimal impact on decoding performance, indicating strong robustness and distributed representation learning in BrainCoDec. Masking scene-selective regions (PPA/OPA/RSC) leads to some performance drop.
Figure 6: Semantic attention patterns in BrainCoDec. Left: Example face and place stimuli used for category-specific analysis. Middle: Comparison of category t -values from an independent NSD functional localizer, with the corresponding attention-weight maps from the final self-attention layer when decoding these stimuli, showing closely matched spatial distributions. Right: UMAP projection of voxelwise attention weights across the full test set. Color-coded clusters separate body/face-selective regions in green (EBA, FFA/aTL-faces) and scene-selective regions in red (RSC, OPA, PPA).
Figure 7: Image-context scaling on BOLD5000. Retrieval Mean rank (lower is better) across three unseen subjects as the number of in-context image–brain pairs increases.
Figure 8: Top-4 image retrieval on BOLD5000. We visualize the retrieval result on a new-scanner unseen subject (CSI1) using 20 images as context. The right columns display the Top- 4 retrieved images. Red boxes indicate correct hits. Our model can generalize to novel datasets and scanning parameters without training.
Backbones
Top-1 Acc.
Top-5 Acc.
Mean Rank
Cosine Sim.
CLIP
31.45 ± 12.80%
81.67 ± 9.42%
3.49 ± 0.76
0.72 ± 0.02
DINOv2
13.99 ± 5.83%
53.33 ± 6.74%
6.78 ± 0.87
0.08 ± 0.01
SigLIP
23.67 ± 8.05%
73.41 ± 8.25%
4.47 ± 0.93
0.66 ± 0.01
Table 2: Quantitative results of BOLD5000. We directly test BrainCoDec using just 20 images as in-context and a different 20 images as the test set on three unseen subjects from BOLD5000. Chance Top-1 Acc is 5% . All metrics are averaged across all folds of all three unseen subjects.
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Model
S1
S2
S5
S7
% Top-1 Accuracy ( ↑ )
MindEye2
4.11±1.41
3.82±1.10
2.87±1.19
2.51±1.64
TGBD
1.27±0.16
0.56±0.12
0.84±0.16
0.39±0.09
BrainCodec-200
25.5±3.02
22.9±2.98
23.2±2.63
19.2±2.42
% Top-5 Accuracy ( ↑ )
MindEye2
12.9±2.55
10.7±3.14
9.58±3.61
6.49±2.87
Appendix
Table S.1 : Quantitative comparison on NSD Subjects 1, 2, 5, and 7.
Model
S3
S4
S6
S8
% Top-1 Accuracy ( ↑ )
MindEye2
3.50±1.13
3.19±1.16
2.69±1.42
2.33±1.86
TGBD
0.65±0.13
0.75±0.15
0.61±0.12
0.17±0.05
BrainCodec-200
19.0±1.86
16.1±1.75
20.1±2.52
14.4±1.56
% Top-5 Accuracy ( ↑ )
MindEye2
10.33±3.30
9.95±3.45
8.04±3.24
4.95±2.50
Appendix
Table S.2 : Quantitative comparison on NSD Subjects 3, 4, 6, and 8.
Figure 13Figure 14Figure 15Figure 16
Figure S.9 : Image-context scaling of BrainCoDec on NSD subjects 3, 4, 6, and 8.
Figure S.10 : Voxel-context scaling of BrainCoDec on NSD subjects 3, 4, 6, and 8.
Figure S.11 : Image-context scaling of BrainCoDec on BOLD5000 subjects.
Figure S.12 : Voxel-context scaling of BrainCoDec on BOLD5000 subjects.
Figure S.13 : Semantic attention patterns in BrainCoDec.
Figure S.14 : Image retrieval results on BOLD5000 unseen subjects from fold 1 using 80 images as context. To note, since BOLD5000 provides only 20 test images, we visualize retrieval results from a pool of 500 images for rigorous evaluation.
Model
S1
S2
S5
S7
% Top-1 Accuracy ( ↑ )
PT only
3.67±0.69
3.24±0.71
2.96±0.64
2.68±0.63
Inversion
1.61±0.73
1.39±0.86
2.04±0.81
1.90±0.77
BrainCoDec-200
25.5±3.02
22.9±2.98
23.2±2.63
19.2±2.42
BrainCoDec-200 no HO
28.3±3.40
27.1±3.21
29.4±3.40
24.0±3.36
% Top-5 Accuracy ( ↑ )
Appendix
Table S.3 : Quantitative comparison on model variants and ablations.
Cross-subject visual decoding aims to reconstruct visual experiences from brain activity across individuals, enabling more scalable and practical brain-computer interfaces. However, existing methods often suffer from degraded performance when adapting to new subjects with limited data, as they struggle to preserve both the semantic consistency of stimuli and the alignment of brain responses. To address these challenges, we propose Duala, a dual-level alignment framework designed to achieve stimulus-level consistency and subject-level alignment in fMRI-based cross-subject visual decoding. (1) At the stimulus level, Duala introduces a semantic alignment and relational consistency strategy that preserves intra-class similarity and inter-class separability, maintaining clear semantic boundaries during adaptation. (2) At the subject level, a distribution-based feature perturbation mechanism is developed to capture both global and subject-specific variations, enabling adaptation to individual neural representations without overfitting. Experiments on the Natural Scenes Dataset (NSD) demonstrate that Duala effectively improves alignment across subjects. Remarkably, even when fine-tuned with only about one hour of fMRI data, Duala achieves over 81.1% image-to-brain retrieval accuracy and consistently outperforms existing fine-tuning strategies in both retrieval and reconstruction. Our code is available at https://github.com/ShumengLI/Duala.
Shumeng Li, Jintao Guo, Jian Zhang +3
State Key Laboratory of Novel Software Technology, Nanjing University · Institute of Brain-Machine Interface, Nanjing University
Cross-subject brain-to-visual decoding remains a core challenge in brain-computer interfaces due to severe inter-individual variability that induces systematic subject-specific functional misalignment. To address this issue, we propose MindAdapter, a parameter-efficient few-shot calibration framework for pretrained brain-to-visual decoding models. MindAdapter adopts a decoupled linear-residual cascade alignment paradigm by freezing a pretrained explicit brain functional alignment backbone (coarse) and introducing a lightweight nonlinear residual adapter (fine), thereby disentangling global cross-subject correspondence from subject-specific residual corrections for fine-grained spatial and semantic calibration. To further preserve global representational stability, we design a topology-anchored dual-stream manifold constraint, where a small set of shared stimuli serves as topological pins with voxel-level paired supervision, while a semantic stream enforces consistency through a frozen vision-language decoder on unpaired brain data. Together, MindAdapter efficiently injects subject-specific corrections while maintaining the global representational geometry learned during pretraining. Experiments on the Natural Scenes Dataset (NSD) demonstrate that MindAdapter substantially improves cross-subject visual reconstruction and retrieval accuracy using only a few shared stimuli, offering a practical and data-efficient solution for personalized brain-to-visual decoding.
Jiaxiang Liu, Jiawei Du, Xupeng Chen +4
Guangdong Institute of Intelligence Science and Technology Hengqin, China · Agency for Science, Technology and Research Singapore · New York University Brooklyn, US +2
Visual decoding from brain signals is a key challenge at the intersection of computer vision and neuroscience, requiring methods that bridge neural representations and computational models of vision. We introduce a tri-modal contrastive framework for EEG-based visual decoding that aligns EEG, visual, and textual representations within a unified latent space. Our approach follows a two-stage design. First, we pre-train an EEG encoder via masked reconstruction on unlabeled trials, learning spatio-temporal regularities that transfer robustly to downstream tasks. Second, we jointly align EEG, image, and LLM-generated textual descriptions through contrastive learning, where text supervision acts as a semantic regularizer that injects linguistic structure into the shared space without overwhelming the primary EEG-image signal. The encoder integrates subject-specific adaptation, graph-attention over channels, and temporal-spatial convolutional embeddings. On the Things-EEG2 200-way zero-shot benchmark, our framework achieves 54.1% Top-1 and 83.4% Top-5 accuracy, substantially exceeding the strongest prior baseline (32.4% / 64.0%), with paired Wilcoxon tests confirming significance (p < 0.01) over all in-subject baselines. We validate generalization on Things-MEG. Analysis reveals that compact embedding geometries (CN-CLIP) outperform much larger backbones, and that decoding aligns with established neurophysiology of visual processing. This work is a critical step towards robust, semantically-grounded visual decoding from non-invasive temporal neural signals. The source code is publicly available in https://github.com/anon-eeg/eeg_image_decoding.
Zexuan Chen, Sichao Liu, Runhao Lu +4
KTH, SWeden · University of Cambridge, UK · EPFL, Switzerland +2