Deep learning has driven substantial progress in modeling fMRI responses, yet most existing approaches are optimized for a single objective---such as neural encoding, stimulus decoding, or post hoc interpretation---making it difficult to obtain a coherent account of neural representations. We introduce a unified framework that connects vision foundation models to brain activity while supporting neural analysis from multiple complementary perspectives. Our approach replaces black-box features with a sparse, interpretable representation composed of visual prototypes, and links these prototypes to voxel responses through a linear readout. This design makes the model transparent at both levels: prototype activations expose the visual concepts present in the stimulus, while the learned linear weights characterize each voxel's selectivity. Using this interface, we identify which visual concepts drive neural responses and where they appear in the image, reveal cortical organization through voxel tuning in prototype space, and perform controlled stimulus synthesis to probe cortical selectivity through targeted interventions. Together, these results establish prototype-based representations as a unified and interpretable foundation for studying visual representations in the human brain.
Figures & tables
Figure 1: Overview of method: (1) In stage-1, we learn interpretable prototypes with a next patch prediction objective. Given patch-level features ( e.g., from DINOv3), we factorize them as a sparse combination of learnable prototypes, which captures distinct semantics and the spatial structure of the visual scene. (2) In stage-2, we train a linear mapping between the activation of prototypes and fMRI responses, enabling brain understanding from complementary perspectives.
OFA
FFA-1
EBA
FBA-1
OPA
PPA
OWFA
VWFA-1
Noise-ceiling
0.561
0.597
0.600
0.541
0.588
0.563
0.530
0.551
Adeli et al. Adeli et al. (2025)
0.430
0.489
0.481
0.432
0.438
0.485
0.399
0.430
JEPA-VIT
0.305
0.381
0.381
0.348
0.348
0.414
0.244
0.317
Proto JEPA-VIT
0.355
0.403
0.404
0.359
0.399
0.419
0.314
0.350
DINOv3-ConvNeXt
0.273
0.366
0.379
0.319
0.324
0.377
0.209
0.304
Proto DINOv3-ConvNeXt
0.335
0.410
0.42
0.347
0.378
0.408
0.268
0.342
Table 1: Comparison of fMRI encoding across 8 subjects. We report the Pearson’s correlation for 8 distinct brain ROIs, averaged across all voxels. See Appendix M for results of other ROIs.
Figure 2: Visualization of learned prototypes. For each prototype, images are ranked by their average prototype activation (Left). We show representative examples among the top-activated images for several prototypes (Right). For visualization, we select 20 representative examples from each set. Additional visualization is provided in Appendix A .
Figure 3: Prototype profiles (visualized as top-4 prototypes with the highest weights) for category-selective regions of the visual cortex. The profiles closely align with the expected selectivity of diverse brain areas. See Appendix C and D for additional results.
Figure 4: Per-prototype cortical activation maps. Strong and weak activations are denoted with yellow and blue colors, respectively. See Appendix J for additional results.
Figure 5: Spatial prototype activation maps for three category-selective regions of the visual cortex. See Appendix I for more examples.
Figure 6: Causal interventions on face prototypes. Each example (from left to right) shows the stimulus, and its reconstruction with original/permuted prototypes. Blue and orange boxes indicate regions with high and low prototype activations that are permuted. See Appendix L for more results.
Figure 7: Changes in predicted fMRI responses for targeted permutations. Gray boxes denote controls.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Supplementary results for the visualization of representative images for different prototypes.
Figure 9: Supplementary results for the visualization of representative images for different prototypes (cont’d).
Figure 10: Histogram for the number of prototypes shared across multiple subjects.
Figure 11: Top prototypes that characterize all 12 selected brain ROIs.
Figure 12: Prototypes with different preference levels for four brain ROIs with distinct selectivity.
Figure 13: Representative images for the same prototypes ( i.e., Top-4 prototypes for different brain ROIs) with and without clustering.
Figure 14: Representative images of top principal components (for PCA features) that characterize different brain ROIs.
Figure 15: Comparison between the proposed prototype method (Prototype) and sparse autoencoder (SAE) for characterizing selectivity of different brain ROIs. Each row visualizes the representative images for top-5 prototypes/SAE features.
OFA
FFA-1
FFA-2
EBA
FBA-1
FBA-2
OPA
PPA
RSC
OWFA
VWFA-1
VWFA-2
Average
Top-4
0.184
0.283
0.358
0.295
0.183
0.327
0.118
0.126
0.262
0.100
0.161
0.169
0.214
Top-8
0.192
0.318
0.375
0.313
0.242
0.329
0.157
0.197
0.333
0.153
0.154
0.213
0.248
Top-16
0.225
0.332
0.393
0.333
0.256
0.359
0.155
0.206
0.334
0.187
0.258
0.228
0.272
Top-32
0.274
0.342
0.411
0.347
0.263
0.372
0.180
0.286
0.342
0.225
0.261
0.261
0.297
Top-64
0.304
0.375
0.427
0.365
0.292
0.394
0.251
0.327
0.365
0.281
0.281
0.279
0.329
Full model
0.371
0.466
0.516
0.506
0.403
0.504
0.414
0.424
0.414
0.299
0.390
0.293
0.416
Appendix
Table 2: fMRI encoding with top-k preferred prototypes, results are reported on subject 1.
Figure 16: Additional examples for spatial selectivity of brain ROIs.
Figure 17: Additional results on prototype weights for different brain ROIs.
Figure 18: Causal stimuli generated results with targeted permutations on body areas.
Figure 19: Additional results on causal stimuli generation with permutations of body areas.
OFA
FFA-1
FFA-2
EBA
FBA-1
FBA-2
OPA
PPA
RSC
OWFA
VWFA-1
VWFA-2
Noise-ceiling
0.561
0.597
0.541
0.600
0.541
0.588
0.563
0.585
0.542
0.530
0.551
0.500
Adeli et al. Adeli et al. (2025)
0.430
0.489
0.428
0.481
0.432
0.476
0.438
0.485
0.444
0.399
0.430
0.363
JEPA-VIT
0.305
0.381
0.324
0.381
0.348
0.381
0.348
0.414
0.368
0.244
0.317
0.259
Proto JEPA-VIT
0.355
0.403
0.346
0.404
0.359
0.400
0.399
0.419
0.353
0.314
0.350
0.270
DINOv3-ConvNeXt
0.273
0.366
0.308
0.379
0.319
0.369
0.324
0.377
0.330
0.209
0.304
0.256
Proto DINOv3-ConvNeXt
0.335
0.410
0.340
0.421
0.347
0.404
0.378
0.408
0.310
0.268
0.342
0.267
Appendix
Table 3: Comparison of fMRI encoding across 8 subjects. We report the voxel-wise Pearson’s correlation for all 12 brain ROIs.
Figure 20: fMRI encoding performance averaged across 12 different brain ROIs. For experiments with different numbers of prototypes (left), we fix the sparsity constraint as k=128 . For experiments with sparsity constraint (right), we fix the number of prototypes as P=1000
Figure 21: Schematic for the design of prototype encoder used in the diffusion models.
Brain encoder models predict cortical fMRI responses from the internal activations of pretrained vision and language networks, and are typically evaluated by held-out prediction accuracy. This is a useful signal for training but a poor one for interpretation: it tells us an encoder fits the data without telling us whether it has internalized the functional organization of the brain. We propose feature visualization -- gradient ascent on the encoder's predicted activation for a target region of interest (ROI) -- as a complementary interpretability technique, and apply it to TRIBE v2 composed with V-JEPA 2 (ViT-G, 40 layers), holding both frozen and synthesizing still images for seven regions spanning the ventral and dorsal visual hierarchies. Under identical hyperparameters, the probe recovers a visible progression of increasing spatial scale and feature complexity across V1 to V4, matching the ventral-stream hierarchy. It also produces three distinctive downstream regimes: radial "frozen-motion" streaks for the middle temporal area (MT) despite static-only optimization, face-like features for the fusiform face area (FFA), and consistent rectilinear line patterns for the parahippocampal place area (PPA). Optimized FFA stimuli drive the predicted region ~4x as much as a natural face photograph, consistent with feature visualization producing adversarial super-stimuli rather than canonical exemplars. The probe is simple, differentiable, and applicable to any brain encoder with a differentiable backbone, allowing for qualitative evaluation of brain encoders.
A central goal in understanding human vision is to uncover the visual features that drive neuronal activity. A growing body of work has used artificial neural networks as encoding models to predict cortical responses to natural images, revealing the visual content that activates category-selective regions. However, existing approaches are largely correlational and treat the encoder as a black box, leaving open which image features drive each voxel's response. We introduce Mechanistically Interpretable Neural Encoding (MINE), a framework that opens this black box by applying mechanistic-interpretability tools to localize the features within natural images that drive millimeter-scale (voxel-level) activity. MINE predicts each voxel's response using language-aligned image representations, and produces semantically interpretable descriptions of the features critical for the voxel's activation. We further generalize these per-image features into per-voxel functional profiles. To validate the per-image descriptions, we show they are sufficient to generate images that elicit voxel responses matching the responses to the original images, more accurately than images generated from random or low-attribution controls. Moreover, counterfactually inserting or removing the predicted features from images shifts activation in the expected direction, providing causal evidence. Counterfactual editing guided by the per-voxel activation profiles produces even stronger activation shifts, indicating that the profiles faithfully capture each voxel's selectivity. Finally, we apply MINE to well-studied category-selective brain regions, showing it recovers their known categorical preferences while revealing fine-grained unique voxel structure within each region. Overall, our results establish mechanistic interpretability as a path to discover and causally validate fine-grained hypotheses about neural function.
Idan Daniel Grosbard, Mor Geva, Galit Yovel
1Sagol School of Neuroscience · 2Blavatnik School of Computer Science and AI · School of Psychological Sciences
A central question in neuroscience is what visual features drive brain responses across regions. Classic approaches rely on handcrafted stimuli to test specific hypotheses, limiting discovery to features defined before experiments. Here we introduce NeuroVolve, a data-driven generative framework for synthesizing images that achieve flexible, user-defined patterns of brain activation in individuals. NeuroVolve recovers established selectivity across visual cortical regions and, paired with subject-specific brain encoding models, generates personalized stimuli reflecting individual differences in neural preference. Beyond single-region selectivity, NeuroVolve uncovers multi-region response selectivity, characterizing how regions jointly represent visual content during naturalistic perception. We further demonstrate generalization beyond visual cortex: in a prospective fMRI experiment, NeuroVolve-generated stimuli targeting the amygdala, a classically non-sensory region, drove measured activation consistent with predictions in novel participants. Together, these results establish NeuroVolve as a less constrained, data-driven tool for probing individual-level neural selectivity across the brain.
Haomiao Chen, Keith W Jamison, Daniel Chong +3
Cornell University · Cornell Tech · Weill Cornell Medicine