An ongoing challenge in sensory neuroscience is to characterize the feature dimensions encoded by cortical populations. Recent approaches probe feature selectivity in a data-driven way, by synthesizing a most-exciting-input (MEI) for a target neural population. While this approach has been successfully applied to human higher visual cortex using fMRI data, generating MEIs for early- and mid-level retinotopic visual areas requires additional modeling constraints due to small receptive field sizes. To address this challenge, we introduce two novel MEI generation frameworks, Receptive Field Diffusion for Visual Exploration (RF-DiVE) and Receptive Field Gradient Optimization (RF-GO). Both methods use a population receptive field (pRF)-constrained voxelwise encoding model; RF-DiVE combines this with a pretrained latent diffusion model, while RF-GO uses regularized gradient ascent. When applied to single voxels in retinotopically defined areas V1-hV4, using data from the Natural Scenes Dataset, we obtain MEIs that exhibit consistent structure within the pRF, suggesting selectivity for local features like contour, color, and texture. We systematically compare MEIs generated by RF-DiVE and RF-GO using two encoding backbones, performing in-silico validation of predicted responses to MEIs using independent encoding models. Across all methods and all visual areas, MEIs elicit higher model-predicted responses than the most activating natural images. We further find that the choice of generation framework and encoding backbone differentially affects MEI properties, including their visual appearance, structural interpretability, and cross-model generalizability. These results offer a new approach for performing data-driven characterization of spatial and feature selectivity across human visual cortex.
Figures & tables
Figure 1: Our stimulus optimization pipeline. (A) Our encoding model predicts activity in early visual cortex using a deep CNN backbone, constrained by a Gaussian that models voxel population receptive field (pRF). (B) Visualization of our RF-DiVE and RF-GO frameworks. In RF-DiVE, the optimization process is constrained by a diffusion model, while RF-GO optimizes the image using MACO. (C) Examples of most-exciting-images (MEIs) for individual voxels in V1, V2, V3, and hV4. Each column corresponds to one visual area; white circle labels the voxel’s pRF
Figure 2: Spatial and feature specificity of V1 MEIs. (A) Top 5 MEIs for an example voxel using both frameworks. The MEIs show consistent patterns within the pRF (white circle, 3σ range of the Gaussian) and greater variation outside it. (B) Full-image and pRF-zoomed views of an example MEI. (C–F) pRF-zoomed top 5 natural images and MEIs for four example V1 voxels, shown for both ADV-RN50 and DINO-RN50 backbones. See Appendix A.11 for more examples.
Figure 3: Feature specificity of hV4 MEIs across generation encoders. Top 5 natural images and MEIs are shown for two example hV4 voxels ( A-B show the same voxel, C-D show the same voxel). Note the pRF is fit separately for each backbone and the rank number of voxels is assigned separately for each backbone. See Appendix A.11 for more examples, including from V2 and V3.
Figure 4: Predicted responses to top 10 images for S1. Top-10 ranking model-selected images for each image group were evaluated by the generation, ranking, and independent test encoders. Each point represents mean prediction across images for one voxel; horizontal bar = mean across voxels in S1. See Appendix A.6 for the other NSD subjects.
Figure 5: Predicted difference score (advantage of MEIs over LAION-fMRI) across subjects. For each voxel, we computed the predicted-response difference between MEIs and natural images. Points reflect the mean of these differences across voxels within each subject and ROI.
Figure 6: Chromatic and spatial-frequency statistics for V1 MEIs. (A) Colorfulness ( Hasler & Suesstrunk, 2003 ) and CIELAB color statistics. (B) Normalized Gabor magnitude profiles across spatial frequencies, shown for RF-DiVE and RF-GO. Similar patterns were observed in V2, V3, and hV4.
Figure 7: Depiction of form in MEIs. Proportion of trials in which participants selected the hV4 images as having more three-dimensional form than the V1 images, using pRF-zoomed images (left) or full images (right). Bar height and error bars indicate mean ± SEM across 32 participants; asterisk indicates a significant difference from 0.50 (one-sample t -test; p < 0.01).
Appendix figures & tables73 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Held-out encoding performance in V1 for training partition 1. Voxelwise R2 is plotted against noise ceiling.
Figure 9: Held-out encoding performance in V1 for training partition 2. Voxelwise R2 is plotted against noise ceiling.
Figure 10: Held-out encoding performance in V2 for training partition 1. Voxelwise R2 is plotted against noise ceiling.
Figure 11: Held-out encoding performance in V2 for training partition 2. Voxelwise R2 is plotted against noise ceiling.
Figure 12: Held-out encoding performance in V3 for training partition 1. Voxelwise R2 is plotted against noise ceiling.
Figure 13: Held-out encoding performance in V3 for training partition 2. Voxelwise R2 is plotted against noise ceiling.
Figure 14: Held-out encoding performance in hV4 for training partition 1. Voxelwise R2 is plotted against noise ceiling.
Figure 15: Held-out encoding performance in hV4 for training partition 2. Voxelwise R2 is plotted against noise ceiling.
MEI − NSD
MEI − LAION
ROI
Backbone
Method
n
t
pBH
t
pBH
S1 — Full image
V1
ADV
RF-DiVE
20
8.67
1.7×10−7
6.76
3.8×10−6
RF-GO
20
11.08
7.9×10−9
11.47
6.1×10−9
DINO
RF-DiVE
20
11.33
6.7×10−9
8.38
2.5×10−7
RF-GO
20
11.25
7.2×10−9
10.81
9.8×10−9
Appendix
Table 1: Paired-voxel tests of MEI responses against natural controls. Positive t favors MEIs; n is the paired-voxel count. Bold adjusted p values indicate significantly higher MEI responses after BH correction across all 256 comparisons. Results continue on the following pages.
MEI − NSD
MEI − LAION
ROI
Backbone
Method
n
t
pBH
t
pBH
S2 — Full image
V1
ADV
RF-DiVE
20
8.54
2.0×10−7
10.00
2.5×10−8
RF-GO
20
8.63
1.8×10−7
10.68
1.2×10−8
DINO
RF-DiVE
20
13.48
1.1×10−9
20.69
4.4×10−12
RF-GO
20
14.30
8.1×10−10
19.06
9.8×10−12
Appendix
Table 17
MEI − NSD
MEI − LAION
ROI
Backbone
Method
n
t
pBH
t
pBH
S5 — Full image
V1
ADV
RF-DiVE
20
5.76
2.5×10−5
8.27
2.9×10−7
RF-GO
20
3.67
0.002
5.50
4.1×10−5
DINO
RF-DiVE
20
10.52
1.3×10−8
10.90
8.9×10−9
RF-GO
20
5.87
2.0×10−5
7.66
7.8×10−7
Appendix
Table 18
MEI − NSD
MEI − LAION
ROI
Backbone
Method
n
t
pBH
t
pBH
S7 — Full image
V1
ADV
RF-DiVE
20
3.67
0.002
3.96
0.001
RF-GO
20
4.55
3.1×10−4
6.35
8.0×10−6
DINO
RF-DiVE
20
9.57
4.6×10−8
9.46
5.4×10−8
RF-GO
20
6.43
6.8×10−6
6.74
3.9×10−6
Appendix
Table 19
Figure 16: Verification of full-image and pRF-weighted luma constraints. Each point summarizes one subject and generation backbone; horizontal bars indicate means across subjects. Statistics were averaged over images, voxels, and ROIs in sequence. Dotted lines and shaded bands indicate targets and tolerances. Measurements use saved-image resolution; subsequent encoder-input resizing can change RMS contrast.
Figure 17: Independent test-encoder predictions across final-constraint conditions for S1. Rows show V1, V2, V3, and hV4; image sources comprise NSD, LAION-fMRI, RF-DiVE, and RF-GO.
Figure 18: Independent test-encoder predictions across final-constraint conditions for S2, using the same layout and conventions as Figure 17 .
Figure 19: Independent test-encoder predictions across final-constraint conditions for S5, using the same layout and conventions as Figure 17 .
Figure 20: Independent test-encoder predictions across final-constraint conditions for S7, using the same layout and conventions as Figure 17 .
Figure 43: CIELAB components and HSV saturation across subjects and visual areas. Points denote subject means; short bars denote means across subjects. Natural references pool the two backbone-conditioned summaries.
Figure 44: Normalized Gabor orientation profiles for each subject and visual area, using generator-pRF pooling. Solid and dashed colored lines denote RF-DiVE and RF-GO; gray denotes the pooled NSD reference.
Figure 45: Colorfulness across subjects and visual areas, using generator-pRF weights. Points denote subject means; bars denote means across subjects.
Figure 46: Per-image orientation concentration and normalized entropy, using generator-pRF weights. Points denote subject means; bars denote means across subjects.
Figure 47: CIELAB components and HSV saturation with uniform full-field analysis weights. Points denote subject means; bars denote means across subjects.
Figure 48: Normalized full-field Gabor orientation profiles for each subject and visual area. Solid and dashed colored lines denote RF-DiVE and RF-GO; gray denotes NSD.
Figure 49: Colorfulness across subjects and visual areas, using uniform full-field weights. Points denote subject means; bars denote means across subjects.
Figure 50: Per-image orientation concentration and normalized entropy with full-field weights. Points denote subject means; bars denote means across subjects.
Figure 51: Bent-Gabor curvature index and effective edge support across visual areas. Points denote subject means of voxel-wise top-10 image statistics; bars denote four-subject means. Statistics use generator soft-pRF weights.
Figure 52: Canny edge fraction and GLCM joint entropy in S1. Pale points denote voxel-wise top-10 image means; bars denote means across 20 voxels per backbone and ROI. Statistics use generator soft-pRF weights.
Figure 53: MEIs of top-8 voxels for Subject 01, visual area V1, using the ADV-RN50 brain-encoder backbone.
Figure 54: MEIs of top-8 voxels for Subject 01, visual area V2, using the ADV-RN50 brain-encoder backbone.
Figure 55: MEIs of top-8 voxels for Subject 01, visual area V3, using the ADV-RN50 brain-encoder backbone.
Figure 56: MEIs of top-8 voxels for Subject 01, visual area hV4, using the ADV-RN50 brain-encoder backbone.
Figure 57: MEIs of top-8 voxels for Subject 01, visual area V1, using the DINO-RN50 brain-encoder backbone.
Figure 58: MEIs of top-8 voxels for Subject 01, visual area V2, using the DINO-RN50 brain-encoder backbone.
Figure 59: MEIs of top-8 voxels for Subject 01, visual area V3, using the DINO-RN50 brain-encoder backbone.
Figure 60: MEIs of top-8 voxels for Subject 01, visual area hV4, using the DINO-RN50 brain-encoder backbone.
Figure 61: MEIs of top-8 voxels for Subject 02, visual area V1, using the ADV-RN50 brain-encoder backbone.
Figure 62: MEIs of top-8 voxels for Subject 02, visual area V2, using the ADV-RN50 brain-encoder backbone.
Figure 63: MEIs of top-8 voxels for Subject 02, visual area V3, using the ADV-RN50 brain-encoder backbone.
Figure 64: MEIs of top-8 voxels for Subject 02, visual area hV4, using the ADV-RN50 brain-encoder backbone.
Figure 65: MEIs of top-8 voxels for Subject 02, visual area V1, using the DINO-RN50 brain-encoder backbone.
Figure 66: MEIs of top-8 voxels for Subject 02, visual area V2, using the DINO-RN50 brain-encoder backbone.
Figure 67: MEIs of top-8 voxels for Subject 02, visual area V3, using the DINO-RN50 brain-encoder backbone.
Figure 68: MEIs of top-8 voxels for Subject 02, visual area hV4, using the DINO-RN50 brain-encoder backbone.
Figure 69: MEIs of top-8 voxels for Subject 05, visual area V1, using the ADV-RN50 brain-encoder backbone.
Figure 70: MEIs of top-8 voxels for Subject 05, visual area V2, using the ADV-RN50 brain-encoder backbone.
Figure 71: MEIs of top-8 voxels for Subject 05, visual area V3, using the ADV-RN50 brain-encoder backbone.
Figure 72: MEIs of top-8 voxels for Subject 05, visual area hV4, using the ADV-RN50 brain-encoder backbone.
Figure 73: MEIs of top-8 voxels for Subject 05, visual area V1, using the DINO-RN50 brain-encoder backbone.
Figure 74: MEIs of top-8 voxels for Subject 05, visual area V2, using the DINO-RN50 brain-encoder backbone.
Figure 75: MEIs of top-8 voxels for Subject 05, visual area V3, using the DINO-RN50 brain-encoder backbone.
Figure 76: MEIs of top-8 voxels for Subject 05, visual area hV4, using the DINO-RN50 brain-encoder backbone.
Figure 77: MEIs of top-8 voxels for Subject 07, visual area V1, using the ADV-RN50 brain-encoder backbone.
Figure 78: MEIs of top-8 voxels for Subject 07, visual area V2, using the ADV-RN50 brain-encoder backbone.
Figure 79: MEIs of top-8 voxels for Subject 07, visual area V3, using the ADV-RN50 brain-encoder backbone.
Figure 80: MEIs of top-8 voxels for Subject 07, visual area hV4, using the ADV-RN50 brain-encoder backbone.
Figure 81: MEIs of top-8 voxels for Subject 07, visual area V1, using the DINO-RN50 brain-encoder backbone.
Figure 82: MEIs of top-8 voxels for Subject 07, visual area V2, using the DINO-RN50 brain-encoder backbone.
Figure 83: MEIs of top-8 voxels for Subject 07, visual area V3, using the DINO-RN50 brain-encoder backbone.
Figure 84: MEIs of top-8 voxels for Subject 07, visual area hV4, using the DINO-RN50 brain-encoder backbone.
A central goal in understanding human vision is to uncover the visual features that drive neuronal activity. A growing body of work has used artificial neural networks as encoding models to predict cortical responses to natural images, revealing the visual content that activates category-selective regions. However, existing approaches are largely correlational and treat the encoder as a black box, leaving open which image features drive each voxel's response. We introduce Mechanistically Interpretable Neural Encoding (MINE), a framework that opens this black box by applying mechanistic-interpretability tools to localize the features within natural images that drive millimeter-scale (voxel-level) activity. MINE predicts each voxel's response using language-aligned image representations, and produces semantically interpretable descriptions of the features critical for the voxel's activation. We further generalize these per-image features into per-voxel functional profiles. To validate the per-image descriptions, we show they are sufficient to generate images that elicit voxel responses matching the responses to the original images, more accurately than images generated from random or low-attribution controls. Moreover, counterfactually inserting or removing the predicted features from images shifts activation in the expected direction, providing causal evidence. Counterfactual editing guided by the per-voxel activation profiles produces even stronger activation shifts, indicating that the profiles faithfully capture each voxel's selectivity. Finally, we apply MINE to well-studied category-selective brain regions, showing it recovers their known categorical preferences while revealing fine-grained unique voxel structure within each region. Overall, our results establish mechanistic interpretability as a path to discover and causally validate fine-grained hypotheses about neural function.
Idan Daniel Grosbard, Mor Geva, Galit Yovel
1Sagol School of Neuroscience · 2Blavatnik School of Computer Science and AI · School of Psychological Sciences
Brain encoder models predict cortical fMRI responses from the internal activations of pretrained vision and language networks, and are typically evaluated by held-out prediction accuracy. This is a useful signal for training but a poor one for interpretation: it tells us an encoder fits the data without telling us whether it has internalized the functional organization of the brain. We propose feature visualization -- gradient ascent on the encoder's predicted activation for a target region of interest (ROI) -- as a complementary interpretability technique, and apply it to TRIBE v2 composed with V-JEPA 2 (ViT-G, 40 layers), holding both frozen and synthesizing still images for seven regions spanning the ventral and dorsal visual hierarchies. Under identical hyperparameters, the probe recovers a visible progression of increasing spatial scale and feature complexity across V1 to V4, matching the ventral-stream hierarchy. It also produces three distinctive downstream regimes: radial "frozen-motion" streaks for the middle temporal area (MT) despite static-only optimization, face-like features for the fusiform face area (FFA), and consistent rectilinear line patterns for the parahippocampal place area (PPA). Optimized FFA stimuli drive the predicted region ~4x as much as a natural face photograph, consistent with feature visualization producing adversarial super-stimuli rather than canonical exemplars. The probe is simple, differentiable, and applicable to any brain encoder with a differentiable backbone, allowing for qualitative evaluation of brain encoders.
The human brain processes dynamic visual input through hierarchically organized, functionally specialized regions. While recent in silico brain encoding models can synthesize optimal stimuli to probe selectivity in different brain regions, prior work has been largely limited to static images, leaving dynamic visual processing underexplored. We introduce a novel neural-guided video synthesis framework that generates stimuli optimized for target brain regions across visual cortex. Our method performs evolutionary search over a structured prompt space, guided by a dynamic encoding model that predicts voxel-level responses to video inputs. By maximizing predicted activity for a target ROI, the framework efficiently discovers hyper-activating dynamic stimuli that consistently surpass handcrafted localizer videos. The synthesized videos recover known selectivities across ventral, dorsal, and lateral pathways, and further reveal systematic differences in sensitivity to temporal dynamics. A searchlight analysis provides new insight into the progression toward increasingly complex social-dynamic features along the lateral stream, further supported by probing with synthesized abstract, non-naturalistic stimuli. Taken together, our framework enables in silico exploration of dynamic visual selectivity, with new predictions for in vivo experiments