A central question in neuroscience is what visual features drive brain responses across regions. Classic approaches rely on handcrafted stimuli to test specific hypotheses, limiting discovery to features defined before experiments. Here we introduce NeuroVolve, a data-driven generative framework for synthesizing images that achieve flexible, user-defined patterns of brain activation in individuals. NeuroVolve recovers established selectivity across visual cortical regions and, paired with subject-specific brain encoding models, generates personalized stimuli reflecting individual differences in neural preference. Beyond single-region selectivity, NeuroVolve uncovers multi-region response selectivity, characterizing how regions jointly represent visual content during naturalistic perception. We further demonstrate generalization beyond visual cortex: in a prospective fMRI experiment, NeuroVolve-generated stimuli targeting the amygdala, a classically non-sensory region, drove measured activation consistent with predictions in novel participants. Together, these results establish NeuroVolve as a less constrained, data-driven tool for probing individual-level neural selectivity across the brain.
Figures & tables
Figure 1: NeuroVolve framework for brain-response-guided image generation. a , Schematic of the NeuroVolve pipeline. An initial image is encoded into Q-Former (Querying Transformer) embeddings by a frozen image encoder and Q-Former. A pretrained subject-specific neural encoding model, Φ , predicts voxelwise brain responses from these embeddings, and a user-defined brain-response objective guides their iterative optimization. The resulting optimized embeddings, q∗ , are combined with a null-text prompt and used to condition the frozen BLIP-Diffusion model for image generation. Snowflake symbols denote frozen model components while fire symbols denote trained model components. The illustrated objective maximizes the predicted response of the fusiform face area (FFA). b , Conceptual illustration of the corresponding optimization trajectory in Q-Former embedding space. Pink dashed contours represent the brain-response objective loss landscape; blue points and arrows indicate successive embedding states from the initial embedding to the optimized embedding. Example images depict the visual content generated at selected points along the trajectory.
Figure 2: NeuroVolve validation via a single-region maximization target objective. a , Optimization and validation trajectories for fusiform face area (FFA) from the initial cat seed image and b human visual area V4 (hV4) with the initial pizza seed image. Upper plots in both panels show the BLIP-based optimization objective value; lower plots show predictions of the subject-specific CLIP-based validation encoder, not used during optimization, for images generated along the trajectory. Points correspond to 20%, 50%, 70% and 100% of the total increase in the optimization objective, and the star marks the maximum value on the validation curve; the checkpoint images are shown between the plots. c , Image sequences generated by maximizing the mean predicted voxel response within six visual regions of interest (ROIs): FFA, extrastriate body area (EBA), parahippocampal place area (PPA), mid-fusiform sulcus word-selective region (mfs-words), primary visual cortex (V1) and hV4. Each row has a different initial image (4 initial images shown at the top) and each column shows the 20%, 50%, 70% and 100% checkpoint images. The last column shows the generated image with the highest prediction from a separate CLIP-based validation encoder (max validation). Images are shown for NSD subject 5; other subjects are in the Supplementary Note 2.
Figure 3: NeuroVolve preserves within- and across-individual differences in optimal images. a , Within- and across-subject similarity of the top 50 NeuroVolve images (blue) and the top 50 predicted natural NSD images (green) for each target region, ranked by the region’s predicted activity from each subject’s CLIP-based validation encoder. Similarity is the mean cosine similarity between image embeddings over all image pairs within or across subjects. Bars show the mean and error bars the s.d. across participants ( n=8 ) or participant pairs ( n=28 ). Brackets show two-tailed exact Wilcoxon signed-rank tests ( n=8 ), with values corrected across the 18 tests in a by the Benjamini–Hochberg false discovery rate procedure; * q<0.05 ; n.s., not significant. b , Across-subject similarity of NeuroVolve versus natural images for all subject pairs and regions ( n=168 ; 28 pairs × 6 regions), z -scored within each region. The dashed line shows the least-squares fit, and r is the pooled Pearson correlation. c , The same data as in b , shown for each region on the original similarity scale ( n=28 pairs per panel). Outlined points mark the pairs shown in d–i . In b and c , p values are from one-tailed exact permutation tests of participant labels. d–i , The top five natural and NeuroVolve images of each participant in the most and least similar pairs in c , for FFA ( d ), EBA ( e ), PPA ( f ), mfs-words ( g ), V1 ( h ) and hV4 ( i ).
Figure 4: Image synthesis under single- and multi-region brain-response objectives. a,b , Example trajectories for maximal co-activation of V1 and FFA ( a ) and maximal activation of PPA with minimal activation of FFA ( b ) starting from the same initial seed image (cat, on far left in row of images). Upper plots show the predicted response of each target region at every iteration of NeuroVolve’s Q-Former embedding optimization, demonstrating optimization of the specific provided objective. Lower plots show regional responses predicted by subject-specific CLIP-based validation encoder, not used during optimization, for images generated along each trajectory. Colored points mark selected checkpoints (initial value, 20, 50, 70 and 100 % of final optimization curve value), with the checkpoints’ corresponding generated images shown between the plots. Dashed horizontal lines correspond to response to that regions’ predicted response to the initial seed image. V1, primary visual cortex; FFA, fusiform face area; PPA, parahippocampal place area. Plus and minus signs denote maximization and minimization of the predicted regional response, respectively. c , NeuroVolve images generated from a single common seed image (cat, far left) under single-region minimization ( − V1, − FFA, and − PPA), two-region co-maximization ( + V1 + FFA and + PPA + FFA), and mixed max/min two-region objectives ( − V1 + FFA, + V1 − FFA, − PPA + FFA and + PPA − FFA).
Figure 5: Construction and in silico validation of amygdala modulating images. a , NeuroVolve optimization curve example of maximizing (red) and minimizing (blue) amygdala responses. Rows of images at top and bottom show corresponding images decoded at checkpoints that are the indicated percentages of the total change in the optimization objective. b,c , Validation model’s predicted amygdala responses for five images generated with the same latent embedding but different guided diffusion seeds at each sampled iteration of the maximization ( b ) and minimization ( c ) trajectories. Thinner, colored lines show individual diffusion samples and thicker black lines their average. Horizontal dashed lines denote 95% of the maximum magnitude predicted response change from baseline; yellow markers and vertical lines identify the earliest iteration reaching this 95% threshold. Images at the bottom show samples from the early stopping selected iteration (top row) and final (bottom row) NeuroVolve iterations, with their validation ensemble predictions listed below them. Colored outlines identify an example image that was selected for the fMRI experiment. d , Five randomly selected images from: (top row) the top 65 natural images yielding the highest amygdala activity from the encoding model (Natural High), (second row) the top 65 natural images yielding the lowest predicted amygdala activity (Natural Low), and (third row) 65 natural images yielding predicted amygdala activity nearest to zero (Natural Zero). NeuroVolve High/Low images (rows 4 and 5) were initialized from Natural Zero images and optimized to increase or decrease the population-predicted amygdala response; images shown were selected based on their values in the CLIP-based validation population model prediction (see panels b, c).
Figure 6: in vivo experimental validation of amygdala modulating images. Responses are shown as beta weights averaged across the amygdala and across six participants for a given image ( n=65 images per condition, with two repeats per image). a , Average amygdala responses per image, across five image conditions. Points represent image averages across participants; vertical bars are interquartile ranges; horizontal lines are medians; white diamonds are means; dashed line shows β=0 . Brackets show significant Tukey HSD comparisons following ANOVA, adjusted across ten comparisons: ∗ corrected p<0.05 . b , The x and y axes show t -statistics of Natural high/low images’ brain responses and NeuroVolve high/low images’ brain responses, contrasted against the responses to Natural Zero images for 378 brain regions (360 HCP–MMP1 cortical parcels and 18 FreeSurfer subcortical structures). Each brain region contributes a High-condition point (red) and a Low-condition point (blue). Stars identify the left and right amygdala. Dotted lines show least-squares fits; r denotes Pearson correlation. Dashed lines mark significance thresholds; shaded areas indicate significantly high or low activity for the corresponding contrasts. Extreme regions are labeled. c , Regional t -statistics of brain activity in response to: NeuroVolve High images versus Natural High images (left) and NeuroVolve Low versus Natural Low (right). Red and blue indicate higher and lower NeuroVolve responses compared to Natural responses within the condition. Green outlines identify significantly different regions after Benjamini–Hochberg FDR correction (corrected p<0.05 ); pink outlines identify the amygdala.
Figure 7: BLIP-based optimization-encoder accuracy for NSD participants S1 - S4. Maps show voxelwise Pearson correlations between predicted and measured responses across the shared NSD evaluation images. Only voxels included in the participant-specific visual-cortex model are displayed.
Figure 8: BLIP-based optimization-encoder accuracy for NSD participants S5 - S8. Maps show voxelwise Pearson correlations between predicted and measured responses across the shared NSD evaluation images. Only voxels included in the participant-specific visual-cortex model are displayed.
Figure 9: Regional encoding-model accuracy in the visual cortex and amygdala. Bars show region-level Pearson correlations between predicted and measured responses to the "shared 1K" evaluation images. CLIP-based validation-encoder accuracy is shown for each participant’s visual regions and left and right amygdala. BLIP-based optimization-encoder accuracy is also shown for the left and right amygdala and mirrors the CLIP-based results.
Figure 10: NeuroVolve’s single-region maximization results for NSD participants S1 and S2. Images were generated from four common initialization images by maximizing the predicted response of FFA, EBA, PPA, mfs-words, V1 or hV4. Columns show images decoded at 20%, 50%, 70% and 100% of the total increase in the BLIP-based optimization objective, with the final column being the image that had the maximum value in the CLIP-based validation encoder.
Figure 11: Single-region image evolution for NSD participants S3 and S4. Images were generated from four common initialization images by maximizing the predicted response of FFA, EBA, PPA, mfs-words, V1 or hV4. Columns show images decoded at 20%, 50%, 70% and 100% of the total increase in the BLIP-based optimization objective, with the final column being the image that had the maximum value in the CLIP-based validation encoder.
Figure 12: Single-region image evolution for NSD participants S5 and S6. Images were generated from four common initialization images by maximizing the predicted response of FFA, EBA, PPA, mfs-words, V1 or hV4. Columns show images decoded at 20%, 50%, 70% and 100% of the total increase in the BLIP-based optimization objective, with the final column being the image that had the maximum value in the CLIP-based validation encoder.
Figure 13: Single-region image evolution for NSD participants S7 and S8. Images were generated from four common initialization images by maximizing the predicted response of FFA, EBA, PPA, mfs-words, V1 or hV4. Columns show images decoded at 20%, 50%, 70% and 100% of the total increase in the BLIP-based optimization objective, with the final column being the image that had the maximum value in the CLIP-based validation encoder.
Image set
FFA (face)
EBA (body)
PPA (scene)
mfs-words (text)
NeuroVolve top 100 (validation prediction)
84.8±14.2
70.9±25.5
97.9±6.0
76.8±27.9
NSD top 100 (validation prediction)
60.0±8.9
69.9±9.6
90.9±10.1
50.0±12.1
NSD top 100 (measured response)
45.5±7.2
68.3±6.4
73.0±13.4
34.6±11.2
Table 1: Semantic specificity across NSD participants. Values are the percentages of images assigned to the category associated with the target ROI by zero-shot CoCa ViT-L/14 classification, reported as mean ± s.d. across eight participants. The measured-response natural reference contains the 100 highest-response images among those viewed by each participant. The predicted-response natural reference contains the 100 highest-ranked images from the full pool of 10,000 available natural images, scored by the participant-specific CLIP-based validation encoder. NeuroVolve sets contain the 100 generated images ranked highest by the same validation encoder.
Category
Prompts
Face
A face facing the camera; A photo of a face; A photo of a human face; A photo of faces; A photo of a person’s face; A person looking at the camera; People looking at the camera; A portrait of a person; A portrait photo
Body
A photo of a torso; A photo of torsos; A photo of limbs; A photo of bodies; A photo of a person; A photo of people
Scene
A photo of a bedroom; A photo of an office; A photo of a hallway; A photo of a doorway; A photo of interior design; A photo of a building; A photo of a house; A photo of nature; A photo of landscape; A landscape photo; A photo of trees; A photo of grass
Text
A photo of words; A photo of glyphs; A photo of a glyph; A photo of text; A photo of numbers; A photo of a letter; A photo of letters; A photo of writing; A photo of text on an object
Table 2: Text prompts for zero-shot semantic classification. The same category-specific prompt sets were used to classify natural and NeuroVolve images with CoCa ViT-L/14.
Figure 14: CLIP-based validation-encoder’s predicted responses to 100 top-ranked natural and NeuroVolve images. Distributions show responses predicted by the participant-specific CLIP-based validation encoder for the 100 highest-ranked natural NSD images and 100 highest-ranked NeuroVolve images in FFA, EBA, PPA, mfs-words, V1 and hV4. Circles indicate medians and thick bars indicate interquartile ranges.
Figure 15: Participants’ top 50 NeuroVolve images do not share initial seed images beyond chance. a , Number of seeds shared by the top 50 NeuroVolve images of each participant pair, for each region; white marks the chance level of 2.5. b , Shared seeds for each pair (small dots; n=28 ) and their mean (large dots). The gray band and dashed line show the 95% range and mean expected by chance (100,000 simulations of random top-50 selection). In no region was the overlap greater than chance (one-tailed uncorrected p>0.05 ), so the fact that NeuroVolve images shared seeds had minimal effect on the subject-specificity results in Fig. 3 .
Figure 16: NeuroVolve preserves within- and across-individual measured differences in optimal images. a , Within- and across-subject similarity of the top 50 NeuroVolve images (blue) and the top 50 measured natural NSD images (red) for each target region, ranked by the region’s measured activity in the NSD dataset. Similarity is the mean cosine similarity between image embeddings over all image pairs within or across subjects. Bars show the mean and error bars the s.d. across participants ( n=8 ) or participant pairs ( n=28 ). Brackets show two-tailed exact Wilcoxon signed-rank tests ( n=8 ), with values corrected across the 18 tests in a by the Benjamini–Hochberg false discovery rate procedure; * q<0.05 ; n.s., not significant. b , Across-subject similarity of NeuroVolve versus measured natural images for all subject pairs and regions ( n=168 ; 28 pairs × 6 regions), z -scored within each region. The dashed line shows the least-squares fit, and r is the pooled Pearson correlation. c , The same data as in b , shown for each region on the original similarity scale ( n=28 pairs per panel).
Figure 17: Individual-level brain responses to amygdala modulating images. Same layout as Fig. 6 , but without averaging image responses across participants. We used a linear mixed-effects model to test for image condition response differences, with participant and image as random effects. a , Amygdala responses for each image condition group, with points are individual participants’ image responses for that condition. Vertical bars are interquartile ranges; horizontal lines are medians; white diamonds are means; dashed line shows β=0 . Top brackets show significant pairwise contrasts from the mixed-effects model, Holm-adjusted across ten comparisons: ∗ corrected p<0.05 . b , The x and y axes show mixed-effects model β for Natural high/low images’ brain responses and NeuroVolve high/low images’ brain responses, contrasted against the responses to Natural Zero images for 378 brain regions, with the model fit separately for each region. Each brain region contributes a High-condition point (red) and a Low-condition point (blue). Solid points are significant for at least one of the two contrasts (Natural or NeuroVolve versus Natural Zero; Benjamini–Hochberg FDR-corrected p<0.05 ); faded points are significant for neither. Stars identify the left and right amygdala. Dotted lines show least-squares fits; r denotes Pearson correlation. Extreme regions are labeled. c , Regional mixed-effects model β of brain activity in response to: NeuroVolve High images versus Natural High images (left) and NeuroVolve Low versus Natural Low (right). Red and blue indicate higher and lower NeuroVolve responses compared to Natural responses within the condition. Green outlines identify significantly different regions after Benjamini–Hochberg FDR correction (corrected p<0.05 ); pink outlines the amygdala.
The human brain processes dynamic visual input through hierarchically organized, functionally specialized regions. While recent in silico brain encoding models can synthesize optimal stimuli to probe selectivity in different brain regions, prior work has been largely limited to static images, leaving dynamic visual processing underexplored. We introduce a novel neural-guided video synthesis framework that generates stimuli optimized for target brain regions across visual cortex. Our method performs evolutionary search over a structured prompt space, guided by a dynamic encoding model that predicts voxel-level responses to video inputs. By maximizing predicted activity for a target ROI, the framework efficiently discovers hyper-activating dynamic stimuli that consistently surpass handcrafted localizer videos. The synthesized videos recover known selectivities across ventral, dorsal, and lateral pathways, and further reveal systematic differences in sensitivity to temporal dynamics. A searchlight analysis provides new insight into the progression toward increasingly complex social-dynamic features along the lateral stream, further supported by probing with synthesized abstract, non-naturalistic stimuli. Taken together, our framework enables in silico exploration of dynamic visual selectivity, with new predictions for in vivo experiments
An ongoing challenge in sensory neuroscience is to characterize the feature dimensions encoded by cortical populations. Recent approaches probe feature selectivity in a data-driven way, by synthesizing a most-exciting-input (MEI) for a target neural population. While this approach has been successfully applied to human higher visual cortex using fMRI data, generating MEIs for early- and mid-level retinotopic visual areas requires additional modeling constraints due to small receptive field sizes. To address this challenge, we introduce two novel MEI generation frameworks, Receptive Field Diffusion for Visual Exploration (RF-DiVE) and Receptive Field Gradient Optimization (RF-GO). Both methods use a population receptive field (pRF)-constrained voxelwise encoding model; RF-DiVE combines this with a pretrained latent diffusion model, while RF-GO uses regularized gradient ascent. When applied to single voxels in retinotopically defined areas V1-hV4, using data from the Natural Scenes Dataset, we obtain MEIs that exhibit consistent structure within the pRF, suggesting selectivity for local features like contour, color, and texture. We systematically compare MEIs generated by RF-DiVE and RF-GO using two encoding backbones, performing in-silico validation of predicted responses to MEIs using independent encoding models. Across all methods and all visual areas, MEIs elicit higher model-predicted responses than the most activating natural images. We further find that the choice of generation framework and encoding backbone differentially affects MEI properties, including their visual appearance, structural interpretability, and cross-model generalizability. These results offer a new approach for performing data-driven characterization of spatial and feature selectivity across human visual cortex.
Junru Zhao, Hanfei Guo, Andrew Luo +1
Carnegie Mellon University · University of Hong Kong
Deep learning has driven substantial progress in modeling fMRI responses, yet most existing approaches are optimized for a single objective---such as neural encoding, stimulus decoding, or post hoc interpretation---making it difficult to obtain a coherent account of neural representations. We introduce a unified framework that connects vision foundation models to brain activity while supporting neural analysis from multiple complementary perspectives. Our approach replaces black-box features with a sparse, interpretable representation composed of visual prototypes, and links these prototypes to voxel responses through a linear readout. This design makes the model transparent at both levels: prototype activations expose the visual concepts present in the stimulus, while the learned linear weights characterize each voxel's selectivity. Using this interface, we identify which visual concepts drive neural responses and where they appear in the image, reveal cortical organization through voxel tuning in prototype space, and perform controlled stimulus synthesis to probe cortical selectivity through targeted interventions. Together, these results establish prototype-based representations as a unified and interpretable foundation for studying visual representations in the human brain.
Shi Chen, Seoyoung Ahn, Doris Tsao
Department of Neuroscience, UC Berkeley · College of AI Convergence, Hankuk University of Foreign Studies