Structured Visual Landscape: Generating Preferred Representations in Biological and Artificial Neural Networks
Authors: Yusong Zhao, Edgar Y. Walker, Hao Wang, Mi Lu
Organizations: University of Illinois Urbana-Champaign Urbana, Illinois, United States · University of Washington Seattle, Washington, United States · Georgia Institute of Technology Atlanta, Georgia, United States
Understanding how neurons respond to visual stimulus is an important question in both deep learning and neuroscience. It has significant implications for enhancing the interpretability of black-box artificial neural networks and for understanding the visual representation in the human brain. We propose a structured visual representations landscape and design an activation-based prior that allows us to effectively regularize the landscape with activations from brain regions or neural network units. Our model Vis-Lens integrates a variational autoencoder and a diffusion model into an image generation framework. It enables the generation of natural, realistic preferred images by directly modifying the activation-regularized latent codes without per-image iterative optimization. We demonstrate the effectiveness of our framework on both artificial neural networks and human visual cortex using functional magnetic resonance imaging (fMRI) data. Our framework improves activation modulation and image realism compared with a recent state-of-the-art method for generating preferred visual representations of these neural targets. Code is available at https://github.com/NeuroAIHub/Vis-Lens.
Figures & tables
Figure 1. Original CLIP Space (Top): Original CLIP embedding space without any activation constraint: image embeddings that elicit high and low neural responses are dispersed and not clustered throughout the space without shared visual structure. Structured Landscape (Bottom): By encoding CLIP embeddings into the regularized latent space constrained with neuronal activations (after z-score normalization) as the prior , our method Vis-Lens could provide a more structured and representative landscape to better identify the representations and preferences of brain regions than sorting top-k images in a dataset. The results are based on brain activations from the NSD dataset ( Allen et al., 2021 ) . In the landscape plots, the color bar and z axis indicate the activation of the target brain region given different images as inputs, while the x and y axes represent two latent dimensions, respectively.
Figure 2. Overview of Vis-Lens . The illustration depicts a two-stage approach integrating a structured landscape and variational autoencoder (VAE) generation with CLIP encoder and diffusion based generative model. Phase I: The input image, itrain , is encoded by a pretrained CLIP encoder E , CLIP embeddings x are encoded into a latent representation z by the VAE encoder. A regularization step using a activation-based prior N(μprior,∣r∣1) modifies the latent space for better alignment with activations, producing a regularized latent z . The VAE is also trained to reconstruct the input CLIP embeddings x with xrecon. Phase II: The test images itest , is encoded by same CLIP encoder E . Then, with fully trained VAE and the structured visual latent representation space, we can directly modify latents constrained by a specific feature represented by the selected regions, obtaining a new latent z′ . This transformed latent is decoded back into an intermediate generated CLIP embeddings xgen , which is used to guild the image generation through a diffusion pipeline, yielding the final output igen as preferred images to modulate the activation r .
Figure 3. Activation change results of our method and BrainACTIV ( Garcia Cerdas et al., 2025 ) (A) Generate preferred representations of multiple ROIs with example images from COCO dataset by our method and baselines. The alpha value in interpolation indicates the strength and direction of preferred semantic in generation. (B) Comparison between predicted activations of ROIs made by DINO-ViT encoder. Means and ranges of activations are presented for different alpha settings. Our method outperforms the baseline in terms of maximizing or minimizing the ROI activations. Activation change results of our method and baseline, showing generated images and predicted activations for different ROIs.
ROIs
OFA
FFA
EBA
VWFA
OPA
PPA
RSC
Activation Increase (original subj.)
0.869
0.851
0.860
0.867
0.731
0.966
0.942
Activation Increase (held-out subj.)
0.861
0.975
0.790
0.648
0.629
1.061
0.940
Activation Decrease (original subj.)
-0.526
-0.382
-0.564
-0.279
-0.557
-0.610
-0.775
Activation Decrease (held-out subj.)
-0.436
-0.302
-0.391
-0.089
-0.485
-0.813
-0.823
Table 2. Cross-subject generalization results on the NSD dataset. The table compares predicted activation changes obtained from the original subject-specific predictor and a held-out-subject predictor across several visual ROIs.
Figure 4. Generate preferred images with increasing unit activations in ANNs. (A) The top 10 images demonstrate a notable pattern of “green plant & insect” represented by a unit in the middle layer of the SimCLR model. Our method generates images that better match this pattern and increase activation more than the BrainACTIV ( Garcia Cerdas et al., 2025 ) . (B) For a mid-layer unit in a more complex Vision Transformer (ViT), our method synthesizes images that can better activate the unit compared with the baseline, although the underlying concept appears more ambiguous.
Figure 5. a) Generating preferred images with different methods on ImageNet Dataset. Left 4 columns show optimizing images to increase their activations. Column 1 are four example input images. Column 2-3 are results of our method while column 4 represent optimizing the latent space with a standard VAE. Right 4 columns are results of optimization in the same methods presenting in the same order, with target of decreasing activations. b) shows the shift in the activation value between and the cosine similarity between randomly chosen 100 original images and their optimized results over different methods. Compared with baseline VAE , our method can modulate activation better as well as maintain similarity with original images, meaning unrelated features are preserved.
Appendix figures & tables30 assets
Supplementary material from the paper’s appendix.
Appendix
ROIs
V1
V2
V3
V4
V8
LO1
LO2
V4t
Activation Increase ( Δr )
Ours
0.257
0.080
0.247
0.486
0.696
0.583
0.498
0.757
BrainACTIV
0.157
0.116
0.226
0.488
0.721
0.582
0.517
0.767
Activation Decrease ( Δr )
Ours
-0.291
-0.272
-0.573
-0.690
-0.406
-0.485
-0.490
-0.601
BrainACTIV
0.102
0.224
0.274
0.501
0.608
0.100
-0.057
-0.511
FID@2k
Ours
164.87
157.84
167.44
173.24
160.07
136.05
186.67
178.62
BrainACTIV
224.29
201.56
240.64
231.60
216.80
208.23
255.06
197.61
Appendix
Table S1. Activation change and realism on additional ROIs. Better values are in bold .
Figure S1. Full result of activation change on region IPS4, positive direction.
Figure S2. Full result of activation change on region IPS4, negative direction.
Figure S3. Full result of activation change on region SPL1, positive direction.
Figure S4. Full result of activation change on region SPL1, negative direction.
Figure S5. Full result of activation change on region 31a, positive direction.
Figure S6. Full result of activation change on region 31a, negative direction.
Figure S7. Full result of activation change on region 31pd, positive direction.
Figure S8. Full result of activation change on region 31pd, negative direction.
Figure S9. Full result of activation change on region V1, positive direction.
Figure S10. Full result of activation change on region V1, negative direction.
Figure S11. Full result of activation change on region V2, positive direction.
Figure S12. Full result of activation change on region V2, negative direction.
Figure S13. Full result of activation change on region V3, positive direction.
Figure S14. Full result of activation change on region V3, negative direction.
Figure S15. Full result of activation change on region V4, positive direction.
Figure S16. Full result of activation change on region V4, negative direction.
Figure S17. One latent modification example to generate preferred images with increasing unit activations on ImageNet-mini Dataset. Though the top 10 images can not demonstrate a notable pattern represented by the unit in the early layer of SimCLR model, our method can generate a uniform representation and compared with original images from ImageNet-mini, generated preferred images based on modifying the latent to the positive prior mean shows a more apparent change in activation compared with original images and representations from BrainACTIV.
Figure S18. One latent modification example to generate preferred images with increasing unit activations on ImageNet-mini Dataset. Though the top 10 images can not demonstrate a notable pattern represented by the unit in the early layer of SimCLR model, our method can generate a uniform representation and compared with original images from ImageNet-mini, generated preferred images based on modifying the latent to the positive prior mean shows a more apparent change in activation compared with original images and representations from BrainACTIV.
Figure S19. One latent modification example to generate preferred images with increasing unit activations on ImageNet-mini Dataset. Though the top 10 images can not demonstrate a notable pattern represented by the unit in the early layer of SimCLR model, our method can generate a uniform representation and compared with original images from ImageNet-mini, generated preferred images based on modifying the latent to the positive prior mean shows a more apparent change in activation compared with original images and representations from BrainACTIV.
Figure S20. One latent modification example to generate preferred images with increasing unit activations on ImageNet-mini Dataset. The top 10 images demonstrate a notable pattern represented by the unit in the deep layer of SimCLR model, that round objects appear in top images. Our method can also generate a uniform representation and compared with original images from ImageNet-mini, generated preferred images based on modifying the latent to the positive prior mean shows a more apparent change in activation compared with original images and representations from BrainACTIV.
Figure S21. One latent modification example to generate preferred images with increasing unit activations on ImageNet-mini Dataset. The top 10 images demonstrate a notable pattern represented by the unit in the deep layer of SimCLR model, that grid objects appear in top images. Our method can also generate a uniform representation and compared with original images from ImageNet-mini, generated preferred images based on modifying the latent to the positive prior mean shows a more apparent change in activation compared with original images and representations from BrainACTIV.
Figure S22. One latent modification example to generate preferred images with increasing unit activations on ImageNet-mini Dataset. Though the top 10 images can not demonstrate a notable pattern represented by the unit in the early layer of SimCLR model, our method can generate a uniform representation and compared with original images from ImageNet-mini, generated preferred images based on modifying the latent to the positive prior mean shows a more apparent change in activation compared with original images and representations from BrainACTIV.
Hyperparameter
NSD
ImageNet
Data Hyperparameters
input_size
1024 dim embeddings
1024 dim embeddings
n_channels
1
1
Model Hyperparameters
latent_dim
2048
2048
Training Settings
Appendix
Table S2. VAE Hyperparameters and Training Settings for NSD and ImageNet-mini datasets.
The emerging paradigm of "thinking with images" embeds visual states into intermediate reasoning steps, defining a new frontier for Vision-Language Models. Existing approaches diverge along two lines. Tool-assisted methods apply explicit visual operations but suffer from high latency and restricted manipulation types. Latent reasoning methods autoregressively produce implicit visual states, but underperform tool-assisted methods, and their latent tokens fail to capture effective visual information. In this work, we propose DeepLatent, a parallel framework for latent visual reasoning. First, we introduce LatentFormer. It uses learnable 2D tokens to generate context-conditioned latent states in parallel, anchoring every visual update directly in the original image features. Second, we design a continuous-space reinforcement learning algorithm. It optimizes latent modulation parameters directly in the embedding space, significantly improving latent representation quality. The framework is trained via knowledge distillation followed by this continuous-space RL algorithm. Furthermore, we contribute DeepLatent-180K, a large-scale dataset tailored for latent visual reasoning. Extensive evaluations across multiple benchmarks demonstrate that DeepLatent achieves state-of-the-art performance.
A central question in neuroscience is what visual features drive brain responses across regions. Classic approaches rely on handcrafted stimuli to test specific hypotheses, limiting discovery to features defined before experiments. Here we introduce NeuroVolve, a data-driven generative framework for synthesizing images that achieve flexible, user-defined patterns of brain activation in individuals. NeuroVolve recovers established selectivity across visual cortical regions and, paired with subject-specific brain encoding models, generates personalized stimuli reflecting individual differences in neural preference. Beyond single-region selectivity, NeuroVolve uncovers multi-region response selectivity, characterizing how regions jointly represent visual content during naturalistic perception. We further demonstrate generalization beyond visual cortex: in a prospective fMRI experiment, NeuroVolve-generated stimuli targeting the amygdala, a classically non-sensory region, drove measured activation consistent with predictions in novel participants. Together, these results establish NeuroVolve as a less constrained, data-driven tool for probing individual-level neural selectivity across the brain.
Haomiao Chen, Keith W Jamison, Daniel Chong +3
Cornell University · Cornell Tech · Weill Cornell Medicine
Interpreting the concepts encoded by individual neurons in deep neural networks is a crucial step towards understanding their complex decision-making processes and ensuring AI safety. Despite recent progress in neuron labeling, existing methods often limit the search space to predefined concept vocabularies or produce overly specific descriptions that fail to capture higher-order, global concepts. We introduce LINE, a novel, training-free iterative approach tailored for open-vocabulary concept labeling in vision models. Operating in a strictly gray-box setting, LINE leverages a large language model and a text-to-image generator to iteratively propose and refine concepts in a closed loop, guided by activation history. We demonstrate that LINE achieves state-of-the-art performance across multiple model architectures, yielding AUC improvements of up to 0.15 on ImageNet and 0.05 on Places365, while discovering, on average, 25% of new concepts missed by predefined vocabularies. Beyond identifying the top concept, LINE provides a complete generation history, enabling polysemanticity evaluation and producing supporting visual explanations that rival gradient-dependent activation maximization methods. The LINE source code is available at https://github.com/CredibleAI/LINE
Vladimir Zaigrajew, Michał Piechota, Gaspar Sekula +2
Centre for Credible AI Warsaw University of Technology · Warsaw University of Technology Warsaw, Poland · Centre for Credible AI, Warsaw University of Technology University of Warsaw, Poland