Structured Visual Landscape: Generating Preferred Representations in Biological and Artificial Neural Networks
Organizations: University of Illinois Urbana-Champaign Urbana, Illinois, United States · University of Washington Seattle, Washington, United States · Georgia Institute of Technology Atlanta, Georgia, United States
Abstract
Understanding how neurons respond to visual stimulus is an important question in both deep learning and neuroscience. It has significant implications for enhancing the interpretability of black-box artificial neural networks and for understanding the visual representation in the human brain. We propose a structured visual representations landscape and design an activation-based prior that allows us to effectively regularize the landscape with activations from brain regions or neural network units. Our model Vis-Lens integrates a variational autoencoder and a diffusion model into an image generation framework. It enables the generation of natural, realistic preferred images by directly modifying the activation-regularized latent codes without per-image iterative optimization. We demonstrate the effectiveness of our framework on both artificial neural networks and human visual cortex using functional magnetic resonance imaging (fMRI) data. Our framework improves activation modulation and image realism compared with a recent state-of-the-art method for generating preferred visual representations of these neural targets. Code is available at https://github.com/NeuroAIHub/Vis-Lens.
Figures & tables
| ROIs | OFA | FFA | EBA | VWFA | OPA | PPA | RSC |
| Activation Increase (original subj.) | 0.869 | 0.851 | 0.860 | 0.867 | 0.731 | 0.966 | 0.942 |
| Activation Increase (held-out subj.) | 0.861 | 0.975 | 0.790 | 0.648 | 0.629 | 1.061 | 0.940 |
| Activation Decrease (original subj.) | -0.526 | -0.382 | -0.564 | -0.279 | -0.557 | -0.610 | -0.775 |
| Activation Decrease (held-out subj.) | -0.436 | -0.302 | -0.391 | -0.089 | -0.485 | -0.813 | -0.823 |
Appendix figures & tables30 assets
Supplementary material from the paper’s appendix.
Appendix
| ROIs | V1 | V2 | V3 | V4 | V8 | LO1 | LO2 | V4t | |
| Activation Increase ( ) | Ours | 0.257 | 0.080 | 0.247 | 0.486 | 0.696 | 0.583 | 0.498 | 0.757 |
| BrainACTIV | 0.157 | 0.116 | 0.226 | 0.488 | 0.721 | 0.582 | 0.517 | 0.767 | |
| Activation Decrease ( ) | Ours | -0.291 | -0.272 | -0.573 | -0.690 | -0.406 | -0.485 | -0.490 | -0.601 |
| BrainACTIV | 0.102 | 0.224 | 0.274 | 0.501 | 0.608 | 0.100 | -0.057 | -0.511 | |
| FID@2k | Ours | 164.87 | 157.84 | 167.44 | 173.24 | 160.07 | 136.05 | 186.67 | 178.62 |
| BrainACTIV | 224.29 | 201.56 | 240.64 | 231.60 | 216.80 | 208.23 | 255.06 | 197.61 |
| Hyperparameter | NSD | ImageNet |
| Data Hyperparameters | ||
| input_size | 1024 dim embeddings | 1024 dim embeddings |
| n_channels | 1 | 1 |
| Model Hyperparameters | ||
| latent_dim | 2048 | 2048 |
| Training Settings | ||
Visual Neuroscience
Visual neuroscience studies how retinal and brain circuits encode, interpret, and guide behavior from visual input.
See all 63 papers