Decoding spatial and semantic information from large-scale visual cortical activity for natural-movie reconstruction
Organizations: MOE Frontiers Center for Brain Science, State Key Laboratory of Brain Function and Disorders, Research Institute of Intelligent and Complex Systems, and Institute of Science, Fudan University, Shanghai, China. · Institute of Science and Technology for Brain-inspired Intelligence, Fudan University, Shanghai, China. · University of Edinburgh Institute International Campus, Zhejiang University, Hangzhou, China. · School of Life Sciences, Fudan University, Shanghai, China. ·
Abstract
Reconstructing natural visual experience from cortical population activity is a central challenge in neural decoding, particularly for calcium imaging, where signals are noisy, temporally filtered, and paired neural-video data are limited. Here we present Ca2REAL (Calcium Activity to Reconstruct Events, Appearance, and Layout), a neural-to-video framework that conditions a pretrained video diffusion model through separate layout and semantic controls derived from calcium activity. Using measured responses from 7,863 V1 neurons in one mouse, Ca2REAL reconstructed held-out natural-movie clips with 81.4% all-pairs stimulus-identification accuracy, with its clearest gains over deterministic baselines in perceptual and semantic measures. We further evaluated the framework in a stimulus-aligned virtual population of 116,705 neurons assembled from encoding-model-predicted responses across 11 independently recorded datasets. Perturbation and ablation analyses showed that reconstruction depends on distributed, stimulus-specific population structure, with layout and semantic pathways providing complementary spatial and identity information. Temporal analyses further showed improved preservation of semantic dynamics across reconstructed clips. These results show that large-scale cortical population activity constrains the spatial, semantic, and temporal structure of natural visual experience, providing a generative framework for high-fidelity neural reconstruction of dynamic visual scenes.
Figures & tables
| Condition | ID Acc. (%) | (pp) |
|---|---|---|
| Synthetic | 92.83 | 0.00 |
| Matched measured | 91.98 | |
| Shuffled measured | 91.10 | |
| Zeroed responses | 91.46 |
| Metric | Formula | Meaning and notation |
|---|---|---|
| MSE | Mean pixel-wise error between reconstruction and target ; is the total number of evaluated pixels. | |
| PSNR | Logarithmic pixel fidelity; is the maximum pixel value. | |
| SSIM | Local structural similarity; are local means, are local variances, is local covariance, and are stabilization constants. | |
| MS-SSIM | Multiscale structural similarity; indexes scale, is the coarsest scale, and , , and denote luminance, contrast, and structure terms weighted by , , and . | |
| LPIPS | Deep-feature perceptual distance; indexes feature layers, denotes channel-normalized features, are their spatial dimensions, is spatial position, contains learned channel weights, and denotes element-wise multiplication. | |
| DISTS | Deep-feature structural-textural distance; and index feature layer and channel, and are structural and texture similarity terms, and are learned weights. |
Neural Decoding
Neural decoding maps brain activity or error-correction syndromes to stimuli, behavior, language, or corrective actions.
See all 201 papers