cs.CVNov 6, 2026

Decoding spatial and semantic information from large-scale visual cortical activity for natural-movie reconstruction

Authors: Yukai Huang, Haoyu Albert Wang, Jiawei Zhang, Jialun Ma, Jiayao Ding, Yiran Lin, Xingyu Chen, Ziqi Gao, 

Organizations: MOE Frontiers Center for Brain Science, State Key Laboratory of Brain Function and Disorders, Research Institute of Intelligent and Complex Systems, and Institute of Science, Fudan University, Shanghai, China. · Institute of Science and Technology for Brain-inspired Intelligence, Fudan University, Shanghai, China. · University of Edinburgh Institute International Campus, Zhejiang University, Hangzhou, China. · School of Life Sciences, Fudan University, Shanghai, China. · 

Abstract

Reconstructing natural visual experience from cortical population activity is a central challenge in neural decoding, particularly for calcium imaging, where signals are noisy, temporally filtered, and paired neural-video data are limited. Here we present Ca2REAL (Calcium Activity to Reconstruct Events, Appearance, and Layout), a neural-to-video framework that conditions a pretrained video diffusion model through separate layout and semantic controls derived from calcium activity. Using measured responses from 7,863 V1 neurons in one mouse, Ca2REAL reconstructed held-out natural-movie clips with 81.4% all-pairs stimulus-identification accuracy, with its clearest gains over deterministic baselines in perceptual and semantic measures. We further evaluated the framework in a stimulus-aligned virtual population of 116,705 neurons assembled from encoding-model-predicted responses across 11 independently recorded datasets. Perturbation and ablation analyses showed that reconstruction depends on distributed, stimulus-specific population structure, with layout and semantic pathways providing complementary spatial and identity information. Temporal analyses further showed improved preservation of semantic dynamics across reconstructed clips. These results show that large-scale cortical population activity constrains the spatial, semantic, and temporal structure of natural visual experience, providing a generative framework for high-fidelity neural reconstruction of dynamic visual scenes.

Figures & tables

Explore similar work

CardsList
  1. Neurons++: A Unified Visual-Stream Framework for Image and Video Reconstruction from fMRI with Neural-to-Visual Interpretability

    Oct 9, 2026Haonan Wang, Yuchen Wang, Jingyu Lu +4fMRI-Based Visual DecodingNeural Decoding

  2. Neuromorphic Decoding and Dynamical Image Reconstruction from Populations of Spiking Retinal Ganglion Cells

    Oct 12, 2026Ning Zhang, Arto NurmikkoNeural DecodingImage Reconstruction

  3. Interpreting V1 Population Activity via Image-Neural Latent Representation Alignment

    May 5, 2026Xin Wang, Zhuangzhi Gao, Hongyi Qin +3Contrastive LearningNeural Processes