Organizations: MOE Frontiers Center for Brain Science, State Key Laboratory of Brain Function and Disorders, Research Institute of Intelligent and Complex Systems, and Institute of Science, Fudan University, Shanghai, China. · Institute of Science and Technology for Brain-inspired Intelligence, Fudan University, Shanghai, China. · University of Edinburgh Institute International Campus, Zhejiang University, Hangzhou, China. · School of Life Sciences, Fudan University, Shanghai, China. · College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China.
Reconstructing natural visual experience from cortical population activity is a central challenge in neural decoding, particularly for calcium imaging, where signals are noisy, temporally filtered, and paired neural-video data are limited. Here we present Ca2REAL (Calcium Activity to Reconstruct Events, Appearance, and Layout), a neural-to-video framework that conditions a pretrained video diffusion model through separate layout and semantic controls derived from calcium activity. Using measured responses from 7,863 V1 neurons in one mouse, Ca2REAL reconstructed held-out natural-movie clips with 81.4% all-pairs stimulus-identification accuracy, with its clearest gains over deterministic baselines in perceptual and semantic measures. We further evaluated the framework in a stimulus-aligned virtual population of 116,705 neurons assembled from encoding-model-predicted responses across 11 independently recorded datasets. Perturbation and ablation analyses showed that reconstruction depends on distributed, stimulus-specific population structure, with layout and semantic pathways providing complementary spatial and identity information. Temporal analyses further showed improved preservation of semantic dynamics across reconstructed clips. These results show that large-scale cortical population activity constrains the spatial, semantic, and temporal structure of natural visual experience, providing a generative framework for high-fidelity neural reconstruction of dynamic visual scenes.
Figures & tables
Figure 1: Overview of the data pipeline and Ca2REAL architecture. a , Dynamic Sensorium data acquisition for the primary measured experiment. Awake, head-fixed mice viewed natural movies presented at 30 Hz while locomoting on a cylindrical treadmill, and V1 calcium activity was recorded with two-photon imaging. b , Neural-response synthesis with digital-twin encoding models. A spatiotemporal neural encoder maps each movie to the corresponding population response through a shared visual feature extractor, temporal processing, and population-specific readout. c , Neural and visual preprocessing. Movie frames were centrally cropped and resized to the generator resolution, while calcium responses were normalized neuron-wise using training-set statistics before decoding. d , Ca2REAL architecture. A multiscale temporal front end provides features to two neural-control pathways. The layout pathway predicts a low-resolution movie for spatial control of Wan2.1-VACE, whereas the semantic pathway predicts global and temporally ordered visual tokens that are adapted to the generator conditioning space.
Figure 2: Natural-movie reconstruction from measured V1 calcium activity in a single mouse. a , Representative held-out target clips (GT) and reconstructions generated by Ca2REAL and baseline methods from measured responses of 7,863 V1 neurons. Selected frames from one representative clip in each of the four target clusters are shown. b , Quantitative reconstruction performance on 178 held-out measured clips.
Figure 3: Neural-to-video decoding from a stimulus-aligned virtual population of 116,705 neurons. a , Construction of the virtual population. Encoding models fitted to 11 independently recorded neural datasets predicted responses to common movie stimuli, and the predictions were concatenated across neurons. The lower panels show the contribution of each dataset and the cortical-area composition of the virtual population. b , Representative held-out target clips (GT) and reconstructions generated by Ca2REAL and baseline methods from the 116,705-dimensional encoding-model-predicted responses. Selected frames from one representative clip in each target cluster are shown. c , Quantitative comparison over 946 held-out segments.
Condition
ID Acc. (%)
Δ (pp)
Synthetic
92.83
0.00
Matched measured
91.98
−0.85
Shuffled measured
91.10
−1.73
Zeroed responses
91.46
−1.37
Table 1: Measured-response substitution tests functional compatibility with the virtual-population decoder. All-pairs identification accuracy after replacing available synthetic responses with matched measured activity or control responses on the full test set ( n=946 clips). Δ denotes the change relative to the fully synthetic condition.
Figure 4: Complementary roles of the layout and semantic conditioning pathways. a , Representative reconstructions from measured and virtual-population activity using the full Ca2REAL model or with the layout or semantic pathway blocked at inference time. b , Mean oriented paired change in reconstruction performance relative to the full model after blocking either pathway. Signs are oriented so that negative values indicate worse performance for both higher-is-better and lower-is-better metrics. Results are shown separately for the measured V1 population ( n=178 clips) and the virtual population ( n=946 clips).
Figure 5: Distributed and stimulus-specific neural dependence of Ca2REAL. a , Effect of removing each source population relative to matched-size random neuron deletions (blue point, source-population removal result; gray line, matched-random 5–95% range; gray tick, median). b , Reconstruction performance as a function of the fraction of retained neurons. c , Robustness to progressive corruption of neuron identity. d , Robustness to replacing neural responses with responses evoked by a different stimulus. e , Robustness to random neuron dropout. f , Structured neuron-identity and cross-sample perturbations compared with magnitude-matched Gaussian perturbations. g , Effect of cross-sample substitution as a function of target–donor similarity defined in frozen SigLIP2 space; the outcome is SigLIP2-based all-pairs identification, and significance was assessed by a cluster-aware permutation test of the global Spearman trend. h , Denoising-objective pairing sensitivity at fixed diffusion noise level ( σ=0.7 ) in the full model and the model trained without Lpair , quantified by Δpair=Lshuffled−Lmatched ; asterisks indicate paired Full versus No- Lpair comparisons. For b–h , vertical bars indicate bootstrap 95% confidence intervals.
Figure 6: Ca2REAL preserves temporal semantic dynamics. a , Semantic trajectory spread retention (points, medians; thin bars, interquartile ranges; thick bars, bootstrap 95% CIs; dashed line, ideal ratio =1 ). b , Semantic path-length retention, with conventions as in a . c , Within-clip temporal identification accuracy (dashed line, chance level =0.5 ). d , Paired temporal-identification advantage of Ca2REAL over each baseline (points, mean paired differences; bars, bootstrap 95% CIs; dashed line, zero). Blue/red denote measured/virtual populations. Significance tests were Holm-corrected; asterisks denote significance and ns denotes non-significance.
Metric
Formula
Meaning and notation
MSE ↓
P1∥R−T∥22
Mean pixel-wise error between reconstruction R and target T ; P is the total number of evaluated pixels.
PSNR ↑
10log10(MSEL2)
Logarithmic pixel fidelity; L is the maximum pixel value.
Local structural similarity; μR,μT are local means, σR2,σT2 are local variances, σRT is local covariance, and C1,C2 are stabilization constants.
MS-SSIM ↑
lMαMm=1∏Mcmβmsmγm
Multiscale structural similarity; m indexes scale, M is the coarsest scale, and lM , cm , and sm denote luminance, contrast, and structure terms weighted by αM , βm , and γm .
Deep-feature perceptual distance; l indexes feature layers, ϕ^l denotes channel-normalized features, Hl,Wl are their spatial dimensions, (h,w) is spatial position, wl contains learned channel weights, and ⊙ denotes element-wise multiplication.
DISTS ↓
1−l,c∑(αlcSlc+βlcTlc)
Deep-feature structural-textural distance; l and c index feature layer and channel, Slc and Tlc are structural and texture similarity terms, and αlc,βlc are learned weights.
Decoding visual experience from fMRI is important for understanding human perception and developing brain-computer interfaces. Recent generative decoding methods can reconstruct plausible images and videos, but visual plausibility alone does not ensure that the output is faithfully grounded in brain activity. In this work, we propose Neurons++, a visual-stream fMRI reconstruction framework for both static image and dynamic video reconstruction. Compared with our conference version Neurons, Neurons++ extends the paradigm from video-only reconstruction to visual-stream reconstruction for both images and videos, and replaces parallel task conditioning with serial cortex-inspired decoding. The model first uses image-fMRI pretraining to obtain a stable spatial-semantic initialization, and then performs subject-specific fine-tuning for image and video reconstruction separately. The serial design organizes fMRI-derived representations into explicit spatial, semantic, language-level, and temporal cues. Extensive experiments show that Neurons++ achieves state-of-the-art video reconstruction and competitive image reconstruction across standard benchmarks. Beyond reconstruction fidelity, neural-to-visual analyses indicate that Neurons++ exhibits qualitative directional controllability under neural perturbation, that its fMRI encoder read-out is sensitive to stimulus-relevant cortical features, and that it supports ROI-guided semantic modulation. Code and model weights are available at: https://github.com/xmed-lab/NeuronsPlusPlus.
Haonan Wang, Yuchen Wang, Jingyu Lu +4
The Hong Kong University of Science and Technology · Shenzhen Loop Area Institute, Shenzhen, China
Decoding visual stimuli from the spiking activity of retinal ganglion cells (RGCs) informs both our understanding of the retinal code and the design of visual prostheses. Natural vision is dynamic, yet state-of-the-art decoders mostly reconstruct single frames with artificial neural networks that process dense, continuous-valued signals rather than the retina's sparse, asynchronous spikes. We asked whether dynamic scenes can be reconstructed and tracked entirely within the spike domain, and at what computational cost. We built an end-to-end spiking neural network (SNN) whose neurons integrate the recorded RGC spikes over time. Populations of up to 1836 RGCs were recorded ex vivo from mouse retina on high-density microelectrode arrays while flashed images, moving bars, drifting gratings and randomly moving handwritten digits were presented. For moving digits, one shared spiking encoder fed two readouts, one reconstructing the image and one decoding the digit's position. Compute energy was estimated from counted synaptic operations. The SNN decoded what and where at once. It reconstructed the stimuli with a structural similarity (SSIM) of up to 0.93, refining the reconstruction as spikes accumulated, and tracked a randomly moving target with a mean error of 1.33 pixels on a 32×32 grid. Its fidelity nearly matched that of the best of 17 larger spiking and artificial decoders (5.9 million to 1.7 billion parameters, versus 5.4 million), and at a mean firing rate of 14.2% it needed an estimated ∼70-fold less compute energy than a dense artificial network of the same architecture. Dynamic visual scenes can thus be decoded from retinal population activity in the spike domain, with a compact network and at a compute cost compatible with implantable and wearable neural interfaces, supporting spike-native decoders for visual prostheses.
Ning Zhang, Arto Nurmikko
School of Engineering, Brown University, Providence, RI 02912 USA
Understanding the neural mechanisms underlying visual computation has long been a central challenge in neuroscience. Recent alignment based approaches have improved the accuracy of decoding visual stimuli from brain activity, yet they provide limited insight into the neural computations that give rise to these improvements. To address this gap, we propose Dual-Tower Image-Neural Alignment (DINA), an interpretable contrastive framework for analyzing population level visual computations in primary visual cortex (V1). DINA jointly trains a biologically motivated dual-tower architecture that aligns visual stimuli and corresponding V1 population responses in a shared latent space at the level of intermediate feature maps, enabling both accurate decoding and direct access to interpretable feature maps. Evaluated on large-scale two-photon calcium imaging data from mouse V1, DINA achieves accurate neural-based decoding while revealing that decoding performance is primarily supported by coarse, low-level visual structure, rather than semantic category information or fine-grained details. Further analysis reveals that alignable feature maps emerge from multiple spatially distributed image regions, capturing both shape and texture cues, and are predominantly reconstructed by sparse subsets of strongly responsive neurons and their functional interactions. Together, these results confirm that, beyond enabling accurate decoding, DINA provides a principled framework for probing the computational mechanisms underlying visual processing in V1.
Xin Wang, Zhuangzhi Gao, Hongyi Qin +3
School of Psychological and Cognitive Sciences, Peking University, Beijing, China · Department of Primary Care and Mental Health, University of Liverpool, Liverpool, United Kingdom · Liverpool Centre for Cardiovascular Science, University of Liverpool, Liverpool, United Kingdom +2