Neuromorphic Decoding and Dynamical Image Reconstruction from Populations of Spiking Retinal Ganglion Cells

Ning Zhang; Arto Nurmikko

cs.ETOct 12, 2026

Neuromorphic Decoding and Dynamical Image Reconstruction from Populations of Spiking Retinal Ganglion Cells

Authors: Ning Zhang, Arto Nurmikko

Organizations: School of Engineering, Brown University, Providence, RI 02912 USA

Abstract

Decoding visual stimuli from the spiking activity of retinal ganglion cells (RGCs) informs both our understanding of the retinal code and the design of visual prostheses. Natural vision is dynamic, yet state-of-the-art decoders mostly reconstruct single frames with artificial neural networks that process dense, continuous-valued signals rather than the retina's sparse, asynchronous spikes. We asked whether dynamic scenes can be reconstructed and tracked entirely within the spike domain, and at what computational cost. We built an end-to-end spiking neural network (SNN) whose neurons integrate the recorded RGC spikes over time. Populations of up to 1836 RGCs were recorded ex vivo from mouse retina on high-density microelectrode arrays while flashed images, moving bars, drifting gratings and randomly moving handwritten digits were presented. For moving digits, one shared spiking encoder fed two readouts, one reconstructing the image and one decoding the digit's position. Compute energy was estimated from counted synaptic operations. The SNN decoded what and where at once. It reconstructed the stimuli with a structural similarity (SSIM) of up to 0.93, refining the reconstruction as spikes accumulated, and tracked a randomly moving target with a mean error of 1.33 pixels on a $32\times32$ grid. Its fidelity nearly matched that of the best of 17 larger spiking and artificial decoders (5.9 million to 1.7 billion parameters, versus 5.4 million), and at a mean firing rate of 14.2% it needed an estimated $\sim$ 70-fold less compute energy than a dense artificial network of the same architecture. Dynamic visual scenes can thus be decoded from retinal population activity in the spike domain, with a compact network and at a compute cost compatible with implantable and wearable neural interfaces, supporting spike-native decoders for visual prostheses.

Figures & tables

Figure 1: End-to-end neuromorphic decoding of dynamic vision from retinal ganglion cell populations. (a) Visual stimuli, flashed natural or synthetic images (400 ms) or moving targets (2-s clips of randomly moving digits, bars and gratings), are projected onto (b) an ex vivo mouse retina on a CMOS high-density microelectrode array, yielding spike-sorted RGC units. (c) Population spikes (raster of one held-out clip; colors mark ON, OFF and ON–OFF cells) are placed at each cell’s receptive-field center with a Gaussian kernel, giving a $[T\times 3\times 32\times 32]$ input tensor with one channel per polarity. (d) A single spike-native decoder, a shared spiking encoder with a reconstruction readout and a tracking readout, built from leaky or non-leaky integrate-and-fire (LIF/IF) neurons and trained with surrogate gradients; its sparse, event-driven computation uses a small fraction of the compute energy of the equivalent ANN decoder (Sec. 4.6 ) and can map onto neuromorphic hardware. (e) Its outputs: the reconstructed stimulus (a flashed synthetic pattern and a moving digit; masked-SSIM 0.81 natural and 0.88 synthetic, SSIM 0.80 for moving digits) and the target position at every time step (decoded, teal; true path, dashed; 1.33 px mean error over 300 held-out clips). The moving-digit example is the same held-out clip as in Figs. 3 and 7 .

Figure 2: Ex vivo retina MEA recording and data preparation. (a) Schematic of ex vivo retina MEA recording testbed. Up to 1836 cells were recorded from a piece of retina (2 mm $\times$ 2 mm) by the array, with visual stimuli delivered by a Digital Light Processor (DLP) at a wavelength of 405 nm. (b) After spike sorting, the mean pairwise correlation coefficient across trials was calculated for each unit, and units with a mean correlation coefficient greater than 0.2 (dashed line) were retained as high-quality units; one curve per recording batch, units ranked by correlation. (c) Receptive fields (RF) of three representative cells of different types: responses over the $20\times 20$ mapping grid in the ON and OFF windows. (d) Receptive-field centers of the 1206 cells of the moving-digit recording on the $32\times 32$ input grid, colored by response polarity (ON, OFF, ON–OFF; the color key of (c) and Figs. 1 c and 7 a). Left: the continuous (sub-pixel) centers, at which each cell’s firing rate is rendered with a unit-mass Gaussian kernel (Gaussian splatting; Methods) to form the decoder input. Right: the same centers snapped to the nearest grid pixel (hard snap), shown for comparison; cells that share a pixel are merged.

Figure 3: Shared-encoder dual-head spiking network architecture. (a) Input spike maps (1206 RGCs, ON/OFF/ON–OFF channels, $T=200$ bins) pass through an optional MA-SNN attention front-end [ 33 ] (purple; moving-digit models only) into a shared SEW-ResNet [ 34 ] spiking U-Net encoder (teal; DS, $2\times 2$ max-pool). Its multiscale features feed (i) the reconstruction submodule (orange; US, nearest-neighbor $\times 2$ with concatenated skips), whose output neuron never fires and whose membrane $V[t]$ is fused over time and passed through a sigmoid, and (ii) the tracking submodule (blue), which adds a full-resolution regression offset to a spike-count centroid. Apart from the input stage (the optional attention module and the first convolution, which read the RF-mapped input), every stage is spiking (LIF/IF neurons; red symbols mark spike tensors); gray blocks are analog readout operations and the dotted line marks the spiking/analog boundary. The same weights are applied at every time step (nested frames). Parameter counts are printed per block: 5.9 M for the moving-digit model and 5.4 M for the flashed-image model (1836 cells, $T=80$ , no attention, 512-pixel output). Thumbnails: running readout $\sigma(V[t])$ at $t=50$ , 100 and 199, and the fused reconstruction of one held-out clip (digit 3). (b) Block definitions and legend.

Figure 4: Image reconstruction of flashed stimuli decoded from 1836 RGCs by the end-to-end spiking U-Net. Ground truth (left) versus spike-native reconstruction (right), $7\times 8$ held-out examples per panel, for (a) synthetic patterns (masked-SSIM 0.88) and (b) natural images (masked-SSIM 0.81). Similarity index is quoted over all 200 held-out images.

Model	Domain	Params (M)	Masked-SSIM
Spiking U-Net (ours)	spiking	5.4	0.81
QKFormer [ 35 ]	spiking	33.0	0.80
STAtten [ 36 ]	spiking	$\sim$ 30	0.81
Meta-SpikeFormer [ 22 ]	spiking	59.8	0.82
SwinIR †	ANN	11.9	0.82
Real-ESRGAN †	ANN	16.7	0.82

Table 1: Reconstruction fidelity versus model size on flashed natural-image decoding from 1836 RGC spiking neurons. All models decode the same held-out receptive-field-mapped input. Parameters are measured by torchinfo for the spiking decoders;

\dagger

denotes published architecture size for the artificial-network refiners. Masked-SSIM values are given on the shared held-out test set.

Figure 5: Reconstruction fidelity versus model size, and compute energy. (a) Reconstruction fidelity (masked-SSIM, window 11, 512 px, $n=200$ held-out images, identical protocol for every point) versus model size for 18 decoders on the same 1836-RGC flashed natural-image task: our spiking U-Net (teal star), other spiking models (light teal) and artificial networks (gray); circles decode the RGC input directly, squares refine the Meta-SpikeFormer decode, diamonds are generative models. Fidelity saturates in a narrow band (shaded, interquartile range of the 18 models) across more than two orders of magnitude in parameter count, suggesting a neural-data-limited rather than capacity-limited regime; the spiking U-Net of this paper (star, $\sim$ 5.4M parameters) reaches this band with $5.5$ – $11\times$ fewer parameters than the spiking-transformer decoders. Models 1–5 lie within 0.003 SSIM of each other and are numbered (markers 2 and 3, both 17M, are offset horizontally for visibility); $\dagger$ published architecture size; spiking-model parameters measured by torchinfo. (b) Compute energy per sample under a 45 nm convention (multiply–accumulate 4.6 pJ, accumulate 0.9 pJ): the ANN-equivalent (dense multiply–accumulates), the same spike-driven operation count at multiply–accumulate cost, and the spike-native network (synaptic operations at accumulate cost); the $\sim$ 70 $\times$ reduction factorizes into $\sim$ 13.6 $\times$ fewer operations (spike sparsity at 14.2% mean firing) and $\sim$ 5.1 $\times$ cheaper operations. (c) Spike-driven energy by module for the reported model ( $\lambda=0$ ) and a model trained with a firing-rate penalty ( $\lambda=0.1$ ): the penalty removes 21% of the synaptic operations ( $\sim$ 70 $\times$ $\to$ $\sim$ 88 $\times$ ) at a reconstruction cost (validation SSIM 0.80 $\to$ 0.73; separate training runs). (d) Firing rate of every spiking layer for the two models: the penalty lowers firing in the decoder, where fan-out is largest, and raises it in the tracking head, so the mean rate is unchanged while synaptic operations fall. The output-head neuron never fires and is not shown. Energy estimates are compute-only (memory movement excluded).

Figure 6: Reconstruction of moving bars and drifting gratings recorded and decoded from spiking RGC populations. (a) Moving bar, 45 ∘ (diagonal) direction, one held-out trial: ground truth (top) and spike-native reconstruction (bottom) at eight times while the bar crosses the recorded patch. (b) SSIM (window 3) versus time within a sweep (bars, top) or condition (gratings, bottom), mean $\pm$ s.d. over directions or conditions and trials; shading marks the bar inside the patch and the 2 s drift (light blue) and the blank after the drift (gray). (c) Bars: SSIM per motion direction over windows with the bar inside the patch (mean $\pm$ s.d., four held-out trials). (d) Gratings: SSIM during the drift (teal) and reconstruction-to-stimulus contrast ratio (open squares) for all 40 conditions (8 directions $\times$ 5 temporal frequencies at a spatial frequency of 0.4 cycles per degree); lines join means over directions. Reconstruction is faithful up to 4 Hz, degrades at 8 Hz and fails at 15 Hz. (e) Ground truth (top) and reconstruction (bottom) mid-drift for one direction at 1, 4, 8 and 15 Hz, with the SSIM of each condition. The reconstructions are the model output at the end of each causal sliding window (10 bins, 100 ms, for the bars; 3 bins, 30 ms, for the gratings; Methods).

Figure 7: Dynamical reconstruction of the image of a single, randomly moving digit projected on mouse retina, acquired from the shared spiking encoder (held-out clip, here for digit “3”), and the sharpening of the reconstruction over the whole held-out set. Panel descriptions continue on the following page.

Figure 9: Consistency of dynamic reconstruction across the test set. Ground truth (left) versus spike-native reconstruction (right) for 64 randomly selected held-out clips of randomly moving digits, each following a distinct random $x$ – $y$ trajectory, displayed as an $8\times 8$ panel. Reconstruction is the temporally fused readout at the end of each clip. The SSIM (window-3) of each reconstruction is printed on its tile; over all 300 held-out clips the SSIM is 0.80 $\pm$ 0.07 (mean $\pm$ s.d.).

Figure 10: The SNN outperforms a calibrated center-of-mass readout in tracking a randomly moving object. (a) Mean $\pm$ s.d. position error (32-grid px) over the 300 held-out clips for each readout: chance (predict-center, 9.10 px), the calibrated center-of-mass (COM) baseline (2.91 px), the spike-count centroid readout (2.14 px) and the full-resolution spiking regression readout ( 1.33 px ), which beats the calibrated COM baseline roughly two-fold. (b) True trajectory (dashed black; circle, start; square, end) overlaid with the decoded trajectory of the spiking regression readout (teal) and of the calibrated COM (dotted gray) for three held-out clips; the error of the spiking readout is printed above each. (c) Per-clip error versus the digit’s pixel content (ink pixels) for the calibrated COM (gray) and the spiking regression readout (teal); lines join the quintile means. Error falls as pixel content grows for both readouts, and the spiking readout stays below the COM baseline at every size. (d) Position error versus time within the clip, mean $\pm$ s.d. over clips, for the spiking regression readout and the calibrated COM; the shaded window is the target’s motion shifted by the 250 ms response latency, and curves are shown where a target position is defined. (e) Cumulative component ablation of the tracking readout (Table 2 ; separate training runs).

Configuration (cumulative)	Reconstruction SSIM	Tracking (px)
F_FC tracking readout	—	4.1
Spiking centroid head	$\sim$ 0.69	2.6
+ temporal integration (de-jitter)	—	2.2
+ channel/spatial/temporal attention	+0.04	2.2
+ temporal-fusion readout	$\sim$ 0.80	2.2
+ full-resolution regression readout (full model)	—	1.33

Table 2: Component ablation on dynamic-MNIST decoding. Rows are separate training runs.

Explore similar work

Jul 21, 2026

Spiking Neural Networks for fMRI-Based Visual Semantic Decoding

Functional magnetic resonance imaging (fMRI)-based visual decoding aims to recover visual information from measured brain activity, commonly by mapping fMRI responses into latent visual features for downstream decoding tasks. Most existing methods learn mappings from fMRI responses to visual features extracted by artificial neural networks (ANNs), yet it remains unclear whether ANN-derived features provide suitable targets for brain decoding. In this study, we investigate spiking neural network (SNN)-derived visual features as alternative targets for fMRI-based visual decoding. We compare an ANN baseline with four SNN variants from the same architectural family, which differ in their spiking dynamics. To isolate the effect of the target features, all models use the same L2-regularized linear fMRI-to-feature decoder, while only the feature vectors used as regression targets are varied. Compared with the ANN baseline, SNN-derived features exhibit stronger alignment with fMRI responses and improve visual semantic decoding performance. For instance, on the GoD dataset, SNN-derived features reduce feature-prediction error from 0.7707 to 0.0282 and improve top-1 semantic decoding accuracy from 0.1800 to 0.4400. Ablation results further indicate that both spiking neural dynamics and temporal simulation steps contribute to the observed advantage. These findings support SNN-derived features as effective brain-decodable visual representations and highlight target feature design as an important component of fMRI-based visual decoding.

Jiahong Zhang, Jinning Zhao, Sijun Shen +3

Institute of Automation, Chinese Academy of Sciences, Beijing 100045, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China · State Key Laboratory of Media Convergence and Communication, Communication University of China, Beijing 100024, China

Visual Representation Learning Neural Decoding Spiking Neural Networks

Ask PDF

Jun 6, 2026

Programmable Silicon Retina on Pixel Processor Array

Standard dynamic vision sensors approximate retinal processing by detecting temporal contrast changes, offering high speed and high dynamic range. In this work, we explore whether incorporating additional biologically inspired processing stages - specifically spatial filtering and gain control - can offer advantages for certain downstream tasks such as saliency prediction. We present the first implementation of a multi-stage Silicon Retina model on the SCAMP-5 Pixel Processor Array, along with a GPU-based simulation framework. We evaluate the performance of our model on Video Intensity Reconstruction and Video Saliency Prediction. While the bio-inspired model is less effective at reconstructing absolute intensity frames, it achieves a 13% reduction in saliency prediction loss in comparison to standard DVS event representation, while reducing the event rate by approximately 47%. These experiments are obtained using a lightweight

\approx 100

k-parameter FireNet-style network, adapted from event-based reconstruction to saliency prediction. These results suggest that the silicon retina's "information distillation" mechanism can achieve a more efficient representation for downstream neural networks, particularly in bandwidth-constrained edge applications.

Maciej Lewandowski, Prince Philip, Alexandre Marcireau +3

Department of Electrical and Electronic Engineering University of Manchester, UK · Department of Electronic Systems Engineering, Indian Institute of Science, Bangalore, India · Department of Computer Science University of Manchester, UK

Event-Based Vision Neuromorphic Computing

Ask PDF

Jun 12, 2026

Test-Time Adaptation of Spiking Neural Networks for Intracortical Neural Decoding using Membrane Potential Alignment

Intracortical brain-computer interfaces suffer from day-to-day neural signal shifts that degrade pretrained decoders. Existing unsupervised adaptation methods rely on deep recurrent or adversarial architectures that are too computationally expensive for implantable hardware. We propose Membrane Potential Alignment (MPA), a test-time adaptation method for spiking neural networks that realigns a pretrained decoder to shifted recordings by only matching membrane potential distributions via KL divergence. By restricting updates to low-rank (LoRA) weights, MPA adapts fewer than 9% of parameters. On a non-human primate reaching task spanning over one month, MPA achieves performance competitive with the state-of-the-art NoMAD method, while using a simpler architecture and finer temporal resolution (4 ms vs. 20 ms). These results show that efficient SNN-based test-time adaptation is a practical path toward long-term, recalibration-free brain-computer interfaces.

Guangzhi Tang

Department of Advanced Computing Sciences, Maastricht University Maastricht, The Netherlands

Test-Time Adaptation Brain-Computer Interfaces Neural Decoding Spiking Neural Networks

Ask PDF