cs.ETOct 12, 2026
SaveNeuromorphic Decoding and Dynamical Image Reconstruction from Populations of Spiking Retinal Ganglion Cells
Organizations: School of Engineering, Brown University, Providence, RI 02912 USA
Abstract
Decoding visual stimuli from the spiking activity of retinal ganglion cells (RGCs) informs both our understanding of the retinal code and the design of visual prostheses. Natural vision is dynamic, yet state-of-the-art decoders mostly reconstruct single frames with artificial neural networks that process dense, continuous-valued signals rather than the retina's sparse, asynchronous spikes. We asked whether dynamic scenes can be reconstructed and tracked entirely within the spike domain, and at what computational cost. We built an end-to-end spiking neural network (SNN) whose neurons integrate the recorded RGC spikes over time. Populations of up to 1836 RGCs were recorded ex vivo from mouse retina on high-density microelectrode arrays while flashed images, moving bars, drifting gratings and randomly moving handwritten digits were presented. For moving digits, one shared spiking encoder fed two readouts, one reconstructing the image and one decoding the digit's position. Compute energy was estimated from counted synaptic operations. The SNN decoded what and where at once. It reconstructed the stimuli with a structural similarity (SSIM) of up to 0.93, refining the reconstruction as spikes accumulated, and tracked a randomly moving target with a mean error of 1.33 pixels on a grid. Its fidelity nearly matched that of the best of 17 larger spiking and artificial decoders (5.9 million to 1.7 billion parameters, versus 5.4 million), and at a mean firing rate of 14.2% it needed an estimated 70-fold less compute energy than a dense artificial network of the same architecture. Dynamic visual scenes can thus be decoded from retinal population activity in the spike domain, with a compact network and at a compute cost compatible with implantable and wearable neural interfaces, supporting spike-native decoders for visual prostheses.
Figures & tables
Figure 1: End-to-end neuromorphic decoding of dynamic vision from retinal ganglion cell populations. (a) Visual stimuli, flashed natural or synthetic images (400 ms) or moving targets (2-s clips of randomly moving digits, bars and gratings), are projected onto (b) an ex vivo mouse retina on a CMOS high-density microelectrode array, yielding spike-sorted RGC units. (c) Population spikes (raster of one held-out clip; colors mark ON, OFF and ON–OFF cells) are placed at each cell’s receptive-field center with a Gaussian kernel, giving a input tensor with one channel per polarity. (d) A single spike-native decoder, a shared spiking encoder with a reconstruction readout and a tracking readout, built from leaky or non-leaky integrate-and-fire (LIF/IF) neurons and trained with surrogate gradients; its sparse, event-driven computation uses a small fraction of the compute energy of the equivalent ANN decoder (Sec. 4.6 ) and can map onto neuromorphic hardware. (e) Its outputs: the reconstructed stimulus (a flashed synthetic pattern and a moving digit; masked-SSIM 0.81 natural and 0.88 synthetic, SSIM 0.80 for moving digits) and the target position at every time step (decoded, teal; true path, dashed; 1.33 px mean error over 300 held-out clips). The moving-digit example is the same held-out clip as in Figs. 3 and 7 .
Figure 2: Ex vivo retina MEA recording and data preparation. (a) Schematic of ex vivo retina MEA recording testbed. Up to 1836 cells were recorded from a piece of retina (2 mm 2 mm) by the array, with visual stimuli delivered by a Digital Light Processor (DLP) at a wavelength of 405 nm. (b) After spike sorting, the mean pairwise correlation coefficient across trials was calculated for each unit, and units with a mean correlation coefficient greater than 0.2 (dashed line) were retained as high-quality units; one curve per recording batch, units ranked by correlation. (c) Receptive fields (RF) of three representative cells of different types: responses over the mapping grid in the ON and OFF windows. (d) Receptive-field centers of the 1206 cells of the moving-digit recording on the input grid, colored by response polarity (ON, OFF, ON–OFF; the color key of (c) and Figs. 1 c and 7 a). Left: the continuous (sub-pixel) centers, at which each cell’s firing rate is rendered with a unit-mass Gaussian kernel (Gaussian splatting; Methods) to form the decoder input. Right: the same centers snapped to the nearest grid pixel (hard snap), shown for comparison; cells that share a pixel are merged.
Figure 3: Shared-encoder dual-head spiking network architecture. (a) Input spike maps (1206 RGCs, ON/OFF/ON–OFF channels, bins) pass through an optional MA-SNN attention front-end [ 33 ] (purple; moving-digit models only) into a shared SEW-ResNet [ 34 ] spiking U-Net encoder (teal; DS, max-pool). Its multiscale features feed (i) the reconstruction submodule (orange; US, nearest-neighbor with concatenated skips), whose output neuron never fires and whose membrane is fused over time and passed through a sigmoid, and (ii) the tracking submodule (blue), which adds a full-resolution regression offset to a spike-count centroid. Apart from the input stage (the optional attention module and the first convolution, which read the RF-mapped input), every stage is spiking (LIF/IF neurons; red symbols mark spike tensors); gray blocks are analog readout operations and the dotted line marks the spiking/analog boundary. The same weights are applied at every time step (nested frames). Parameter counts are printed per block: 5.9 M for the moving-digit model and 5.4 M for the flashed-image model (1836 cells, , no attention, 512-pixel output). Thumbnails: running readout at , 100 and 199, and the fused reconstruction of one held-out clip (digit 3). (b) Block definitions and legend.
Figure 4: Image reconstruction of flashed stimuli decoded from 1836 RGCs by the end-to-end spiking U-Net. Ground truth (left) versus spike-native reconstruction (right), held-out examples per panel, for (a) synthetic patterns (masked-SSIM 0.88) and (b) natural images (masked-SSIM 0.81). Similarity index is quoted over all 200 held-out images.
| Model | Domain | Params (M) | Masked-SSIM |
|---|---|---|---|
| Spiking U-Net (ours) | spiking | 5.4 | 0.81 |
| QKFormer [ 35 ] | spiking | 33.0 | 0.80 |
| STAtten [ 36 ] | spiking | 30 | 0.81 |
| Meta-SpikeFormer [ 22 ] | spiking | 59.8 | 0.82 |
| SwinIR † | ANN | 11.9 | 0.82 |
| Real-ESRGAN † | ANN | 16.7 | 0.82 |
Table 1: Reconstruction fidelity versus model size on flashed natural-image decoding from 1836 RGC spiking neurons. All models decode the same held-out receptive-field-mapped input. Parameters are measured by torchinfo for the spiking decoders; denotes published architecture size for the artificial-network refiners. Masked-SSIM values are given on the shared held-out test set.
Figure 5: Reconstruction fidelity versus model size, and compute energy. (a) Reconstruction fidelity (masked-SSIM, window 11, 512 px, held-out images, identical protocol for every point) versus model size for 18 decoders on the same 1836-RGC flashed natural-image task: our spiking U-Net (teal star), other spiking models (light teal) and artificial networks (gray); circles decode the RGC input directly, squares refine the Meta-SpikeFormer decode, diamonds are generative models. Fidelity saturates in a narrow band (shaded, interquartile range of the 18 models) across more than two orders of magnitude in parameter count, suggesting a neural-data-limited rather than capacity-limited regime; the spiking U-Net of this paper (star, 5.4M parameters) reaches this band with – fewer parameters than the spiking-transformer decoders. Models 1–5 lie within 0.003 SSIM of each other and are numbered (markers 2 and 3, both 17M, are offset horizontally for visibility); published architecture size; spiking-model parameters measured by torchinfo. (b) Compute energy per sample under a 45 nm convention (multiply–accumulate 4.6 pJ, accumulate 0.9 pJ): the ANN-equivalent (dense multiply–accumulates), the same spike-driven operation count at multiply–accumulate cost, and the spike-native network (synaptic operations at accumulate cost); the 70 reduction factorizes into 13.6 fewer operations (spike sparsity at 14.2% mean firing) and 5.1 cheaper operations. (c) Spike-driven energy by module for the reported model ( ) and a model trained with a firing-rate penalty ( ): the penalty removes 21% of the synaptic operations ( 70 88 ) at a reconstruction cost (validation SSIM 0.80 0.73; separate training runs). (d) Firing rate of every spiking layer for the two models: the penalty lowers firing in the decoder, where fan-out is largest, and raises it in the tracking head, so the mean rate is unchanged while synaptic operations fall. The output-head neuron never fires and is not shown. Energy estimates are compute-only (memory movement excluded).
Figure 6: Reconstruction of moving bars and drifting gratings recorded and decoded from spiking RGC populations. (a) Moving bar, 45 ∘ (diagonal) direction, one held-out trial: ground truth (top) and spike-native reconstruction (bottom) at eight times while the bar crosses the recorded patch. (b) SSIM (window 3) versus time within a sweep (bars, top) or condition (gratings, bottom), mean s.d. over directions or conditions and trials; shading marks the bar inside the patch and the 2 s drift (light blue) and the blank after the drift (gray). (c) Bars: SSIM per motion direction over windows with the bar inside the patch (mean s.d., four held-out trials). (d) Gratings: SSIM during the drift (teal) and reconstruction-to-stimulus contrast ratio (open squares) for all 40 conditions (8 directions 5 temporal frequencies at a spatial frequency of 0.4 cycles per degree); lines join means over directions. Reconstruction is faithful up to 4 Hz, degrades at 8 Hz and fails at 15 Hz. (e) Ground truth (top) and reconstruction (bottom) mid-drift for one direction at 1, 4, 8 and 15 Hz, with the SSIM of each condition. The reconstructions are the model output at the end of each causal sliding window (10 bins, 100 ms, for the bars; 3 bins, 30 ms, for the gratings; Methods).
Figure 7: Dynamical reconstruction of the image of a single, randomly moving digit projected on mouse retina, acquired from the shared spiking encoder (held-out clip, here for digit “3”), and the sharpening of the reconstruction over the whole held-out set. Panel descriptions continue on the following page.
Figure 9: Consistency of dynamic reconstruction across the test set. Ground truth (left) versus spike-native reconstruction (right) for 64 randomly selected held-out clips of randomly moving digits, each following a distinct random – trajectory, displayed as an panel. Reconstruction is the temporally fused readout at the end of each clip. The SSIM (window-3) of each reconstruction is printed on its tile; over all 300 held-out clips the SSIM is 0.80 0.07 (mean s.d.).
Figure 10: The SNN outperforms a calibrated center-of-mass readout in tracking a randomly moving object. (a) Mean s.d. position error (32-grid px) over the 300 held-out clips for each readout: chance (predict-center, 9.10 px), the calibrated center-of-mass (COM) baseline (2.91 px), the spike-count centroid readout (2.14 px) and the full-resolution spiking regression readout ( 1.33 px ), which beats the calibrated COM baseline roughly two-fold. (b) True trajectory (dashed black; circle, start; square, end) overlaid with the decoded trajectory of the spiking regression readout (teal) and of the calibrated COM (dotted gray) for three held-out clips; the error of the spiking readout is printed above each. (c) Per-clip error versus the digit’s pixel content (ink pixels) for the calibrated COM (gray) and the spiking regression readout (teal); lines join the quintile means. Error falls as pixel content grows for both readouts, and the spiking readout stays below the COM baseline at every size. (d) Position error versus time within the clip, mean s.d. over clips, for the spiking regression readout and the calibrated COM; the shaded window is the target’s motion shifted by the 250 ms response latency, and curves are shown where a target position is defined. (e) Cumulative component ablation of the tracking readout (Table 2 ; separate training runs).
| Configuration (cumulative) | Reconstruction SSIM | Tracking (px) |
|---|---|---|
| F_FC tracking readout | — | 4.1 |
| Spiking centroid head | 0.69 | 2.6 |
| + temporal integration (de-jitter) | — | 2.2 |
| + channel/spatial/temporal attention | +0.04 | 2.2 |
| + temporal-fusion readout | 0.80 | 2.2 |
| + full-resolution regression readout (full model) | — | 1.33 |
Table 2: Component ablation on dynamic-MNIST decoding. Rows are separate training runs.