Toward On-Chip Training of Spiking Neural Networks for Dense Event-Based Vision
Authors: Maxime Vaillant, Axel Carlier, Lai Xing Ng, Christophe Hurter, Benoit R. Cottereau
Organizations: CNRS, IPAL IRL 2955, Singapore · Université de Toulouse, IRIT, France · Institute for Infocomm Research, A*STAR, Singapore · Fédération ENAC ISAE-SUPAERO ONERA, Université de Toulouse, France · CerCo, CNRS UMR 5549, Université de Toulouse, France
Event cameras provide low-latency, asynchronous visual sensing for resource-constrained robotics. Spiking neural networks (SNNs) process event streams naturally, but training deep SNNs with backpropagation through time (BPTT) requires substantial memory and remains difficult on neuromorphic hardware. Local learning avoids this by restricting error propagation to local blocks, but existing methods mainly target classification rather than dense prediction. We introduce DELL (Dense Event-driven Local Learning), a block wise scheme for dense event-based vision that replaces global gradient propagation with local dense supervision. Learnable, spatially structured local heads supervise each block at its appropriate resolution while preserving temporal dynamics within blocks. We evaluate DELL on optical-flow regression and semantic segmentation with a fully spiking U-shaped architecture. On DSEC optical flow, DELL reduces peak training memory by 39.6% relative to end-to-end BPTT while improving accuracy, reaching 1.670 px endpoint error on the official test benchmark versus 1.941 px for the same backbone trained end-to-end. Block detachment behaves more like a regularizer than a constraint. DECOLLE, the existing local-learning baseline, relies on fixed random local read-outs poorly suited to dense regression, resulting in a 3.9x higher endpoint error; learnable local heads recover this loss and outperform end-to-end training across all optical-flow metrics. On segmentation, they recover most of the performance gap, although DELL remains a few mIoU points behind end-to-end training. With 2.3M parameters, 24x fewer than the strongest SNN baseline, the backbone remains competitive with the SNN state of the art on DSEC. These results extend local learning to dense event-based prediction while substantially reducing training memory.
Figures & tables
Fig. 1: Block-wise local learning: each block has its own local head and local loss Li , with gradients detached at every block boundary (symbols per the inset legend). In DELL , encoder and bottleneck blocks reach their local head through a training-only decoder module that restores the resolution of the block’s input. Only the decoder and read-out heads’ predictions are combined into the final output map at inference; encoder and bottleneck local heads are training-only. Shown with a reduced block count for legibility; the full network uses 11 blocks (Section III-B ).
Model
Train.
EPE (px) ↓
AE ( ∘ ) ↓
Out. (%) ↓
Par. (M) ↓
Cuadrado et al. [ 4 ]
BPTT
1.707
6.338
10.31%
1.2
SDformerFlow [ 20 ]
BPTT
1.602
4.871
10.05%
∼ 54.9
Ours ( E2E )
BPTT
1.941
6.324
14.63%
2.3
Ours ( DELL )
B-W
1.670
5.347
10.98%
2.3
TABLE I: Optical flow on the official DSEC test benchmark . Outliers (Out.) are pixels with EPE > 3 px. Train. is end-to-end BPTT or our block-wise scheme (B-W, Section III-B ). Every row is a submission to the DSEC evaluation server. Bold: best; underline: second best.
Fig. 2: Qualitative optical flow on DSEC. Each row is one validation sample: (a) the event input, (b) the ground truth, (c, d) DELL (block-wise, ours), (e, f) s-DECOLLE (frozen random local heads), and (g, h) E2E (end-to-end BPTT). Flow is color-coded with the usual hue–magnitude wheel, hue giving direction and brightness magnitude. The DSEC ground truth is sparse, so pixels without a valid target stay black; the masked columns restrict each prediction to those same pixels, which is where the metrics of Table II are computed.
EPE
AE
Out.
Peak memory
Par. (M)
Config.
(px) ↓
( ∘ ) ↓
(%) ↓
(MB) ↓
saved ↑
train → inf.
E2E
1.250
5.86
6.44
13501
ref.
2.3 → 2.3
s-DECOLLE
4.923
14.30
48.38
6316
53.2%
2.4 → 2.3
DELL
1.197
5.54
5.42
8157
39.6%
7.9 → 2.3
TABLE II: Accuracy and peak training memory on validation split DSEC optical flow , at identical backbone and detachment schedule. s-DECOLLE freezes every local head at its random initialization, DELL learns them. Peak memory is measured at equal batch size; saved is the reduction relative to the E2E reference (green). The test-benchmark figures are in Table I .
Depth
Par. (M)
E2E ↓
DELL ↓
Gap
Mem. saved ↑
3
0.8
1.377
1.366
0.011
39.3%
4 (main paper)
2.3
1.250
1.197
0.053
39.6%
5
8.7
1.424
1.165
0.259
37.8%
TABLE III: Backbone depth ablation: EPE (px) on the validation split , E2E against DELL , with the inference parameter count of each backbone and the peak-memory reduction of block-wise training over E2E . Gap is E2E minus DELL ; it is positive at every depth, meaning DELL is the more accurate (green).
Model
Params (M) ↓
mIoU ↑
SpikingEDN (events only) [ 21 ]
8.5
52.71%
Ours ( E2E )
2.3
48.4%
Ours ( s-DECOLLE )
2.3
40.7%
Ours ( DELL )
2.3
44.7%
TABLE IV: Semantic segmentation on DSEC-Semantic (11 classes [ 19 ] ), against the SNN state of the art. SpikingEDN’s events-only LIF configuration runs at T=1 against our T=16 . Bold: best; underline: second best.
Model
Acc ↑
mAcc ↑
mIoU ↑
fIoU ↑
Ours ( E2E )
81.0
47.9
39.0
69.2
Ours ( s-DECOLLE )
77.4
41.5
32.9
64.1
Ours ( DELL )
81.2
47.0
38.3
69.4
TABLE V: Semantic segmentation on M3ED (11 classes [ 19 ] ), the first SNN results on this benchmark.
Fig. 3: Qualitative semantic segmentation on M3ED, in the style of Figure 2 : (a) the event input, (b) DELL (block-wise, ours), (c) E2E (end-to-end BPTT), and (d) the ground truth, in the 11-class scheme [ 19 ] .
Energy (J)
Model
Ops / inf. ↓
Loihi 1
Eyeriss v1
E-RAFT [ 9 ] (ANN)
247.5 GMAC
n/a
4.950
Cuadrado et al. [ 4 ] (SNN)
41.0 GAC
0.967
n/a
Ours (SNN)
24.2 GAC
0.571
n/a
TABLE VI: Theoretical energy per inference on DSEC optical flow . MACs for the non-spiking baseline, spike-gated ACs for the SNNs at a uniform 10% firing rate, priced on the accelerator matching each architecture [ 5 , 3 ] ; n/a marks the pairings that do not apply. Ours at T=16 .
Spiking neural networks process information through discrete, time-dependent neural events, aiming for efficient computation and biologically inspired learning.
Neuromorphic sensing and computing promise low latency and high energy efficiency for robotics, yet most event-based visual place recognition (VPR) pipelines reintroduce dense, frame-like computation downstream of an inherently sparse sensor, forfeiting the sensor's latency advantage. In this paper, we address the problem of resolving an event-based VPR query in tens of microseconds on a commodity CPU. We train a two-layer spiking neural network (SNN) with additive spike-timing-dependent plasticity (STDP), which drives the learned weight matrix towards a strongly bimodal distribution, amenable to aggressive magnitude pruning. At inference, our pipeline exploits this sparsity by reading only the pruned weights corresponding to the query's active pixels. As surviving weights share a near-uniform magnitude, each neuron's score reduces to an integer count of matching synapses instead of a weighted sum, lowering per-query latency. This computation is small enough to remain resident in a CPU's fast on-chip cache. On three event-based VPR benchmarks, our sparse SNN exceeds the recall of LENS, an existing event-based SNN, while reducing per-query latency by 5.5-6.2x to tens of microseconds. It matches the recall of a dense variant of our network and of the sum-of-absolute-differences (SAD) baseline, while reading 10x fewer weights than the dense variant. We further characterise how recall and latency scale with the number of active pixels and reference places. These results suggest that the sparsity of both event-based input and learned weights can already be exploited for low-latency inference on conventional processors.
We propose a new spiking neural network (SNN) design to process static images and event streams using time-to-first-spike (TTFS) latencies. Our key research question is which architectural choices best accommodate local and online learning in multi-layer convolutional SNNs. This question is addressed via an original framework combining residual-like connections with multi-depth feature aggregation and consensus. The full SNN pipeline features an early-vision front end, to convert raw visual data into sparse spike latencies, a four-layer convolutional backbone trained layerwise with unsupervised spike-timing-dependent plasticity (STDP), a deterministic Multi-Depth Temporal Fusion (MDTF) and a final classifier trained with reward-modulated spike-timing-dependent plasticity (R-STDP). Rather than replacing early features in deeper layers, the proposed MDTF preserves early temporal evidence, adding sparse residual events from intermediate layers, and incorporating deeper features only when they agree in time with earlier representations. The resulting architecture is experimentally validated across MNIST, Fashion-MNIST, CIFAR-10, and N-MNIST, delivering strong classification performance under a fully local learning regime. Selective multi-depth fusion significantly outperforms traditional STDP/R-STDP baselines on higher-variability visual tasks (achieving +18.2 pp on Fashion-MNIST and +29.2 pp on CIFAR-10). Furthermore, activity-budget analyses show that the network retains high accuracy even when removing a large fraction of late or weak spike events, confirming its high data efficiency and reduced event-processing requirements. The codebase is publicly available at github.com/aidinattar/multi-depth-temporal-fusion-snn.
Aidin Attar, Eleonora Cicciarella, Michele Rossi
Department of Information Engineering, University of Padua Padua, Italy
Event cameras follow a retina-inspired sensing principle, reporting local intensity changes asynchronously with hightemporal resolution and a wide dynamic range. Spiking Neural Networks (SNNs) complement these sparse event streams through brain-inspired dynamics, using sparse spikes and leaky membrane potentials to integrate information over time. However, many SNN object detectors process isolated event intervals with a single label and reset the network state after each prediction, thereby underusing temporal information in continuous event streams. We introduce Sequence-SOD, a sequence-aware SNN object detector that processes extended event sequences containing labels at multiple time points. Events are accumulated into short intervals, discretized into temporal steps, and fed sequentially to an SSD-style Spiking DenseNet while preserving membrane potentials across intervals within a sequence, so that detection is driven by an evolving neural state instead of independently reset input windows. On the Gen1 Automotive Detection Dataset, sequence-aware training improves mAP from 23.38 for single-interval training to 25.30 without augmentation and to 26.88 withevent-data augmentation. The model achieves a theoretical prediction frequency of 40 Hz. Training and evaluating SNN object detectors on extended event sequences improves their ability to exploit temporal cues while preserving the energy-efficiency benefits of sparse spiking computation. The results highlight sequence-aware training as a complementary direction to architectural improvements for event-based SNN detection.
Katharina Bendig, René Schuster, Didier Stricker
RPTU – Technical University Kaiserslautern-Landau, Gottlieb-Daimler-Straße 47, Kaiserslautern 67663, Germany · DFKI – German Research Center for Artificial Intelligence, Trippstadter Str. 122, Kaiserslautern 67663, Germany