Multi-Depth Temporal Fusion for Feedforward, Locally Trained Spiking Neural Networks

Aidin Attar; Eleonora Cicciarella; Michele Rossi

cs.NESep 29, 2026

Multi-Depth Temporal Fusion for Feedforward, Locally Trained Spiking Neural Networks

Authors: Aidin Attar, Eleonora Cicciarella, Michele Rossi

Organizations: Department of Information Engineering, University of Padua Padua, Italy

Abstract

We propose a new spiking neural network (SNN) design to process static images and event streams using time-to-first-spike (TTFS) latencies. Our key research question is which architectural choices best accommodate local and online learning in multi-layer convolutional SNNs. This question is addressed via an original framework combining residual-like connections with multi-depth feature aggregation and consensus. The full SNN pipeline features an early-vision front end, to convert raw visual data into sparse spike latencies, a four-layer convolutional backbone trained layerwise with unsupervised spike-timing-dependent plasticity (STDP), a deterministic Multi-Depth Temporal Fusion (MDTF) and a final classifier trained with reward-modulated spike-timing-dependent plasticity (R-STDP). Rather than replacing early features in deeper layers, the proposed MDTF preserves early temporal evidence, adding sparse residual events from intermediate layers, and incorporating deeper features only when they agree in time with earlier representations. The resulting architecture is experimentally validated across MNIST, Fashion-MNIST, CIFAR-10, and N-MNIST, delivering strong classification performance under a fully local learning regime. Selective multi-depth fusion significantly outperforms traditional STDP/R-STDP baselines on higher-variability visual tasks (achieving +18.2 pp on Fashion-MNIST and +29.2 pp on CIFAR-10). Furthermore, activity-budget analyses show that the network retains high accuracy even when removing a large fraction of late or weak spike events, confirming its high data efficiency and reduced event-processing requirements. The codebase is publicly available at github.com/aidinattar/multi-depth-temporal-fusion-snn.

Figures & tables

Figure 1: Schematic overview of the proposed locally trained TTFS architecture. The model uses a four-layer spiking convolutional backbone (

S_{1}

–

S_{4}

). The MDTF merges representations at different depths. It preserves the early code

P=H^{(1)}

, incorporates the sparse residual contribution

\Delta_{\mathrm{res}}

derived from the intermediate representation

I=H^{(2)}

, and adds the agreement contribution

\Delta_{\mathrm{agree}}

obtained from temporally consistent events in

I

and the deep representation

D=H^{(4)}

. These components are concatenated into the final latency representation

H=[P,\Delta_{\mathrm{res}},\Delta_{\mathrm{agree}}]

, which is classified by a reward-modulated multi-prototype R-STDP readout through global winner-take-all spike timing. The earliest output spike determines the class score, silent neurons are assigned the maximum time, and membrane potential is used only for tie breaking. Labels are used only to define the reward and punishment signals at the decision stage.

Stage	Plasticity	Role in the pipeline
Visual front end	None; deterministic preprocessing	Converts static images or event streams into label-free latency representations. Static-input statistics are estimated once from the training split, whereas event-based normalization is applied independently to each sample.
Early convolutional stages	Unsupervised STDP in $S_{1}$ - $S_{2}$ ; deterministic $C_{1}$ - $C_{2}$ export	To learn early temporal features and produce the preserved representation $P=H^{(1)}$ and the intermediate representation $I=H^{(2)}$ ; each trained spiking stage is frozen before subsequent stages are fitted.
Deep convolutional path	Unsupervised STDP in $S_{3}$ - $S_{4}$	Further processes $I$ to produce the deep representation $D=H^{(4)}$ , used together with $I$ by the temporal agreement mechanism.
Sparse Multi-Depth Temporal Fusion	None; deterministic selection and agreement	Preserves $P$ , extracts the sparse residual contribution $\Delta_{\mathrm{res}}$ from $I$ , and adds the agreement contribution $\Delta_{\mathrm{agree}}$ obtained from temporally consistent events in $I$ and $D$ .
Multi-prototype readout	Reward-modulated STDP	Learns class decisions from the frozen fused representation $H=[P,\Delta_{\mathrm{res}},\Delta_{\mathrm{agree}}]$ using target-side reinforcement, hard-negative anti-STDP, and multiple prototypes per class.

Table 1: Training schedule of the proposed locally trained pipeline.

Dataset	Input structure	Code dim.	Events/sample	Density	Accuracy (%)
MNIST	$28{\times}28$ grayscale digits	$17\,408$	$994.5$	$5.71\%$	$96.6\,{\scriptstyle\pm\,0.6}$
Fashion-MNIST	$28{\times}28$ grayscale apparel	$17\,408$	$1\,182.3$	$6.79\%$	$86.3\,{\scriptstyle\pm\,0.7}$
CIFAR-10	$32{\times}32$ RGB natural images	$24\,704$	$3\,100.6$	$12.6\%$	$62.5\,{\scriptstyle\pm\,0.4}$
N-MNIST	$34{\times}34$ polarity event streams	$24\,704$	$1\,992.7$	$8.07\%$	$95.1\,{\scriptstyle\pm\,0.6}$

Table 2: Main classification results and frozen-code statistics. Accuracy is reported in percent on the test split for validation-selected operating points; multi-seed rows report mean

\pm

standard deviation over readout seeds. Code dimension, mean active events, and active density are measured on the sparse latency representation delivered to the readout. All rows use the same four-layer locally trained residual/agreement spiking backbone and a reward-modulated temporal R-STDP readout.

Figure 2: Training dynamics of the reward-modulated readout on the main datasets. Solid black curves report test accuracy, dashed gray curves report training accuracy, and the vertical dotted line marks the best validation epoch. The panels show stable readout regimes, with test accuracy remaining close to its best value over the final training phase.

Dataset	Front-end configuration	Main configuration change	Accuracy
MNIST	simple latency	local decorrelation and polarity processing	$9.80$
MNIST	full without signed context	signed-context gate	$95.41$
MNIST	full without polarity balancing	local polarity-balance gates	$95.02$
MNIST	full static front end	-	$96.68$
Fashion-MNIST	simple latency	local decorrelation and polarity processing	$10.00$
Fashion-MNIST	full without signed context	signed-context gate	$86.23$

Table 3: Front-end component configurations. Table entries report test accuracy in percent on the full train/test protocol, using the same four-layer backbone architecture and R-STDP readout framework.

Dataset	Local STDP/R-STDP baseline	Proposed full model	$\Delta$ pp	$p_{\mathrm{Holm}}$
MNIST	$97.0\,{\scriptstyle[96.6,97.3]}$	$96.7\,{\scriptstyle[96.3,97.1]}$	$-0.3$	$0.18$
Fashion-MNIST	$68.2\,{\scriptstyle[67.5,69.1]}$	$86.3\,{\scriptstyle[85.6,87.0]}$	$+18.2$	$<10^{-16}$
CIFAR-10	$33.3\,{\scriptstyle[32.4,34.2]}$	$62.5\,{\scriptstyle[61.6,63.5]}$	$+29.2$	$<10^{-16}$
N-MNIST	$22.0^{\dagger}\,{\scriptstyle[21.2,22.8]}$	$95.0\,{\scriptstyle[94.6,95.5]}$	$+73.0$	$<10^{-16}$

Table 4: Comparison with a local STDP/R-STDP baseline from [ 26 ] . Accuracies are reported in percent with

95\%

Wilson confidence intervals over the test split. The

p

-values are Holm-corrected two-proportion tests at the aggregate accuracy level. Very small corrected values are clipped at

10^{-16}

for readability.

Dataset	100	500	1k	5k	10k	20k	30k
MNIST	$41.45$	$78.41$	$84.08$	$91.90$	$93.17$	$94.28$	$94.93$
Fashion-MNIST	$49.59$	$71.20$	$74.42$	$80.73$	$82.45$	$84.37$	$85.02$
CIFAR-10	$12.57$	$34.98$	$41.66$	$53.54$	$56.40$	$59.31$	$61.16$
N-MNIST	$27.67$	$51.43$	$68.11$	$88.50$	$90.78$	$92.44$	$93.29$

Table 5: Data-efficiency for the proposed pipeline. Entries are test accuracies in percent vs number of training examples.

Panel of Figure 3 — Figure 3: Data-efficiency behaviour of the locally trained pipeline. The figure uses line pattern and marker shape rather than color: circles denote MNIST, squares denote Fashion-MNIST, triangles denote CIFAR-10, and diamonds denote N-MNIST.

Panel of Figure 4 — Figure 4: Spike-budget Pareto and event-operation proxy for the final sparse latency code. (a) Test accuracy relative to the matched full-code readout after retaining only the strongest latency events in each sample and retraining the R-STDP readout with the backbone fixed. The horizontal dotted line marks the untruncated code normalized to $100\%$ . (b) Zoom on the high-accuracy operating regime. Filled markers indicate the interpolated retained activity required to preserve $95\%$ of full-code accuracy, yielding a compact proxy for the event-operation budget of the readout code.

Panel of Figure 5 — Figure 5: Residual routing and temporal-evidence preservation. (a) Percentage-point changes relative to the preserved representation $P=H^{(1)}$ for the residual route $[P,\Delta_{\mathrm{res}}]$ , its direct deep extension $[P,\Delta_{\mathrm{res}},\Delta_{D}]$ , and the proposed agreement-filtered code $[P,\Delta_{\mathrm{res}},\Delta_{\mathrm{agree}}]$ . (b) Mean retained accuracy across the four datasets, normalized by the $P$ -only route; shaded envelopes indicate across-dataset dispersion. At the three processing depths, the serial curve evaluates $P$ , $I$ , and $D$ , whereas the residual curve evaluates $P$ , $[P,\Delta_{\mathrm{res}}]$ , and $[P,\Delta_{\mathrm{res}},\Delta_{\mathrm{agree}}]$ , respectively.

Panel of Figure 6 — Figure 6: Decision transitions induced by the full fused representation $H=[P,\Delta_{\mathrm{res}},\Delta_{\mathrm{agree}}]$ relative to the $P$ -only readout. (a) Fraction of test samples corresponding to repaired errors ( $P$ wrong, $H$ correct), new errors ( $P$ correct, $H$ wrong), and changed but unresolved errors (both predictions wrong, but with different predicted classes). (b) Fate of the errors produced by the $P$ -only readout, partitioned into repaired errors, changed but unresolved errors, and errors whose predicted class remains unchanged.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Parameter	MNIST/Fashion-MNIST	CIFAR-10	N-MNIST
Input source	Raw data	grayscale images	RGB images	polarity event stream
Latency encoder	Representation	calibrated local-decorrelation polarity TTFS	calibrated local-decorrelation polarity TTFS	event-map TTFS
Latency encoder	Output channels	2	6	10
Local decorrelation	fit-patch $k$ , stride, $\epsilon$	$7$ , $2$ , $10^{-2}$	$9$ , $2$ , $10^{-2}$	–
Local decorrelation	PCA component fraction, max. patches, fit batch	$1.0$ , $10^{6}$ , $64$	$1.0$ , $10^{6}$ , $64$	–
Polarity calibration	split and scaling	positive/negative; training-set min–max per channel and location	positive/negative; training-set min–max per channel and location	native ON/OFF channels

Appendix

Table 6: Front-end and shared optimization settings. The whitening stride is used only to sample fitting patches; the learned whitening operator is applied densely with unit stride. Static calibration is fitted on the training split and held fixed thereafter.

Block	Setting	Input	Output	$k/s/p$ or op	$\theta_{0}$ or $m_{\mathrm{agree}}$	Top- $k_{\mathrm{map/fus}}$	$g,\sigma$	$A^{+}/A^{-}/\beta$	Ep.
$S_{1}{+}C_{1}$	MNIST/Fashion-MNIST	$2{\times}28{\times}28$	$128{\times}6{\times}6$	$5/1/0$ ; pool $4/4$	5.0	0	-	$.100/.100/1.00$	8
$S_{1}{+}C_{1}$	CIFAR-10	$6{\times}32{\times}32$	$128{\times}7{\times}7$	$5/1/0$ ; pool $4/4$	10.0	0	-	$.100/.100/1.00$	18
$S_{1}{+}C_{1}$	N-MNIST	$10{\times}34{\times}34$	$128{\times}7{\times}7$	$5/1/0$ ; pool $4/4$	4.0	0	-	$.100/.080/.85$	2
$S_{2}{+}C_{2}$	MNIST/Fashion-MNIST	$128{\times}6{\times}6$	$256{\times}5{\times}5$	$1/1/0$ ; pool $2/1$	24.0	32	-	$.100/.100/1.00$	3
$S_{2}{+}C_{2}$	CIFAR-10	$128{\times}7{\times}7$	$256{\times}6{\times}6$	$1/1/0$ ; pool $2/1$	24.0	32	-	$.100/.100/1.00$	3
$S_{2}{+}C_{2}$	N-MNIST	$128{\times}7{\times}7$	$256{\times}6{\times}6$	$1/1/0$ ; pool $2/1$	8.0	32	-	$.100/.100/1.00$	2

Appendix

Table 7: Backbone and fusion parameters for the four-layer locally trained spiking pipeline. The output column reports the latency representation consumed by the following stage. The

C_{2}

representations were exported with pooling kernel/stride

2/1

. The deep rows describe the adaptive path used by the final model.

Dataset	$d_{\mathrm{feat}}$	$N_{\mathrm{out}}$	Proto./class	$\theta_{0}$	$K$	Updated proto.	Non-target scale	$\lambda_{\max}$	Correct scale
MNIST	$17\,408$	80	8	120	7	$1/3$	0.25	0.0035	0.0075
Fashion-MNIST	$17\,408$	40	4	223	5	$1/2$	0.32	0.0050	0.1000
CIFAR-10	$24\,704$	40	4	400	5	$1/2$	0.32	0.0050	0.1000
N-MNIST	$24\,704$	60	6	243	9	$1/3$	0.35	0.0030	0.0050

Appendix

Table 8: Temporal R-STDP readout operating points for the main reported models. All rows use additive STDP, fixed per-neuron

\ell_{1}

weight normalization, global winner-take-all decoding by output spike time,

\tau_{\max}=1

m_{\mathrm{ro}}=.005

, prototype decay

.5

, target scale

2

, floor

0

, target guard

1.25

, anti-guard

0

, and learning-rate annealing

.75

. “Updated prototypes” gives the number selected from the target population and from each selected hard-negative class, respectively.

Explore similar work

SMM Transformer: Leveraging Spiking Neural Networks for Multimodal Tasks
Aug 3, 2026Xiubo Liang, Jinxing Han, Yuke Li +3Spiking Neural Networks Detection Transformer
School of Software Technology, Zhejiang University, Ningbo, China · NetEase Yidun AI Lab, Hangzhou, China
Hybrid ANN-SNN Pipeline with Local Plasticity
Jun 18, 2026Denis Larionov, Khairutin Shtanchaev, Mikhail Kiselev +2Spiking Neural Networks Time-To-First-Spike
Chuvash State University, Cheboksary, Russia · LLC NForm, Moscow, Russia · Dagestan State Technical University, Makhachkala, Russia +1
Scalable Training of Continuous-Time Spiking Neural Networks with Differentiable Spike-Time Discretization
Jul 16, 2026Yusuke Sakemi, Tomoya Takeuchi, Takeo Hosomi +1Spiking Neural Networks Time-To-First-Spike
Research Center for Mathematical Engineering, Chiba Institute of Technology, Narashino, Japan · International Research Center for Neurointelligence (WPI-IRCN), The University of Tokyo, Tokyo, Japan · NEC Corporation, Kawasaki, Japan

Aug 3, 2026, cs.NEJ/K move · Enter open · S save

SMM Transformer: Leveraging Spiking Neural Networks for Multimodal Tasks

Xiubo Liang, Jinxing Han, Yuke Li, Haoqi Zhu+2

School of Software Technology, Zhejiang University, Ningbo, China · NetEase Yidun AI Lab, Hangzhou, China

Spiking Neural Networks Detection Transformer Transformer Architectures Multimodal Deep Learning Multimodal Understanding Neural Network

Open paper AskPDF arXiv

Spiking Neural Networks (SNNs) enable event-driven computation with sparse activations, but building multimodal Transformers on SNNs is hindered by unstable training in deep spiking stacks and the mismatch between dense softmax attention and spike-based communication. We propose SMM Transformer, an SNN-based multimodal Transformer framework that combines (i)PLMP, a Parallel LIF with Multistage Learnable Parameters neuron and a tailored P-STBP algorithm for stable deep SNN training, (ii) SMSA, an attention-inspired spike-driven token-mixing module that replaces dense pairwise softmax attention with channel-wise spike co-activation and self-compensation, and (iii)SMoE, a spiking mixture-of-experts module for modality-aware fusion. Across visual and multimodal benchmarks, SMM Transformer achieves competitive accuracy compared to ANN baselines. Under a standard MAC/AC arithmetic model, SMSA reduces the estimated operator-level compute energy of the attention module by up to 97%, while whole-model profiling shows more moderate but consistent efficiency gains.

Abstract

Explore similar work

SMM Transformer: Leveraging Spiking Neural Networks for Multimodal Tasks

Hybrid ANN-SNN Pipeline with Local Plasticity

Scalable Training of Continuous-Time Spiking Neural Networks with Differentiable Spike-Time Discretization