Spiking neural networks (SNNs) offer a compelling pathway toward energy-efficient artificial intelligence, fundamentally anchored in presynaptic input-driven sparsity. However, a substantial yet chronically overlooked source of computational overhead persists: the continuous subthreshold dynamics of postsynaptic neurons. Quantitative analysis reveals that up to 44% of neurons in an ImageNet-trained ResNet-18 model reside in a deeply inhibited state, imposing a persistent computational burden despite having a negligible probability of firing. To address this inefficiency, the Dormant Spiking Neuron (DSN) model is introduced. By incorporating a state-driven dynamic gate, the DSN allows a neuron to enter a dormant state when its membrane potential falls below a predefined negative threshold, dynamically pruning redundant computations at the instance level. Crucially, this dormancy mechanism is directly integrated into the neuronal dynamics and trained end-to-end via backpropagation through time using surrogate gradients, enabling the network to actively learn a robust dormancy policy that adaptively preserves critical information flow. Extensive experiments across static and neuromorphic datasets demonstrate that the DSN framework consistently reduces computational energy by 40% to over 70%. Notably, on the large-scale ImageNet dataset, the DSN outperforms state-of-the-art static pruning techniques at comparable reported cost reductions and simultaneously improves the corresponding baseline SNN accuracy by 1.34%. By establishing a trainable, state-driven sparsity paradigm orthogonal to conventional input-driven methods, the DSN unlocks synergistic efficiency gains, presenting an effective approach for advancing neuromorphic computing.
Figures & tables
Figure 1: The four-step dynamics of the Dormant Spiking Neuron (DSN). At each timestep t , a DSN executes a sequence of operations. (1) Dormancy Gate Update: The neuron first checks whether its previous membrane potential Vnt−1 has fallen below the dormancy threshold Dth . If so, it enters a permanent dormant state, and all subsequent stages are bypassed for the current and all future timesteps. (2) Potential Integration: If active ( Dnt=1 ), the neuron integrates presynaptic inputs with its decayed membrane potential. (3) Gated Spike Firing: A spike Snt is generated only if the neuron is active and its integrated potential Hnt surpasses the firing threshold Vth . (4) Potential Update & Reset: The membrane potential Vnt is updated for the next timestep, incorporating a reset mechanism.
Figure 2: Orthogonal sparsity mechanisms in the DSN model. (a) In dense ANNs, all operations are active. (b) Traditional SNNs exploit input-driven sparsity, wherein computations involving inactive input rows are skipped. (c) The dormancy mechanism introduces state-driven sparsity, wherein computations involving inactive output columns are skipped. (d) The DSN combines both mechanisms, yielding a multiplicative reduction in the number of active synaptic operations.
Dataset
Model
Architecture
T
GSOPs/ GFLOPs
Energy (mJ)
Acc. (%)
CIFAR10
ANN
ResNet-18
-
0.56
2.55
95.93
Transformer
2-512
-
1.24
5.72
96.82
MS-ResNet
ResNet-18
4
0.30
0.28
95.64
SD-Transformer
2-512
4
0.51
0.46
95.60
DSN (ITO)
ResNet-18
4
0.15
0.14
94.90
2-512
4
0.31
0.30
94.29
Table 1: Main results on static and neuromorphic classification benchmarks. The DSN is evaluated under two paradigms: inference-time optimization (ITO) applied to pre-trained SNNs, and direct training with BPTT from scratch. Results show reductions in estimated energy ranging from 40% to over 70%, while maintaining accuracy comparable to or better than that of the corresponding baseline SNNs. Notably, the DSN outperforms state-of-the-art static pruning methods on ImageNet, delivering comparable reported cost reductions and improved accuracy.
Figure 3: Ablation studies on DSN dynamics. (a) Impact of the dormancy threshold Dth on accuracy and energy savings evaluated across three random seeds. (b) Energy savings of the DSN compared with the baseline SNN as a function of the number of simulation timesteps T . (c) ITO on network representations. Layer-wise comparison of the SSIM and spike train accuracy (STA) metrics between a pre-trained baseline ResNet-18 SNN and the same network modified using the DSN ITO paradigm.
CIFAR10
CIFAR100
ImageNet
DVS-Gesture
DVS-CIFAR10
Accuracy (%)
95.30±0.10
76.61±0.13
64.36±0.19
96.65±0.40
77.83±0.12
Table 2: Three-seed accuracy results for the BPTT-trained DSN. Values are reported as mean ± standard deviation over three runs.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Vanilla Spike-Driven Transformer
Spike-Driven Transformer with DSN
SPS
First Conv
Emac⋅MACsConv1
Emac⋅MACsConv1
Other Conv
Eac⋅MACsConvn⋅∑t=1TRn−1t
Eac⋅MACsConvn⋅∑t=1T(Rn−1t⋅Dnt)
Self-Attention
Q
Eac⋅ND2⋅∑t=1TRint
Eac⋅ND2⋅∑t=1T(Rint⋅DQt)
K
Eac⋅ND2⋅∑t=1TRint
Eac⋅ND2⋅∑t=1T(Rint⋅DKt)
V
Eac⋅ND2⋅∑t=1TRint
Eac⋅ND2⋅∑t=1T(Rint⋅DVt)
f(Q,K,V)
Eac⋅ND⋅∑t=1TRKVt
Eac⋅ND⋅∑t=1TRKVt
Appendix
Table 3: Energy calculation example for a Spike-Driven Transformer architecture.
Figure 4: (a) The cumulative distribution function (CDF) of membrane potentials in a ResNet-18 SNN trained on ImageNet reveals a tripartite split. A substantial fraction (44%) of neurons reside in the deeply inhibited region ( Vm≤0 ). (b) The Heaviside step function, whose derivative is zero almost everywhere, preventing gradient flow during backpropagation. (c) To enable gradient-based training, the true derivative is replaced by a smooth surrogate function σ′(⋅) during the backward pass.
Dth
DVS-CIFAR10 (SD-Transformer)
DVS-Gesture (SD-Transformer)
ImageNet (ResNet-18)
CIFAR100 (SD-Transformer)
−0.01
75.9% ( ↓71% )
94.44% ( ↓65% )
64.20% ( ↓73% )
79.22% ( ↓44% )
−0.05
76.6% ( ↓70% )
95.48% ( ↓63% )
64.44% ( ↓73% )
79.36% ( ↓43% )
−0.1
77.2% ( ↓68% )
96.18% ( ↓60% )
64.34% ( ↓72% )
79.40% ( ↓42% )
−0.5
77.9% ( ↓52% )
96.88% ( ↓42% )
64.50% ( ↓54% )
79.50% ( ↓32% )
−1
78.0% ( ↓38% )
96.88% ( ↓30% )
64.66% ( ↓37% )
79.55% ( ↓21% )
Appendix
Table 4: Threshold-sweep results across datasets and architectures for BPTT-trained DSN models. Each entry reports accuracy (%) and estimated energy reduction.
Figure 5: (a) Layer-wise firing rates, non-dormant rates, and the Neuronal Computational Utilization (NCU) on a ResNet-18 architecture. (b) Layer-wise Jaccard similarity of dormancy masks for a ResNet-34 model, showing their heterogeneity across inputs.
Dataset
Architecture
Learning Rate
Weight Decay
Batch Size
Epochs
Optimizer
CIFAR
MS-ResNet
2e-2
5e-4
64
300+10
SGD
SD-Transformer-2-512
3e-4
6e-2
64
300+10
AdamW
ImageNet
MS-ResNet
5e-2
2e-5
64
150+5
SGD
SD-Transformer-8-384
7.5e-4
1e-2
28
300+10
LAMB
DVS-Gesture
SD-Transformer-2-256
1e-4
6e-2
16
200+10
LAMB
DVS-CIFAR10
SD-Transformer-2-256
2e-3
1e-4
32
200+10
AdamW
Appendix
Table 5: Training hyperparameters. A summary of the key hyperparameters utilized across all datasets and network architectures.
Spiking neural networks process information through discrete, time-dependent neural events, aiming for efficient computation and biologically inspired learning.
Deploying spiking neural networks (SNNs) on neuromorphic hardware demands aggressive synaptic pruning while preserving temporal computation integrity. Existing strategies either neglect neuronal criticality or rely on convex relaxations of the inherently combinatorial pruning problem whose fractional masks, upon binarisation, destroy accuracy at moderate-to-high sparsity. We present Criticality-Constrained Quadratic Pruning (CQP), a native PyTorch pipeline that fuses weight magnitude with surrogate-gradient criticality into an analytically exact importance metric, eliminating the rounding artefacts endemic to solver-based approaches. We formally characterise a continuous-relaxation trap wherein OSQP-solver fractional masks overshoot the intended sparsity by up to 12 percentage points (pp), precipitating a 44 pp accuracy collapse. We identify and remediate a zombie-weight failure mode in which Adam's first-moment tensors resurrect pruned synapses, violating the binary sparsity guarantee. An iterative schedule - prune, fine-tune with gradient masking, recompute criticality, and repeat - eliminates gradient staleness at high sparsity. A KL-divergence temporal analysis identifies a redundant simulation timestep, enabling a free 10% theoretical energy reduction without weight modification. On MNIST (60,000 training examples), CQP yields 95.6% accuracy at 90% sparsity versus 93.4% for magnitude pruning (+2.2 pp). A criticality-threshold sweep reveals an empirical criticality cliff: accuracy falls from 87.0% to 14.4% as the threshold reaches tau = 0.9, constituting a quantitative SNN-level analogue of the Critical Brain Hypothesis. Combined weight sparsification and temporal truncation yield a compound 73% reduction in per-inference energy at 70% sparsity, confirming the practical value of the proposed pipeline for neuromorphic deployment.
Muhammad Hamza
Indian Institute of Technology Kharagpur, West Bengal 721302, India
The performance of deep spiking neural networks (SNNs) often relies on batch normalization (BN). However, the advanced dynamic BN variants used in state-of-the-art models introduce runtime multiplications, which weaken the hardware-efficiency motivation of SNNs. To address this tension, we identify catastrophic firing-rate decay as a primary cause of severe performance degradation in normalization-free SNNs. Guided by this insight, this work proposes the Intrinsically Stable SNN (IS-SNN) architecture, which removes activation-normalization layers by enforcing signal homeostasis through topology-aware weight standardization and modified residual connections. By folding the standardization operations into static weights offline, IS-SNN removes the runtime statistics tracking and multiplications introduced by activation normalization, restoring an accumulation-oriented inference datapath. Comprehensive experiments show that IS-SNN achieves performance competitive with or superior to computationally expensive dynamic BN techniques across VGG, ResNet, and Transformer-based models. Notably, it achieves a competitive accuracy of 68.05% on ImageNet and overcomes the severe depth limitations of prior BN-free attempts. Together with a 96.4% reduction in FPGA lookup table resource consumption for neuron implementations, these results support IS-SNN as a practical framework for building accurate and hardware-friendly deep neuromorphic systems.
Ruichen Ma, Xiaoyang Zhang, Jian Bai +5
University of Electronic Science and Technology of China (UESTC) · Beijing Institute of Remote-Sensing Equipment · Shenzhen Institute for Advanced Study, UESTC
Spiking neural networks (SNNs) offer inherent energy efficiency due to their event-driven computation model, making them promising for edge AI deployment. However, their practical adoption is limited by the computational overhead of deep architectures and the absence of input-adaptive control. This work presents SPARQ, a unified framework that integrates spiking computation, quantization-aware training, and reinforcement learning-guided early exits for efficient and adaptive inference. Evaluations across MLP, LeNet, and AlexNet architectures demonstrated that the proposed Quantised Dynamic SNNs (QDSNN) consistently outperform conventional SNNs and QSNNs, achieving up to 5.15% higher accuracy over QSNNs, over 330 times lower system energy compared to baseline SNNs, and over 90 percent fewer synaptic operations across different datasets. These results validate SPARQ as a hardware-friendly, energy-efficient solution for real-time AI at the edge.
Parth Patne, Mahdi Taheri, Ali Mahani +3
Brandenburg Technical University, Cottbus, Germany · Tallinn University of Technology, Tallinn, Estonia · Shahid Bahonar University of Kerman, Iran +2