Ternary Spiking Neural Networks Enhanced by Complemented Neurons and Membrane Potential Aggregation
Authors: Boxuan Zhang, Jiaxin Wang, Zhen Xu, Kuan Tao
Organizations: School of Sports Engineering, Beijing Sport University, Beijing, China · School of Mathematics, Statistics and Mechanics, Beijing University of Technology, Beijing, China
Spiking Neural Networks (SNNs) are promising energy-efficient models and powerful framworks of modeling neuron dynamics. However, existing binary spiking neurons exhibit limited biological plausibilities and low information capacity. Recently developed ternary spiking neuron possesses higher consistency with biological principles (i.e. excitation-inhibition balance mechanism). Despite of this, the ternary spiking neuron suffers from defects including iterative information loss, temporal gradient vanishing and irregular distributions of membrane potentials. To address these issues, we propose Complemented Ternary Spiking Neuron (CTSN), a novel ternary spiking neuron model that incorporates an learnable complemental term to store information from historical inputs. CTSN effectively improves the deficiencies of ternary spiking neuron, while the embedded learnable factors enable CTSN to adaptively adjust neuron dynamics, providing strong neural heterogeneity. Furthermore, based on the temporal evolution features of ternary spiking neurons' membrane potential distributions, we propose the Temporal Membrane Potential Regularization (TMPR) training method. TMPR introduces time-varying regularization strategy utilizing membrane potentials, furhter enhancing the training process by creating extra backpropagation paths. We validate our methods through extensive experiments on various datasets, demonstrating remarkable performance advances.
Figures & tables
Figure 1: The overall workflow of CTSN and TMPR.
Figure 2: Membrane potential distributions of the first and the last timestep. Upper left: Ternary spiking ResNet20 on CIFAR-100 (4 timesteps). Upper right: Ternary spiking ResNet20 on CIFAR10-DVS (10 timesteps). Middle left: Spiking ResNet20 on CIFAR-100 (4 timesteps) with CTSN. Middle right: Spiking ResNet20 on CIFAR10-DVS (10 timesteps) with CTSN. Lower left: Spiking ResNet20 on CIFAR100 (4 timesteps) with CTSN and TMPR. Lower right: Spiking ResNet20 on CIFAR10-DVS (10 timesteps) with CTSN and TMPR. See C for detailed visualizations.
Reset Mechanism
Dataset
T
Hard
Soft
CIFAR-100
4
73.95
73.56
CIFAR10-DVS
10
80.30
80.20
Table 1: Comparative accuracy of ternary spiking ResNet20 with different reset mechanisms (%).
Figure 3: Influence of the hyperparameter λ on performance.
Method
Dataset
T
Ternary
w/ C
w/ T
w/ C&T
CIFAR-100
4 6
73.9574.12
74.1874.48
74.5774.73
74.8275.12
CIFAR10-DVS
10
80.30
80.57
80.50
81.23
Table 2: Ablation study results of the proposed method (%). w/ C: with CTSN. w/ T: with TMPR. w/ C&T: with CTSN & TMPR.
Figure 4: Timestep ablation study of CTSN.
Figure 5: Tracking of learnable parameters in each layer of ResNet20. Upper: Network trained on CIFAR10-DVS with 10 timesteps. Lower: Network trained on CIFAR-100 with 4 timesteps.
Dataset
Methods
Architecture
T
Accuracy (%)
CIFAR-10
PLIF ( 2021b )
PLIF Net
6
94.25
GLIF ( 2022 )
ResNet18
6
94.88
TET ( 2022 )
ResNet19
4
94.44
RMP-Loss ( 2023 )
ResNet19
4
95.51
ASGL ( 2023 )
ResNet18
4
95.35
CLIF ( 2024 )
ResNet18
4
96.01
Table 3: Comparison with other state-of-the-art methods. We compare our method with the results reported in the respective literature. *: self-implementation results. † : implementation of our methods with open source code ( Fang et al., 2023 ) .
Methods
Architecture
T
Accuracy (%)
PLIF ( 2021b )
PLIF Net ResNet20 *
20 10
74.8080.70
GLIF ( 2022 )
7B-wideNet ResNet20 *
16 10
78.1077.65
CLIF ( 2024 )
VGG11 ResNet20 *
16 10
79.0079.53
Ternary Spike ( 2024 )
ResNet20 ResNet20 *
10 10
79.8080.30
Ours
VGG16
10
79.06
ResNet20
10
81.23
Table 4: Comparison with the existing neuron models on CIFAR10-DVS dataset. * denotes self-implementation results. † denotes implementation of our methods directly in open source code provided by Fang et al. (2023) .
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Membrane potential Distributions for the third layer of ResNet20 across all timesteps. Left: Hard-resetting neuron. Right: Soft-resetting neuron. Dashed lines indicate the spiking thresholds.
Figure 7: Membrane potential Distributions for the third layer of Ternary Spiking ResNet20. (a) CIFAR-100. (b) CIFAR10-DVS.
Figure 8: Membrane potential Distributions for the third layer of CTSN Spiking ResNet20. (a) CIFAR-100. (b) CIFAR10-DVS.
Figure 9: Membrane potential Distributions for the third layer of CTSN Spiking ResNet20. (a) CIFAR-100 trained by TMPR. (b) CIFAR10-DVS.
Figure 10: The computation graph of (a) forward propagation and (b) backpropagation for CTSN model.
Figure 11: Influence of the hyperparameter λ on ResNet32. The orange dash line indicates the performance of the baseline ternary spiking network.
Dataset
Method
Training Time (hours)
Memory Usage (MB)
Inference Time (s)
CIFAR100 (T=4)
Ternary
5.50
2213
3.55
w/ C
6.48
3087
3.84
w/ C&T
8.75
4041
3.84
CIFAR10-DVS (T=10)
Ternary
3.56
11381
2.02
w/ C
4.63
17975
2.48
w/ C&T
5.68
22809
2.48
Appendix
Table 5: Comparison of training time, memory usage, and inference time. w/ C: with CTSN. w/ C&T: with CTSN & TMPR.
Figure 12: Spike firing rate results. For ternary spiking neuron and our CTSN model, each bar consists of two segments: the lower segment represents +1 spike firing rate, while the upper segment represents -1 spike firing rate.
Dataset
T
Neuron
ACs (M)
MACs (M)
SOP Energy ( μ J)
CIFAR-100
4
LIF Ternary w/ C&T
81.99 216.61 201.30
1.33 1.33 3.89
79.90 201.06 199.04
CIFAR10-DVS
10
LIF Ternary w/ C&T
54.83 196.47 204.48
3.20 3.20 9.59
64.07 191.55 228.15
Appendix
Table 6: Estimated energy consumption. The parameters contained in the ResNet20 model are 10.8M. w/ C&T: with CTSN & TMPR
Spiking neural networks process information through discrete, time-dependent neural events, aiming for efficient computation and biologically inspired learning.
Spiking Neural Networks (SNNs) have emerged with promising energy-efficient property, yet a substantial performance gap persists compared to Artificial Neural Networks (ANNs). This gap stems from at least two key limitations: first, conventional spiking neurons offer limited information representation capacity, underutilizing the rich dynamics of membrane potentials; second, fixed surrogate gradient (SG) functions across time steps leads to imprecise gradient propagation, impeding effective direct training. To address these two challenges, we propose a new direct training algorithm with three core innovations: first, a circulate-firing spiking neuron model that enhances information representation capacity by leveraging membrane potentials more effectively; second, a time-step-wise learnable surrogate gradient function, enabling accurate gradient estimation during backpropagation; third, a positive-negative balanced loss function to achieve equilibrium between positive and negative membrane potentials and further boost SNN performance. Extensive experiments demonstrate that our methods achieve competitive performance across multiple datasets. Our methods can generalize seamlessly to advanced architectures of Transformer, consistently outperforming existing methods. Our work highlights the effectiveness of further harnessing intrinsic membrane dynamics of SNNs for performance improvement, and thus open a new avenue for advancing high-performance spiking neural architectures.
Feifan Zhou, Xiang Wei, Yang Liu +1
School of Artificial and Intelligence, Tianjin University, Tianjin 300072, China
Continuous-time spiking neural networks (SNNs) provide an event-driven framework for temporal computation, computational neuroscience, and neuromorphic hardware. However, training deep continuous-time SNNs is severely constrained by the memory required for exact spike-time computation, which evaluates and retains candidate firing times over intervals determined by presynaptic spike ordering. Here we introduce a memory-efficient training framework based on differentiable spike-time discretization (DSTD) for leaky integrate-and-fire neurons with general membrane and synaptic time constants. DSTD maps irregular presynaptic spikes onto differentiable weighted events at fixed time points, replacing the input-dependent candidate dimension with M fixed time intervals while accurately approximating continuous-time membrane-potential dynamics. This reduces candidate-related activation memory from O(NoutNin) to O(NoutM) in the case of time-to-first-spike (TTFS) coding, where Nin and Nout denote the numbers of presynaptic and postsynaptic neurons, respectively. We further introduce synfire-chain-inspired temporal regularization that organizes layer-wise firing windows, mitigates dead-neuron failures, and enables pipeline-like processing. In dense LIF layers, DSTD reduced peak memory consumption by up to approximately 100-fold and training time by up to approximately 20-fold compared with exact spike-time computation. Together, these methods allowed us to train 9-layer convolutional SNNs on CIFAR-10 and 20-layer convolutional SNNs on Fashion-MNIST on a single GPU.
Yusuke Sakemi, Tomoya Takeuchi, Takeo Hosomi +1
Research Center for Mathematical Engineering, Chiba Institute of Technology, Narashino, Japan · International Research Center for Neurointelligence (WPI-IRCN), The University of Tokyo, Tokyo, Japan · NEC Corporation, Kawasaki, Japan
Spiking neural networks (SNNs) promise low-power event-driven computation for temporally rich tasks, but commonly used neuron models often trade off gradient-based trainability, dynamical richness, and high activity sparsity. These limitations are acute in regression, where approximation error, noise and spike discretization can severely degrade continuous-valued outputs. Indeed, many state-of-the-art (SOTA) SNNs rely on simple phenomenological dynamics trained with surrogate gradients and offer limited control over spiking diversity and sparsity. To overcome such limitations, we introduce multi-timescale conductance spiking networks, a gradient-trainable framework in which neural dynamics emerge from shaping the current-voltage (I-V) curve by tuning fast, slow and ultra-slow conductances. This parametrization allows systematic control over excitability, can be implemented efficiently in analog circuits, and yields rich firing regimes including tonic, phasic and bursting responses within a single model. We derive a discrete-time formulation of these differentiable dynamics, enabling direct backpropagation through time without surrogate-gradient approximations. To probe both trainability and accuracy, we evaluate feedforward networks of these neurons at the predictability limit of Mackey-Glass time-series regression and compare them to baseline LIF and SOTA AdLIF networks. Our model outperforms LIF and AdLIF networks, while exhibiting substantially sparser activity from both communication and computational perspectives. These results highlight multi-timescale conductance spiking neurons as a promising building block for energy-aware temporal processing and neuromorphic implementation.
Alex Fulleda-Garcia, Saray Soldado-Magraner, Josep Maria Margarit-Taulé
Instituto de Microelectrónica de Barcelona (IMB-CNM), Consejo Superior de Investigaciones Científicas (CSIC) Cerdanyola del Vallès, Spain · Department of Neurobiology, University of California Los Angeles (UCLA) Los Angeles, USA · Joint Senior authors