Ternary Spiking Neural Networks Enhanced by Complemented Neurons and Membrane Potential Aggregation
Authors: Boxuan Zhang, Jiaxin Wang, Zhen Xu, Kuan Tao
Organizations: School of Sports Engineering, Beijing Sport University, Beijing, China · School of Mathematics, Statistics and Mechanics, Beijing University of Technology, Beijing, China
Spiking Neural Networks (SNNs) are promising energy-efficient models and powerful framworks of modeling neuron dynamics. However, existing binary spiking neurons exhibit limited biological plausibilities and low information capacity. Recently developed ternary spiking neuron possesses higher consistency with biological principles (i.e. excitation-inhibition balance mechanism). Despite of this, the ternary spiking neuron suffers from defects including iterative information loss, temporal gradient vanishing and irregular distributions of membrane potentials. To address these issues, we propose Complemented Ternary Spiking Neuron (CTSN), a novel ternary spiking neuron model that incorporates an learnable complemental term to store information from historical inputs. CTSN effectively improves the deficiencies of ternary spiking neuron, while the embedded learnable factors enable CTSN to adaptively adjust neuron dynamics, providing strong neural heterogeneity. Furthermore, based on the temporal evolution features of ternary spiking neurons' membrane potential distributions, we propose the Temporal Membrane Potential Regularization (TMPR) training method. TMPR introduces time-varying regularization strategy utilizing membrane potentials, furhter enhancing the training process by creating extra backpropagation paths. We validate our methods through extensive experiments on various datasets, demonstrating remarkable performance advances.
Figures & tables
Figure 1: The overall workflow of CTSN and TMPR.
Figure 2: Membrane potential distributions of the first and the last timestep. Upper left: Ternary spiking ResNet20 on CIFAR-100 (4 timesteps). Upper right: Ternary spiking ResNet20 on CIFAR10-DVS (10 timesteps). Middle left: Spiking ResNet20 on CIFAR-100 (4 timesteps) with CTSN. Middle right: Spiking ResNet20 on CIFAR10-DVS (10 timesteps) with CTSN. Lower left: Spiking ResNet20 on CIFAR100 (4 timesteps) with CTSN and TMPR. Lower right: Spiking ResNet20 on CIFAR10-DVS (10 timesteps) with CTSN and TMPR. See C for detailed visualizations.
Reset Mechanism
Dataset
T
Hard
Soft
CIFAR-100
4
73.95
73.56
CIFAR10-DVS
10
80.30
80.20
Table 1: Comparative accuracy of ternary spiking ResNet20 with different reset mechanisms (%).
Figure 3: Influence of the hyperparameter λ on performance.
Method
Dataset
T
Ternary
w/ C
w/ T
w/ C&T
CIFAR-100
4 6
73.9574.12
74.1874.48
74.5774.73
74.8275.12
CIFAR10-DVS
10
80.30
80.57
80.50
81.23
Table 2: Ablation study results of the proposed method (%). w/ C: with CTSN. w/ T: with TMPR. w/ C&T: with CTSN & TMPR.
Figure 4: Timestep ablation study of CTSN.
Figure 5: Tracking of learnable parameters in each layer of ResNet20. Upper: Network trained on CIFAR10-DVS with 10 timesteps. Lower: Network trained on CIFAR-100 with 4 timesteps.
Dataset
Methods
Architecture
T
Accuracy (%)
CIFAR-10
PLIF ( 2021b )
PLIF Net
6
94.25
GLIF ( 2022 )
ResNet18
6
94.88
TET ( 2022 )
ResNet19
4
94.44
RMP-Loss ( 2023 )
ResNet19
4
95.51
ASGL ( 2023 )
ResNet18
4
95.35
CLIF ( 2024 )
ResNet18
4
96.01
Table 3: Comparison with other state-of-the-art methods. We compare our method with the results reported in the respective literature. *: self-implementation results. † : implementation of our methods with open source code ( Fang et al., 2023 ) .
Methods
Architecture
T
Accuracy (%)
PLIF ( 2021b )
PLIF Net ResNet20 *
20 10
74.8080.70
GLIF ( 2022 )
7B-wideNet ResNet20 *
16 10
78.1077.65
CLIF ( 2024 )
VGG11 ResNet20 *
16 10
79.0079.53
Ternary Spike ( 2024 )
ResNet20 ResNet20 *
10 10
79.8080.30
Ours
VGG16
10
79.06
ResNet20
10
81.23
Table 4: Comparison with the existing neuron models on CIFAR10-DVS dataset. * denotes self-implementation results. † denotes implementation of our methods directly in open source code provided by Fang et al. (2023) .
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Membrane potential Distributions for the third layer of ResNet20 across all timesteps. Left: Hard-resetting neuron. Right: Soft-resetting neuron. Dashed lines indicate the spiking thresholds.
Figure 7: Membrane potential Distributions for the third layer of Ternary Spiking ResNet20. (a) CIFAR-100. (b) CIFAR10-DVS.
Figure 8: Membrane potential Distributions for the third layer of CTSN Spiking ResNet20. (a) CIFAR-100. (b) CIFAR10-DVS.
Figure 9: Membrane potential Distributions for the third layer of CTSN Spiking ResNet20. (a) CIFAR-100 trained by TMPR. (b) CIFAR10-DVS.
Figure 10: The computation graph of (a) forward propagation and (b) backpropagation for CTSN model.
Figure 11: Influence of the hyperparameter λ on ResNet32. The orange dash line indicates the performance of the baseline ternary spiking network.
Dataset
Method
Training Time (hours)
Memory Usage (MB)
Inference Time (s)
CIFAR100 (T=4)
Ternary
5.50
2213
3.55
w/ C
6.48
3087
3.84
w/ C&T
8.75
4041
3.84
CIFAR10-DVS (T=10)
Ternary
3.56
11381
2.02
w/ C
4.63
17975
2.48
w/ C&T
5.68
22809
2.48
Appendix
Table 5: Comparison of training time, memory usage, and inference time. w/ C: with CTSN. w/ C&T: with CTSN & TMPR.
Figure 12: Spike firing rate results. For ternary spiking neuron and our CTSN model, each bar consists of two segments: the lower segment represents +1 spike firing rate, while the upper segment represents -1 spike firing rate.
Dataset
T
Neuron
ACs (M)
MACs (M)
SOP Energy ( μ J)
CIFAR-100
4
LIF Ternary w/ C&T
81.99 216.61 201.30
1.33 1.33 3.89
79.90 201.06 199.04
CIFAR10-DVS
10
LIF Ternary w/ C&T
54.83 196.47 204.48
3.20 3.20 9.59
64.07 191.55 228.15
Appendix
Table 6: Estimated energy consumption. The parameters contained in the ResNet20 model are 10.8M. w/ C&T: with CTSN & TMPR
Spiking neural networks process information through discrete, time-dependent neural events, aiming for efficient computation and biologically inspired learning.
Research Center for Mathematical Engineering, Chiba Institute of Technology, Narashino, Japan · International Research Center for Neurointelligence (WPI-IRCN), The University of Tokyo, Tokyo, Japan · NEC Corporation, Kawasaki, Japan
Instituto de Microelectrónica de Barcelona (IMB-CNM), Consejo Superior de Investigaciones Científicas (CSIC) Cerdanyola del Vallès, Spain · Department of Neurobiology, University of California Los Angeles (UCLA) Los Angeles, USA · Joint Senior authors