On the second-order optimization for spiking neural networks

Ngoc Phu Doan; Ihsen Alouani

cs.LGSep 24, 2026

On the second-order optimization for spiking neural networks

Authors: Ngoc Phu Doan, Ihsen Alouani

Organizations: Centre for Secure Information Technologies Queen’s University Belfast

Abstract

Spiking Neural Networks (SNNs) offer an energy-efficient alternative to conventional neural networks by exploiting sparse, binary spikes, and event-driven computation. However, the training of SNNs remains challenging, as spiking activations create a sharp loss landscape that hinders training, and diagonal-curvature optimizers such as the Adam family may fail to capture this geometry. The extension of curvature-based optimization methods to SNNs is further complicated by the sparse, discrete, and temporally recurrent nature of their underlying dynamics. To address these limitations, we propose SpiKFAX, a second-order optimization method that formulates a computationally tractable, Kronecker-factored approximation of the Fisher information matrix specifically adapted to the structure of SNNs. Empirical evaluation across five architectures and seven datasets demonstrates that SpiKFAX consistently yields improvements in test accuracy and training stability relative to other popular optimizers.

Figures & tables

Figure 1: Loss landscape sharpness between VGG11 and S-VGG11 on the CIFAR10 dataset.

	N-MNIST				CIFAR10-DVS				DVS128 Gesture
Architecture	SGD	Adam	AdamW	SpiKFAX	SGD	Adam	AdamW	SpiKFAX	SGD	Adam	AdamW	SpiKFAX
S-LeNet5	$92.66_{1.27}$	$93.76_{1.32}$	$93.97_{0.63}$	$\textbf{98.24}_{0.11}$	$16.6_{0.0}$	$40.6_{0.6}$	$42.7_{0.3}$	$\textbf{46.2}_{1.14}$	$50.37_{1.3}$	$67.42_{0.76}$	$69.7_{0.38}$	$\textbf{76.13}_{1.89}$
S-VGG11	$96.06_{0.74}$	$96.93_{0.55}$	$97.35_{0.21}$	$\textbf{99.22}_{0.01}$	$50.7_{1.25}$	$47.9_{3.2}$	$43.15_{0.25}$	$\textbf{56.15}_{0.85}$	$53.78_{2.27}$	$52.84_{0.19}$	$46.4_{0.95}$	$\textbf{71.78}_{0.57}$
S-VGG16	$97.13_{0.32}$	$95.57_{0.45}$	$96.68_{0.12}$	$\textbf{98.49}_{0.47}$	$45.8_{1.0}$	$29.0_{3.8}$	$31.95_{10.05}$	$\textbf{50.5}_{1.3}$	$55.11_{0.19}$	$51.89_{3.79}$	$54.92_{3.78}$	$\textbf{67.05}_{0.76}$
S-ResNet18	$50.39_{9.14}$	$89.32_{1.03}$	$93.43_{0.89}$	$\textbf{99.13}_{0.16}$	$31.3_{2.8}$	$29.75_{3.35}$	$31.7_{0.4}$	$\textbf{46.95}_{2.75}$	$43.18_{0.39}$	$49.24_{1.5}$	$48.86_{4.2}$	$\textbf{70.27}_{2.46}$

Table 1: Performance comparison of different optimizers across neuromorphic datasets. All results are reported as mean

\pm

std.

Figure 2: Performance comparison of different optimizers and models across static image datasets.

Figure 3: Training loss (over steps) and testing accuracy (over epochs) of S-VGG11 on the DVS128-Gesture dataset.

Figure 4: Effect of learning rate (left) and training time (right) of S-VGG11 on the DVS128-Gesture dataset.

Explore similar work

Advancing Direct Training for Spiking Neural Networks with Circulate-Firing Neurons and Learnable Gradients
May 14, 2026Feifan Zhou, Xiang Wei, Yang Liu +1Spiking Neural Networks Neural Network
School of Artificial and Intelligence, Tianjin University, Tianjin 300072, China
A2SG:Adaptive and Asymmetric Surrogate Gradients for Training Deep Spiking Neural Networks
May 29, 2026Yechan Kang, Yongjin Kweon, Mingyeong Seo +8Spiking Neural Networks Surrogate Gradient
Center for Semiconductor Technology, Korea Institute of Science and Technology (KIST), Seoul, Republic of Korea · Department of Computer Science and Engineering, Korea University, Seoul, Republic of Korea · LIG Defense&Aerospace, Seongnam, Republic of Korea +1
Globally Optimal Training of Spiking Neural Networks via Parameter Reconstruction
May 8, 2026Himanshu Udupi, Xiaocong Yang, ChengXiang ZhaiSpiking Neural Networks Recurrent Neural Networks
University of Illinois Urbana-Champaign

May 14, 2026, cs.NEJ/K move · Enter open · S save

Advancing Direct Training for Spiking Neural Networks with Circulate-Firing Neurons and Learnable Gradients

Feifan Zhou, Xiang Wei, Yang Liu, Qiang Yu

School of Artificial and Intelligence, Tianjin University, Tianjin 300072, China

Spiking Neural Networks Neural Network Surrogate Gradient Neurons

Open paper AskPDF arXiv

Spiking Neural Networks (SNNs) have emerged with promising energy-efficient property, yet a substantial performance gap persists compared to Artificial Neural Networks (ANNs). This gap stems from at least two key limitations: first, conventional spiking neurons offer limited information representation capacity, underutilizing the rich dynamics of membrane potentials; second, fixed surrogate gradient (SG) functions across time steps leads to imprecise gradient propagation, impeding effective direct training. To address these two challenges, we propose a new direct training algorithm with three core innovations: first, a circulate-firing spiking neuron model that enhances information representation capacity by leveraging membrane potentials more effectively; second, a time-step-wise learnable surrogate gradient function, enabling accurate gradient estimation during backpropagation; third, a positive-negative balanced loss function to achieve equilibrium between positive and negative membrane potentials and further boost SNN performance. Extensive experiments demonstrate that our methods achieve competitive performance across multiple datasets. Our methods can generalize seamlessly to advanced architectures of Transformer, consistently outperforming existing methods. Our work highlights the effectiveness of further harnessing intrinsic membrane dynamics of SNNs for performance improvement, and thus open a new avenue for advancing high-performance spiking neural architectures.

Abstract

Explore similar work

Advancing Direct Training for Spiking Neural Networks with Circulate-Firing Neurons and Learnable Gradients

A2SG:Adaptive and Asymmetric Surrogate Gradients for Training Deep Spiking Neural Networks

Globally Optimal Training of Spiking Neural Networks via Parameter Reconstruction