Linearized Bregman Iterations for Sparse Spiking Neural Networks

Daniel Windhager; Bernhard A. Moser; Michael Lunglmayr

eess.SPMar 17, 2026

Linearized Bregman Iterations for Sparse Spiking Neural Networks

Authors: Daniel Windhager, Bernhard A. Moser, Michael Lunglmayr

Organizations: Silicon Austria Labs, Linz, Austria · Software Competence Center Hagenberg, Hagenberg, Austria · Institute of Signal Processing, Johannes Kepler University, Linz, Austria

Abstract

Spiking Neural Networks (SNNs) offer an energy efficient alternative to conventional Artificial Neural Networks (ANNs) but typically still require a large number of parameters. This work introduces Linearized Bregman Iterations (LBI) as an optimizer for training SNNs, enforcing sparsity through iterative minimization of the Bregman distance and proximal soft thresholding updates. To improve convergence and generalization, we employ the AdaBreg optimizer, a momentum and bias corrected Bregman variant of Adam. Experiments on three established neuromorphic benchmarks, i.e. the Spiking Heidelberg Digits (SHD), the Spiking Speech Commands (SSC), and the Permuted Sequential MNIST (PSMNIST) datasets, show that LBI based optimization reduces the number of active parameters by about 50% while maintaining accuracy comparable to models trained with the Adam optimizer, demonstrating the potential of convex sparsity inducing methods for efficient neuromorphic learning.

Figures & tables

Dataset	Inputs	Hidden Layer 1	Hidden Layer 2	Hidden Layer 3	Outputs
SHD	$140^{\lx@sectionsign}$	$256$	$256^{\dagger}$	-	$20$
SSC	$140^{\lx@sectionsign}$	$256^{\dagger}$	$256^{\dagger}$	$256^{\dagger}$	$35{\ddagger}$
PSMNIST	$1$	$64^{\dagger}$	$212^{\dagger}$	$212^{\dagger}$	$10{\ddagger}$

Table 1: Number of neurons and layers of the networks used for evaluations of Linearized Bregman iterations on the SHD, SSC and PSMNIST datasets.

Panel of Figure 1 — Figure 1: Loss curves for the training on SHD, SSC and PSMNIST dataset with different values for $\lambda$ . All curves were averaged over three separate training runs with different seeds.

Panel of Figure 2 — Figure 2: Number of non-zero values in networks for SHD, SSC and PSMNIST datasets during the training process plotted as a function of current epoch for different $\lambda$ values. Results represent the mean across three independent training runs.

Figure 3: Peak validation accuracy across SHD, SSC, and PSMNIST datasets as a function of regularization parameter $\lambda$ , averaged over multiple training runs. The optimal $\lambda$ is slightly higher for SHD and SSC, while a lower value of $\lambda$ achieves the best results for PSMNIST.

Panel of Figure 4 — Figure 4: Loss curves for the training on SHD, SSC and PSMNIST dataset with different values for $\lambda$ . All curves represent means over three independent training runs with different random seeds.

Figure 5: Peak validation accuracy without learning rate scheduling across SHD, SSC, and PSMNIST datasets versus regularization parameter $\lambda$ , averaged over multiple training runs.

Dataset	SHD	SSC	PSMNIST
Queant et al. (2025)	93.39%	82.58%	96.21%
Ours	92.98%	81.86%	95.59%
Ours (no LR scheduling)	92.28%	80.67%	95.11%

Table 2: Test set accuracy comparison. Baseline results from Queant et al. (2025) (Adam optimizer) versus AdaBreg results with and without learning rate scheduling across SHD, SSC, and PSMNIST datasets.

Explore similar work

On the second-order optimization for spiking neural networks
Sep 24, 2026Ngoc Phu Doan, Ihsen AlouaniNeural Network Optimization Second-Order Optimization
Centre for Secure Information Technologies Queen’s University Belfast
Adaptive Spiking Neurons for Vision and Language Modeling
Apr 14, 2026Chenlin Zhou, Sihang Guo, Jiaqi Wang +5Spiking Neural Networks
School of Electronic and Computer Engineering, Peking University, Shenzhen, China · Harbin Institute of Technology, Shenzhen, China · School of Computer Science, Peking University, Beijing, China +1
Adaptive Speech-to-Spike Encoding for Spiking Neural Networks
Jun 17, 2026Taharim Rahman Anon, Jakaria Islam EmonKeyword Spotting Speech Processing
1PI LLC, Sapporo, Hokkaido, Japan

Sep 24, 2026J/K move · Enter open · S save

On the second-order optimization for spiking neural networks

Ngoc Phu Doan, Ihsen Alouani

Centre for Secure Information Technologies Queen’s University Belfast

Neural Network Optimization Second-Order Optimization Spiking Neural Networks

Open paper AskPDF arXiv

Spiking Neural Networks (SNNs) offer an energy-efficient alternative to conventional neural networks by exploiting sparse, binary spikes, and event-driven computation. However, the training of SNNs remains challenging, as spiking activations create a sharp loss landscape that hinders training, and diagonal-curvature optimizers such as the Adam family may fail to capture this geometry. The extension of curvature-based optimization methods to SNNs is further complicated by the sparse, discrete, and temporally recurrent nature of their underlying dynamics. To address these limitations, we propose SpiKFAX, a second-order optimization method that formulates a computationally tractable, Kronecker-factored approximation of the Fisher information matrix specifically adapted to the structure of SNNs. Empirical evaluation across five architectures and seven datasets demonstrates that SpiKFAX consistently yields improvements in test accuracy and training stability relative to other popular optimizers.

Abstract

Explore similar work

On the second-order optimization for spiking neural networks

Adaptive Spiking Neurons for Vision and Language Modeling

Adaptive Speech-to-Spike Encoding for Spiking Neural Networks