Not All Timesteps Matter Equally: Selective Alignment Knowledge Distillation for Spiking Neural Networks
Authors: Kai Sun, Peibo Duan, Yongsheng Huang, Guowei Zhang, Benjamin Smith, Nanxu Gong, Levin Kuhlmann
Organizations: Faculty of Information Technology, Monash University, Australia · School of Software, Northeastern University, China · Department of Medicine, National University of Singapore, Singapore
Spiking neural networks (SNNs), which are brain-inspired and spike-driven, achieve high energy efficiency. However, a performance gap between SNNs and artificial neural networks (ANNs) still remains. Knowledge distillation (KD) is commonly adopted to improve SNN performance, but existing methods typically enforce uniform alignment across all timesteps, either from a teacher network or through inter-temporal self-distillation, implicitly assuming that per-timestep predictions should be treated equally. In practice, SNN predictions vary and evolve over time, and intermediate timesteps need not all be individually correct even when the final aggregated output is correct. Under such conditions, effective distillation should not force every timestep toward the same supervision target, but instead provide corrective guidance to erroneous timesteps while preserving useful temporal dynamics. To address this issue, we propose Selective Alignment Knowledge Distillation (SeAl-KD), which selectively aligns class-level and temporal knowledge by equalizing competing logits at erroneous timesteps and reweighting temporal alignment based on confidence and inter-timestep similarity. Extensive experiments on static image and neuromorphic event-based datasets demonstrate consistent improvements over existing distillation methods. The code is available at https://github.com/KaiSUN1/SeAl
SNNs promise energy-efficient and low-latency inference, but their performance still trails that of ANNs. ANN-to-SNN knowledge distillation helps narrow this gap, yet the original training data are often unavailable in practical deployment settings. Existing data-free knowledge distillation (DFKD) methods synthesize surrogate data by matching teacher-side priors, especially BN statistics, but these ANN-oriented constraints mainly regularize mean and variance and therefore remain under-constrained for SNN students whose responses depend on threshold-crossing dynamics. In this paper, we propose Spike Tail-Aware Relational Synthesis (STARS), a plug-and-play method for ANN-to-SNN DFKD that augments standard BN-guided synthesis with two complementary objectives: Relational Consistency Alignment, which preserves cross-sample relational consistency between teacher and student, and Tail-Aware Regularization, which regularizes threshold-relevant tail probabilities through soft exceedance over teacher-derived thresholds. Together, these objectives generate synthetic batches that remain teacher-valid while becoming more informative for SNN students. Experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet across multiple ANN-SNN pairs show that our method consistently improves conventional DFKD baselines and even surpasses several KD methods, with gains of up to 4.6% on CIFAR-10 and 6.7% on CIFAR-100, highlighting the importance of complementing BN matching with relational and tail-aware constraints in SNN-oriented DFKD.
Continuous-time spiking neural networks (SNNs) provide an event-driven framework for temporal computation, computational neuroscience, and neuromorphic hardware. However, training deep continuous-time SNNs is severely constrained by the memory required for exact spike-time computation, which evaluates and retains candidate firing times over intervals determined by presynaptic spike ordering. Here we introduce a memory-efficient training framework based on differentiable spike-time discretization (DSTD) for leaky integrate-and-fire neurons with general membrane and synaptic time constants. DSTD maps irregular presynaptic spikes onto differentiable weighted events at fixed time points, replacing the input-dependent candidate dimension with M fixed time intervals while accurately approximating continuous-time membrane-potential dynamics. This reduces candidate-related activation memory from O(NoutNin) to O(NoutM) in the case of time-to-first-spike (TTFS) coding, where Nin and Nout denote the numbers of presynaptic and postsynaptic neurons, respectively. We further introduce synfire-chain-inspired temporal regularization that organizes layer-wise firing windows, mitigates dead-neuron failures, and enables pipeline-like processing. In dense LIF layers, DSTD reduced peak memory consumption by up to approximately 100-fold and training time by up to approximately 20-fold compared with exact spike-time computation. Together, these methods allowed us to train 9-layer convolutional SNNs on CIFAR-10 and 20-layer convolutional SNNs on Fashion-MNIST on a single GPU.
Supervised learning in spiking neural networks (SNNs) typically requires either gradient-based backpropagation, which sacrifices the Hebbian, spike-driven character of biological plasticity, or reward-modulated Spike-Timing-Dependent Plasticity (STDP), in which class supervision enters only as a scalar gate on an otherwise class-agnostic correlation signal. We propose Supervised Spike Agreement-Dependent Plasticity (Supervised SADP), a gradient-free supervised Hebbian learning algorithm in which class information is embedded directly into the Hebbian plasticity computation rather than introduced through reward modulation. SADP trains the output layer via a supervised Hebbian rule that encodes class labels into output spike patterns, then trains the hidden layer by measuring each hidden neuron's chance-corrected temporal agreement, Cohen's kappa, with the correct-class output spike train produced by the forward pass without gradient computation or external reward. A K-shift extension aggregates agreement over temporal offsets, providing robustness to spike-timing jitter at linear computational cost. We evaluate Supervised SADP against reward-modulated STDP across six benchmark and medical imaging datasets, four input encoding strategies, K_shift in {5,25}, and three reward modes (none, binary, margin). Supervised SADP outperforms STDP in a significant majority of comparisons. Under Poisson encoding, SADP achieves 86.46% on MNIST and 76.62% on Fashion-MNIST, outperforming the best STDP configurations by 23.66 and 23.29 percentage points, respectively. Across the encodings tested, including CNN-extracted features, SADP outperforms STDP in the large majority of cells and trains 1.47x faster on average, with up to 2.86x speedup under Poisson inputs. These results position Supervised SADP as a stable, efficient, gradient-free alternative to reward-modulated STDP for supervised SNN learning.