Organizations: The Chinese University of Hong Kong, Shenzhen · Agency for Science, Technology and Research (A*STAR), Singapore · Nanyang Technological University, Singapore · Peking University · National University of Singapore · Rice University
Spiking neural networks (SNNs) offer a path to energy-efficient language modeling through sparse encoding and event-driven computation, but training capable spiking language models from scratch remains difficult. A practical alternative is ANN-to-SNN migration through knowledge distillation (KD), where a pretrained artificial neural network (ANN) teacher supervises an SNN student. Existing migration approaches distill on fixed corpus prefixes, whereas autoregressive inference conditions on self-generated prefixes, creating prefix-source mismatch. It manifests as output-policy mismatch with the ANN teacher and internal spiking-dynamics drift between self-generated and matched corpus prefixes. On-policy distillation (OPD) offers a natural way to mitigate both manifestations by continuing teacher supervision on self-generated prefixes. We evaluate a teacher-only full-KL variant, Vanilla OPD, via a controlled stress test and observe it may suffer from delayed rollout-feedback collapse. This result shows that on-policy coverage alone does not ensure stable adaptation. Motivated by these findings, we propose SpikeOPD, a stable on-policy distillation framework for autoregressive SNNs that learns from self-generated prefixes while maintaining rollout stability. It applies full-KL teacher correction to reduce output-policy mismatch, while matched-prefix policy anchoring constrains policy departure from the frozen reference SNN on the same prefixes. Layerwise spike regularization further limits firing-rate deviations during on-policy adaptation. Across three model scales, SpikeOPD improves average accuracy over the corresponding KD SNNs by 0.8, 1.7, and 2.9 points at 0.125B, 0.35B, and 1.3B, respectively, while preserving their sparse-compute profiles.
Figures & tables
Figure 1: Paired-prefix output-policy diagnostic on 256 prompts. (A) Teacher-to-SNN KL on self-generated (solid) and corpus (dashed) prefixes. (B) Prompt-internal Self − Corpus gap.
Figure 2: Internal effects under matched prompts, positions, and lengths: (A) spike-rate gap, (B) hidden-state gap, and (C) aggregate membrane-potential gap. Smaller is better.
Method
LR
Clip
Step
C/T
Onset
Avg. ↑
Adj. ↓
Dist.-4 ↑
Vanilla OPD
η0
-
500
10/10
168.5
34.8
39.1
83.7
Small-LR
η0/3
-
1000
10/10
527
34.8
36.0
85.5
Clipped
η0
1.0
500
10/10
327
34.5
31.5
87.8
SpikeOPD
η0
-
1000
0/10
-
35.5
0.1
99.2
Table 1: Stability controls from one KD checkpoint. C/T reports collapsed/tested runs, and Onset averages first-collapse updates. Collapse uses 20-update Adj. and 1−Dist.-4 thresholds of 0.10 and 0.05. Avg. is the last pre-onset eight-task average or final average.
Figure 3: Delayed rollout-feedback collapse in one of 10 matched trajectories. Adjacent repetition measures equal neighboring-token pairs, while 1−Dist.-4 measures repeated contiguous four-token sequences. Vanilla OPD becomes repetitive, whereas SpikeOPD remains stable. Lower is better.
Figure 4: Overview of SpikeOPD. On the same detached self-generated prefixes, teacher correction transfers the ANN distribution, policy anchoring constrains departure from the frozen reference SNN, and spike regularization controls selected-layer firing rates. Only θ is updated.
Zero-shot Accuracy (%)
Analytical Efficiency
Model
Scale
ARC-e
ARC-c
WG
BQ
PIQA
HS
OBQA
HQA
Avg. ↑
OPs ↓
Rate ↓
Energy ↓
OPT Teacher
0.125B
43.4
19.3
52.4
54.2
62.4
31.9
20.3
23.5
38.4
125.6
NA
126.0
KD
0.125B
37.1
19.1
51.2
47.8
53.8
27.6
18.4
22.2
34.7
27.7
0.17
9.3
SpikeGPT
0.046B
32.3
16.2
50.2
45.7
54.6
25.3
15.7
20.6
32.6
3.7
0.17
3.3
SpikeGPT
0.216B
35.2
17.7
50.7
47.3
55.1
27.6
17.3
23.1
34.3
18.3
0.17
16.5
SpikeOPD
0.125B
39.1
19.2
51.9
48.5
55.3
28.1
19.0
22.6
35.5
27.5
0.17
9.2
Table 2: Zero-shot accuracy and analytical efficiency. For SpikeOPD, all task accuracies, Avg., and Rate are three-seed means; the corresponding sample standard deviations for Avg. and Rate are reported in the text. Avg. is unweighted. OPs and Energy are reported in G and mJ. Bold marks the best matched-scale SNN accuracy, including ties.
Zero-shot Accuracy (%)
Rollout Statistics
Method
ARC-e
ARC-c
WG
BQ
PIQA
HS
OBQA
HQA
Avg.
Adj. ↓
Dist.-4 ↑
Run ↓
KD
37.1
19.1
51.2
47.8
53.8
27.6
18.4
22.2
34.7
0.16
99.83
1.0433
Continued SpAD
37.3
19.1
49.6
48.0
54.2
27.3
18.4
22.0
34.5
0.16
99.77
1.0457
SFT
36.8
16.9
50.6
38.4
49.1
25.9
20.0
18.8
32.1
0.09
7.28
1.0275
Vanilla OPD
36.0
17.0
49.0
38.4
49.9
26.1
22.0
20.3
32.3
39.09
83.67
6.9041
EOPD
37.3
17.5
51.3
46.2
54.1
27.3
21.2
21.7
34.6
24.77
87.21
5.8413
Table 3: Controlled 0.125B comparison from one checkpoint and 500 updates. Adj., Dist.-4, and Run denote adjacent repetition (%), unique four-token fraction (%), and maximum identical-token run length. Bold marks the best accuracy, including ties.
Figure 5: Component ablation at 0.125B. “w/o reference KL” removes matched-prefix policy anchoring. The other labels remove the named term in Eq. 10 .
Figure 6: Initialization robustness across 1k, 2k, and final 4k offline checkpoints. O and S denote the matched KD and SpikeOPD results. All three pairs improve.
K
Upd.
Gen./upd.
Avg. ↑
Adj. ↓
Dist.-4 ↑
8
500
8B
34.9
0.2
99.0
16
500
16B
35.3
0.1
99.3
32
500
32B
35.5
0.1
99.2
64
500
64B
35.2
0.2
99.1
128
500
128B
34.8
0.3
98.8
Table 4: Rollout-length sensitivity at 0.125B. Metrics are percentages. The shaded default uses three seeds and other rows use one fixed seed.
Layers
Avg. ↑
Adj. ↓
Dist.-4 ↑
Rate
ΔL=1
35.0
0.2
99.0
0.176
ΔL=2
35.3
0.1
99.3
0.171
ΔL=3
35.5
0.1
99.2
0.173
ΔL=4
35.2
0.1
99.1
0.168
ΔL=6
34.9
0.2
99.0
0.164
ΔL=12
34.6
0.3
98.8
0.158
Table 5: Layer-selection sensitivity at 0.125B. None removes spike regularization. Metrics are percentages except Rate. The shaded default uses three seeds and other rows use one.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Setting
Value
Stage 1 checkpoint
4,000 updates
Prompt length
480 tokens
Rollout length K
32 tokens
Rollout temperature
1.0
Simulation steps Ts
4
Diagnostic prompt count
256
Appendix
Table 1: Stage 2 reproducibility record. The objective weights are fixed throughout each run, including the 1,000-update stability runs.
Stage 1 setting
Value
Teacher / student
OPT / BiSpikCLM
Tokenizer and vocabulary
Shared with OPT
Updates used for θ0
4,000
Simulation steps Ts
4
Optimizer
Adam
Learning rate
5×10−4
Appendix
Table 2: Stage 1 initialization record. The public recipe’s effective batch of 256 is device-local ( 16×16 ); under eight-way data parallelism the corresponding global effective batch is 2,048 sequences per optimizer step.
Spiking Neural Networks (SNNs) offer promising energy-efficient alternatives to large language models (LLMs) due to their event-driven nature and ultra-low power consumption. However, to preserve capacity, most existing spiking LLMs still incur intensive floating-point matrix multiplication (MatMul) and nonlinearities, or training difficulties arising from the complex spatiotemporal dynamics. To address these challenges, we propose BiSpikCLM, the first fully binary spiking MatMul-free causal language model. BiSpikCLM introduces Softmax-Free Spiking Attention (SFSA), eliminating softmax and floating-point operations in autoregressive language modeling. For efficient training, we introduce Spike-Aware Alignment Distillation (SpAD), which aligns ANN teacher and SNN student across embeddings, attention maps, intermediate features, and output logits. SpAD framework allows BiSpikCLM to reach comparable performance to ANN counterparts using substantially fewer training tokens (e.g., only 5.6% of the tokens for the 1.3B model). As a result, BiSpikCLM achieves competitive performance at only 4.16% - 5.87% of the computational cost on natural language generation tasks. Our results highlight the feasibility and effectiveness of fully binary spike-driven LLMs and establish the distillation as a promising pathway for brain-inspired spiking NLP.
Sihang Guo, Chenlin Zhou, Jiaqi Wang +3
Harbin Institute of Technology, Shenzhen, China · School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University, Shenzhen, China · Peng Cheng Laboratory, Shenzhen, China
Spiking neural networks (SNNs), which are brain-inspired and spike-driven, achieve high energy efficiency. However, a performance gap between SNNs and artificial neural networks (ANNs) still remains. Knowledge distillation (KD) is commonly adopted to improve SNN performance, but existing methods typically enforce uniform alignment across all timesteps, either from a teacher network or through inter-temporal self-distillation, implicitly assuming that per-timestep predictions should be treated equally. In practice, SNN predictions vary and evolve over time, and intermediate timesteps need not all be individually correct even when the final aggregated output is correct. Under such conditions, effective distillation should not force every timestep toward the same supervision target, but instead provide corrective guidance to erroneous timesteps while preserving useful temporal dynamics. To address this issue, we propose Selective Alignment Knowledge Distillation (SeAl-KD), which selectively aligns class-level and temporal knowledge by equalizing competing logits at erroneous timesteps and reweighting temporal alignment based on confidence and inter-timestep similarity. Extensive experiments on static image and neuromorphic event-based datasets demonstrate consistent improvements over existing distillation methods. The code is available at https://github.com/KaiSUN1/SeAl
Kai Sun, Peibo Duan, Yongsheng Huang +4
Faculty of Information Technology, Monash University, Australia · School of Software, Northeastern University, China · Department of Medicine, National University of Singapore, Singapore
On-Policy distillation (OPD) transfers teacher capabilities by supervising student-sampled trajectories with dense token-level teacher signals. Recent selective OPD methods improve this process by prioritizing signals that are confident, informative, or learnable. However, the assumptions overlook a fundamental failure mode of language models: their token-level judgments can be driven by input-agnostic language priors, formatting conventions, or stereotyped reasoning templates rather than task-specific evidence. We refer to such optimization-relevant but weakly input-grounded supervision as spurious signals in OPD, which may produce large gradients while contributing little task-improving direction. To mitigate this issue, we propose SA-OPD, a Spurious-Signal-Aware On-Policy Distillation framework that identifies and filters misleading token-level supervision based on input-groundedness and optimization impact. SA-OPD introduces a lightweight input-groundedness proxy estimating whether a token-level distillation signal truly depends on the input. It then filters only tokens that simultaneously exhibit low input-groundedness and extreme distillation divergence, thereby removing high-impact spurious updates and achieving fine-grained OPD optimization. Extensive experiments on both large language model (LLM) and vision-language model (VLM) settings demonstrate that SA-OPD consistently outperforms Vanilla OPD and competitive selective methods. These results establish input-groundedness as a key dimension for OPD supervision selection and offer a simple, effective strategy for mitigating spurious updates.
Yinuo Jiang, Yongjie Ye, Zhou Tao +4
Zhejiang University · ByteDance · Shanghai Artificial Intelligence Laboratory