SpikeOPD: Stable On-Policy Distillation for Autoregressive Spiking Language Models

Enqiao Lu; Xingrui Yu; Yiwei Fu; Zhenglin Wan; Pengfei Zhou; Wangbo Zhao; Muqing Jian; Xueyi Zhang

cs.AIAug 28, 2026

SpikeOPD: Stable On-Policy Distillation for Autoregressive Spiking Language Models

Authors: Enqiao Lu, Xingrui Yu, Yiwei Fu, Zhenglin Wan, Pengfei Zhou, Wangbo Zhao, Muqing Jian, Xueyi Zhang,

Organizations: The Chinese University of Hong Kong, Shenzhen · Agency for Science, Technology and Research (A*STAR), Singapore · Nanyang Technological University, Singapore · Peking University ·

Abstract

Spiking neural networks (SNNs) offer a path to energy-efficient language modeling through sparse encoding and event-driven computation, but training capable spiking language models from scratch remains difficult. A practical alternative is ANN-to-SNN migration through knowledge distillation (KD), where a pretrained artificial neural network (ANN) teacher supervises an SNN student. Existing migration approaches distill on fixed corpus prefixes, whereas autoregressive inference conditions on self-generated prefixes, creating prefix-source mismatch. It manifests as output-policy mismatch with the ANN teacher and internal spiking-dynamics drift between self-generated and matched corpus prefixes. On-policy distillation (OPD) offers a natural way to mitigate both manifestations by continuing teacher supervision on self-generated prefixes. We evaluate a teacher-only full-KL variant, Vanilla OPD, via a controlled stress test and observe it may suffer from delayed rollout-feedback collapse. This result shows that on-policy coverage alone does not ensure stable adaptation. Motivated by these findings, we propose SpikeOPD, a stable on-policy distillation framework for autoregressive SNNs that learns from self-generated prefixes while maintaining rollout stability. It applies full-KL teacher correction to reduce output-policy mismatch, while matched-prefix policy anchoring constrains policy departure from the frozen reference SNN on the same prefixes. Layerwise spike regularization further limits firing-rate deviations during on-policy adaptation. Across three model scales, SpikeOPD improves average accuracy over the corresponding KD SNNs by 0.8, 1.7, and 2.9 points at 0.125B, 0.35B, and 1.3B, respectively, while preserving their sparse-compute profiles.

Figures & tables

Figure 1: Paired-prefix output-policy diagnostic on 256 prompts. (A) Teacher-to-SNN KL on self-generated (solid) and corpus (dashed) prefixes. (B) Prompt-internal Self $-$ Corpus gap.

Figure 2: Internal effects under matched prompts, positions, and lengths: (A) spike-rate gap, (B) hidden-state gap, and (C) aggregate membrane-potential gap. Smaller is better.

Method	LR	Clip	Step	C/T	Onset	Avg. $\uparrow$	Adj. $\downarrow$	Dist.-4 $\uparrow$
Vanilla OPD	$\eta_{0}$	-	500	$10/10$	168.5	34.8	39.1	83.7
Small-LR	$\eta_{0}/3$	-	1000	$10/10$	527	34.8	36.0	85.5
Clipped	$\eta_{0}$	1.0	500	$10/10$	327	34.5	31.5	87.8
SpikeOPD	$\eta_{0}$	-	1000	$0/10$	-	35.5	0.1	99.2

Table 1: Stability controls from one KD checkpoint. C/T reports collapsed/tested runs, and Onset averages first-collapse updates. Collapse uses 20-update Adj. and

1-\text{Dist.-4}

thresholds of 0.10 and 0.05. Avg. is the last pre-onset eight-task average or final average.

Figure 3: Delayed rollout-feedback collapse in one of 10 matched trajectories. Adjacent repetition measures equal neighboring-token pairs, while

1-\text{Dist.-4}

measures repeated contiguous four-token sequences. Vanilla OPD becomes repetitive, whereas SpikeOPD remains stable. Lower is better.

Figure 4: Overview of SpikeOPD. On the same detached self-generated prefixes, teacher correction transfers the ANN distribution, policy anchoring constrains departure from the frozen reference SNN, and spike regularization controls selected-layer firing rates. Only $\theta$ is updated.

		Zero-shot Accuracy (%)									Analytical Efficiency
Model	Scale	ARC-e	ARC-c	WG	BQ	PIQA	HS	OBQA	HQA	Avg. $\uparrow$	OPs $\downarrow$	Rate $\downarrow$	Energy $\downarrow$
OPT Teacher	0.125B	43.4	19.3	52.4	54.2	62.4	31.9	20.3	23.5	38.4	125.6	NA	126.0
KD	0.125B	37.1	19.1	51.2	47.8	53.8	27.6	18.4	22.2	34.7	27.7	0.17	9.3
SpikeGPT	0.046B	32.3	16.2	50.2	45.7	54.6	25.3	15.7	20.6	32.6	3.7	0.17	3.3
SpikeGPT	0.216B	35.2	17.7	50.7	47.3	55.1	27.6	17.3	23.1	34.3	18.3	0.17	16.5
SpikeOPD	0.125B	39.1	19.2	51.9	48.5	55.3	28.1	19.0	22.6	$\mathbf{35.5}$	27.5	$0.17$	9.2

Table 2: Zero-shot accuracy and analytical efficiency. For SpikeOPD, all task accuracies, Avg., and Rate are three-seed means; the corresponding sample standard deviations for Avg. and Rate are reported in the text. Avg. is unweighted. OPs and Energy are reported in G and mJ. Bold marks the best matched-scale SNN accuracy, including ties.

	Zero-shot Accuracy (%)									Rollout Statistics
Method	ARC-e	ARC-c	WG	BQ	PIQA	HS	OBQA	HQA	Avg.	Adj. $\downarrow$	Dist.-4 $\uparrow$	Run $\downarrow$
KD	37.1	19.1	51.2	47.8	53.8	27.6	18.4	22.2	34.7	0.16	99.83	1.0433
Continued SpAD	37.3	19.1	49.6	48.0	54.2	27.3	18.4	22.0	34.5	0.16	99.77	1.0457
SFT	36.8	16.9	50.6	38.4	49.1	25.9	20.0	18.8	32.1	0.09	7.28	1.0275
Vanilla OPD	36.0	17.0	49.0	38.4	49.9	26.1	22.0	20.3	32.3	39.09	83.67	6.9041
EOPD	37.3	17.5	51.3	46.2	54.1	27.3	21.2	21.7	34.6	24.77	87.21	5.8413

Table 3: Controlled 0.125B comparison from one checkpoint and 500 updates. Adj., Dist.-4, and Run denote adjacent repetition (%), unique four-token fraction (%), and maximum identical-token run length. Bold marks the best accuracy, including ties.

Figure 5: Component ablation at 0.125B. “w/o reference KL” removes matched-prefix policy anchoring. The other labels remove the named term in Eq. 10 .

Figure 6: Initialization robustness across 1k, 2k, and final 4k offline checkpoints. O and S denote the matched KD and SpikeOPD results. All three pairs improve.

$K$	Upd.	Gen./upd.	Avg. $\uparrow$	Adj. $\downarrow$	Dist.-4 $\uparrow$
8	500	$8B$	34.9	0.2	99.0
16	500	$16B$	35.3	0.1	99.3
32	500	$32B$	$\mathbf{35.5}$	0.1	99.2
64	500	$64B$	35.2	0.2	99.1
128	500	$128B$	34.8	0.3	98.8

Table 4: Rollout-length sensitivity at 0.125B. Metrics are percentages. The shaded default uses three seeds and other rows use one fixed seed.

Layers	Avg. $\uparrow$	Adj. $\downarrow$	Dist.-4 $\uparrow$	Rate
$\Delta L=1$	35.0	0.2	99.0	0.176
$\Delta L=2$	35.3	0.1	99.3	0.171
$\Delta L=3$	$\mathbf{35.5}$	0.1	99.2	$0.173$
$\Delta L=4$	35.2	0.1	99.1	0.168
$\Delta L=6$	34.9	0.2	99.0	0.164
$\Delta L=12$	34.6	0.3	98.8	0.158

Table 5: Layer-selection sensitivity at 0.125B. None removes spike regularization. Metrics are percentages except Rate. The shaded default uses three seeds and other rows use one.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Value
Stage 1 checkpoint	4,000 updates
Prompt length	480 tokens
Rollout length $K$	32 tokens
Rollout temperature	1.0
Simulation steps $T_{s}$	4
Diagnostic prompt count	256

Appendix

Table 1: Stage 2 reproducibility record. The objective weights are fixed throughout each run, including the 1,000-update stability runs.

Stage 1 setting	Value
Teacher / student	OPT / BiSpikCLM
Tokenizer and vocabulary	Shared with OPT
Updates used for $\theta_{0}$	4,000
Simulation steps $T_{s}$	4
Optimizer	Adam
Learning rate	$5\times 10^{-4}$

Appendix

Table 2: Stage 1 initialization record. The public recipe’s effective batch of 256 is device-local (

16\times 16

); under eight-way data parallelism the corresponding global effective batch is 2,048 sequences per optimizer step.

Spiking Language Models

See all 12 papers

BiSpikCLM: A Spiking Language Model integrating Softmax-Free Spiking Attention and Spike-Aware Alignment Distillation
Apr 14, 2026Sihang Guo, Chenlin Zhou, Jiaqi Wang +3Spiking Language Models Spiking Neural Networks
Harbin Institute of Technology, Shenzhen, China · School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University, Shenzhen, China · Peng Cheng Laboratory, Shenzhen, China
Not All Timesteps Matter Equally: Selective Alignment Knowledge Distillation for Spiking Neural Networks
May 14, 2026Kai Sun, Peibo Duan, Yongsheng Huang +4Selective Knowledge Distillation Spiking Neural Networks
Faculty of Information Technology, Monash University, Australia · School of Software, Northeastern University, China · Department of Medicine, National University of Singapore, Singapore
When Teachers Mislead: Spurious-Signal-Aware On-Policy Distillation
Aug 4, 2026Yinuo Jiang, Yongjie Ye, Zhou Tao +4VLM Distillation Language Model Distillation
Zhejiang University · ByteDance · Shanghai Artificial Intelligence Laboratory

Apr 14, 2026J/K move · Enter open · S save

BiSpikCLM: A Spiking Language Model integrating Softmax-Free Spiking Attention and Spike-Aware Alignment Distillation

Sihang Guo, Chenlin Zhou, Jiaqi Wang, Kehai Chen+2

Harbin Institute of Technology, Shenzhen, China · School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University, Shenzhen, China · Peng Cheng Laboratory, Shenzhen, China

Spiking Language Models Spiking Neural Networks Language Modeling LLM Inference Acceleration Spiking Transformers Binary Neural Networks+1

Open paper AskPDF arXiv

Spiking Neural Networks (SNNs) offer promising energy-efficient alternatives to large language models (LLMs) due to their event-driven nature and ultra-low power consumption. However, to preserve capacity, most existing spiking LLMs still incur intensive floating-point matrix multiplication (MatMul) and nonlinearities, or training difficulties arising from the complex spatiotemporal dynamics. To address these challenges, we propose BiSpikCLM, the first fully binary spiking MatMul-free causal language model. BiSpikCLM introduces Softmax-Free Spiking Attention (SFSA), eliminating softmax and floating-point operations in autoregressive language modeling. For efficient training, we introduce Spike-Aware Alignment Distillation (SpAD), which aligns ANN teacher and SNN student across embeddings, attention maps, intermediate features, and output logits. SpAD framework allows BiSpikCLM to reach comparable performance to ANN counterparts using substantially fewer training tokens (e.g., only 5.6% of the tokens for the 1.3B model). As a result, BiSpikCLM achieves competitive performance at only 4.16% - 5.87% of the computational cost on natural language generation tasks. Our results highlight the feasibility and effectiveness of fully binary spike-driven LLMs and establish the distillation as a promising pathway for brain-inspired spiking NLP.

Abstract

Spiking Language Models

Explore similar work

BiSpikCLM: A Spiking Language Model integrating Softmax-Free Spiking Attention and Spike-Aware Alignment Distillation

Not All Timesteps Matter Equally: Selective Alignment Knowledge Distillation for Spiking Neural Networks

When Teachers Mislead: Spurious-Signal-Aware On-Policy Distillation