Large Language Models with At Most One Spike per Neuron

Zhuoya Zhao; Parsa Omidi; Aref Jafari; Richard Naud

cs.NESep 4, 2026

Large Language Models with At Most One Spike per Neuron

Authors: Zhuoya Zhao, Parsa Omidi, Aref Jafari, Richard Naud

Abstract

Leveraging their inherent sparse event-driven computation, spiking neural networks (SNNs) offer a promising path toward energy-efficient large language models (LLMs). Time-to-first-spike (TTFS) coding generates at most one spike per neuron within a time window, yielding extremely low firing rates. However, conventional TTFS SNNs are restricted to specific structures, making it challenging to encode certain blocks in LLM -- such as layer normalization and matrix multiplication --using TTFS. To overcome this limitation, we introduce a reference-based strategy specifically to encode the four core LLM components: embedding layers, layer normalization, attention-related operations and dropout. We construct a fully TTFS-based SNN architecture and train it end-to-end. Experiments on modern LLMs like BERT and GPT-2 demonstrate that our approach achieves performance comparable to ANN counterparts on natural language understanding and common-sense reasoning, while a clear gap remains on language modeling perplexity. To the best of our knowledge, this is the first work to scale a spiking LLM to 1.5 billion parameters using TTFS coding. We also report an estimate of spike-related energy; this is a spike-count proxy under an established cost model rather than a measurement on neuromorphic hardware.

Figures & tables

Figure 1: Comparison of performance, estimated spike-related energy consumption (refer to Section 6.1 ), and parameter size for TTFS-BERT, standard BERT, and other spiking BERTs on the GLUE benchmark.

Figure 2: A. Spikes propagation. B. Integral of membrane potential $V_{i}^{(n)}$ over time. The blue background represents $t\in(t_{min}^{(n-1)},t_{max}^{(n-1)})$ , and the orange background represents $t\in(t_{min}^{(n)},t_{max}^{(n)})$ . The color change (from blue to orange) of the slope corresponds to the variation of the gates. C. Gating signal. D. ReLU and its variant.

Figure 3: Left. A set of neurons representing the vocabulary or positions are waiting to be activated by input tokens. The time window is set to $1000ms$ . A spike is generated if the corresponding token ID or position exists. The spike timings are set to $1ms$ . Right. Each token or position in the input sequence activates the corresponding neuron.

Figure 5: Fitting $\frac{1}{\sqrt{x}}$ by using two fully connected layer.

Model	Size	MNLI-(m/mm)	QQP	QNLI	SST-2	STS-B	MRPC	RTE	Avg.
BERT	Base	84.0/79.6	87.4	86.2	92.5	88.8	85.2	68.6	84.0
SpikeBERT		71.4/71.0	68.2	66.4	85.4	18.7	82.0	57.5	65.1
SpikingBERT		78.1/-	86.8	85.2	88.2	81.9	79.2	66.1	-
SpikeLM		77.1/77.2	83.9	85.3	87.0	84.9	85.7	69.0	81.3
BPSN		74.1/-	83.9	83.9	88.0	83.9	85.3	64.6	-
TTFS-BERT		77.1/77.6	87.0	85.2	86.3	82.0	81.6	66.8	80.5

Table 1: Performance comparison on the GLUE benchmark. F1 scores are reported for QQP and MRPC, Spearman correlations are reported for STS-B, and accuracy scores are reported for the other tasks.

Model	Size	Wiki.	LMB.	LMB.	PIQA	Hella.	Wino.	ARC-e	ARC-c	SIQA	BoolQ	Avg.
		ppl $\downarrow$	ppl $\downarrow$	acc $\uparrow$	acc $\uparrow$	acc_n $\uparrow$	acc $\uparrow$	acc $\uparrow$	acc_n $\uparrow$	acc $\uparrow$	acc $\uparrow$
GPT-2	Small	37.4	40.1	32.6	62.5	31.1	51.6	39.5	22.7	36.6	48.7	40.7
NSLLM		-	-	-	59.3	-	51.0	39.9	23.3	-	-	-
NSLLM*		-	-	-	60.5	-	50.8	38.3	22.8	-	-	-
TTFS-GPT-2		40.4	92.6	25.4	63.9	34.3	50.6	47.0	25.7	37.9	55.1	42.5
GPT-2	XL	20.4	10.6	51.2	70.5	50.9	58.3	51.1	28.5	40.3	61.8	51.6

Table 2: Performance comparison of conventional and TTFS-based GPT2 models on language modeling and common-sense reasoning benchmarks. All evaluations are performed using lm-evaluation-harness.

	SpikeLM	SpikingBERT	TTFS-BERT
Spike-related cost	$1.2T_{r}$	$3.2T_{r}$	$0.80\,T_{r}$

Table 3: Estimated average spike-related energy cost per neuron per inference. Baseline entries are obtained as (firing rate)

\times

(number of time steps) from the settings reported in the respective papers: SpikeLM

0.3\times 4

and SpikingBERT

0.2\times 16

. The TTFS-BERT entry is derived from Table 4 in Appendix B .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Transformer Block	Sub-Block	Num. Neurons	Avg. Spike Count per Neuron per Time Window
TTFS_LayerNorm	mean	1	1
	subtraction	$d$	1
	square	$d$	1
	variance	1	1
	$\frac{1}{\sqrt{x}}$	512	0.25
	norm	$d$	1

Appendix

Table 4: Average Spike Count across Transformer Blocks per Time Window

Figure 6: Comparsion between Vanilla-BERT-Base and TTFS-BERT-Base.

Figure 7: Comparsion between Vanilla-BERT-Large and TTFS-BERT-Large.

Figure 8: Schematic illustration of the distribution of first spike timings.

Explore similar work

Spike-driven Large Language Model
Apr 11, 2026Han Xu, Xuerui Qiu, Baiyu Chen +7Spiking Neural Networks LLM Inference Acceleration
Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China · Brain-Inspired Models Group, Beijing Academy of Artificial Intelligence, Beijing 100862, China +7
QuantaSpike: Short-Window Spike-Driven Quantization for Large Language Models
Sep 28, 2026Bang Hu, Guowei Zhu, Changze Lv +4LLM Quantization Spiking Neural Networks
School of Computer Science, Fudan University, Shanghai, China
BiSpikCLM: A Spiking Language Model integrating Softmax-Free Spiking Attention and Spike-Aware Alignment Distillation
Apr 14, 2026Sihang Guo, Chenlin Zhou, Jiaqi Wang +3Spiking Neural Networks Language Modeling
Harbin Institute of Technology, Shenzhen, China · School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University, Shenzhen, China · Peng Cheng Laboratory, Shenzhen, China

Apr 11, 2026J/K move · Enter open · S save

Spike-driven Large Language Model

Han Xu, Xuerui Qiu, Baiyu Chen, Xinhao Luo+6

Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China · Brain-Inspired Models Group, Beijing Academy of Artificial Intelligence, Beijing 100862, China+7

Spiking Neural Networks LLM Inference Acceleration Spiking Transformers LLM Quantization

Open paper AskPDF arXiv

Current Large Language Models (LLMs) are primarily based on large-scale dense matrix multiplications. Inspired by the brain's information processing mechanism, we explore the fundamental question: how to effectively integrate the brain's spiking-driven characteristics into LLM inference. Spiking Neural Networks (SNNs) possess spike-driven characteristics, and some works have attempted to combine SNNs with Transformers. However, achieving spike-driven LLMs with billions of parameters, relying solely on sparse additions, remains a challenge in the SNN field. To address the issues of limited representational capacity and sparsity in existing spike encoding schemes at the LLM level, we propose SDLLM, a spike-driven large language model that eliminates dense matrix multiplications through sparse addition operations. Specifically, we use the plug-and-play gamma-SQP two-step spike encoding method to ensure that the quantization process aligns with the model's semantic space, mitigating representation degradation caused by binary spikes. Furthermore, we introduce bidirectional encoding under symmetric quantization and membrane potential clipping mechanisms, leading to spike trains with no or low firing counts dominating, significantly reducing the model's spike firing rate, while halving the number of time steps. Experimental results show that SDLLM not only significantly reduces inference costs but also achieves state-of-the-art task performance under the spike-based paradigm. For example, compared to previous spike-based LLMs, SDLLM reduces energy consumption by 7x and improves accuracy by 4.2%. Our model provides inspiration for the architecture design of the next generation of event-driven neuromorphic chips.

Abstract

Explore similar work

Spike-driven Large Language Model

QuantaSpike: Short-Window Spike-Driven Quantization for Large Language Models

BiSpikCLM: A Spiking Language Model integrating Softmax-Free Spiking Attention and Spike-Aware Alignment Distillation