Authors: Han Xu, Xuerui Qiu, Baiyu Chen, Xinhao Luo, Xingrun Xing, Jiahong Zhang, Bo Lei, Tiejun Huang, +2 more
Organizations: Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China · Brain-Inspired Models Group, Beijing Academy of Artificial Intelligence, Beijing 100862, China · School of Future Technology, University of Chinese Academy of Sciences, Beijing 101408, China · Zhongguancun Academy, Beijing 100094, China · Pengcheng Laboratory, Guangdong 518055, China · School of Advanced Inter-disciplinary Sciences, University of Chinese Academy of Sciences, Beijing 101408, China · School of Computer Science, Peking University, Beijing 100871, China · Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, Beijing 100190, China · Spiking Intelligence Lab, Tianqiao & Chirssy Chen Institute, Shanghai 201203, China
Current Large Language Models (LLMs) are primarily based on large-scale dense matrix multiplications. Inspired by the brain's information processing mechanism, we explore the fundamental question: how to effectively integrate the brain's spiking-driven characteristics into LLM inference. Spiking Neural Networks (SNNs) possess spike-driven characteristics, and some works have attempted to combine SNNs with Transformers. However, achieving spike-driven LLMs with billions of parameters, relying solely on sparse additions, remains a challenge in the SNN field. To address the issues of limited representational capacity and sparsity in existing spike encoding schemes at the LLM level, we propose SDLLM, a spike-driven large language model that eliminates dense matrix multiplications through sparse addition operations. Specifically, we use the plug-and-play gamma-SQP two-step spike encoding method to ensure that the quantization process aligns with the model's semantic space, mitigating representation degradation caused by binary spikes. Furthermore, we introduce bidirectional encoding under symmetric quantization and membrane potential clipping mechanisms, leading to spike trains with no or low firing counts dominating, significantly reducing the model's spike firing rate, while halving the number of time steps. Experimental results show that SDLLM not only significantly reduces inference costs but also achieves state-of-the-art task performance under the spike-based paradigm. For example, compared to previous spike-based LLMs, SDLLM reduces energy consumption by 7x and improves accuracy by 4.2%. Our model provides inspiration for the architecture design of the next generation of event-driven neuromorphic chips.
Figures & tables
Figure 1: (a) SDLLM replaces dense matrix multiplication with spike-driven sparse addition in Transformer-based LLMs through novel LLM-level spike encoding. (b) To address insufficient representation capacity and sparsity in existing LLM-level spike encoding, we propose γ -SQP two-step spike encoding to reduce semantic quantization loss and mitigate binary spike representation degradation, combined with bidirectional encoding and membrane potential clipping to significantly reduce firing rate (e.g., 0.5 → 0.2, 0.5 → 0.1) and halve temporal steps ( D→D /2). (c) Performance comparison of SDLLM vs. SpikeLLM (top) and low-bit edge solutions (bottom).
Figure 2: Comparison of attention activation distributions before and after γ -semantic space alignment (QuaRot vs. SDLLM). Compared to QuaRot, SDLLM exhibits a more balanced and more compact attention activation distribution.
Figure 3: Different methods of spike quantization methods. Clipped method has adjustable 0-1 boundary, the other thresholds are uniformly split among 0 and saturation value D.
Figure 4: Unlike rate coding, our count-to-spike paradigm with T×D timesteps ( D as unfolded dimension) supports both synchronous firing (multi-step latency) and asynchronous firing (single-step latency) modes on neuromorphic chips.
Figure 5: Replacing dense matrix multiplication with sparse addition via spike encoding.
Figure 6: Bidirectional encoding under symmetric quantization significantly reduces spike count (e.g., 7.52→1.78 ) while halving unfolded time steps (e.g., 15→8 ), dramatically lowering firing rate (e.g., 0.5→0.2 ). More results can be found in Fig. A2 .
Figure 7: Spike count is further reduced by membrane potential clipping via quantile-based ReLU (e.g., 7.52→0.97 ), achieving lower firing rate (e.g., 0.5→0.1 ).
Figure A1 : Implementation of rotational sparse training for enhancing spike sparsity.
Figure A2 : Bidirectional encoding under symmetric quantization consistently reduces average spike counts while halving unfolded time steps, dramatically lowering firing rate.
Figure A3 : SDLLM spike visualization in LLaMA2-7B with single-layer neuron sampling.
Leveraging their inherent sparse event-driven computation, spiking neural networks (SNNs) offer a promising path toward energy-efficient large language models (LLMs). Time-to-first-spike (TTFS) coding generates at most one spike per neuron within a time window, yielding extremely low firing rates. However, conventional TTFS SNNs are restricted to specific structures, making it challenging to encode certain blocks in LLM -- such as layer normalization and matrix multiplication --using TTFS. To overcome this limitation, we introduce a reference-based strategy specifically to encode the four core LLM components: embedding layers, layer normalization, attention-related operations and dropout. We construct a fully TTFS-based SNN architecture and train it end-to-end. Experiments on modern LLMs like BERT and GPT-2 demonstrate that our approach achieves performance comparable to ANN counterparts on natural language understanding and common-sense reasoning, while a clear gap remains on language modeling perplexity. To the best of our knowledge, this is the first work to scale a spiking LLM to 1.5 billion parameters using TTFS coding. We also report an estimate of spike-related energy; this is a spike-count proxy under an established cost model rather than a measurement on neuromorphic hardware.
Large language models (LLMs) achieve strong performance across many tasks but rely on dense multiply-accumulate (MAC) operations during inference, resulting in high energy cost. Spiking neural networks (SNNs) offer an event-driven alternative in which synaptic integration uses lightweight accumulation. However, spike-driven LLM inference remains difficult because outlier-heavy activations typically require long firing windows or auxiliary non-spiking paths. We propose QuantaSpike, a short-window spike-driven quantization framework for LLMs built around Logarithmic Ternary Integrate-and-Fire (LTIF) neurons. LTIF uses ternary events with power-of-two membrane-response quanta, improving the information represented by each firing step while retaining shift-ACC-compatible computation. QuantaSpike combines this neuron with group-adaptive gain and selective outlier admission: normal values use residual LTIF steps, whereas admitted outliers receive one additional onset spike before entering the same residual dynamics. Across OPT and Llama-2, QuantaSpike achieves state-of-the-art or competitive perplexity and zero-shot accuracy among spike-driven LLM quantization methods. It also transfers to newer dense LLMs, remaining close to the FP16 reference on Llama-3-8B and Qwen3-8B under the same four-step firing window. Analytical linear-energy projections show that QuantaSpike reduces the energy of one linear transformation by about 80.0% on OPT models and 67.1% on Llama-2 models relative to SpikeQuant, providing an accurate and energy-efficient spike-driven path for LLM inference.
Bang Hu, Guowei Zhu, Changze Lv +4
School of Computer Science, Fudan University, Shanghai, China
Spiking Neural Networks (SNNs) offer promising energy-efficient alternatives to large language models (LLMs) due to their event-driven nature and ultra-low power consumption. However, to preserve capacity, most existing spiking LLMs still incur intensive floating-point matrix multiplication (MatMul) and nonlinearities, or training difficulties arising from the complex spatiotemporal dynamics. To address these challenges, we propose BiSpikCLM, the first fully binary spiking MatMul-free causal language model. BiSpikCLM introduces Softmax-Free Spiking Attention (SFSA), eliminating softmax and floating-point operations in autoregressive language modeling. For efficient training, we introduce Spike-Aware Alignment Distillation (SpAD), which aligns ANN teacher and SNN student across embeddings, attention maps, intermediate features, and output logits. SpAD framework allows BiSpikCLM to reach comparable performance to ANN counterparts using substantially fewer training tokens (e.g., only 5.6% of the tokens for the 1.3B model). As a result, BiSpikCLM achieves competitive performance at only 4.16% - 5.87% of the computational cost on natural language generation tasks. Our results highlight the feasibility and effectiveness of fully binary spike-driven LLMs and establish the distillation as a promising pathway for brain-inspired spiking NLP.
Sihang Guo, Chenlin Zhou, Jiaqi Wang +3
Harbin Institute of Technology, Shenzhen, China · School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University, Shenzhen, China · Peng Cheng Laboratory, Shenzhen, China