cs.NEApr 14, 2026
SaveBiSpikCLM: A Spiking Language Model integrating Softmax-Free Spiking Attention and Spike-Aware Alignment Distillation
Organizations: Harbin Institute of Technology, Shenzhen, China · School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University, Shenzhen, China · Peng Cheng Laboratory, Shenzhen, China
Abstract
Spiking Neural Networks (SNNs) offer promising energy-efficient alternatives to large language models (LLMs) due to their event-driven nature and ultra-low power consumption. However, to preserve capacity, most existing spiking LLMs still incur intensive floating-point matrix multiplication (MatMul) and nonlinearities, or training difficulties arising from the complex spatiotemporal dynamics. To address these challenges, we propose BiSpikCLM, the first fully binary spiking MatMul-free causal language model. BiSpikCLM introduces Softmax-Free Spiking Attention (SFSA), eliminating softmax and floating-point operations in autoregressive language modeling. For efficient training, we introduce Spike-Aware Alignment Distillation (SpAD), which aligns ANN teacher and SNN student across embeddings, attention maps, intermediate features, and output logits. SpAD framework allows BiSpikCLM to reach comparable performance to ANN counterparts using substantially fewer training tokens (e.g., only 5.6% of the tokens for the 1.3B model). As a result, BiSpikCLM achieves competitive performance at only 4.16% - 5.87% of the computational cost on natural language generation tasks. Our results highlight the feasibility and effectiveness of fully binary spike-driven LLMs and establish the distillation as a promising pathway for brain-inspired spiking NLP.
Figures & tables
Figure 1: Overview of Softmax-Free Spiking Attention (SFSA) . Left: Comparison between Vanilla Causal Self-Attention (CSA) (bottom) and SFSA (top). CSA uses softmax and additive masks, while SFSA employs spike-based activation and binary causal masking. Right: Detailed SFSA pipeline, showing spike-form Q, K, V computation, masked integer attention, spiking activation, and spike-based output, enabling fully discrete and energy-efficient attention modeling.
| Model | Params | Spike Form | Training Method | Architecture | Softmax-Free | Exp-Free | FP-Mul-Free |
| SpikeBERT ( Lv et al., 2023 ) | 109M | Binary | Knowledge distillation | Encoder-only | ✓ | ✓ | |
| SpikingBERT ( Bal and Sengupta, 2024 ) | 50M | Binary | Knowledge distillation | Encoder-only | |||
| SpikeLM (BERT) ( Xing et al., 2024b ) | 110M | Ternary | Train from scratch | Encoder-only | |||
| SpikeLM (BART) ( Xing et al., 2024b ) | 139M | Ternary | Train from scratch | Encoder–Decoder | |||
| SpikeLM (mBART) ( Xing et al., 2024b ) | 680M | Ternary | Train from scratch | Encoder–Decoder | |||
| SpikeGPT ( Zhu et al., 2023 ) | 216M | Binary | Train from scratch | Decoder-only | ✓ |
Table 1: Comparison of spiking language models. The properties “Softmax-Free”, “Exp-Free (exponential-function-free)”, and “FP-Mul-Free (floating-point multiplication-free)” are assessed at the attention-module level.
Figure 2: Left depicts the BiSpikCLM framework, detailing the operations of the Softmax-Free Spiking Attention (SFSA) module and the Spiking Feed-Forward Network (SFFN) module. Right compares the computational process of vanilla Causal Self-Attention (CSA), Spiking Self-Attention (SSA), and SFSA, where red spikes represent binary values of 1 and all other values are 0.
Figure 3: Overview of our Spike-Aware Alignment Distillation (SpAD) framework. Knowledge is transferred from a frozen ANN teacher to a trainable SNN student via five alignment modules: (1) Embedding Alignment (EA); (2) Spike-Attention Alignment (SAA); (3) Spike-Feature Alignment (SFA); (4) Soft-Target Alignment (STA); and (5) Hard-Target Alignment (HTA). Losses include MSE, CE, and spike-aware temporal strategies. In particular, our proposed Rate-MSE loss in Eq. ( 9 ) aligns the attention dynamics between ANN and SNN models over time. Dashed arrows indicate loss paths; spike-related operations are denoted with icons.
| Model | Params (B) | Tokens (B) | Spike Form | Time Step | OPs (G) | Firing Rate | Energy (mJ) | Zero - shot Accuracy (%) | ||||||||
| ARC-e | ARC-c | WG | BQ | PIQA | HS | OBQA | HQA | Avg. | ||||||||
| OPT | 0.125 | 180 | — | 125.6 | — | 125.95 | 43.6 | 19.3 | 52.3 | 54.6 | 62.4 | 32.1 | 20.2 | 23.7 | 38.60 | |
| Pythia | 0.160 | 300 | — | 125.7 | — | 126.01 | 43.7 | 19.8 | 52.8 | 55.1 | 62.7 | 33.6 | 20.1 | 24.2 | 39.00 | |
| SpikeGPT | 0.046 | 16.5 | Binary | — | 3.66 | 0.174 | 3.29 | 32.3 | 16.2 | 50.2 | 45.7 | 54.6 | 25.3 | 15.7 | 20.6 | 32.58 |
| SpikeGPT | 0.216 | 16.5 | Binary | — | 18.3 | 0.168 | 16.53 | 35.2 | 17.7 | 50.7 | 47.3 | 55.1 | 27.6 | 17.3 | 23.1 | 34.25 |
| BiSpikCLM | 0.125 | 1.0 | Binary | 2 | 12.1 | 0.196 | 5.24 | 39.1 | 18.9 | 50.3 | 52.7 | 56.7 | 28.1 | 19.8 | 22.9 | 36.05 |
Table 2: Comparison of performance and estimated energy efficiency between BiSpikCLM, TriSpikCLM and conventional ANN baselines on the ACC benchmark. BiSpikCLM adopts classic LIF neurons (see Eq. ( 1 )) implemented via SpikingJelly ( Fang et al., 2023 ) , while TriSpikCLM employs ternary-valued spiking neurons with amplitude encoding (see Eq. ( 2 )). All energy estimates are calculated under a uniform FP32-based energy model for fair comparison. Time Step indicates the number of discrete simulation steps during SNN inference.
Figure 4: Visualization of ablation experiments.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Hyperparameter | Value |
| Teacher ANN model | OPT-family |
| Student SNN model | BiSpikCLM |
| Tokenizer / Vocabulary | Aligned with OPT |
| Batch size (per GPU) | 16 |
| Gradient accumulation steps | 16 |
| Effective batch size | 256 |
Table 3: Summary of training hyperparameters and configurations used for BiSpikCLM, including optimization settings, distillation parameters, and hardware specifications.
| Model | Params (B) | Tokens (B) | Spike Form | Time Step | Avg. Acc.(%) | SNN/ANN Ratio (%) |
| BiSpikCLM | 0.125 | 1.0 | Binary | 2 | 36.05 | 93.39 |
| TriSpikCLM | 0.125 | 1.0 | Ternary | 2 | 36.11 | 93.55 |
| BiSpikCLM | 0.125 | 1.0 | Binary | 4 | 36.50 | 94.56 |
| TriSpikCLM | 0.125 | 1.0 | Ternary | 4 | 36.27 | 93.96 |
| BiSpikCLM | 0.350 | 2.0 | Binary | 2 | 38.48 | 90.16 |
| TriSpikCLM | 0.350 | 2.0 | Ternary | 2 | 38.14 | 89.36 |
Table 4: Comparison of BiSpikCLM and SpikeLLM across different model scales, neuron/spike formats, and time steps. Avg. Acc. reports zero-shot accuracy (%), and SNN/ANN Ratio shows the performance of spiking models relative to their ANN counterparts.
Figure 5: Weight distribution comparison between Artificial Neural Networks (ANNs) and Spiking Neural Networks (SNNs). ANNs typically exhibit a more concentrated weight distribution around zero, especially in early layers (e.g., Layer 0). In deeper layers (e.g., Layer 11), their weight distribution becomes slightly more spread out but remains relatively compact, indicating tightly clustered weights that contribute to stability and ease of training. In contrast, SNNs show a broader and more dispersed weight distribution, with weights less tightly clustered around zero. This broader spread is particularly notable in deeper layers (e.g., Layer 11), reflecting the dynamic and diverse weight updates characteristic of their spiking nature.
| Model | Total Outliers |
| ANN | 18,663 |
| SNN | 5,834 |
Table 5: Total number of weight outliers across all layers and components.
| Layer | Component | Model | Mean | Std | Max | Min | Num_Outliers |
| 0 | q_proj | ANN | -3.15E-05 | 7.87E-02 | 0.77 | -0.80 | 596 |
| 0 | q_proj | SNN | 1.30E-04 | 1.14E-02 | 0.06 | -0.06 | 6 |
| 0 | k_proj | ANN | 6.12E-05 | 7.26E-02 | 0.77 | -0.70 | 1154 |
| 0 | k_proj | SNN | -3.36E-05 | 1.10E-02 | 0.06 | -0.06 | 24 |
| 0 | v_proj | ANN | -1.55E-05 | 1.28E-02 | 0.17 | -0.19 | 163 |
| 0 | v_proj | SNN | 2.97E-05 | 1.31E-02 | 0.08 | -0.08 | 34 |
Table 6: Comparison of weight statistics between ANN and SNN across various layers and components.
Figure 6: Layer-wise firing rate heatmaps under different temporal resolutions. (a) At , firing activity is sparse and concentrated in a small subset of layers. (b) At , elevated firing rates begin to propagate across more layers, indicating increased temporal integration. (c) With , spiking activity becomes more distributed, engaging a larger portion of the network. (d) At , firing patterns are largely uniform across layers, suggesting a temporally saturated regime with broad layer participation.
Figure 7: This figure presents the spiking activity across token positions for four representative layers (Layer 0, Layer 3, Layer 7, and Layer 11) at an early inference time step. The firing patterns are relatively sparse and uniformly distributed, particularly in the lower layers. This reflects the initial stage of neuronal processing, where the model begins encoding input signals with limited temporal context. Notably, deeper layers such as Layer 11 exhibit subdued activation, suggesting that higher-level abstractions have not yet emerged.
Figure 8: At , the firing distributions become slightly more structured across token positions and layers. While lower layers maintain broadly distributed activity, deeper layers begin to display early signs of selective activation. Compared to , this figure reveals the onset of temporal refinement, indicating that additional time steps allow the model to initiate more context-sensitive computation, particularly in the upper layers.
Figure 9: With four time steps, the model exhibits more pronounced spatiotemporal differentiation in firing behavior. Activity becomes more variable across token positions, and certain regions in deeper layers start to display concentrated firing. This suggests that the network is engaging in increasingly specialized processing, distributing its computation more selectively based on both input semantics and accumulated temporal evidence.
Figure 10: By , the firing patterns exhibit substantial temporal evolution and structural complexity. Deeper layers, in particular, show heightened and more focused activation for specific token regions, reflecting refined internal representations. This level of activity suggests that the model has transitioned into a more stable and semantically rich encoding phase. The marked increase in firing diversity and intensity across layers highlights the model’s capacity to utilize extended temporal windows for deeper contextual integration and task-specific computation.
| Model | Params (B) | Time Step | Zero-shot Accuracy (%) | ||||||||
| ARC-e | ARC-c | WG | BQ | PIQA | HS | OBQA | HQA | Avg. | |||
| BiSpikCLM-OPT | 0.125 | 2 | 39.1 | 18.9 | 50.3 | 52.7 | 56.7 | 28.1 | 19.8 | 22.9 | 36.05 |
| BiSpikCLM-OPT | 0.125 | 4 | 39.4 | 19.0 | 51.2 | 53.0 | 57.5 | 29.2 | 19.7 | 23.1 | 36.50 |
| BiSpikCLM-Llama | 0.165 | 2 | 39.3 | 19.2 | 51.3 | 52.8 | 56.8 | 28.3 | 20.3 | 23.2 | 36.40 |
| BiSpikCLM-Llama | 0.165 | 4 | 39.6 | 19.6 | 51.7 | 53.2 | 57.3 | 29.5 | 20.7 | 23.7 | 36.91 |
| BiSpikCLM-OPT | 1.300 | 2 | 45.7 | 23.5 | 54.2 | 56.3 | 62.3 | 40.2 | 24.5 | 24.0 | 41.33 |
Table 7: Performance comparison of BiSpikCLM models across different architectures and time steps.
| Model | Params (B) | Zero-shot Accuracy (%) | ||||||||
| ARC-e | ARC-c | WG | BQ | PIQA | HS | OBQA | HQA | Avg. | ||
| BiSpikCLM (T=2) | 1.3 | 45.7 | 23.5 | 54.2 | 56.3 | 62.3 | 40.2 | 24.5 | 24.0 | 41.33 |
| BiSpikCLM (T=4) | 1.3 | 46.3 | 24.3 | 55.6 | 56.8 | 63.4 | 41.7 | 25.2 | 24.3 | 42.19 |
| BiSpikCLM-Llama (T=2) | 1.2 | 45.8 | 23.9 | 54.7 | 56.1 | 63.0 | 40.6 | 24.7 | 24.1 | 41.61 |
| BiSpikCLM-Llama (T=4) | 1.2 | 46.5 | 24.4 | 55.9 | 56.6 | 63.6 | 41.8 | 25.4 | 24.4 | 42.33 |
| BitNet (1.58-bit) | 1.3 | 48.7 | 24.1 | 56.8 | 57.4 | 64.2 | 40.6 | 24.6 | 24.9 | 42.66 |
Table 8: Performance comparison of BiSpikCLM with quantized LLMs.
| Model | Params (B) | Tokens (B) | Zero-shot Accuracy (%) | ||||||||
| ARC-e | ARC-c | WG | BQ | PIQA | HS | OBQA | HQA | Avg. | |||
| BiSpikCLM (T=2) | 0.125 | 1.0 | 39.1 | 18.9 | 50.3 | 52.7 | 56.7 | 28.1 | 19.8 | 22.9 | 36.05 |
| BiSpikCLM (T=2) | 0.125 | 5.0 | 41.4 | 19.2 | 51.4 | 53.4 | 58.2 | 30.2 | 19.9 | 23.1 | 37.10 |
| BiSpikCLM (T=2) | 1.300 | 10.0 | 45.7 | 23.5 | 54.2 | 56.3 | 62.3 | 40.2 | 24.5 | 24.0 | 41.33 |
| BiSpikCLM (T=2) | 1.300 | 25.0 | 48.3 | 26.4 | 57.8 | 58.6 | 65.1 | 44.9 | 27.3 | 26.7 | 44.39 |
Table 9: Performance of BiSpikCLM with varying training token counts.
Figure 11: Training loss curves for the BiSpikCLM-1.3B model, comparing the 10B and 25B token training regimes. The smooth downward trend confirms stable convergence.
| Prompt | BiSpikCLM (10B Tokens) | BiSpikCLM (25B Tokens) |
| Hi there! How are you doing today? | Hi! How are you? I’m doing well. How about you? | I’m doing great! I’m just tired as hell. I went to bed at 11pm last night, and now I’m up at 4am. I have a class tomorrow morning at 7:30, so I have to get up and go to class. |
| Where is the capital of the United States? | It’s in the middle of the country. | Washington, D.C. |
| How about the cat? | I think it’s a good idea to have a cat in the house. | I have a cat and he doesn’t like the camera. I’ll send you a picture when he is in the mood. |
Table 10: Comparison of conversational abilities for BiSpikCLM-1.3B trained on different token counts.
| Model | Params (B) | Firing Threshold | Context Length | ||||
| 512 | 1024 | 2048 | 4096 | 8192 | |||
| OPT | 1.300 | - | 16.26 | 13.58 | 11.13 | - | - |
| Llama-3.2 | 1.200 | - | 12.93 | 10.96 | 9.76 | 9.02 | 8.54 |
| BiSpikCLM | 1.300 | 0.70 | 36.72 (0.184) | 32.49 (0.183) | 29.34 (0.181) | - | - |
| 0.85 | 39.75 (0.177) | 36.30 (0.175) | 32.18 (0.174) | - | - | ||
| 1.00 | 43.17 (0.163) | 39.88 (0.162) | 34.62 (0.160) | - | - | ||
Table 11: Perplexity (PPL) on WikiText-2 with varying context lengths and firing thresholds. The firing rate (sparsity) is shown in parentheses.