Spiking neural networks target energy-efficient language modeling through sparse, binary activations, but the efficiency-accuracy tradeoff of spiking language models is uncharacterized at scale. We present the first Chinchilla-style IsoFLOP scaling study of a spiking language model, sweeping SpikeGPT-family models (RWKV backbone) from width 384 to 1536 across compute budgets from 3×1017 to 1019 FLOPs on FineWeb-EDU. We report two findings. First, the compute-optimal frontier is Nopt∝C0.56, close to the 0.50 that Hoffmann et al. report for dense Transformers, so spiking does not change how parameters and data should be traded off under a compute budget. Second, against non-spiking RWKV and Transformer baselines trained the same way, the loss gap splits into a backbone cost of about 0.16 nats that does not change with compute and a spiking cost that falls from 0.34 to about 0.23 nats and then levels off. These results give the first compute-optimal recipe for spiking language models and quantify the accuracy cost of their sparsity.
Figures & tables
Figure 1: (a) IsoFLOP profiles: validation loss L versus model size N at each compute budget C , with an interior compute-optimal minimum per tier. (b) The same runs as learning curves (loss versus compute C=6N\ED , colored by budget); their lower envelope is the compute-optimal frontier Lopt(C) , which extrapolates a decade beyond the fit grid to the out-of-sample hero run at C=1020 (measured 2.76 vs. predicted 2.78 nats).
Convention
N
α (IsoFLOP)
α (Approach 3)
D/N ( 3×1017→1019 )
Strict non-embedding (Kaplan)
13ld2
0.748
0.745
25.1→4.7
FLOP-matmul (grid build)
13ld2+vd
0.624
0.644
9.2→4.1
Total (Chinchilla, primary)
13ld2+2vd
0.556
0.595
4.9→3.5
Table 1: Compute-optimal exponent α ( Nopt∝Cα ) under three parameter-counting conventions, estimated in two ways: the IsoFLOP vertex fit (Sec. 4.1 ) and a Chinchilla Approach 3 parametric fit L=E+A/Na+B/Db (Appendix G ). The two estimators agree within each convention. D/N is the compute-optimal token-to-parameter ratio at the smallest and largest budget using IsoFLOP fit.
Figure 2: Spiking RWKV compared with non-spiking baselines. (a) Compute-optimal loss Lopt(C) for each architecture. (b) Backbone cost (dense RWKV minus Transformer) and spiking cost (spiking minus dense RWKV) at each budget.
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
C (FLOPs)
Transformer
Dense RWKV
Spiking RWKV
backbone cost
spiking cost
3×1017
3.195
3.363
3.701
0.168
0.338
1×1018
3.012
3.177
3.441
0.166
0.263
3×1018
2.878
3.030
3.256
0.151
0.227
1×1019
2.671
2.835
3.061
0.164
0.225
Appendix
Table 2: Compute-optimal loss Lopt (nats) for spiking RWKV and the two non-spiking baselines (Section 4.5 ), with backbone cost (dense RWKV minus Transformer) and spiking cost (spiking minus dense RWKV). All runs share data, batch size, schedule, and evaluation; the learning rate is tuned per architecture. The 1019 dense RWKV fit includes a width- 2048 run above the compute-optimal size.
Compute budget C (FLOPs)
d / l
N
3×1017
1×1018
3×1018
1×1019
384/10
57.8M
3.8049
3.7049
–
–
512/12
92.4M
3.7002
3.5283
3.4866
–
640/14
138.9M
3.7384
3.4413
3.3480
3.3141
768/18
215.2M
3.8867
3.4621
3.2978
3.2030
1024/20
375.6M
–
3.5627
3.2558
3.1041
Appendix
Table 3: Final validation loss across the IsoFLOP grid. N is the total parameter count. d is the embedding dimension. l is the number of layers. Bold marks the per-tier minimum; dashes indicate configurations outside the grid at that budget.
C (FLOPs)
Nopt
Dopt
D/N
Lopt
3×1017
101M
0.49B
4.9
3.7014
1×1018
175M
0.95B
5.5
3.4406
3×1018
349M
1.43B
4.1
3.2564
1×1019
693M
2.40B
3.5
3.0619
Appendix
Table 4: Compute-optimal allocation from the IsoFLOP parabola vertices (total parameters).
Figure 3: Fitted learning-rate power law for SpikeGPT.
Dropped tier
α
αtotal
c3e17
0.674
0.615
c1e18
0.616
0.553
c3e18
0.625
0.560
c1e19
0.605
0.532
Full-data
0.624
0.556
Jackknife mean
0.630±0.046
0.565±0.053
Appendix
Table 5: Leave-one-group-out jackknife on the IsoFLOP tiers.
Figure 4: Mean block spike rate versus tokens per parameter; width curves collapse onto one trend.
Spiking language models face a tradeoff between representing continuous semantic features over short temporal windows and retaining costly nonlinear attention operations. We introduce Spora, which jointly designs spike encodings and attention operators. Binary temporal weights let T spikes represent compositional values with up to T bits of capacity, compared with O(log2T) bits for spike-count readout. Unipolar Binary Spiking (UBS) uses thresholds and spike-triggered residual decay to produce non-negative integer codes; Bipolar Binary Spiking (BBS) separates sign and magnitude and learns a scale for signed activations. These representations support accumulation-and-shift dot products and integer-exponent mappings in attention. With four time steps, Spora achieves 76.6 average GLUE score and 44.1 CoLA MCC, improving over SpikeLM by 1.2 and 6.2 points, respectively. Extending BBS to six steps raises these scores to 78.2 and 47.4. Conditional-decay analysis, matched-budget activation-quantization comparisons, event-workload statistics, and fixed-point evaluation further characterize the connection between encoding fidelity and computational cost.
Hanfei Liu, Shuchang Feng, Yanxia Chen +2
Northeastern University, 110819, Shenyang, China · Hebei Key Laboratory of Marine Perception Network and Data Processing, 066004, Qinhuangdao, China
Leveraging their inherent sparse event-driven computation, spiking neural networks (SNNs) offer a promising path toward energy-efficient large language models (LLMs). Time-to-first-spike (TTFS) coding generates at most one spike per neuron within a time window, yielding extremely low firing rates. However, conventional TTFS SNNs are restricted to specific structures, making it challenging to encode certain blocks in LLM -- such as layer normalization and matrix multiplication --using TTFS. To overcome this limitation, we introduce a reference-based strategy specifically to encode the four core LLM components: embedding layers, layer normalization, attention-related operations and dropout. We construct a fully TTFS-based SNN architecture and train it end-to-end. Experiments on modern LLMs like BERT and GPT-2 demonstrate that our approach achieves performance comparable to ANN counterparts on natural language understanding and common-sense reasoning, while a clear gap remains on language modeling perplexity. To the best of our knowledge, this is the first work to scale a spiking LLM to 1.5 billion parameters using TTFS coding. We also report an estimate of spike-related energy; this is a spike-count proxy under an established cost model rather than a measurement on neuromorphic hardware.
Binary spike activations allow a language-model runtime to read only active weight columns and replace multiplications by weight sums. We implement this execution strategy in C++ for an 874M-parameter spike-gated language model. Sparse projections use column-major INT8 weights, integer accumulation, and one scale application per output channel; dense projections retain row-major access and FP32 activations. In a single-thread comparison using an early checkpoint, INT8 achieves 23.31 tokens/s versus 9.82 for FP32, while reducing weight storage from 3355.2 to 1087.4 MiB. A variant using INT4 on dense projections saves a further 17.4% of storage but reduces decode throughput by 46.6%. On an AMD Ryzen 7 5800X, the final INT8 checkpoint achieves 22.63 tokens/s on one thread and 47.90 on four threads; 512-token prefill reaches 94.68 tokens/s on eight threads. A separate ARM output-head case study records higher trimmed decode-window energy metrics for two candidate-verification configurations. The results characterize how activation-specific layouts and quantized kernels support CPU deployment of a spike-gated language model.