Compute-Optimal Scaling Laws for Spiking Language Models

Dan Saunders; Benjamin Ren; Catherine He

cs.LGOct 10, 2026

Compute-Optimal Scaling Laws for Spiking Language Models

Authors: Dan Saunders, Benjamin Ren, Catherine He

Organizations: Luma AI · X School

Abstract

Spiking neural networks target energy-efficient language modeling through sparse, binary activations, but the efficiency-accuracy tradeoff of spiking language models is uncharacterized at scale. We present the first Chinchilla-style IsoFLOP scaling study of a spiking language model, sweeping SpikeGPT-family models (RWKV backbone) from width 384 to 1536 across compute budgets from $3\times10^{17}$ to $10^{19}$ FLOPs on FineWeb-EDU. We report two findings. First, the compute-optimal frontier is $N_{opt} \propto C^{0.56}$ , close to the 0.50 that Hoffmann et al. report for dense Transformers, so spiking does not change how parameters and data should be traded off under a compute budget. Second, against non-spiking RWKV and Transformer baselines trained the same way, the loss gap splits into a backbone cost of about 0.16 nats that does not change with compute and a spiking cost that falls from 0.34 to about 0.23 nats and then levels off. These results give the first compute-optimal recipe for spiking language models and quantify the accuracy cost of their sparsity.

Figures & tables

Panel of Figure 1 — Figure 1: (a) IsoFLOP profiles: validation loss $L$ versus model size $N$ at each compute budget $C$ , with an interior compute-optimal minimum per tier. (b) The same runs as learning curves (loss versus compute $C=6N_{\backslash E}D$ , colored by budget); their lower envelope is the compute-optimal frontier $L_{\mathrm{opt}}(C)$ , which extrapolates a decade beyond the fit grid to the out-of-sample hero run at $C=10^{20}$ (measured $2.76$ vs. predicted $2.78$ nats).

Convention	$N$	$\alpha$ (IsoFLOP)	$\alpha$ (Approach 3)	$D/N$ ( $3{\times}10^{17}{\to}10^{19}$ )
Strict non-embedding (Kaplan)	$13ld^{2}$	0.748	0.745	$25.1\to 4.7$
FLOP-matmul (grid build)	$13ld^{2}+vd$	0.624	0.644	$9.2\to 4.1$
Total (Chinchilla, primary)	$13ld^{2}+2vd$	0.556	0.595	$4.9\to 3.5$

Table 1: Compute-optimal exponent

\alpha

(

N_{\mathrm{opt}}\propto C^{\alpha}

) under three parameter-counting conventions, estimated in two ways: the IsoFLOP vertex fit (Sec. 4.1 ) and a Chinchilla Approach 3 parametric fit

L=E+A/N^{a}+B/D^{b}

(Appendix G ). The two estimators agree within each convention.

D/N

is the compute-optimal token-to-parameter ratio at the smallest and largest budget using IsoFLOP fit.

Figure 2: Spiking RWKV compared with non-spiking baselines. (a) Compute-optimal loss $L_{\mathrm{opt}}(C)$ for each architecture. (b) Backbone cost (dense RWKV minus Transformer) and spiking cost (spiking minus dense RWKV) at each budget.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

$C$ (FLOPs)	Transformer	Dense RWKV	Spiking RWKV	backbone cost	spiking cost
$3\times 10^{17}$	3.195	3.363	3.701	0.168	0.338
$1\times 10^{18}$	3.012	3.177	3.441	0.166	0.263
$3\times 10^{18}$	2.878	3.030	3.256	0.151	0.227
$1\times 10^{19}$	2.671	2.835	3.061	0.164	0.225

Appendix

Table 2: Compute-optimal loss

L_{\mathrm{opt}}

(nats) for spiking RWKV and the two non-spiking baselines (Section 4.5 ), with backbone cost (dense RWKV minus Transformer) and spiking cost (spiking minus dense RWKV). All runs share data, batch size, schedule, and evaluation; the learning rate is tuned per architecture. The

10^{19}

dense RWKV fit includes a width-

2048

run above the compute-optimal size.

		Compute budget $C$ (FLOPs)
$d$ / $l$	$N$	$3\times 10^{17}$	$1\times 10^{18}$	$3\times 10^{18}$	$1\times 10^{19}$
384/10	57.8M	3.8049	3.7049	–	–
512/12	92.4M	3.7002	3.5283	3.4866	–
640/14	138.9M	3.7384	3.4413	3.3480	3.3141
768/18	215.2M	3.8867	3.4621	3.2978	3.2030
1024/20	375.6M	–	3.5627	3.2558	3.1041

Appendix

Table 3: Final validation loss across the IsoFLOP grid.

N

is the total parameter count.

d

is the embedding dimension.

l

is the number of layers. Bold marks the per-tier minimum; dashes indicate configurations outside the grid at that budget.

$C$ (FLOPs)	$N_{\mathrm{opt}}$	$D_{\mathrm{opt}}$	$D/N$	$L_{\mathrm{opt}}$
$3\times 10^{17}$	101M	0.49B	4.9	3.7014
$1\times 10^{18}$	175M	0.95B	5.5	3.4406
$3\times 10^{18}$	349M	1.43B	4.1	3.2564
$1\times 10^{19}$	693M	2.40B	3.5	3.0619

Appendix

Table 4: Compute-optimal allocation from the IsoFLOP parabola vertices (total parameters).

Figure 3: Fitted learning-rate power law for SpikeGPT.

Dropped tier	$\alpha$	$\alpha_{\mathrm{total}}$
c3e17	0.674	0.615
c1e18	0.616	0.553
c3e18	0.625	0.560
c1e19	0.605	0.532
Full-data	0.624	0.556
Jackknife mean	$0.630\pm 0.046$	$0.565\pm 0.053$

Appendix

Table 5: Leave-one-group-out jackknife on the IsoFLOP tiers.

Figure 4: Mean block spike rate versus tokens per parameter; width curves collapse onto one trend.

Explore similar work

Rethinking the Tradeoff Between Temporal Encoding and Nonlinear Computation in Spiking Language Models
Oct 7, 2026Hanfei Liu, Shuchang Feng, Yanxia Chen +2Language Modeling Spiking Transformers
Northeastern University, 110819, Shenyang, China · Hebei Key Laboratory of Marine Perception Network and Data Processing, 066004, Qinhuangdao, China
Large Language Models with At Most One Spike per Neuron
Sep 4, 2026Zhuoya Zhao, Parsa Omidi, Aref Jafari +1Spiking Neural Networks Spiking Transformers
Spike-Aware INT8 Execution for Spiking Language Models on Commodity CPUs
Jun 2, 2026Ting LiuLLM Quantization On-Device Language Model Inference
SymbolicLight Research Foshan, Guangdong, China

Oct 7, 2026J/K move · Enter open · S save

Rethinking the Tradeoff Between Temporal Encoding and Nonlinear Computation in Spiking Language Models

Hanfei Liu, Shuchang Feng, Yanxia Chen, Changzeng Fu+1

Northeastern University, 110819, Shenyang, China · Hebei Key Laboratory of Marine Perception Network and Data Processing, 066004, Qinhuangdao, China

Language Modeling Spiking Transformers Neural Encoding

Open paper AskPDF arXiv

Spiking language models face a tradeoff between representing continuous semantic features over short temporal windows and retaining costly nonlinear attention operations. We introduce Spora, which jointly designs spike encodings and attention operators. Binary temporal weights let

T

spikes represent compositional values with up to

T

bits of capacity, compared with

O(\log_2 T)

bits for spike-count readout. Unipolar Binary Spiking (UBS) uses thresholds and spike-triggered residual decay to produce non-negative integer codes; Bipolar Binary Spiking (BBS) separates sign and magnitude and learns a scale for signed activations. These representations support accumulation-and-shift dot products and integer-exponent mappings in attention. With four time steps, Spora achieves 76.6 average GLUE score and 44.1 CoLA MCC, improving over SpikeLM by 1.2 and 6.2 points, respectively. Extending BBS to six steps raises these scores to 78.2 and 47.4. Conditional-decay analysis, matched-budget activation-quantization comparisons, event-workload statistics, and fixed-point evaluation further characterize the connection between encoding fidelity and computational cost.

Abstract

Explore similar work

Rethinking the Tradeoff Between Temporal Encoding and Nonlinear Computation in Spiking Language Models

Large Language Models with At Most One Spike per Neuron

Spike-Aware INT8 Execution for Spiking Language Models on Commodity CPUs