NeuronSpark: Selective Spiking State Space Models with Adaptive Computation for Language Modeling

Zhengzheng Tang

cs.AIMar 17, 2026

NeuronSpark: Selective Spiking State Space Models with Adaptive Computation for Language Modeling

Authors: Zhengzheng Tang

Organizations: Boston University

Abstract

Spiking neural networks promise brain-inspired neural dynamics, yet native large-scale spiking language-model pretraining remains largely unexplored relative to conversion, distillation, and backbone-adaptation approaches. We diagnose a central obstacle: standard spiking neurons apply the same firing sensitivity to every token, creating dense- or silent-firing regimes that are difficult to resolve with globally tuned parameters alone. This problem is particularly acute in language modeling, where a single sequence contains tokens of vastly different informational roles, making a uniform firing regime poorly matched to the data. We address this with NeuronSpark, a self-attention-free language model built around input-dependent spiking membrane dynamics, where each neuron's decay rate, gain, and firing threshold adapt to the current token content. Together with gated associative memory and adaptive temporal aggregation, this selectivity enables billion-scale pretraining from random initialization without backbone transfer, conversion, or distillation. We relate the neuron to fixed-parameter spiking models and the memory to linear attention. At 1.1B parameters and 2.1B tokens, NeuronSpark reaches a 40.0 eight-task zero-shot average, exceeds comparable-budget Pythia/Mamba checkpoints, retains 95.2%/93.2% of their fully trained scores, and spontaneously develops a fast--slow hierarchy. At 2,048 tokens, parameter-matched VHDL points reduce operation-energy proxy by 90.1%/67.0% and cycles by 94.1%/80.3% versus Transformer/Mamba; a physical FPGA path reaches 24.000s TTFT and 0.082440 TPS.

Figures & tables

Figure 1: Architecture of NeuronSpark . Left: token embeddings are repeated over $K$ frames, processed by mixed SNN layers, and decoded through an output PLIF neuron, $K$ -frame aggregation, and a tied LM head. Right: the expanded layer view shows the SNNDecoderLayer and SNNAttentionLayer, the selective SNNBlock membrane recurrence, and the PLIF-gated associative-memory path used in place of conventional self-attention.

Figure 2: Token-efficiency views.

Figure 3: Training-loss and matched-step stability diagnostics.

Figure 4: Post-warmup architecture-stability ablation traces and matched-step summary.

Figure 6: Emergent fast–slow temporal organization in the logged pretraining trace.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Status	Prior art	What is new
Input-dependent decay $\beta_{t}$	Adapted	Mamba ( Gu and Dao, 2023 )	Applied to PLIF membrane, not continuous SSM
Input-dependent threshold $V_{\mathrm{th}}(t)$	New	N/A	Per-token threshold; resolves firing-regime rigidity
Input-dependent gain $\alpha_{t}$	New	N/A	Per-token input gain; separate from decay modulation
Soft reset	Known	PLIF ( Fang et al., 2021 )	Used as-is
$K$ -frame temporal expansion	Adapted	PSN ( Fang et al., 2023 )	PonderNet soft aggregation instead of fixed pooling
PonderNet soft aggregation	Adapted	PonderNet ( Banino et al., 2021 )	Applied to SNN temporal frames, not layer depth

Appendix

Table 2: Novelty decomposition. Each component is characterized as: New in this combination, Adapted (known ingredient with a specific modification), or Known (used as-is from prior work).

Setting	Value
Model parameters	1.16B
Layers ( $L$ )	24
Model dimension ( $D$ )	1024
FFN dimension ( $D_{\mathrm{ff}}$ )	3072
Temporal frames ( $K$ )	12
State expansion ( $N$ )	8

Appendix

Table 3: Complete training configuration.

Model	$D$	$D_{\mathrm{ff}}$	$L$	Mem. int.	$K$	$N$	Params	Tokens
NeuronSpark -1.1B	1024	3072	24	4	12	8	1.16B	2.1B

Appendix

Table 4: Model configurations.

Item	Value
Training objective	Next-token prediction
Reported checkpoint	NeuronSpark -1.1B
Training tokens	2.1B
Available large-scale trace endpoint	479K steps
Terminal smoothed loss in available trace	3.5
Precision	bf16

Appendix

Table 5: Training and evaluation summary.

Variant	End step	Final loss	Best loss	Accepted ref.	Rejection signal
MPD-AGL w/o phase-2	4.79K	7.24	6.78 @ 3.76K	6.86	Reversed after early descent
w/o input-dependent PLIF	5.08K	7.69	7.06 @ 3.83K	6.85	Reversed after early descent
HC $\alpha$ decoupled	0.16K	7.72	7.60 @ 0.15K	7.68	No sustained descent
Sinkhorn health	2.05K	7.60	7.41 @ 1.19K	7.14	Stayed in high-loss band
Cortical lateral	4.11K	7.68	7.25 @ 1.19K	6.89	Reversed after early descent

Appendix

Table 6: Matched-step development-stability runs.

Variant	What changed	Diagnostic purpose
MPD-AGL w/o phase-2	Removed cross-layer equalization from modulation-gradient compensation	Tests whether local compensation alone stabilizes selective PLIF parameters
w/o input-dependent PLIF	Replaced token-dependent PLIF modulation by fixed-parameter PLIF dynamics	Tests whether neuron-level input dependence is required for stable membrane updates
HC $\alpha$ decoupled	Used a separate $\alpha$ parameterization from the health-control path	Tests whether input gain should be decoupled from health-control modulation
Sinkhorn health	Replaced the health signal by a Sinkhorn-projected health score	Tests whether projected global health balancing stabilizes neuron utilization
Cortical lateral	Added cross-token causal spike propagation	Tests whether cortical-style lateral propagation helps contextual exchange

Appendix

Table 7: Rejected-variant definitions.

Datapath	Parameters	Energy proxy	Cycles
Transformer, matched	1,264,129,912	$2.768430{\times}10^{12}$	42,242,890
Mamba, matched	1,260,206,464	$0.829721{\times}10^{12}$	12,660,736
NeuronSpark , energy point	1,260,062,732	$0.273821{\times}10^{12}$	17,463,296
NeuronSpark , speed point	1,260,062,732	$0.273847{\times}10^{12}$	2,496,512

Appendix

Table 8: Same-parameter VHDL cost accounting.

Model	Execution path	TTFT (s)	Decode TPS
Transformer	Cortex-A72	73.509	0.024262
Mamba	Cortex-A72	70.683	0.024125
NeuronSpark	Hybrid Arm/PL	24.000	0.082440

Appendix

Table 9: Physical FPGA inference measurements.

Figure 9: Per-token soft temporal-depth diagnostic.

Bucket	$n$	Mean	Std	Median	P10	P90
Articles/determiners	507	3.84	0.70	3.65	3.14	4.87
Pronouns	221	5.01	0.28	5.05	4.60	5.28
Prep./conj./particles	468	4.41	0.71	4.31	3.61	5.45
Aux./copula/modal	242	4.87	0.47	4.76	4.32	5.62
Content word starts	1982	5.80	0.58	5.87	4.94	6.47
Subword pieces	114	5.73	0.40	5.77	5.25	6.21

Appendix

Table 10: Token-category soft temporal-depth statistics.

Explore similar work

Sep 4, 2026

Large Language Models with At Most One Spike per Neuron

Leveraging their inherent sparse event-driven computation, spiking neural networks (SNNs) offer a promising path toward energy-efficient large language models (LLMs). Time-to-first-spike (TTFS) coding generates at most one spike per neuron within a time window, yielding extremely low firing rates. However, conventional TTFS SNNs are restricted to specific structures, making it challenging to encode certain blocks in LLM -- such as layer normalization and matrix multiplication --using TTFS. To overcome this limitation, we introduce a reference-based strategy specifically to encode the four core LLM components: embedding layers, layer normalization, attention-related operations and dropout. We construct a fully TTFS-based SNN architecture and train it end-to-end. Experiments on modern LLMs like BERT and GPT-2 demonstrate that our approach achieves performance comparable to ANN counterparts on natural language understanding and common-sense reasoning, while a clear gap remains on language modeling perplexity. To the best of our knowledge, this is the first work to scale a spiking LLM to 1.5 billion parameters using TTFS coding. We also report an estimate of spike-related energy; this is a spike-count proxy under an established cost model rather than a measurement on neuromorphic hardware.

Zhuoya Zhao, Parsa Omidi, Aref Jafari +1

Spiking Neural Networks Spiking Transformers Language Modeling

Ask PDF

Apr 14, 2026

BiSpikCLM: A Spiking Language Model integrating Softmax-Free Spiking Attention and Spike-Aware Alignment Distillation

Spiking Neural Networks (SNNs) offer promising energy-efficient alternatives to large language models (LLMs) due to their event-driven nature and ultra-low power consumption. However, to preserve capacity, most existing spiking LLMs still incur intensive floating-point matrix multiplication (MatMul) and nonlinearities, or training difficulties arising from the complex spatiotemporal dynamics. To address these challenges, we propose BiSpikCLM, the first fully binary spiking MatMul-free causal language model. BiSpikCLM introduces Softmax-Free Spiking Attention (SFSA), eliminating softmax and floating-point operations in autoregressive language modeling. For efficient training, we introduce Spike-Aware Alignment Distillation (SpAD), which aligns ANN teacher and SNN student across embeddings, attention maps, intermediate features, and output logits. SpAD framework allows BiSpikCLM to reach comparable performance to ANN counterparts using substantially fewer training tokens (e.g., only 5.6% of the tokens for the 1.3B model). As a result, BiSpikCLM achieves competitive performance at only 4.16% - 5.87% of the computational cost on natural language generation tasks. Our results highlight the feasibility and effectiveness of fully binary spike-driven LLMs and establish the distillation as a promising pathway for brain-inspired spiking NLP.

Sihang Guo, Chenlin Zhou, Jiaqi Wang +3

Harbin Institute of Technology, Shenzhen, China · School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University, Shenzhen, China · Peng Cheng Laboratory, Shenzhen, China

Spiking Neural Networks Language Modeling LLM Inference Acceleration Spiking Transformers Binary Neural Networks Language Model Distillation

Ask PDF

Oct 10, 2026

Compute-Optimal Scaling Laws for Spiking Language Models

Spiking neural networks target energy-efficient language modeling through sparse, binary activations, but the efficiency-accuracy tradeoff of spiking language models is uncharacterized at scale. We present the first Chinchilla-style IsoFLOP scaling study of a spiking language model, sweeping SpikeGPT-family models (RWKV backbone) from width 384 to 1536 across compute budgets from

3\times10^{17}

10^{19}

FLOPs on FineWeb-EDU. We report two findings. First, the compute-optimal frontier is

N_{opt} \propto C^{0.56}

, close to the 0.50 that Hoffmann et al. report for dense Transformers, so spiking does not change how parameters and data should be traded off under a compute budget. Second, against non-spiking RWKV and Transformer baselines trained the same way, the loss gap splits into a backbone cost of about 0.16 nats that does not change with compute and a spiking cost that falls from 0.34 to about 0.23 nats and then levels off. These results give the first compute-optimal recipe for spiking language models and quantify the accuracy cost of their sparsity.

Dan Saunders, Benjamin Ren, Catherine He

Luma AI · X School

Language Model Scaling Laws Spiking Neural Networks Language Modeling

Ask PDF