NeuronSpark: Selective Spiking State Space Models with Adaptive Computation for Language Modeling

Zhengzheng Tang

cs.AIMar 17, 2026

NeuronSpark: Selective Spiking State Space Models with Adaptive Computation for Language Modeling

Authors: Zhengzheng Tang

Organizations: Boston University

Abstract

Spiking neural networks promise brain-inspired neural dynamics, yet native large-scale spiking language-model pretraining remains largely unexplored relative to conversion, distillation, and backbone-adaptation approaches. We diagnose a central obstacle: standard spiking neurons apply the same firing sensitivity to every token, creating dense- or silent-firing regimes that are difficult to resolve with globally tuned parameters alone. This problem is particularly acute in language modeling, where a single sequence contains tokens of vastly different informational roles, making a uniform firing regime poorly matched to the data. We address this with NeuronSpark, a self-attention-free language model built around input-dependent spiking membrane dynamics, where each neuron's decay rate, gain, and firing threshold adapt to the current token content. Together with gated associative memory and adaptive temporal aggregation, this selectivity enables billion-scale pretraining from random initialization without backbone transfer, conversion, or distillation. We relate the neuron to fixed-parameter spiking models and the memory to linear attention. At 1.1B parameters and 2.1B tokens, NeuronSpark reaches a 40.0 eight-task zero-shot average, exceeds comparable-budget Pythia/Mamba checkpoints, retains 95.2%/93.2% of their fully trained scores, and spontaneously develops a fast--slow hierarchy. At 2,048 tokens, parameter-matched VHDL points reduce operation-energy proxy by 90.1%/67.0% and cycles by 94.1%/80.3% versus Transformer/Mamba; a physical FPGA path reaches 24.000s TTFT and 0.082440 TPS.

Figures & tables

Figure 1: Architecture of NeuronSpark . Left: token embeddings are repeated over $K$ frames, processed by mixed SNN layers, and decoded through an output PLIF neuron, $K$ -frame aggregation, and a tied LM head. Right: the expanded layer view shows the SNNDecoderLayer and SNNAttentionLayer, the selective SNNBlock membrane recurrence, and the PLIF-gated associative-memory path used in place of conventional self-attention.

Figure 2: Token-efficiency views.

Figure 3: Training-loss and matched-step stability diagnostics.

Figure 4: Post-warmup architecture-stability ablation traces and matched-step summary.

Figure 6: Emergent fast–slow temporal organization in the logged pretraining trace.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Status	Prior art	What is new
Input-dependent decay $\beta_{t}$	Adapted	Mamba ( Gu and Dao, 2023 )	Applied to PLIF membrane, not continuous SSM
Input-dependent threshold $V_{\mathrm{th}}(t)$	New	N/A	Per-token threshold; resolves firing-regime rigidity
Input-dependent gain $\alpha_{t}$	New	N/A	Per-token input gain; separate from decay modulation
Soft reset	Known	PLIF ( Fang et al., 2021 )	Used as-is
$K$ -frame temporal expansion	Adapted	PSN ( Fang et al., 2023 )	PonderNet soft aggregation instead of fixed pooling
PonderNet soft aggregation	Adapted	PonderNet ( Banino et al., 2021 )	Applied to SNN temporal frames, not layer depth

Appendix

Table 2: Novelty decomposition. Each component is characterized as: New in this combination, Adapted (known ingredient with a specific modification), or Known (used as-is from prior work).

Setting	Value
Model parameters	1.16B
Layers ( $L$ )	24
Model dimension ( $D$ )	1024
FFN dimension ( $D_{\mathrm{ff}}$ )	3072
Temporal frames ( $K$ )	12
State expansion ( $N$ )	8

Appendix

Table 3: Complete training configuration.

Model	$D$	$D_{\mathrm{ff}}$	$L$	Mem. int.	$K$	$N$	Params	Tokens
NeuronSpark -1.1B	1024	3072	24	4	12	8	1.16B	2.1B

Appendix

Table 4: Model configurations.

Item	Value
Training objective	Next-token prediction
Reported checkpoint	NeuronSpark -1.1B
Training tokens	2.1B
Available large-scale trace endpoint	479K steps
Terminal smoothed loss in available trace	3.5
Precision	bf16

Appendix

Table 5: Training and evaluation summary.

Variant	End step	Final loss	Best loss	Accepted ref.	Rejection signal
MPD-AGL w/o phase-2	4.79K	7.24	6.78 @ 3.76K	6.86	Reversed after early descent
w/o input-dependent PLIF	5.08K	7.69	7.06 @ 3.83K	6.85	Reversed after early descent
HC $\alpha$ decoupled	0.16K	7.72	7.60 @ 0.15K	7.68	No sustained descent
Sinkhorn health	2.05K	7.60	7.41 @ 1.19K	7.14	Stayed in high-loss band
Cortical lateral	4.11K	7.68	7.25 @ 1.19K	6.89	Reversed after early descent

Appendix

Table 6: Matched-step development-stability runs.

Variant	What changed	Diagnostic purpose
MPD-AGL w/o phase-2	Removed cross-layer equalization from modulation-gradient compensation	Tests whether local compensation alone stabilizes selective PLIF parameters
w/o input-dependent PLIF	Replaced token-dependent PLIF modulation by fixed-parameter PLIF dynamics	Tests whether neuron-level input dependence is required for stable membrane updates
HC $\alpha$ decoupled	Used a separate $\alpha$ parameterization from the health-control path	Tests whether input gain should be decoupled from health-control modulation
Sinkhorn health	Replaced the health signal by a Sinkhorn-projected health score	Tests whether projected global health balancing stabilizes neuron utilization
Cortical lateral	Added cross-token causal spike propagation	Tests whether cortical-style lateral propagation helps contextual exchange

Appendix

Table 7: Rejected-variant definitions.

Datapath	Parameters	Energy proxy	Cycles
Transformer, matched	1,264,129,912	$2.768430{\times}10^{12}$	42,242,890
Mamba, matched	1,260,206,464	$0.829721{\times}10^{12}$	12,660,736
NeuronSpark , energy point	1,260,062,732	$0.273821{\times}10^{12}$	17,463,296
NeuronSpark , speed point	1,260,062,732	$0.273847{\times}10^{12}$	2,496,512

Appendix

Table 8: Same-parameter VHDL cost accounting.

Model	Execution path	TTFT (s)	Decode TPS
Transformer	Cortex-A72	73.509	0.024262
Mamba	Cortex-A72	70.683	0.024125
NeuronSpark	Hybrid Arm/PL	24.000	0.082440

Appendix

Table 9: Physical FPGA inference measurements.

Figure 9: Per-token soft temporal-depth diagnostic.

Bucket	$n$	Mean	Std	Median	P10	P90
Articles/determiners	507	3.84	0.70	3.65	3.14	4.87
Pronouns	221	5.01	0.28	5.05	4.60	5.28
Prep./conj./particles	468	4.41	0.71	4.31	3.61	5.45
Aux./copula/modal	242	4.87	0.47	4.76	4.32	5.62
Content word starts	1982	5.80	0.58	5.87	4.94	6.47
Subword pieces	114	5.73	0.40	5.77	5.25	6.21

Appendix

Table 10: Token-category soft temporal-depth statistics.

Explore similar work

Large Language Models with At Most One Spike per Neuron
Sep 4, 2026Zhuoya Zhao, Parsa Omidi, Aref Jafari +1Spiking Neural Networks Spiking Transformers
BiSpikCLM: A Spiking Language Model integrating Softmax-Free Spiking Attention and Spike-Aware Alignment Distillation
Apr 14, 2026Sihang Guo, Chenlin Zhou, Jiaqi Wang +3Spiking Neural Networks Language Modeling
Harbin Institute of Technology, Shenzhen, China · School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University, Shenzhen, China · Peng Cheng Laboratory, Shenzhen, China
Compute-Optimal Scaling Laws for Spiking Language Models
Oct 10, 2026Dan Saunders, Benjamin Ren, Catherine HeLanguage Model Scaling Laws Spiking Neural Networks
Luma AI · X School

Sep 4, 2026J/K move · Enter open · S save

Large Language Models with At Most One Spike per Neuron

Zhuoya Zhao, Parsa Omidi, Aref Jafari, Richard Naud

Spiking Neural Networks Spiking Transformers Language Modeling

Open paper AskPDF arXiv

Leveraging their inherent sparse event-driven computation, spiking neural networks (SNNs) offer a promising path toward energy-efficient large language models (LLMs). Time-to-first-spike (TTFS) coding generates at most one spike per neuron within a time window, yielding extremely low firing rates. However, conventional TTFS SNNs are restricted to specific structures, making it challenging to encode certain blocks in LLM -- such as layer normalization and matrix multiplication --using TTFS. To overcome this limitation, we introduce a reference-based strategy specifically to encode the four core LLM components: embedding layers, layer normalization, attention-related operations and dropout. We construct a fully TTFS-based SNN architecture and train it end-to-end. Experiments on modern LLMs like BERT and GPT-2 demonstrate that our approach achieves performance comparable to ANN counterparts on natural language understanding and common-sense reasoning, while a clear gap remains on language modeling perplexity. To the best of our knowledge, this is the first work to scale a spiking LLM to 1.5 billion parameters using TTFS coding. We also report an estimate of spike-related energy; this is a spike-count proxy under an established cost model rather than a measurement on neuromorphic hardware.

Abstract

Explore similar work

Large Language Models with At Most One Spike per Neuron

BiSpikCLM: A Spiking Language Model integrating Softmax-Free Spiking Attention and Spike-Aware Alignment Distillation

Compute-Optimal Scaling Laws for Spiking Language Models