Spiking Language Models

Follow this topic Ask about this topic

Latest papers

12 papers3 papers in the last four weeks, level with the four weeks before. 0.0% of all new papers.

Compute-Optimal Scaling Laws for Spiking Language Models
Oct 10, 2026Dan Saunders, Benjamin Ren, Catherine HeSpiking Language Models Language Model Scaling Laws
Luma AI · X School
Rethinking the Tradeoff Between Temporal Encoding and Nonlinear Computation in Spiking Language Models
Oct 7, 2026Hanfei Liu, Shuchang Feng, Yanxia Chen +2Spiking Language Models Language Modeling
Northeastern University, 110819, Shenyang, China · Hebei Key Laboratory of Marine Perception Network and Data Processing, 066004, Qinhuangdao, China
QuantaSpike: Short-Window Spike-Driven Quantization for Large Language Models
Sep 28, 2026Bang Hu, Guowei Zhu, Changze Lv +4LLM Quantization Spiking Neural Networks
School of Computer Science, Fudan University, Shanghai, China
URCHIN: A Horizontal Spiking Language Model for Data-Constrained Pretraining
Sep 12, 2026Po-Han ChiangSpiking Language Models Language Modeling
National Yang Ming Chiao Tung University
Large Language Models with At Most One Spike per Neuron
Sep 4, 2026Zhuoya Zhao, Parsa Omidi, Aref Jafari +1Spiking Language Models Spiking Neural Networks
SpikeOPD: Stable On-Policy Distillation for Autoregressive Spiking Language Models
Aug 28, 2026Enqiao Lu, Xingrui Yu, Yiwei Fu +7Spiking Language Models Spiking Neural Networks
The Chinese University of Hong Kong, Shenzhen · Agency for Science, Technology and Research (A*STAR), Singapore · Nanyang Technological University, Singapore +3
Spike-Aware INT8 Execution for Spiking Language Models on Commodity CPUs
Jun 2, 2026Ting LiuLLM Quantization On-Device Language Model Inference
SymbolicLight Research Foshan, Guangdong, China
SymbolicLight V1: Spike-Gated Dual-Path Language Modeling at High Encoder Spike Sparsity
May 20, 2026Ting LiuLanguage Model Pretraining Spiking Neural Networks
SymbolicLight Research Foshan, Guangdong, China
BiSpikCLM: A Spiking Language Model integrating Softmax-Free Spiking Attention and Spike-Aware Alignment Distillation
Apr 14, 2026Sihang Guo, Chenlin Zhou, Jiaqi Wang +3Spiking Language Models Spiking Neural Networks
Harbin Institute of Technology, Shenzhen, China · School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University, Shenzhen, China · Peng Cheng Laboratory, Shenzhen, China
Winner-Take-All Spiking Transformer for Language Modeling
Apr 13, 2026Chenlin Zhou, Sihang Guo, Jiaqi Wang +6Spiking Transformers Spiking Language Models
School of Electronic and Computer Engineering, Shenzhen Graduate School, Peking University · Pengcheng Laboratory · Harbin Institute of Technology +1
Spike-driven Large Language Model
Apr 11, 2026Han Xu, Xuerui Qiu, Baiyu Chen +7Spiking Language Models Spiking Neural Networks
Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China · Brain-Inspired Models Group, Beijing Academy of Artificial Intelligence, Beijing 100862, China +7
NeuronSpark: Selective Spiking State Space Models with Adaptive Computation for Language Modeling
Mar 17, 2026Zhengzheng TangSpiking Language Models Spiking Neural Networks
Boston University

Oct 10, 2026J/K move · Enter open · S save

Compute-Optimal Scaling Laws for Spiking Language Models

Dan Saunders, Benjamin Ren, Catherine He

Luma AI · X School

Spiking Language Models Language Model Scaling Laws Spiking Neural Networks Language Modeling

Open paper AskPDF arXiv

Spiking neural networks target energy-efficient language modeling through sparse, binary activations, but the efficiency-accuracy tradeoff of spiking language models is uncharacterized at scale. We present the first Chinchilla-style IsoFLOP scaling study of a spiking language model, sweeping SpikeGPT-family models (RWKV backbone) from width 384 to 1536 across compute budgets from

3\times10^{17}

10^{19}

FLOPs on FineWeb-EDU. We report two findings. First, the compute-optimal frontier is

N_{opt} \propto C^{0.56}

, close to the 0.50 that Hoffmann et al. report for dense Transformers, so spiking does not change how parameters and data should be traded off under a compute budget. Second, against non-spiking RWKV and Transformer baselines trained the same way, the loss gap splits into a backbone cost of about 0.16 nats that does not change with compute and a spiking cost that falls from 0.34 to about 0.23 nats and then levels off. These results give the first compute-optimal recipe for spiking language models and quantify the accuracy cost of their sparsity.