URCHIN: A Horizontal Spiking Language Model for Data-Constrained Pretraining

Po-Han Chiang

q-bio.NCSep 12, 2026

URCHIN: A Horizontal Spiking Language Model for Data-Constrained Pretraining

Authors: Po-Han Chiang

Organizations: National Yang Ming Chiao Tung University

Abstract

The BabyLM challenge measures how much language a model can learn from developmentally-plausible, child-scale data rather than internet-scale corpora, yet prior language models forgo the biological constraints of the neural circuitry that acquires human language: spiking neurons separated into excitatory and inhibitory populations wired by a recurrent lateral connectome. This paper presents URCHIN (Unified Recurrent Connectome with Horizontal Integrate-and-fire Neurons), which applies the Parallelized Hierarchical Connectome Spiking State-space Model (PHCSSM) to language modeling: leaky integrate-and-fire neurons coupled by a Dale's-law lateral connectome resolve each token through a multi-transmission loop that recirculates activity to a fixed point. The instantiation is deliberately minimal: a single horizontal layer of 128 neurons, no attention, and 4.23M parameters. Two implementations share one set of weights and produce identical benchmark scores, so URCHIN is trained once and deployed either way with no conversion step: a parallel state-space model (SSM) scan that is GPU-efficient for training, or an event-driven recurrent spiking neural network (RSNN) with constant-cost inference for CPU or neuromorphic edge deployment. Across all three BabyLM tracks (Strict-100M, Strict-Small, and Multilingual), URCHIN offers a biologically plausible, efficient, and directly deployable reference point.

Figures & tables

Figure 1: URCHIN architecture. The embedding of the current token is injected as input current into a single layer of excitatory (red) and inhibitory (blue) LIF neurons joined only by lateral connections; a linear LM head reads the membrane voltage of the same neurons out to the 16,384-entry vocabulary. Each token is processed by a multi-transmission loop, a fixed number of lateral transmission steps that settles to a fixed point. Parallel URCHIN evaluates this with parallel scans for training; Serial URCHIN runs the identical dynamics event-driven for deployment.

Component	Trainable	Total (stored)
Token embedding	2.10M	2.10M
LM head	2.10M	2.10M
Spiking core	17K	17K
Fixed Dale + input masks	0	16.6K
Total	4.21M	4.23M

Table 1: Parameter budget of the submitted model. Trainable parameters against total stored tensors. The fixed Dale’s-law connectivity mask is stored in the checkpoint but is not trained.

Metric	URCHIN	Baseline- GPT2	Baseline- Interaction
Overall Average	37.01	37.38	38.71
BLiMP	61.29	65.23	63.07
BLiMP-Supplement	56.05	57.25	58.13
EWoK	50.29	50.63	51.45
Entity-Tracking (deb.)	18.16	19.10	19.62
COMPS	50.70	51.81	51.29

Table 2: Strict-Small official results: URCHIN against the two official BabyLM baselines, the GPT-2 baseline ( BabyLM-2026-Baseline-GPT2-Strict-Small ) and the interaction baseline ( BabyLM-2026-Baseline-Strict-Small-Interaction ). Bold marks the metrics on which URCHIN matches or exceeds both baselines. AoA is decoupled to zero in URCHIN by design.

Metric	URCHIN	Baseline- GPT2	Baseline- Interaction
Overall Average	37.54	40.73	39.89
BLiMP	63.08	74.73	72.89
BLiMP-Supplement	52.57	65.00	65.09
EWoK	49.97	54.37	54.23
Entity-Tracking (deb.)	18.39	16.91	15.89
COMPS	52.73	55.85	55.37

Table 3: Strict-100M official results: URCHIN against the two official BabyLM baselines, the GPT-2 baseline ( BabyLM-2026-Baseline-GPT2-Strict ) and the interaction baseline ( BabyLM-2026-Baseline-Strict-Interaction ), formatted as in Table 2 . Bold marks the metrics on which URCHIN matches or exceeds both baselines.

URCHIN
	En	Nl	Zh	Overall
Zero-shot avg.	48.60	48.72	46.70	48.01
Fine-tune avg.	39.28	39.55	41.47	40.10
Language avg.	43.27	43.83	43.57	43.56
Baseline-GPT2 (en/nl/zh)
Zero-shot avg.	53.63	55.05	47.25	51.98

Table 4: Multilingual official results: per-language zero-shot, fine-tune, and language averages for URCHIN and the two official GPT-2 baselines, BabyLM-2026-Baseline-GPT2-en_nld_zho_equal (en/nl/zh) and babylm-multilingual-gpt2-baseline (multilingual). All figures are from the official leaderboard. The Overall column is the mean of the three language averages; the zero-shot average includes the GlobalPIQA surprise task and the fine-tune average includes the POS surprise task. Bold marks cells where URCHIN matches or exceeds both baselines.

Figure 2: Single-sequence inference throughput of Parallel URCHIN (24 transmission steps, evaluation setting) and Serial URCHIN (event-driven), the mean of the three homogeneous tracks. On the H200 GPU (a) parallel throughput scales with context length while serial is flat; on a single CPU core (b) the event-driven serial path is about seven times faster. Batch size one.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Pretraining
Hyperparameter	Strict-Small	Strict-100M	Multilingual
Corpus	babylm_10m	babylm_100m	babylm_ml_100m
Tokenizer vocabulary size	16,384	16,384	16,384
Context length	2,048	2,048	2,048
Synaptic delay	1	1	1
Sensory drive scale ( drive_alpha )	0.3	0.3	0.3

Appendix

Table 5: Pretraining and fine-tuning hyperparameters for the three tracks. The architecture (a single 128-neuron layer, 80:20 E/I, full [1,1,1,1] lateral connectivity, LIF neurons, no STDP) is identical across tracks. Fine-tuning tasks: the strict tracks use the seven (Super)GLUE tasks (boolq, mnli, mrpc, multirc, qqp, rte, wsc); the Multilingual track uses nine tasks (arc, belebele, bmlama, include, mnli, sib200, truthfulqa, xnli, pos), each in English, Dutch, and Chinese. The Multilingual POS fine-tuning matches the Multilingual classification fine-tuning except for the learning rate and batch size marked in POS, and additionally uses a maximum sequence length of 128 and a warmup ratio and weight decay of 0.1 and 0.01.

Track	Task	N	Flips	$\Delta$
Strict-Small	BLiMP (ZS)	59,875	4	$+0.000033$
Strict-Small	EWoK (ZS)	7,618	1	$-0.000131$
Strict-Small	MNLI (FT)	4,908	15	$+0.000204$
Strict-Small	QQP (FT)	20,215	9	$-0.000049$
Strict-Small	BoolQ (FT)	1,635	4	$-0.001223$
Strict-Small	RTE (FT)	139	1	$+0.007194$

Appendix

Table 6: Item-level parallel-serial disagreements. Across the three tracks the two modes are compared on 66 benchmarks and tasks, together covering 445,447 individual predictions. 48 of the 66 are perfectly item-identical between the parallel and serial modes; only the 18 listed below disagree at all, and only on 90 of the 445,447 predictions (0.02 percent). Each row gives the item count N, the number of flipped items, and the change in the reported score

\Delta

(serial minus parallel); every benchmark or task not listed has zero flips and

\Delta=0.000000

. The entire Multilingual zero-shot suite falls in the identical set: all 16 of its benchmarks, covering 170,759 of these predictions over English, Dutch, and Chinese, agree item for item. Reading is excluded, as it is scored by surprisal correlation rather than accuracy. ZS = zero-shot, FT = fine-tune.

Spiking Language Models

See all 12 papers

NeuronSpark: Selective Spiking State Space Models with Adaptive Computation for Language Modeling
Mar 17, 2026Zhengzheng TangSpiking Language Models Spiking Neural Networks
Boston University
Large Language Models with At Most One Spike per Neuron
Sep 4, 2026Zhuoya Zhao, Parsa Omidi, Aref Jafari +1Spiking Language Models Spiking Neural Networks
Compute-Optimal Scaling Laws for Spiking Language Models
Oct 10, 2026Dan Saunders, Benjamin Ren, Catherine HeSpiking Language Models Language Model Scaling Laws
Luma AI · X School

Mar 17, 2026J/K move · Enter open · S save

NeuronSpark: Selective Spiking State Space Models with Adaptive Computation for Language Modeling

Zhengzheng Tang

Boston University

Spiking Language Models Spiking Neural Networks Language Model Pretraining Language Modeling Efficient Language Model Inference State Space Models

Open paper AskPDF arXiv

Spiking neural networks promise brain-inspired neural dynamics, yet native large-scale spiking language-model pretraining remains largely unexplored relative to conversion, distillation, and backbone-adaptation approaches. We diagnose a central obstacle: standard spiking neurons apply the same firing sensitivity to every token, creating dense- or silent-firing regimes that are difficult to resolve with globally tuned parameters alone. This problem is particularly acute in language modeling, where a single sequence contains tokens of vastly different informational roles, making a uniform firing regime poorly matched to the data. We address this with NeuronSpark, a self-attention-free language model built around input-dependent spiking membrane dynamics, where each neuron's decay rate, gain, and firing threshold adapt to the current token content. Together with gated associative memory and adaptive temporal aggregation, this selectivity enables billion-scale pretraining from random initialization without backbone transfer, conversion, or distillation. We relate the neuron to fixed-parameter spiking models and the memory to linear attention. At 1.1B parameters and 2.1B tokens, NeuronSpark reaches a 40.0 eight-task zero-shot average, exceeds comparable-budget Pythia/Mamba checkpoints, retains 95.2%/93.2% of their fully trained scores, and spontaneously develops a fast--slow hierarchy. At 2,048 tokens, parameter-matched VHDL points reduce operation-energy proxy by 90.1%/67.0% and cycles by 94.1%/80.3% versus Transformer/Mamba; a physical FPGA path reaches 24.000s TTFT and 0.082440 TPS.

Abstract

Spiking Language Models

Explore similar work

NeuronSpark: Selective Spiking State Space Models with Adaptive Computation for Language Modeling

Large Language Models with At Most One Spike per Neuron

Compute-Optimal Scaling Laws for Spiking Language Models