Intermittent ``spikes'' in loss, during the pre-training of Large Language Models (LLMs), remains a mysterious phenomenon both theoretically and empirically. In this paper, we show that simple quadratic models with heavy-tailed, power-law distributed inputs can replicate these spikes. We introduce a theoretical framework and simulation algorithm to model per-layer training dynamics of Deep Neural Networks (DNNs), using such quadratic models. In a case study on an Adam hyperparameter configuration (where β2 is set close to 1) known to induce spikes in Transformer LMs, our simulation successfully predicts bumps indicating spiky behavior from a given checkpoint. Furthermore, we derive an appropriate β2 schedule from observed metrics, which empirically improves performance.
Figures & tables
Figure 1: Training of a Transformer LM. Comparing a “Normal” training with two “Spiky” trainings. “Normal” and “Spiky1” differ in the β2 hyperparameter of Adam only, while “Spiky1” and “Spiky2” differ in the random-seed of model initialization only. Training spikes can occur at different steps between “Spiky1” and “Spiky2”, while “normal fluctuations” in training loss remain consistent.
Figure 2: The validation loss of PLRF, from training step 0 to 1k. We compare α∈{0.3,0.9} , for SGD and Adam, with different learning-rates. Each plot shows 3 random seeds (to distinguish initialization and data-shuffle separately), and learning-rates are chosen such that training converges, nearly diverges, or diverges, respectively. For SGD, the required learning-rates are calculated from the threshold γ given in formula ( 2 ). For Adam, the learning-rates are found by manual grid search.
Experiment
Train
Eval δt,sw
Eval E(htht⊤)
E(∇Wt) & E(∥∇BWt∥2)
\trace(E(∇2Wt))
Simulation
Data size
25.5B tokens
336M tokens
336M tokens
336M tokens
336M tokens
1500 steps
Model size
1.3B (24 blocks)
24 ffn_0
all layers
all layers
6 ffn_0
1 layer
Time
40 hrs
1.1 hrs
8 min
15 min
1.7 hrs
5 sec
Machine
TPUv4 ×32
TPUv4 ×32
TPUv4 ×32
TPUv4 ×32
TPUv4 ×32
CPU
Table 1: Summary of compute resources for running experiments in this paper.
Figure 3: Simulation of the layer-wise quadratic approximation of validation loss at 6 layers (the feed-forward input of block_{0,4,8,12,16,20}), from training step 21k to 22.5k. We compare 3 settings of training (Normal, Spiky1 and Spiky2) as in Figure 1 , and plot two simulations of different seeds, together with the actual 2nd-order term measured from corresponding training trajectories. The scale of the y -axis is kept the same for all plots, so that the fluctuations can be compared.
Figure 4: Plots of the 1st-order term of δt,sw .
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Simulating PLRF by Algorithm 1 . We apply SGD and Adam at different batch-sizes. Each plot includes one PLRF training and two simulation runs (with different random seeds).
B=1,LR=0.03,TH=0.3
B=8,LR=0.1,TH=0.5
B=64,LR=0.5,TH=1.0
Q90
RAT
MRLAT
Q90
RAT
MRLAT
Q90
RAT
MRLAT
PLRF
0.89
0.18
8.2
1.1
0.051
2.2
3.8
0.035
1.3
Simulation1
0.59
0.14
4.9
1.4
0.096
2.7
0.38
4.6
Simulation2
0.53
0.18
5.1
2.5
0.19
3.7
0.36
6.9
Appendix
Table 2: Statistics of training spikes, for three configurations of batch-sizes B={1,8,64} , with respectively picked learning-rate ( LR ) and threshold ( TH ).
Figure 6: The learning-rate schedule used for our Transformer LM training.
Figure 7: Log-log plot with truncated power-law fit, for ranked eigenvalues of correlation matrices.
Figure 8: Evolution of power-law fitting statistics, across layers and training steps.
Figure 9: Empirical verification of Equation ( 6 ).
Figure 10: Breakdown of the equivalent batch factor.
β2=0.95
β2=0.982
β2=0.992
β2 schedule
2.4421
2.4405
2.4502
2.4404
Appendix
Table 3: Final validation loss for different β2 settings.
Figure 11: Training loss for different β2 settings.
Figure 12: Estimating an appropriate β2 schedule.
Random feature methods approximate kernels and neural networks with finite randomized embeddings to reduce computation while retaining useful accuracy.
Large language models (LLMs) achieve strong performance across many tasks but rely on dense multiply-accumulate (MAC) operations during inference, resulting in high energy cost. Spiking neural networks (SNNs) offer an event-driven alternative in which synaptic integration uses lightweight accumulation. However, spike-driven LLM inference remains difficult because outlier-heavy activations typically require long firing windows or auxiliary non-spiking paths. We propose QuantaSpike, a short-window spike-driven quantization framework for LLMs built around Logarithmic Ternary Integrate-and-Fire (LTIF) neurons. LTIF uses ternary events with power-of-two membrane-response quanta, improving the information represented by each firing step while retaining shift-ACC-compatible computation. QuantaSpike combines this neuron with group-adaptive gain and selective outlier admission: normal values use residual LTIF steps, whereas admitted outliers receive one additional onset spike before entering the same residual dynamics. Across OPT and Llama-2, QuantaSpike achieves state-of-the-art or competitive perplexity and zero-shot accuracy among spike-driven LLM quantization methods. It also transfers to newer dense LLMs, remaining close to the FP16 reference on Llama-3-8B and Qwen3-8B under the same four-step firing window. Analytical linear-energy projections show that QuantaSpike reduces the energy of one linear transformation by about 80.0% on OPT models and 67.1% on Llama-2 models relative to SpikeQuant, providing an accurate and energy-efficient spike-driven path for LLM inference.
Bang Hu, Guowei Zhu, Changze Lv +4
School of Computer Science, Fudan University, Shanghai, China
Leveraging their inherent sparse event-driven computation, spiking neural networks (SNNs) offer a promising path toward energy-efficient large language models (LLMs). Time-to-first-spike (TTFS) coding generates at most one spike per neuron within a time window, yielding extremely low firing rates. However, conventional TTFS SNNs are restricted to specific structures, making it challenging to encode certain blocks in LLM -- such as layer normalization and matrix multiplication --using TTFS. To overcome this limitation, we introduce a reference-based strategy specifically to encode the four core LLM components: embedding layers, layer normalization, attention-related operations and dropout. We construct a fully TTFS-based SNN architecture and train it end-to-end. Experiments on modern LLMs like BERT and GPT-2 demonstrate that our approach achieves performance comparable to ANN counterparts on natural language understanding and common-sense reasoning, while a clear gap remains on language modeling perplexity. To the best of our knowledge, this is the first work to scale a spiking LLM to 1.5 billion parameters using TTFS coding. We also report an estimate of spike-related energy; this is a spike-count proxy under an established cost model rather than a measurement on neuromorphic hardware.
The design of modern neural architectures has converged through incremental empirical choices, yet the mechanisms governing their training dynamics remain only partially understood. We identify and analyze a negative weight drift induced by the interaction between standard losses and positively biased activation functions. We prove that under MSE or cross-entropy loss, the gradient with respect to positive pre-activations is non-negative in expectation at initialization, driving downstream weights toward negative values during early training. The drift is intrinsic to optimization rather than data, and persists across architectures and asymmetric activation functions (ReLU, GELU, SiLU). Coupled with ReLU, weight drift produces activation sparsity reaching nearly 90% in GPT-nano. We further analyze whether this natural sparsity is damaging and find that, on average, models sharply degrade above ∼70% activation sparsity. We then evaluate sparsity-inducing activation functions, which reveals activation spikes in intermediate transformer layers. Code is available at https://github.com/On-Point-RND/BugOrFeature.
Egor Shvetsov, Aleksandr Serkov, Shokorov Viacheslav +3