Loss Spike

Loss spikes, sudden and dramatic increases in training loss, represent a significant challenge in training various neural network architectures, particularly large language models and recurrent neural networks used for solving partial differential equations. Current research focuses on understanding the underlying causes, such as exploding gradients, non-uniform parameter norms, and the interplay between different loss functions, and developing mitigation strategies through improved initialization techniques, modified training algorithms, and careful loss function design. Addressing loss spikes is crucial for improving the efficiency and reliability of training complex neural networks, ultimately impacting the performance and scalability of numerous machine learning applications.

Papers

October 7, 2024

Initialization of Large Language Models via Reparameterization to Mitigate Loss Spikes
Kosuke Nishida, Kyosuke Nishida, Kuniko Saito
Large Language Model Transformer Model New Initialization Transformer Decoder Reparameterization Method Loss Spike

May 6, 2024

Loss Jump During Loss Switch in Solving PDEs with Neural Networks
Zhiwei Wang, Lulu Zhang, Zhongwang Zhang, Zhi-Qin John Xu
Neural Network Loss Function Detection Loss Loss Spike

December 28, 2023

Spike No More: Stabilizing the Pre-training of Large Language Models
Sho Takase, Shun Kiyono, Sosuke Kobayashi, Jun Suzuki
Large Language Model Jina Embeddings Pre Training Natural Gradient Spike Protein Intermediate Layer Loss Spike

October 26, 2023

Bifurcations and loss jumps in RNN training
Lukas Eisenmann, Zahra Monfared, Niclas Alexander Göring, Daniel Durstewitz
Recurrent Neural Network \Sigma}{\Delta}$ Low Pas RNN Bifurcation Diagram RNN T Training Loss Spike

May 20, 2023

Loss Spike in Training Neural Networks
Xiaolong Li, Zhi-Qin John Xu, Zhongwang Zhang
Neural Network Strong Generalization Hessian Matrix Generalization Characteristic Loss Spike

Loss Spike

Papers

Initialization of Large Language Models via Reparameterization to Mitigate Loss Spikes

Loss Jump During Loss Switch in Solving PDEs with Neural Networks

Spike No More: Stabilizing the Pre-training of Large Language Models

Bifurcations and loss jumps in RNN training

Loss Spike in Training Neural Networks