Loss of Plasticity

Loss of plasticity describes how neural networks can become harder to train as they learn, limiting adaptation in continual learning, online reinforcement learning, and transfer. Researchers diagnose it through weak gradients, dormant neurons, neural tangent kernel rank collapse, and growing spectral anisotropy, and test trainability across task shifts and ongoing learning settings. Approaches include targeted reinitialization, update regulation, sample weight decay, isometry and spectral regularization, and growing or evolutionary networks. A central challenge is restoring learning capacity without erasing useful knowledge, while evidence suggests that different architectures and training dynamics may lose plasticity for different reasons.