Neural networks trained on nonstationary tasks frequently lose the ability to fit new targets, a phenomenon referred to as loss of plasticity. We identify a novel source of plasticity loss due to the growing anisotropy of weight matrices' singular values during training, and analyze this phenomenon both empirically and theoretically. To mitigate this issue, we introduce SingularClip, a procedure that periodically clips the singular values of all weight matrices. We show that SingularClip performs strongly against baselines across a range of tasks in both continual supervised learning and deep reinforcement learning.
Figures & tables
Figure 1: Evolution of the condition numbers for each layer in a network training on Random Label CIFAR task. Error bars show 95% bootstrapped confidence intervals across 10 runs.
Figure 2: Evolution of the singular value distribution for the first convolutional layer in a run of Random Label CIFAR. Note that the largest singular value increases by a factor of roughly 3, as the parameter norm growth is controlled by weight decay.
Figure 3: Measuring the number of steps to reach perfect train accuracy as a function of initialization anisotropy, across 20 independent runs (left). Measuring initial weight condition number against average task accuracy in the Random Label MNIST task; datapoints are collected in a sequence of 35 tasks, across 10 independent runs (right).
Figure 4: Comparison of average online accuracy ( Equation 1 ) on continual supervised learning tasks. Shaded regions represent bootstrapped estimates of 95% confidence intervals. Each experiment is repeated over 30 independent runs.
Figure 5: Performance of SAC-based agents aggregated across DMC suite. Performance is aggregated across 10 runs per task. Error bars indicate 95% bootstrapped confidence intervals (left). Performance curves of SAC-based agents in the humanoid-stand environment averaged across 10 independent runs. Shaded regions correspond to 95% bootstrapped confidence intervals (right).
Figure 6: Performance of BRO-based agents across the DMC suite, across 20 independent runs. Shaded areas indicate bootstrapped 95% confidence intervals.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Ablation of c over a subset of continual learning tasks.
Figure 8: Ablation of c over a subset of RL settings.
Figure 9: Ablation of one-sided clipping across a subset of continual learning tasks. The baseline curve corresponds to a network with no intervention applied.
Figure 10: Comparison of condition number growth between SingularClip and spectral regularization on the Random Label CIFAR task, across different network layers.
Shape
Relative Frobenius Error (%)
(128, 128)
8.850×10−3±1.9×10−3
(256, 256)
1.019×10−2±8.8×10−4
(512, 512)
1.118×10−2±3.6×10−4
(1024, 1024)
1.076×10−2±1.3×10−4
(256, 512)
7.119×10−3±9.9×10−5
Appendix
Table 1: Relative Frobenius error of Newton-Schulz clipping for c=2 across different matrix shapes. Each value is computed by averaging over 30 runs. Uncertainties indicate two standard errors.
Shape
SVD (ms)
Newton-Schulz (ms)
(128, 128)
2.527 ± 0.075
2.219 ± 0.013
(256, 256)
6.282 ± 0.006
2.540 ± 0.653
(512, 512)
17.390 ± 0.017
2.287 ± 0.008
(1024, 1024)
51.230 ± 0.259
4.887 ± 0.029
(256, 512)
6.515 ± 0.009
2.417 ± 0.022
Appendix
Table 2: Wallclock time comparisons of singular clipping methods on a NVIDIA L40S GPU. Each value is computed by averaging over 30 runs after 10 warmup iterations. Uncertainties indicate two standard errors.
Figure 11: Comparison of adding SingularClip to Muon on the Continual ImageNet task. We plot average weight condition number (left) and online accuracy (right). Error bars represent bootstrapped 95% confidence intervals across 10 runs.
School of Computer Software, Tianjin University · International Joint Institute of Tianjin University, Fuzhou · The Hong Kong University of Science and Technology