cs.LGOct 14, 2026
SaveThe Accuracy Well: Nonmonotone Accuracy Dynamics at Vanishing Initialization
Organizations: Applied AI Institute, Moscow, Russia · Steklov Mathematical Institute of Russian Academy of Sciences, Moscow, Russia
Abstract
We study classification dynamics under small initialization and identify a pronounced non-monotone behavior of accuracy that we call the accuracy well: accuracy rises rapidly early in training, subsequently drops, and only much later recovers. We observe this phenomenon in deep linear networks, fully connected ReLU networks on MNIST, deep convolutional networks on CIFAR-10, and controlled synthetic tasks, across different losses and optimizers and in both training and test accuracy. We propose a three-stage explanation in terms of target-driven organization, a degeneracy bottleneck, and subsequent recruitment and recovery. We develop a rigorous theory of these dynamics for shallow and deep linear networks, showing in particular that high accuracy can emerge while the predictor and the change in loss remain asymptotically vanishing, before continued optimization causes accuracy to deteriorate. For a simple synthetic classification task, we prove an extreme instance in which accuracy asymptotically rises to 100%, collapses to 0, and later recovers permanently to 100%. Our results show that classification accuracy can reveal aspects of learning dynamics that are not captured by the training loss alone.
Figures & tables
Figure 1: Schematic accuracy well: non-monotone accuracy dynamics at small initialization.
Figure 2: Accuracy-well phenomenology on CIFAR-10 and MNIST. Left : train and test dynamics of a two-hidden-layer CIFAR-10 CNN under an initialization-scale sweep. Middle : MNIST ReLU networks without biases under cross-entropy, varying initialization scale (top) and width (bottom). Right : optimizer comparison for the MNIST ReLU network (top) and a comparison of linear networks on MNIST with (bottom).
Figure 3: Representative unary-operation experiments. Left : a deep linear network ( layers) trained on a permutation operation develops a pronounced accuracy well as width increases. Middle : a shallow network ( layers) trained on a random unary operation develops a well in which the early classifier can substantially outperform the late-time predictor. Right : decreasing the initialization scale separates the rise, collapse, and recovery over increasingly distinct time scales.
Figure 4: Diagnostics of the proposed accuracy-well mechanism. Left : evolution of a two-layer linear network on a random unary operation. Snapshots of the normalized score matrix show how the model reorganizes along the accuracy-well trajectory. Notably, snapshots with the same accuracy ( ) have qualitatively different score-matrix structure. Right : test accuracy, effective hidden-neuron participation fraction, and output diversity in a one-hidden-layer ReLU network on MNIST. The participation fraction measures the effective fraction of hidden neurons contributing to class-dependent scores, while output diversity measures the effective fraction of output classes used by the predictions; see equations ( 897 ), ( 901 ) in Appendix C.2 .
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Depth | Setting | Result |
|---|---|---|
| General | such that for all . Any accuracy drawdown is at most . All classifier dynamics occur in a microscopic boundary layer. | |
| General | As , then , such that, with probability tending to one, and . The trajectory through is microscopic; for fixed , it eventually recovers permanently to . | |
| Random unary, | As , then , such that and in probability. The trajectory is microscopic through . If , fixed- recovery to is permanent with probability tending to one. | |
| Permutation, | As , then , accuracy is nondecreasing with probability tending to one and reaches during the microscopic regime, remaining thereafter. | |
| Gaussian, | As , then , such that and . The trajectory through is microscopic. The probability tends to for , and is asymptotically at least for . Generic recovery remains open. | |
| Spectrally aligned | If a leading-mode bottleneck occurs, its accuracy is . Subsequent mode recruitment gives , hence permanent recovery to . Thus yields a bottleneck followed by recovery. |
Table 1: Summary of the deep-linear accuracy theory. “Microscopic” means that the predictor norm and loss change vanish uniformly up to the stated time. Recovery statements are for fixed .
| Dataset | |||
|---|---|---|---|
| MNIST | 0.859 | 0.209 | Yes |
| Fashion-MNIST | 0.831 | 0.200 | Yes |
| KMNIST | 0.770 | 0.196 | Yes |
| CIFAR-10 | 0.487 | 0.180 | Yes |
| CIFAR-100 | 0.442 | 0.019 | Yes |
Table 2: Accuracy of the full effective target and its complete top singular block on centered and whitened training data. In all cases .
Figure 5: Controlled unary-operation experiments. Columns correspond to the shallow permutation model (left), the three-factor permutation model (middle), and the shallow model on an iid random unary target (right). The top row varies the initialization scale ; the bottom row varies the hidden width . Darker curves denote smaller in the top row and larger in the bottom row. Blue curves show full-population MSE and red curves show training accuracy. The shallow permutation model reaches high accuracy while the loss remains near its initial value, without developing a well. Changing to the three-factor parameterization produces a pronounced well for the same flat-spectrum target, while a heterogeneous random unary target produces a well already in the shallow model, consistently with the heterogeneous-spectrum mechanism analyzed theoretically.
Figure 6: Initialization-scale sweeps for the MNIST ReLU bias ablation under squared loss. Columns distinguish the absence (left) and presence (right) of an output bias, while rows distinguish the absence (top) and presence (bottom) of a hidden bias. All panels use and , with darker curves denoting smaller initialization. Blue curves show minibatch MSE and red curves show minibatch top-one accuracy. Without an output bias, sufficiently small initialization produces a pronounced accuracy well regardless of whether a hidden bias is present. Introducing an output bias suppresses the pronounced well in both corresponding architectures.
Figure 7: Width sweeps for the MNIST ReLU bias ablation under squared loss. The panel arrangement is the same as in Figure 6 . All runs use and , with darker curves denoting larger width. The no-bias and hidden-bias-only networks exhibit pronounced nonmonotone accuracy trajectories, whereas architectures containing an output bias do not develop a comparably pronounced well.
Figure 8: Linear-network controls on MNIST under squared loss. Left : comparison of the direct model with two- and three-factor parameterizations , using , , and . Although all three parameterizations represent the same class of end-to-end linear maps, the factorized models develop pronounced accuracy wells while the direct model has much simpler accuracy dynamics. Right : initialization-scale sweep for the direct classifier with . Even when the direct model is initialized extremely close to zero, its accuracy evolves comparatively monotonically and does not develop the pronounced well observed in the factorized models. Blue curves show minibatch MSE and red curves show minibatch top-one accuracy.
Figure 9: Robustness and bias ablation for the MNIST accuracy well. Left : seed robustness for the bias-free ReLU network trained with cross-entropy at , , showing raw minibatch accuracy trajectories for five independent initializations and minibatch streams. All five runs exhibit the same hill–well–recovery pattern. Right : four-way bias ablation under squared loss at , . The pronounced well is preserved with no bias or a hidden bias alone, but is suppressed when an output bias is present.
Figure 10: Accuracy-well dynamics in a two-hidden-layer CNN on CIFAR-10. Columns correspond to the initialization-scale sweep (left) and the model-capacity sweep (right), while rows show training dynamics (top) and held-out test dynamics (bottom). In the left column, the channel widths are fixed to , and darker curves correspond to smaller initialization scale . In the right column, is fixed, and darker curves correspond to larger channel widths. Blue curves show cross-entropy loss and red curves show top-one accuracy. The horizontal dashed blue line denotes the zero-logit cross-entropy . Each displayed trajectory is a 25-step moving average of the corresponding per-minibatch measurements. Small initialization exposes a pronounced early accuracy hill followed by an accuracy well, with substantial accuracy dynamics occurring while the cross-entropy remains close to its zero-logit value. In the capacity sweep, larger models form a stronger early classifier and undergo an earlier and more pronounced hill–well–recovery trajectory. The corresponding held-out test curves closely reproduce the qualitative training dynamics.
Figure 11: Dynamics of a deeper convolutional network on CIFAR-10. Training trajectories are shown on the left and held-out test trajectories on the right. The network has five convolutional hidden layers with , followed by a fully connected ten-class output layer. All weights use entrywise Gaussian initialization with ; optimization uses minibatch SGD with learning rate and batch size . Blue curves show cross-entropy loss and red curves show top-one accuracy; the horizontal dashed blue line denotes the zero-logit cross-entropy . Curves are 25-step moving averages of the per-minibatch measurements.
More in this topic
Neural Network Training Dynamics
Neural network training dynamics studies how parameters and representations change during optimization and how those changes shape learning, stability, and generalization.
See all 499 papers