We study classification dynamics under small initialization and identify a pronounced non-monotone behavior of accuracy that we call the accuracy well: accuracy rises rapidly early in training, subsequently drops, and only much later recovers. We observe this phenomenon in deep linear networks, fully connected ReLU networks on MNIST, deep convolutional networks on CIFAR-10, and controlled synthetic tasks, across different losses and optimizers and in both training and test accuracy. We propose a three-stage explanation in terms of target-driven organization, a degeneracy bottleneck, and subsequent recruitment and recovery. We develop a rigorous theory of these dynamics for shallow and deep linear networks, showing in particular that high accuracy can emerge while the predictor and the change in loss remain asymptotically vanishing, before continued optimization causes accuracy to deteriorate. For a simple synthetic classification task, we prove an extreme instance in which accuracy asymptotically rises to 100%, collapses to 0, and later recovers permanently to 100%. Our results show that classification accuracy can reveal aspects of learning dynamics that are not captured by the training loss alone.
Figures & tables
Figure 1: Schematic accuracy well: non-monotone accuracy dynamics at small initialization.
Figure 2: Accuracy-well phenomenology on CIFAR-10 and MNIST. Left : train and test dynamics of a two-hidden-layer CIFAR-10 CNN under an initialization-scale sweep. Middle : MNIST ReLU networks without biases under cross-entropy, varying initialization scale (top) and width (bottom). Right : optimizer comparison for the MNIST ReLU network (top) and a comparison of linear networks f=WL⋯W1 on MNIST with L=1,2,3 (bottom).
Figure 3: Representative unary-operation experiments. Left : a deep linear network ( L=3 layers) trained on a permutation operation develops a pronounced accuracy well as width increases. Middle : a shallow network ( L=2 layers) trained on a random unary operation develops a well in which the early classifier can substantially outperform the late-time predictor. Right : decreasing the initialization scale σ separates the rise, collapse, and recovery over increasingly distinct time scales.
Figure 4: Diagnostics of the proposed accuracy-well mechanism. Left : evolution of a two-layer linear network M=W2W1 on a random unary operation. Snapshots of the normalized score matrix show how the model reorganizes along the accuracy-well trajectory. Notably, snapshots with the same accuracy ( A=0.6 ) have qualitatively different score-matrix structure. Right : test accuracy, effective hidden-neuron participation fraction, and output diversity in a one-hidden-layer ReLU network on MNIST. The participation fraction measures the effective fraction of hidden neurons contributing to class-dependent scores, while output diversity measures the effective fraction of output classes used by the predictions; see equations ( 897 ), ( 901 ) in Appendix C.2 .
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
Depth
Setting
Result
L=1
General
∃τ∗=O(σ) such that A(M(τ))=AT for all τ≥τ∗ . Any accuracy drawdown is at most 1−AT . All classifier dynamics occur in a microscopic O(σ) boundary layer.
L=2
General
As σ↓0 , then H→∞ , ∃τorg<τbot such that, with probability tending to one, A(M(τorg))=AT and A(M(τbot))=Ak⋆ . The trajectory through τbot is microscopic; for fixed σ>0 , it eventually recovers permanently to AT .
Random unary, H/p→α
As σ↓0 , then p,H→∞ , ∃τorg<τbot such that A(M(τorg))→1 and A(M(τbot))→0 in probability. The trajectory is microscopic through τbot . If α>1−e−1 , fixed- σ recovery to 1 is permanent with probability tending to one.
Permutation, H/p→α
As σ↓0 , then p,H→∞ , accuracy is nondecreasing with probability tending to one and reaches 1 during the microscopic regime, remaining 1 thereafter.
L≥3
Gaussian, s1>s2
As σ↓0 , then H→∞ , ∃τorg<τbot such that A(M(τorg))=AT and A(M(τbot))=A1 . The trajectory through τbot is microscopic. The probability tends to 1 for L=3 , and is asymptotically at least 1/2 for L>3 . Generic recovery remains open.
Spectrally aligned
If a leading-mode bottleneck occurs, its accuracy is A1 . Subsequent mode recruitment gives M(τ)→T , hence permanent recovery to AT . Thus AT>A1 yields a bottleneck followed by recovery.
Appendix
Table 1: Summary of the deep-linear accuracy theory. “Microscopic” means that the predictor norm and loss change vanish uniformly up to the stated time. Recovery statements are for fixed σ>0 .
Dataset
AT
Ak⋆
AT>Ak⋆
MNIST
0.859
0.209
Yes
Fashion-MNIST
0.831
0.200
Yes
KMNIST
0.770
0.196
Yes
CIFAR-10
0.487
0.180
Yes
CIFAR-100
0.442
0.019
Yes
Appendix
Table 2: Accuracy of the full effective target and its complete top singular block on centered and whitened training data. In all cases k⋆=1 .
Figure 5: Controlled unary-operation experiments. Columns correspond to the shallow permutation model (left), the three-factor permutation model (middle), and the shallow model on an iid random unary target (right). The top row varies the initialization scale σ ; the bottom row varies the hidden width H . Darker curves denote smaller σ in the top row and larger H in the bottom row. Blue curves show full-population MSE and red curves show training accuracy. The shallow permutation model reaches high accuracy while the loss remains near its initial value, without developing a well. Changing to the three-factor parameterization produces a pronounced well for the same flat-spectrum target, while a heterogeneous random unary target produces a well already in the shallow model, consistently with the heterogeneous-spectrum mechanism analyzed theoretically.
Figure 6: Initialization-scale sweeps for the MNIST ReLU bias ablation under squared loss. Columns distinguish the absence (left) and presence (right) of an output bias, while rows distinguish the absence (top) and presence (bottom) of a hidden bias. All panels use H=8192 and σ∈{10−1,10−2,10−4} , with darker curves denoting smaller initialization. Blue curves show minibatch MSE and red curves show minibatch top-one accuracy. Without an output bias, sufficiently small initialization produces a pronounced accuracy well regardless of whether a hidden bias is present. Introducing an output bias suppresses the pronounced well in both corresponding architectures.
Figure 7: Width sweeps for the MNIST ReLU bias ablation under squared loss. The panel arrangement is the same as in Figure 6 . All runs use σ=10−4 and H∈{32,512,8192} , with darker curves denoting larger width. The no-bias and hidden-bias-only networks exhibit pronounced nonmonotone accuracy trajectories, whereas architectures containing an output bias do not develop a comparably pronounced well.
Figure 8: Linear-network controls on MNIST under squared loss. Left : comparison of the direct L=1 model with two- and three-factor parameterizations L=2,3 , using σL=1=10−9 , σL=2=10−3 , and σL=3=10−1 . Although all three parameterizations represent the same class of end-to-end linear maps, the factorized models develop pronounced accuracy wells while the direct model has much simpler accuracy dynamics. Right : initialization-scale sweep for the direct classifier with σ∈{10−1,10−5,10−9} . Even when the direct model is initialized extremely close to zero, its accuracy evolves comparatively monotonically and does not develop the pronounced well observed in the factorized models. Blue curves show minibatch MSE and red curves show minibatch top-one accuracy.
Figure 9: Robustness and bias ablation for the MNIST accuracy well. Left : seed robustness for the bias-free ReLU network trained with cross-entropy at H=8192 , σ=10−4 , showing raw minibatch accuracy trajectories for five independent initializations and minibatch streams. All five runs exhibit the same hill–well–recovery pattern. Right : four-way bias ablation under squared loss at H=8192 , σ=10−4 . The pronounced well is preserved with no bias or a hidden bias alone, but is suppressed when an output bias is present.
Figure 10: Accuracy-well dynamics in a two-hidden-layer CNN on CIFAR-10. Columns correspond to the initialization-scale sweep (left) and the model-capacity sweep (right), while rows show training dynamics (top) and held-out test dynamics (bottom). In the left column, the channel widths are fixed to (C1,C2)=(128,256) , and darker curves correspond to smaller initialization scale σ . In the right column, σ=10−4 is fixed, and darker curves correspond to larger channel widths. Blue curves show cross-entropy loss and red curves show top-one accuracy. The horizontal dashed blue line denotes the zero-logit cross-entropy log10 . Each displayed trajectory is a 25-step moving average of the corresponding per-minibatch measurements. Small initialization exposes a pronounced early accuracy hill followed by an accuracy well, with substantial accuracy dynamics occurring while the cross-entropy remains close to its zero-logit value. In the capacity sweep, larger models form a stronger early classifier and undergo an earlier and more pronounced hill–well–recovery trajectory. The corresponding held-out test curves closely reproduce the qualitative training dynamics.
Figure 11: Dynamics of a deeper convolutional network on CIFAR-10. Training trajectories are shown on the left and held-out test trajectories on the right. The network has five convolutional hidden layers with (C1,…,C5)=(256,256,256,256,256) , followed by a fully connected ten-class output layer. All weights use entrywise Gaussian initialization with σ=8×10−3 ; optimization uses minibatch SGD with learning rate 0.025 and batch size 128 . Blue curves show cross-entropy loss and red curves show top-one accuracy; the horizontal dashed blue line denotes the zero-logit cross-entropy log10 . Curves are 25-step moving averages of the per-minibatch measurements.
Neural network training dynamics studies how parameters and representations change during optimization and how those changes shape learning, stability, and generalization.
Delayed generalization, where training accuracy becomes nearly perfect substantially before test accuracy, has often been studied in the context of the depth and compositional structure of deep networks. In this paper, we ask whether delayed generalization can occur in simpler settings. We show that it can arise in a linear model trained with gradient descent and an exponential-tail loss on linearly separable data. We characterize the finite-time hyperplane dynamics into three phases: a population phase, a support vector dominated unlearning phase and a generalization phase. The convergence of the residual beyond the maximum-margin direction is sufficient for the unlearning phase to begin. In the support vector dominated phase, we characterize the hyperplane dynamics. The time scale in these dynamics causes delayed generalization. We validate our theory through controlled experiments varying population and support-vector distributions, and further verify the predicted bias behavior in a deep model with a single output neuron on MNIST. Our results provide a unified characterization of the asymptotic and finite-time dynamics underlying delayed generalization in linear models.
Deep visual recognition models are usually trained and evaluated using metrics such as loss and accuracy. While these measures show whether a model is improving, they reveal very little about how its internal representations change during training. This paper introduces a complementary way to study that process by examining training through the lens of dynamical systems. Drawing on ideas from signal analysis originally used to study biological neural activity, we define three measures from layer activations collected across training epochs: an integration score that reflects long-range coordination across layers, a metastability score that captures how flexibly the network shifts between more and less synchronised states, and a combined dynamical stability index. We apply this framework to nine combinations of model architecture and dataset, including several ResNet variants, DenseNet-121, MobileNetV2, VGG-16, and a pretrained Vision Transformer on CIFAR-10 and CIFAR-100. The results suggest three main patterns. First, the integration measure consistently distinguishes the easier CIFAR-10 setting from the more difficult CIFAR-100 setting. Second, changes in the volatility of the stability index may provide an early sign of convergence before accuracy fully plateaus. Third, the relationship between integration and metastability appears to reflect different styles of training behaviour. Overall, this study offers an exploratory but promising new way to understand deep visual training beyond loss and accuracy.
Hai La Quang, Hassan Ugail, Newton Howard +3
Posts and Telecommunications Institute of Technology Vietnam · Centre for Visual Computing and Intelligent Systems University of Bradford United Kingdom · School of Individualized Study Rochester Institute of Technology United States
Weight perturbation (WP) estimates learning directions from random parameter perturbations and a scalar change in loss, without backpropagating derivatives. We examine how batch size, training duration, and estimator stabilization affect its attainable accuracy in fully connected MNIST networks. In the shallow configuration, large-batch training reaches approximately 97% test accuracy, compared with an early backpropagation (BP) reference near 97.5--98%. The same comparison exposes a substantial efficiency gap: under a ten-epoch budget, shallow WP reaches 90.2%, whereas the reported BP reference reaches 97.4%. Epoch-indexed curves show diminishing returns from increasing batch size beyond 4,000. Learning-rate reduction and averaging independent perturbation-based updates also stabilize deeper networks, although their recorded results do not establish BP-level accuracy. We distinguish these empirical findings from a finite-horizon approximation argument: under stated regularity conditions, population WP can approximate a fixed gradient-descent trajectory arbitrarily closely at any fixed finite depth. Positive classification margins then suffice to preserve the reference predictions on a finite test set with arbitrarily high probability. Together, the results identify near-BP accuracy as a configuration- and budget-dependent possibility, rather than evidence of equal efficiency or an unconditional accuracy guarantee across depth.
Patrick Inoue
The MNIST experiments originate from the author’s master’s thesis, submitted under the name Patrick Stricker at Albstadt-Sigmaringen University in March 2023 [4].