Institute for Theoretical Physics, Leipzig University · 2ScaDS.AI Dresden/Leipzig
Hard-label classification is usually trained with smooth surrogate losses, most prominently softmax cross-entropy. We isolate an asymptotic mechanism by which this mismatch between smooth surrogate and discrete labels produces power-law learning curves in an online teacher-student model. After subtracting the mean logit, the thermodynamic-limit dynamics close in centered variables: a growing centered student-teacher alignment
D and the residual student variance
Δ. At late times, examples away from teacher decision boundaries are already classified confidently and contribute exponentially little. Only boundary layers of width
O(D−1) remain active, while the noise of fixed-learning-rate online gradient descent maintains a nonzero
Δ. As a function of the training time
α the late-time solution yields a
α−1/3 power law not only for the test loss but also for the generalization error
εg, i.e., one minus test accuracy. This is much slower than the
α−1 Bayes-optimal reference for the same model. We further show that learning-rate schedules can improve the generalization error towards a
εg∼α−1/2 power law. Simulations support the predicted order parameter dynamics and learning curves. Controlled experiments with correlated Gaussian inputs and whitened pretrained features show that data structure can dominate transients. Therefore, our result is an asymptotic, complementary mechanism rather than an alternative to spectral explanations of neural scaling laws.