cs.LGOct 24, 2026
SaveWhen is early stopping beneficial under label noise? A phase diagram for kernel gradient flow
Organizations: Statistical Physics of Computation Laboratory, École Polytechnique Fédérale de Lausanne (EPFL) CH-1015 Lausanne
Abstract
Early stopping is commonly used to prevent fitting label noise from degrading generalization. Does this mean that, when a positive fraction of the training labels is intrinsically unpredictable, the best test error must occur before interpolation? Experiments on a CIFAR10 classification task with randomly labeled examples motivate the opposite possibility: although fixed-feature and shallow models exhibit late-time deterioration of the test error, the need for early stopping essentially disappears in larger, deeper convolutional networks even after the random labels have all been classified correctly. We determine when late-time deterioration disappears entirely in a solvable setting. For kernel gradient flow on a synthetic noisy classification task, we derive sharp high-dimensional learning curves and obtain a phase diagram controlled by the balance between the linear kernel component that learns the shared relation between inputs and labels and the higher-order components that enable fitting the label noise. In one regime, generalization is optimized at finite time and early stopping is necessary. In another, generalization improves until interpolation, but the endpoint is suboptimal across kernels. At the boundary, the interpolating endpoint attains the lowest generalization error across the considered kernel family and all training times. Thus, even with extensive irreducible label noise, interpolation can outperform early stopping. The neural-network experiments leave open how learned representations acquire an analogous favorable geometry.
Figures & tables
Figure 1: Generalization along the path to memorization. Test error on 2000 cats and dogs ( -axes) versus classification error on the randomly labelled automobiles ( -axes) taken from the CIFAR10 dataset. The fixed training set has images: cats, dogs with their respective labels, and automobiles with independent random labels. Panels vary the overparametrization ratio , where counts optimized parameters. Curves average about 30 completed seeds; a centered arithmetic average of up to seven neighboring recorded times is applied to both coordinates for display, preserving the initial and final values. These parametric curves trace the two errors as training time increases. Triangles mark initialization, squares the last iterate and crosses the minimum displayed test error. All trainable networks use square loss and constant-rate Nesterov SGD, without augmentation or weight decay, through 1500 epochs; RF-CNN uses full-batch readout GD through steps. In Appendix F we provide all numerical details and the unsmoothed trajectories, including training loss.
Figure 2: The generalization cost of training to interpolation. For , , and , colors show from Eq. ( 16 ) in percentage points, comparing interpolation with optimal stopping for the same kernel. The gray region has exactly zero stopping benefit. The dashed line, (for ), separates it from a region with a finite optimal stopping time, independently of . Contours mark benefits of , , , and percentage points.
Figure 3: Test error at interpolation and optimal stopping. We fix , , and and evaluate Eqs. ( 9 ) and ( 10 ). The solid curve is ; the dashed and dash-dotted curves are for and . For , Eq. ( 15 ) gives , so all three curves coincide. The star marks the unique global optimum at from Eq. ( 14 ). For larger angles, finite optimal stopping remains strictly above this value; the gap is more visible for . The inset zooms on the increasing excess for the dashed blue curve at , to highlight that the minimum is truly attained only at .
Figure 4: Kernel geometry controls the gradient-flow trajectory. We fix , , , and . Arrows indicate increasing time; the cross marks test-error minima and the blue dot marks convergence. The blue trajectory benefits from early stopping. Red and green improve until interpolation, but only the critical red kernel attains the test-error bound in Eq. ( 18 ) (vertical line).
Figure 5: Gradient flow and ridge follow different parametric paths. We fix , , and , with , ( ), and . For each kernel, we plot the test-error difference between unregularized kernel gradient flow and converged kernel ridge regression at the same memorization error. Negative values favor gradient flow. Squares mark the shared and classification limits; the circle marks the shared interpolating endpoint. Curves are numerical evaluations of Eqs. ( 9 )–( 11 ); we use the ERM ridge result of Farné et al. (2026) . More details in Appendix E .
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Numerical validation across four learning regimes. Lines show the theoretical generalization and memorization errors; circles and squares show the corresponding means from RBF simulations, with one standard error bars. All panels use and retain the kernel’s nonzero constant term. Parameters are shown above the panels. (a) Delayed generalization after factual memorization. (b) Delayed generalization followed by late overfitting. (c) Optimal stopping before factual memorization. (d) Continued improvement toward the critical interpolating endpoint. Dotted lines mark and dashed lines finite . The corresponding reduced times are and .
Figure 7: Kernel-dependent generalization and memorization on CIFAR10. Full RBF-kernel gradient flow with zero initialization and zero ridge. Curves show the unsmoothed mean errors over five repetitions; arrows indicate increasing time. Stars mark the minimum mean test error on the evaluated grid, including the exact convergence endpoint and choosing it when tied. Filled squares mark interpolation. The different bandwidths reproduce the qualitative contrast between trajectories that benefit from early stopping and trajectories with little or no observed deterioration near interpolation.
Figure 8: Unsmoothed dynamics behind Figure 1 . Test error, memorization error and full training MSE (top to bottom), averaged over – completed seeds per curve through epochs at . Shading gives pointwise confidence intervals for the mean across seeds. Crosses mark each raw mean test curve’s minimum and the corresponding values of the other metrics; squares mark the last observation. No temporal averaging is applied. The models are two-, four- and six-layer CNNs, a residual network with nominal depth (ResNet-16), and a wide residual network with nominal depth (WRN-28).
Figure 9: Generalization while fitting random labels with augmented cross-entropy. Clean cat-dog test error versus classification error on randomly labelled automobiles, for the -image dataset and approximately million optimized parameters per network. Coordinates are averaged over at most five neighboring saved observations within each seed, then over five seeds. Triangles mark initialization, arrows indicate increasing time, crosses mark the minimum displayed mean test error and squares mark the reporting endpoints. Trajectories extend to epochs, except WRN-28, which is shown through epochs.
Figure 10: Unsmoothed trajectories under augmented cross-entropy training. Clean test error, automobile memorization error and mean binary cross-entropy on all original training images. Lines show five-seed means; shading gives pointwise confidence intervals for these means. Crosses mark the minimum raw mean test error and the corresponding values of the other metrics; squares mark the reporting endpoints. The comparison includes CNN4, ResNet-16, CNN6, WRN-28 and MobileNetV2. Trajectories extend to epochs, except WRN-28, which is shown through epochs. Time is displayed as with epoch labels; the cross-entropy axis is logarithmic.
Figure 11: Sensitivity to the optimization protocol. Difference between the final test error and the lowest test error recorded during each run, averaged over five seeds and expressed in percentage points. Results compare two-layer CNNs, four-layer CNNs and ResNet-16 at intermediate capacity (top) and high capacity (bottom). Error bars give confidence intervals for the mean. The larger deep models retain a smaller gap across the displayed optimizer, learning-rate and batch-size choices.
Figure 12: Extended ResNet controls with augmented cross-entropy. Columns vary training-set size, batch size and random-label fraction under the -epoch recipe of Section F.3 . Means are unsmoothed; bars show one sample standard deviation over ten seeds at and five otherwise. “Fixed LR” and “linear LR” refer to peak rates. Memorization error is omitted at .
Figure 13: ResNet controls with constant-rate square loss. Conditions follow Figure 12 , with five seeds. Per-seed arithmetic averaging uses a symmetric window of half-width at most in , shrinking at the endpoints. Lines and shading give means and one sample standard deviation. Horizons are epochs for , for , and otherwise; time is labelled in epochs.
Figure 14: Memorization paths with random labels on cats and dogs. Square-loss training at , averaged over ten seeds. Both coordinates use the endpoint-preserving average over at most seven saved times described in Section F.1 . Triangles mark initialization, crosses the minimum displayed mean test error and squares the endpoints. Trainable networks run for epochs; RF readout GD runs through full-batch steps.
Figure 15: Unsmoothed square-loss trajectories with random labels on cats and dogs. Test error, memorization error and training square loss for the five trainable architectures, through epochs. Lines show ten-seed means; shading gives pointwise confidence intervals. Crosses mark the minimum raw mean test error and corresponding other metrics; squares mark endpoints. Time uses with epoch labels, and loss is shown logarithmically.
Figure 16: Memorization paths with augmented cross-entropy and randomized cats and dogs. Five-seed means after endpoint-preserving averaging over at most five saved times; markers follow Figure 14 . Reporting horizons are epochs for CNN4, for ResNet-16, for WRN-28 and for MobileNetV2. Each network has approximately million optimized parameters.
Figure 17: Unsmoothed cross-entropy trajectories with randomized cats and dogs. Five-seed means of test error, memorization error and training cross-entropy on original images, with pointwise confidence intervals. Horizons follow Figure 16 ; CNN6 is additionally shown through epochs. Markers and axes follow Figure 15 . No temporal averaging is applied.
More in this topic
Early Stopping
Early stopping uses signals during training or inference to halt computation before a fixed budget, balancing generalization, quality, and cost.
See all 46 papers