cs.LGOct 2, 2025

PENEX: AdaBoost-Inspired Neural Network Regularization

Authors: Klaus-Rudolf Kladny, Bernhard Schölkopf, Michael Muehlebach

Organizations: MPI for Intelligent Systems, Tübingen · Tübingen AI Center · ELLIS Institute Tübingen

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

AdaBoost sequentially fits so-called weak learners to minimize an exponential loss, which penalizes misclassified data points more severely than other loss functions like cross-entropy. Paradoxically, AdaBoost generalizes well in practice as the number of weak learners grows. In the present work, we introduce Penalized Exponential Loss (PENEX), a new formulation of the multi-class exponential loss that is theoretically grounded and, in contrast to the existing formulation, amenable to optimization via first-order methods, making it a practical objective for training neural networks. We demonstrate that PENEX effectively increases margins of data points, which can be translated into a generalization bound. Empirically, across computer vision and language tasks, PENEX improves neural network generalization in low-data regimes, matching and in some settings outperforming established regularizers at comparable computational cost. Our results highlight the potential of the exponential loss beyond its application in AdaBoost.

Figures & tables

Figure 1: AdaBoost vs. Gradient Descent on PENEX. AdaBoost (left) builds a strong learner by sequentially fitting weak learners such as decision stumps and linearly combining them. Prior work has shown that this algorithm tends to be resilient to “overfitting”, meaning that validation performance does not worsen as many decision stumps are added. PENEX (right) extends this phenomenon to deep neural network training by re-formulating the exponential loss into a formulation that is amenable to optimization via first-order methods. Just as for AdaBoost, the neural network does not overfit and keeps generalizing better in contrast to using cross-entropy loss.
Figure 2: Comparison of Margins. Neural networks trained with PENEX (center) tend to implicitly increase the smallest margins (here, geometric margins, indicated for example point in green), in a similar way to support vector machines (right), here trained with a RBF kernel ( Schölkopf and Smola, 2002 , p. 46) . Training via cross-entropy loss (left), in contrast, typically leads to smaller margins.
Figure 3: CE vs. PENEX. We consider the binary case ( K=2K=2 ) with f(2)(x)≡0f^{(2)}(x)\equiv 0 , for a single xx and y=1y=1 . PENEX penalizes errors more than cross-entropy.
Figure 4: Performance Analysis on CIFAR-100. Larger means better . Results are computed from validation data. All hyperparameters have been tuned individually. PENEX (thick red) is an effective regularizer with often better generalization than other common regularization techniques (thin), and shows no signs of “overfitting” like cross-entropy training (orange)
MethodMetricCIFAR-10Noisy CIFAR-10CIFAR-100PathMNISTImageNetBBC News
CEACC0.785±0.0040.785\pm 0.0040.724±0.0040.724\pm 0.0040.443±0.0040.443\pm 0.0040.826±0.0040.826\pm 0.0040.648±0.0000.648\pm 0.0000.967±0.0070.967\pm 0.007
-ECE−0.162±0.003-0.162\pm 0.003−0.179±0.003-0.179\pm 0.003−0.287±0.003-0.287\pm 0.003−0.151±0.004-0.151\pm 0.004−0.200±0.000-0.200\pm 0.000−0.032±0.006\bf{-0.032}\pm 0.006
-CE−1.004±0.024-1.004\pm 0.024−1.125±0.019-1.125\pm 0.019−3.072±0.034-3.072\pm 0.034−2.018±0.130-2.018\pm 0.130−2.175±0.000-2.175\pm 0.000−0.109±0.024-0.109\pm 0.024
-BRIER−0.346±0.006-0.346\pm 0.006−0.424±0.006-0.424\pm 0.006−0.794±0.006-0.794\pm 0.006−0.300±0.007-0.300\pm 0.007−0.536±0.000-0.536\pm 0.000−0.051±0.011-0.051\pm 0.011
label smoothingACC0.789±0.0040.789\pm 0.0040.747±0.0040.747\pm 0.0040.451±0.0050.451\pm 0.0050.829±0.0040.829\pm 0.0040.651±0.000\bf{0.651}\pm 0.0000.970±0.0060.970\pm 0.006
-ECE−0.112±0.002-0.112\pm 0.002−0.183±0.003-0.183\pm 0.003−0.147±0.002\bf{-0.147}\pm 0.002−0.109±0.002-0.109\pm 0.002−0.159±0.000-0.159\pm 0.000−0.033±0.006-0.033\pm 0.006
Table 1: Test Set Performance. Larger means better, best is bold . Mean test result ±1\pm 1 standard deviation after 200200 training epochs (for ImageNet: 300300 epochs). Standard deviation is obtained by running 100100 bootstrap evaluations ( Efron and Tibshirani, 1993 ) . Hyperparameters are tuned on validation data (except for ImageNet, see ).
Figure 5: Ablation Studies on CIFAR-10. Larger means better . a) Optimizing the exponential loss without any constraint or with the constraint encoded into the model architectures break down early on during training (no results are shown for the hard constraint method starting at around epoch 8080 because of NaN model outputs). b) While constrained optimization algorithms based on CONEX 2 are trainable in principle, they reach worse optima and suffer from less stable training, in addition to requiring tuning more parameters (see ).
Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

 wm+1(i)  ←  wm(i)exp⁡(−ηm\mathbbm1{gm(xi)≠yi})Zm,w_{m+1}(i)\;\leftarrow\;\frac{w_{m}(i)\exp(-\eta_{m}\mathbbm{1}\{g_{m}(\mathbf{x}_{i})\neq y_{i}\})}{Z_{m}}, 
Appendix
Algorithm 2 SAMME (Multi-class AdaBoost)
Figure 6: Simplified PyTorch-like code for the PENEX loss.
Table 2: Ablation of clipping and the EMA factor used by the automatic ρ\rho estimator. Performance remains nearly unchanged when clipping is removed or when the EMA factor is set to β=0.01\beta=0.01 .
More in this topic

Neural Network Generalization

Neural network generalization studies why trained models perform well on unseen data and how architecture, data, and training shape that ability.

See all 311 papers

Explore similar work

CardsList
Mar 3, 2025

The Mathematical Theory of Double Descent as Shrinkage, not Overfitting: Matching Bounds for Path-Regularized Multilayer Neural Networks

We give a nonasymptotic generalization theory for multilayer fully connected neural networks trained with path regularization, for general Lipschitz losses and without assuming the loss bounded, the architecture sparse, or any width or depth limit. Approximation error is modelled explicitly, and along the way we settle an open problem of E et al. on approximation rates in generalized Barron spaces. We treat the underparameterized and overparameterized regimes separately, which is what allows the resulting curve to be non-monotone in the width. We then prove lower bounds on the error itself, valid at every target and at every local minimizer, and combine them with the upper bounds into a theorem asserting that the error - not merely our bound on it - descends, rises and descends again as the width grows, with a peak exceeding the final plateau by a factor diverging in the sample size. Having both sides of the bound also identifies what drives the ascent, and it is not overfitting: once the regularization exceeds the dual norm of the noise, which the prescribed schedule does at every width, a slow-rate oracle inequality controls the error with no variance term at all. The rise is instead the shrinkage demanded by the regularization that the underparameterized regime requires, and we show it cannot be removed by any width-independent regularization under which the theory's guarantees hold. This mechanism is a property of the estimator analyzed here; we do not claim it accounts for the double descent reported for unregularized training.

Hao Yu

Department of Automation, Tsinghua University, Beijing, 100086, China

Double DescentNeural Network GeneralizationNeural Network Approximation TheoryGeneralization Bounds
AskPDF
Oct 16, 2026

"Deep lifts": Neural networks can generalize even with the dumbest optimizer

Given a labeled dataset {(xi,yi)}i=1n\{(x_i, y_i)\}_{i = 1}^n, consider drawing a parameter vector θ^\hatθ uniformly at random from those that interpolate the data within a width-WW depth-DD ReLU architecture with continuous weights bounded within [−M,M][-M, M]. We study the generalization properties of θ^\hatθ in the case where the dataset can, in fact, be explained by a smaller ww-by-dd ReLU network. Despite the obliviousness of the selection rule for θ^\hatθ, it adaptively generalizes at the rate O~(wdWDn)\tilde O\left(\frac{wdWD}{n}\right), rather than being governed by the VC dimension of the full WW-by-DD hypothesis class. The reason is due to a particular inductive bias in ReLU networks: the ww-by-dd parameter witnesses an exponential volume of WW-by-DD interpolating parameters, which dwarfs the volume of poorly generalizing parameters (conditioned on interpolation). We explicitly construct this set of near-equivalent WW-by-DD parameters by a proof technique called deep lifting, and then tie the achievable volume to stability properties of the ww-by-dd witness. Examples show that these techniques allow us to convert approximation-theoretic results into generalization guarantees for nonparametric classes. We further show that generalization of uniform-random interpolators can survive label corruptions.

Sunil Madhow, Yu-Xiang Wang

UC San Diego

Neural Network GeneralizationGeneralization BoundsOverparameterized Neural NetworkReLU Neural NetworksBenign Overfitting
AskPDF
May 28, 2026

Gradient Perturbation: Learning to Perturb Gradients for Adaptive Training

Deep neural network training involves both forward propagation (from features through logits to loss) and backward propagation (from loss through gradients to parameter updates). While perturbations along the forward chain, including feature perturbation, logit perturbation, and label perturbation, have been extensively studied, the backward chain's gradient perturbation has received little systematic investigation. In this paper, we establish a unified framework for gradient perturbation, revealing that existing methods such as Sharpness-Aware Minimization (SAM), gradient clipping, and gradient noise injection can all be interpreted as imposing specific forms of gradient perturbation. Analogous to the recently proposed Logit Perturbation Learning (LPL), we conjecture that amplifying the gradient norm for a class acts as positive augmentation (enhancing learning), while dampening it acts as negative augmentation (suppressing overfitting). Based on these observations, we propose Learning to Perturb Gradients (LPG), which adaptively perturbs logit-level gradients at the class level to achieve category-aware training. We also establish theoretical connections between gradient perturbation bounds and generalization guarantees via PAC-Bayesian analysis. Experiments on balanced classification, long-tail classification, and noisy label learning demonstrate that LPG consistently outperforms existing methods and can be combined with them as a plug-in module.

Hua Li

Henan University

Topic pending
AskPDF