stat.MLOct 12, 2026
SaveFeature Normalization under Finite Training Budgets
Organizations: Department of Mathematical Sciences, University of Copenhagen, Universitetsparken 5, 2100 Copenhagen, Denmark
Abstract
Standardizing a rare feature can help stochastic gradient descent learn its effect within a finite budget, but it also changes the noise cost of learning. We characterize the lowest rare-subgroup risks attainable by standardized and unscaled training when each chooses a nonnegative constant learning rate and a deterministic stopping time under the same observation cap. For two independent binary features, we derive both optimized risks as prevalence vanishes while the expected number of rare observations remains finite. Uniform control over rates and stopping times reduces each limiting oracle to a scalar minimization. With positive noise, the preferred scaling changes exactly once as we move signal to the rare feature while the limiting risk at initialization stays fixed and positive. The formulas also explain how noise and budget affect the comparison. Separate finite-prevalence bounds favor standardization on a rare-dominant signal family and unscaled training on a common-dominant family. The results concern final iterates with population preprocessing and fresh observations. Exact-moment searches assess the approximation at finite prevalence. In a controlled study, both orderings persist under observation reuse, estimated preprocessing, and intercept fitting when tuning choices are fixed before evaluation. Selecting a predictor after observing training can reverse an ordering, even when selection uses exact population risk.
Figures & tables
Figure 1: Normalization changes learning paths, and its effect on risk depends on the signal.
Figure 2: Tuning does not erase the effect of normalization. Independent binary features have prevalences and , with (left) or (right) and noise variance one. Under the stated population setup, curves show the lowest exact rare risk over a finite rate grid and every integer stop within each cap, including zero. Both panels use the same axes, and lower risk is better. This illustrates \cref thm:conference-predictive, which covers all nonnegative constant rates. See \cref sec:predictive-intro-figure for numerical details.
Figure 3: Normalization changes the balance between incomplete learning and training variability, even after tuning. Panels use (left) and (right), with prevalence , noise variance one, and observation cap . Each method selects its rate and stop to minimize exact rare-subgroup risk over the search grid used in \cref fig:conference-tuned-intro. Bars show the common and rare coefficients’ contributions to that risk. Colors distinguish incomplete learning, feature-sampling variation, and observation noise, as in \cref eq:conference-decomposition. Each method’s two bars sum to its total rare-subgroup risk.
| Rare risk | Reduction from calibration | SD-minus-U gap | ||||
| Selection | U | SD | U | SD | Mean | MCSE |
| Fresh observations, fitted intercept | ||||||
| Separate calibration | — | — | ||||
| Validation (2048 rows) | ||||||
| Exact population risk | ||||||
| Fresh observations, fixed zero intercept (baseline) | ||||||
Table 1: Rare risk and gains from selection with , , and , with fresh observations and population preprocessing. The fitted-intercept arm and baseline (arms 2 and 1 in \cref tab:bridge-arms) share the observation cap and differ only in whether an intercept is fitted; all coefficients are initialized at zero. Positive reductions indicate improvement over the separately calibrated choice for the same scaling; negative SD-minus-U gaps favor SD. Means use 4096 paired evaluation runs. MCSEs refer to the paired gaps and condition on the calibration choices. \Cref sec:bridge-exact-calibration gives exact expected-risk calibration benchmarks for the same candidate family.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Family | Maximum prevalence | Gap bound | Certificate leaves |
|---|---|---|---|
| Rare | |||
| Common | |||
| Rare | |||
| Common | |||
| Rare | |||
| Common |
Table 2: Certified gap bounds for , , and . Each bound holds throughout the corresponding signal family. Certificate leaves count the accepted boxes in the interval certificates.
| Preferred scaling | |||
|---|---|---|---|
| 0 | 0.067668 | 0.069359 | U |
| 0.1 | 0.109877 | 0.103572 | S |
| 1 | 0.256267 | 0.232599 | S |
| 100 | 0.500000 | 0.502306 | U |
Table 3: Limiting oracle risks across noise variances for , , and . The preferred scaling has the smaller risk: U at the smallest and largest noise variances shown, and S at the intermediate values.
Figure 4: Additional common features change the cost of normalization at fixed signal strength. The sums of squared common and rare coefficients are (left) or (right), with noise variance one. Caps are at and at , giving . Curves show SD-minus-U rare risk after separate searches over finite stable-rate grids and all integer stops; negative values favor SD. Each panel has its own vertical scale. \Cref sec:common-block-numerics gives all settings and checks.
Figure 5: Dimension comparisons across all three prevalences and budgets. The top row uses and the bottom row uses , holding total squared signal fixed within each row. Negative SD-minus-U rare risk favors standardization. Each scaling searches its own finite grid of stable rates and every integer stop through the cap. The rows have different vertical scales. At the larger prevalences, the size of SD’s rare-signal advantage can decrease after initially increasing with dimension.
Figure 6: Exact finite-grid selections approach the limiting oracle predictions. Colored curves connect finite-grid results; black dashed curves connect limiting predictions at the sampled settings. Panels (b)–(d) use nominal target budgets, so coarse caps also reflect rounding. Lines do not certify intervening crossing locations.
| Arm | Sampling | Estimated preprocessing | Intercept |
|---|---|---|---|
| 1 | Fresh | None | Fixed zero |
| 2 | Fresh | None | Fitted from zero |
| 3 | Reuse | None | Fixed zero |
| 4 | Reuse | Scales | Fixed zero |
| 5 | Reuse | None | Fitted from zero |
| 6 | Reuse | Scales | Fitted from zero |
Table 4: Matched training arms, in the order shown in \cref fig:bridge-controlled. Centering and scales use population values unless listed as estimated. “None” denotes no estimated preprocessing. U uses divisor one in every arm. Every fitted intercept is unscaled and uses the same learning rate as the transformed coefficients.
Figure 7: Both calibrated risk orderings persist across the eight training arms. Negative SD-minus-U rare risk favors SD. Colors distinguish the three caps; bars show two paired Monte Carlo standard errors over 4096 evaluation replications, conditional on the calibration choices. The last two rows successively add empirical centering and response-mean initialization to reuse with estimated scales and a fitted intercept. These comparisons concern two signal pairs and finite searches, not the full theorem families.
Figure 8: Changing selection on the same candidate paths at the largest cap for each prevalence. Calibrated choices are fixed before evaluation; the other points select by rare validation loss with the indicated sample size. Negative gaps favor SD. The blue curve includes all changes through arm 8. Bars show two paired Monte Carlo standard errors, conditional on the calibration choices. All panels share the same vertical scale and limits, with linear spacing between and and logarithmic spacing outside this interval to keep both small and large gaps visible. Tick labels give the original risk differences. Lines connect discrete selection procedures and do not imply monotonic behavior with sample size.
| Selection | Scaling | Median | 5th pct. | 95th pct. | Worse (%) |
|---|---|---|---|---|---|
| Fitted intercept | |||||
| Exact risk | U | ||||
| Exact risk | SD | ||||
| Validation (2048) | U | ||||
| Validation (2048) | SD | ||||
| Fixed zero intercept (baseline) | |||||
Table 5: Selection-gain distributions for the example in \cref tab:bridge-selection: fresh observations, , , and . Quantiles use 4096 evaluation runs. “Worse” is the percentage with negative gain, meaning higher population rare risk than the calibrated predictor of the same scaling.
Figure 9: Empirical cumulative distributions of selection gains for \cref tab:bridge-gain-distributions. A curve farther to the right indicates larger gains. The horizontal scale is symmetric logarithmic, linear for absolute gains at most ; negative gains mean selection worsens risk. Both arms and scalings use the same 4096 evaluation streams.
| Selection | Scaling | Common | Intercept | Rare | Cross | Total |
|---|---|---|---|---|---|---|
| Fitted intercept | ||||||
| Calibrated | U | |||||
| Calibrated | SD | |||||
| Exact risk | U | |||||
| Exact risk | SD | |||||
| Validation (2048) | U | |||||
Table 6: Mean rare-risk contributions for the primary example in \cref tab:bridge-selection, using the same 4096 paired evaluation runs. Columns give , , , , and their sum. Rounding can obscure exact addition. The baseline’s centered intercept and cross terms are exactly zero.
More in this topic
Statistical Learning Theory
Statistical learning theory studies when data-driven algorithms generalize, how many samples they need, and what statistical or computational limits apply.
See all 584 papers