Benign Overfitting under Heterogeneous Input Fusion

Houzhen Liu; Xiaobo Xia

cs.LGOct 7, 2026

Benign Overfitting under Heterogeneous Input Fusion

Authors: Houzhen Liu, Xiaobo Xia

Organizations: University of Science and Technology of China

Abstract

Benign overfitting is extensively studied when learning from a single high-dimensional input, but its behavior under heterogeneous input fusion remains largely unexplored. We study this question for minimum-norm linear interpolation under a heterogeneous Gaussian design, comparing two statistically dependent input blocks with their fusion while holding the underlying population task fixed. For regression, we identify a full-spectrum covariance certificate whose asymptotic status is independent of the cutoff threshold and prove that it is preserved by every positive-semidefinite joint covariance consistent with the two marginals. This protection is sharp, yet it does not extend to all benign regression problems: outside the certified regime, two benign marginals can have a harmful fusion. For one-sparse Gaussian classification, benignity in the regular regime is characterized by the balance between surviving predictive signal and nuisance contamination. Fusion can move these two quantities in opposite directions, and within this model class every marginal-to-joint benign/non-benign pattern is attainable. We further show that the same fused input can have qualitatively different effects on regression and classification. These results establish that benign overfitting under heterogeneous fusion is determined by the joint signal and spectral geometry created by input interaction, rather than by marginal benignity alone.

Figures & tables

Pattern	$\operatorname{spec}(\Sigma_{1})$	$\operatorname{spec}(\Sigma_{2})$	Spectral status
$\mathsf{B}^{\mathrm{FS}}\!+\!\mathsf{B}^{\mathrm{FS}}\!\to\!\mathsf{B}^{\mathrm{FS}}$	$1\uplus\epsilon^{[p]}$	$1\uplus\epsilon^{[p]}$	1: cert.; 2: cert.; $J$ : cert.
$\mathsf{B}^{\mathrm{FS}}\!+\!\mathsf{B}^{\mathrm{FS}}\!\to\!\mathsf{N}^{\mathrm{FS}}$	arbitrary $\mathsf{B}^{\mathrm{FS}}$	arbitrary $\mathsf{B}^{\mathrm{FS}}$	Impossible by Corollary 4
$\mathsf{B}^{\mathrm{FS}}\!+\!\mathsf{N}^{\mathrm{FS}}\!\to\!\mathsf{B}^{\mathrm{FS}}$	$1\uplus\epsilon^{[p]}$	$1\uplus(n^{-2})^{[m]}\uplus\rho^{[p-m]}$	1: cert.; 2: (iii); $J$ : cert.
$\mathsf{B}^{\mathrm{FS}}\!+\!\mathsf{N}^{\mathrm{FS}}\!\to\!\mathsf{N}^{\mathrm{FS}}$	$1\uplus\eta^{[p]}$	$1\uplus(n^{-2})^{[m]}\uplus\eta^{[p-m]}$	1: cert.; 2: (iii); $J$ : (iii).
$\mathsf{N}^{\mathrm{FS}}\!+\!\mathsf{N}^{\mathrm{FS}}\!\to\!\mathsf{B}^{\mathrm{FS}}$	$1\uplus\epsilon^{[m]}\uplus\rho^{[p-m]}$	$\epsilon^{[p+1]}$	1: (iii); 2: (i); $J$ : cert.
$\mathsf{N}^{\mathrm{FS}}\!+\!\mathsf{N}^{\mathrm{FS}}\!\to\!\mathsf{N}^{\mathrm{FS}}$	$1\uplus(n^{-2})^{[m]}\uplus\rho^{[p-m]}$	$1\uplus(n^{-2})^{[m]}\uplus\rho^{[p-m]}$	1: (iii); 2: (iii); $J$ : (iii).

Table 1: Complete spectral realization of the interaction pattern for the threshold-free full-spectrum surrogate 1 1 1 The labels

1,2,J

refer to

\Sigma_{1},\Sigma_{2},\Sigma_{J}

, respectively; cert. means membership in

\mathsf{B}^{\mathrm{FS}}

, while (i) or (iii) identifies the corresponding violated condition in ( 29 ). These are covariance-level indicators only, not task-level benign/non-benign conclusions. . The two marginal states are interchangeable. The impossible row is excluded by Corollary 4 . Every other row is witnessed by the displayed spectra and the nondegenerate couplings constructed and verified in Appendix A.4 .

Pattern	Nuisance spectrum 1	Nuisance spectrum 2	$(\Delta_{1},\Delta_{2},\Delta_{J})$
$\mathsf{B}^{\mathrm{cls}}+\mathsf{B}^{\mathrm{cls}}\to\mathsf{B}^{\mathrm{cls}}$	$(n^{-1})^{[n^{2}]}$	$(n^{-1})^{[n^{2}]}$	$(n^{-1},n^{-1},n^{-1})$
$\mathsf{B}^{\mathrm{cls}}+\mathsf{N}^{\mathrm{cls}}\to\mathsf{B}^{\mathrm{cls}}$	$(n^{-1})^{[n^{2}]}$	$(n^{-1})^{[p_{n}]}$	$(n^{-1},\rho^{-1},n^{-1})$
$\mathsf{B}^{\mathrm{cls}}+\mathsf{N}^{\mathrm{cls}}\to\mathsf{N}^{\mathrm{cls}}$	$(n^{-2})^{[n^{2}]}$	$1^{[n^{2}]}$	$(n^{-1},n,n)$
$\mathsf{N}^{\mathrm{cls}}+\mathsf{N}^{\mathrm{cls}}\to\mathsf{N}^{\mathrm{cls}}$	$(n^{-1})^{[p_{n}]}$	$(n^{-1})^{[p_{n}]}$	$(\rho^{-1},\rho^{-1},(2\rho)^{-1})$
$\mathsf{B}^{\mathrm{cls}}+\mathsf{B}^{\mathrm{cls}}\to\mathsf{N}^{\mathrm{cls}}$	$(n^{-2})^{[n^{4}]}$	$(n^{2})^{[q_{n}]}\uplus(n^{-2})^{[n^{2}]}$	$(n^{-1},n^{-1/2},n^{3/2})$
$\mathsf{N}^{\mathrm{cls}}+\mathsf{N}^{\mathrm{cls}}\to\mathsf{B}^{\mathrm{cls}}$	$(n^{\ell-1/2})^{[n^{2}]}$	$a_{n}^{[p_{n}]}\uplus b_{n}^{[n^{2}-p_{n}]}$	$(d_{n},\rho^{-1},j_{n})$

Table 2: Complete spectral realization of the six classification patterns in Theorem 10 . The two middle columns specify the target marginal nuisance spectra. The entrywise nonzero coupling maps, common-label constructions, signal checks, and all

S_{k},R_{k},\Delta

calculations are given in Appendix B.3 . The two marginal states are interchangeable, and only orders relevant to

\Delta

are displayed.

Figure 1: Population regression excess risk for the five attainable interaction patterns. The panels follow the ordering in Table 1 ; $\mathsf{B}^{\mathrm{reg}}$ and $\mathsf{N}^{\mathrm{reg}}$ denote task-level benign and non-benign regression behavior for the displayed response. Curves show unsmoothed Monte Carlo medians, and shaded bands show the interquartile range. The corresponding repetition counts are specified in Section 6.1 .

Figure 2: The two regression case studies. Panels (a)-(b) illustrate how vanishing innovation energy can be associated with a nonvanishing variance obstruction. Panels (c)-(d) probe Bartlett’s logarithmic bias gap: the two marginal risks decrease while the fused risk stays separated, and the marginal quantities $r_{0}/n$ and $r_{0}/(n\log n)$ move in opposite directions. The logarithmic denominator in panel (d) is the finite-sample proxy for $\log(1+r_{0})$ , which is asymptotically equivalent in this construction. Risk curves are unsmoothed medians with interquartile bands.

Figure 3: Population classification excess risk for all six unordered interaction patterns. Each construction uses one common latent label mechanism and input-specific one-sparse signal directions. The six panels illustrate the absence of a nontrivial Boolean closure law based only on the two marginal benign/non-benign states. Curves are unsmoothed medians with interquartile bands.

Figure 4: Cross-task separation under the same heterogeneous fusion and common latent score. Panels (a)–(b) use the first construction of Theorem 11 : the displayed finite-sample trends are consistent with regression non-benignity and classification rescue. Panels (c)–(d) use the second construction: the displayed trends are consistent with benign marginal inputs for both tasks, regression degradation after fusion, and preserved classification benignity. The inset reports the vanishing marginal Bayes advantages in panel (d). All synthetic panels extend through $n=15000$ . All points use $50$ repetitions. Curves are unsmoothed medians with interquartile bands.

Figure 5: Constant-factor robustness of the small-innovation mechanism. Panels (a)–(e) report population regression excess risk for the five fixed shoulder multipliers $a$ , with one problem instance per panel; the $X_{1}$ curve is repeated as a common reference because its law does not depend on $a$ . Panel (f) compares the finite-sample fixed-tail quantity at $n=15000$ against the analytic limits $6a^{2}/(1+6a)^{2}$ for $X_{2}$ and $6a^{2}/(2+6a)^{2}$ for $X_{J}$ ; dashed curves are analytic, not fitted. Every shaded band is the Monte Carlo interquartile range for its own fixed problem instance; no band aggregates variation across multipliers.

Figure 6: Out-of-model sanity check on the UCI Multiple Features data.

X_{1}

is the profile-correlation view,

X_{2}

the pixel-average view, and

X_{J}

their concatenation. Solid curves are median clean-test errors and shaded bands are interquartile ranges over 30 full-row-rank stratified splits. Dashed magenta curves use the right axis and report

\Delta_{\mathrm{fusion}}

. Only training labels are flipped.

Explore similar work

Sep 27, 2026cs.LG

Benign Overfitting for General Norms and Distributions

Understanding why predictors can generalize despite interpolating noisy training data is a central puzzle in machine learning. Most work on such "benign overfitting" studies minimum-2-norm linear regression, reflecting the inductive bias of gradient descent. However, modern optimizers such as Adam and Muon use non-Euclidean update geometries, favoring solutions associated with other norms. Analyzing regression for non-Euclidean norms is substantially more difficult, with known results essentially limited to Gaussians. In this paper, we develop a method to analyze benign overfitting in linear regression for general norms and general (sub-Gaussian) distributions. As a special case, we prove that minimum-p-norm interpolation with p>1 can benignly overfit even for non-Gaussian distributions, under suitable conditions. Perhaps surprisingly, for the 1-norm, benign overfitting does not hold in general for well-behaved (but non-Gaussian) distributions, showing that existing positive 1-norm results rely crucially on Gaussianity. Our proof analyzes the geometry of the dual optimization problem, using concentration and central limit tools to show it is approximately Euclidean in many high-dimensional cases.

Daniel Barzilai, Ohad Shamir

Weizmann Institute of Science · University of Toronto & Vector Institute

Benign Overfitting Statistical Learning Theory

Ask PDF

Jan 17, 2025cs.LG

Universality of Benign Overfitting in Binary Linear Classification

The practical success of deep learning has led to the discovery of several surprising phenomena. One of these phenomena, that has spurred intense theoretical research, is ``benign overfitting'': deep neural networks seem to generalize well in the over-parametrized regime even though the networks show a perfect fit to noisy training data. It is now known that benign overfitting also occurs in various classical statistical models. For linear maximum margin classifiers, benign overfitting has been established theoretically in a class of mixture models with very strong assumptions on the covariate distribution. However, even in this simple setting, many questions remain open. For instance, most of the existing literature focuses on the noiseless case where all true class labels are observed without errors, whereas the more interesting noisy case remains poorly understood. We provide a comprehensive study of benign overfitting for linear maximum margin classifiers. We discover a phase transition in test error bounds for the noisy model which was previously unknown and provide some geometric intuition behind it. We further considerably relax the required covariate assumptions in both the noisy and noiseless cases. Our results demonstrate that benign overfitting of maximum margin classifiers holds in a much wider range of scenarios than was previously known and provide new insights into the underlying mechanisms.

Ichiro Hashimoto, Stanislav Volgushev, Piotr Zwiernik

Department of Statistical Sciences, University of Toronto

Neural Network Generalization Benign Overfitting Binary Classification Classification Generalization Bounds

Ask PDF

May 7, 2026cs.LG

When Does $\ell_2$ -Boosting Overfit Benignly? High-Dimensional Risk Asymptotics and the $\ell_1$ Implicit Bias

Benign overfitting is well-characterized in

\ell_2

geometries, but its behavior under the

\ell_1

implicit bias of greedy ensembles remains challenging. The analytical barrier stems from the non-linear coupling of coordinate selection thresholds, which invalidates standard spectral resolvent tools. To isolate this algorithmic bias, we characterize the high-dimensional risk of continuous-time

\ell_2

-Boosting over

p

features and

n

samples. By coupling the Convex Gaussian Minimax Theorem with delicate asymptotic expansions of double-sided truncated Gaussian moments, we analytically resolve the non-smooth

\ell_1

interpolant. Under an isotropic pure-noise model, we prove that benign overfitting fails at the linear rate: greedy selection localizes noise into sparse active sets, and the excess variance decays at a logarithmic rate

Θ(σ^2/\log(p/n))

for noise variance

σ^2

. We remark that while this localization mechanism should persist in the presence of signals, the exact signal-noise decomposition remains an open problem. For spiked-isotropic designs with

k^*

head eigenvalues and

r_2 = p - k^*

tail dimensions, the risk converges to zero when

r_{2} \gg n

, but only at a logarithmic rate

Θ(σ^2/\log(r_2/n))

, which is slower than the linear decay observed in

\ell_2

geometries. To avoid this slow convergence, we analyze the non-smooth subdifferential dynamics of the boosting flow. This yields a tuning-free early stopping rule that, under a bounded

\ell_1

-path condition, recovers the Lasso basic inequality and attains the minimax-optimal empirical prediction rate for

\ell_1

-bounded signals.

Ye Su, Jian Li, Yong Liu

Shenzhen Institutes of Advanced Technology Chinese Academy of Sciences Shenzhen 518055, China · School of Artificial Intelligence Beijing Normal University Beijing, 100875, China · Gaoling School of Artificial Intelligence Renmin University of China Beijing 100872, China

Benign Overfitting

Ask PDF

Abstract

Explore similar work

Benign Overfitting for General Norms and Distributions

Universality of Benign Overfitting in Binary Linear Classification

When Does ℓ2\ell_2ℓ2​-Boosting Overfit Benignly? High-Dimensional Risk Asymptotics and the ℓ1\ell_1ℓ1​ Implicit Bias

When Does $\ell_2$ -Boosting Overfit Benignly? High-Dimensional Risk Asymptotics and the $\ell_1$ Implicit Bias