Benign Overfitting under Heterogeneous Input Fusion

Houzhen Liu; Xiaobo Xia

cs.LGOct 7, 2026

Benign Overfitting under Heterogeneous Input Fusion

Authors: Houzhen Liu, Xiaobo Xia

Organizations: University of Science and Technology of China

Abstract

Benign overfitting is extensively studied when learning from a single high-dimensional input, but its behavior under heterogeneous input fusion remains largely unexplored. We study this question for minimum-norm linear interpolation under a heterogeneous Gaussian design, comparing two statistically dependent input blocks with their fusion while holding the underlying population task fixed. For regression, we identify a full-spectrum covariance certificate whose asymptotic status is independent of the cutoff threshold and prove that it is preserved by every positive-semidefinite joint covariance consistent with the two marginals. This protection is sharp, yet it does not extend to all benign regression problems: outside the certified regime, two benign marginals can have a harmful fusion. For one-sparse Gaussian classification, benignity in the regular regime is characterized by the balance between surviving predictive signal and nuisance contamination. Fusion can move these two quantities in opposite directions, and within this model class every marginal-to-joint benign/non-benign pattern is attainable. We further show that the same fused input can have qualitatively different effects on regression and classification. These results establish that benign overfitting under heterogeneous fusion is determined by the joint signal and spectral geometry created by input interaction, rather than by marginal benignity alone.

Figures & tables

Pattern	$\operatorname{spec}(\Sigma_{1})$	$\operatorname{spec}(\Sigma_{2})$	Spectral status
$\mathsf{B}^{\mathrm{FS}}\!+\!\mathsf{B}^{\mathrm{FS}}\!\to\!\mathsf{B}^{\mathrm{FS}}$	$1\uplus\epsilon^{[p]}$	$1\uplus\epsilon^{[p]}$	1: cert.; 2: cert.; $J$ : cert.
$\mathsf{B}^{\mathrm{FS}}\!+\!\mathsf{B}^{\mathrm{FS}}\!\to\!\mathsf{N}^{\mathrm{FS}}$	arbitrary $\mathsf{B}^{\mathrm{FS}}$	arbitrary $\mathsf{B}^{\mathrm{FS}}$	Impossible by Corollary 4
$\mathsf{B}^{\mathrm{FS}}\!+\!\mathsf{N}^{\mathrm{FS}}\!\to\!\mathsf{B}^{\mathrm{FS}}$	$1\uplus\epsilon^{[p]}$	$1\uplus(n^{-2})^{[m]}\uplus\rho^{[p-m]}$	1: cert.; 2: (iii); $J$ : cert.
$\mathsf{B}^{\mathrm{FS}}\!+\!\mathsf{N}^{\mathrm{FS}}\!\to\!\mathsf{N}^{\mathrm{FS}}$	$1\uplus\eta^{[p]}$	$1\uplus(n^{-2})^{[m]}\uplus\eta^{[p-m]}$	1: cert.; 2: (iii); $J$ : (iii).
$\mathsf{N}^{\mathrm{FS}}\!+\!\mathsf{N}^{\mathrm{FS}}\!\to\!\mathsf{B}^{\mathrm{FS}}$	$1\uplus\epsilon^{[m]}\uplus\rho^{[p-m]}$	$\epsilon^{[p+1]}$	1: (iii); 2: (i); $J$ : cert.
$\mathsf{N}^{\mathrm{FS}}\!+\!\mathsf{N}^{\mathrm{FS}}\!\to\!\mathsf{N}^{\mathrm{FS}}$	$1\uplus(n^{-2})^{[m]}\uplus\rho^{[p-m]}$	$1\uplus(n^{-2})^{[m]}\uplus\rho^{[p-m]}$	1: (iii); 2: (iii); $J$ : (iii).

Table 1: Complete spectral realization of the interaction pattern for the threshold-free full-spectrum surrogate 1 1 1 The labels

1,2,J

refer to

\Sigma_{1},\Sigma_{2},\Sigma_{J}

, respectively; cert. means membership in

\mathsf{B}^{\mathrm{FS}}

, while (i) or (iii) identifies the corresponding violated condition in ( 29 ). These are covariance-level indicators only, not task-level benign/non-benign conclusions. . The two marginal states are interchangeable. The impossible row is excluded by Corollary 4 . Every other row is witnessed by the displayed spectra and the nondegenerate couplings constructed and verified in Appendix A.4 .

Pattern	Nuisance spectrum 1	Nuisance spectrum 2	$(\Delta_{1},\Delta_{2},\Delta_{J})$
$\mathsf{B}^{\mathrm{cls}}+\mathsf{B}^{\mathrm{cls}}\to\mathsf{B}^{\mathrm{cls}}$	$(n^{-1})^{[n^{2}]}$	$(n^{-1})^{[n^{2}]}$	$(n^{-1},n^{-1},n^{-1})$
$\mathsf{B}^{\mathrm{cls}}+\mathsf{N}^{\mathrm{cls}}\to\mathsf{B}^{\mathrm{cls}}$	$(n^{-1})^{[n^{2}]}$	$(n^{-1})^{[p_{n}]}$	$(n^{-1},\rho^{-1},n^{-1})$
$\mathsf{B}^{\mathrm{cls}}+\mathsf{N}^{\mathrm{cls}}\to\mathsf{N}^{\mathrm{cls}}$	$(n^{-2})^{[n^{2}]}$	$1^{[n^{2}]}$	$(n^{-1},n,n)$
$\mathsf{N}^{\mathrm{cls}}+\mathsf{N}^{\mathrm{cls}}\to\mathsf{N}^{\mathrm{cls}}$	$(n^{-1})^{[p_{n}]}$	$(n^{-1})^{[p_{n}]}$	$(\rho^{-1},\rho^{-1},(2\rho)^{-1})$
$\mathsf{B}^{\mathrm{cls}}+\mathsf{B}^{\mathrm{cls}}\to\mathsf{N}^{\mathrm{cls}}$	$(n^{-2})^{[n^{4}]}$	$(n^{2})^{[q_{n}]}\uplus(n^{-2})^{[n^{2}]}$	$(n^{-1},n^{-1/2},n^{3/2})$
$\mathsf{N}^{\mathrm{cls}}+\mathsf{N}^{\mathrm{cls}}\to\mathsf{B}^{\mathrm{cls}}$	$(n^{\ell-1/2})^{[n^{2}]}$	$a_{n}^{[p_{n}]}\uplus b_{n}^{[n^{2}-p_{n}]}$	$(d_{n},\rho^{-1},j_{n})$

Table 2: Complete spectral realization of the six classification patterns in Theorem 10 . The two middle columns specify the target marginal nuisance spectra. The entrywise nonzero coupling maps, common-label constructions, signal checks, and all

S_{k},R_{k},\Delta

calculations are given in Appendix B.3 . The two marginal states are interchangeable, and only orders relevant to

\Delta

are displayed.

Figure 1: Population regression excess risk for the five attainable interaction patterns. The panels follow the ordering in Table 1 ; $\mathsf{B}^{\mathrm{reg}}$ and $\mathsf{N}^{\mathrm{reg}}$ denote task-level benign and non-benign regression behavior for the displayed response. Curves show unsmoothed Monte Carlo medians, and shaded bands show the interquartile range. The corresponding repetition counts are specified in Section 6.1 .

Figure 2: The two regression case studies. Panels (a)-(b) illustrate how vanishing innovation energy can be associated with a nonvanishing variance obstruction. Panels (c)-(d) probe Bartlett’s logarithmic bias gap: the two marginal risks decrease while the fused risk stays separated, and the marginal quantities $r_{0}/n$ and $r_{0}/(n\log n)$ move in opposite directions. The logarithmic denominator in panel (d) is the finite-sample proxy for $\log(1+r_{0})$ , which is asymptotically equivalent in this construction. Risk curves are unsmoothed medians with interquartile bands.

Figure 3: Population classification excess risk for all six unordered interaction patterns. Each construction uses one common latent label mechanism and input-specific one-sparse signal directions. The six panels illustrate the absence of a nontrivial Boolean closure law based only on the two marginal benign/non-benign states. Curves are unsmoothed medians with interquartile bands.

Figure 4: Cross-task separation under the same heterogeneous fusion and common latent score. Panels (a)–(b) use the first construction of Theorem 11 : the displayed finite-sample trends are consistent with regression non-benignity and classification rescue. Panels (c)–(d) use the second construction: the displayed trends are consistent with benign marginal inputs for both tasks, regression degradation after fusion, and preserved classification benignity. The inset reports the vanishing marginal Bayes advantages in panel (d). All synthetic panels extend through $n=15000$ . All points use $50$ repetitions. Curves are unsmoothed medians with interquartile bands.

Figure 5: Constant-factor robustness of the small-innovation mechanism. Panels (a)–(e) report population regression excess risk for the five fixed shoulder multipliers $a$ , with one problem instance per panel; the $X_{1}$ curve is repeated as a common reference because its law does not depend on $a$ . Panel (f) compares the finite-sample fixed-tail quantity at $n=15000$ against the analytic limits $6a^{2}/(1+6a)^{2}$ for $X_{2}$ and $6a^{2}/(2+6a)^{2}$ for $X_{J}$ ; dashed curves are analytic, not fitted. Every shaded band is the Monte Carlo interquartile range for its own fixed problem instance; no band aggregates variation across multipliers.

Figure 6: Out-of-model sanity check on the UCI Multiple Features data.

X_{1}

is the profile-correlation view,

X_{2}

the pixel-average view, and

X_{J}

their concatenation. Solid curves are median clean-test errors and shaded bands are interquartile ranges over 30 full-row-rank stratified splits. Dashed magenta curves use the right axis and report

\Delta_{\mathrm{fusion}}

. Only training labels are flipped.

Explore similar work

Benign Overfitting for General Norms and Distributions
Sep 27, 2026Daniel Barzilai, Ohad ShamirBenign Overfitting Statistical Learning Theory
Weizmann Institute of Science · University of Toronto & Vector Institute
Universality of Benign Overfitting in Binary Linear Classification
Jan 17, 2025Ichiro Hashimoto, Stanislav Volgushev, Piotr ZwiernikNeural Network Generalization Benign Overfitting
Department of Statistical Sciences, University of Toronto
When Does $\ell_2$ -Boosting Overfit Benignly? High-Dimensional Risk Asymptotics and the $\ell_1$ Implicit Bias
May 7, 2026Ye Su, Jian Li, Yong LiuBenign Overfitting
Shenzhen Institutes of Advanced Technology Chinese Academy of Sciences Shenzhen 518055, China · School of Artificial Intelligence Beijing Normal University Beijing, 100875, China · Gaoling School of Artificial Intelligence Renmin University of China Beijing 100872, China

Sep 27, 2026, cs.LGJ/K move · Enter open · S save

Benign Overfitting for General Norms and Distributions

Daniel Barzilai, Ohad Shamir

Weizmann Institute of Science · University of Toronto & Vector Institute

Benign Overfitting Statistical Learning Theory

Open paper AskPDF arXiv

Understanding why predictors can generalize despite interpolating noisy training data is a central puzzle in machine learning. Most work on such "benign overfitting" studies minimum-2-norm linear regression, reflecting the inductive bias of gradient descent. However, modern optimizers such as Adam and Muon use non-Euclidean update geometries, favoring solutions associated with other norms. Analyzing regression for non-Euclidean norms is substantially more difficult, with known results essentially limited to Gaussians. In this paper, we develop a method to analyze benign overfitting in linear regression for general norms and general (sub-Gaussian) distributions. As a special case, we prove that minimum-p-norm interpolation with p>1 can benignly overfit even for non-Gaussian distributions, under suitable conditions. Perhaps surprisingly, for the 1-norm, benign overfitting does not hold in general for well-behaved (but non-Gaussian) distributions, showing that existing positive 1-norm results rely crucially on Gaussianity. Our proof analyzes the geometry of the dual optimization problem, using concentration and central limit tools to show it is approximately Euclidean in many high-dimensional cases.

Abstract

Explore similar work

Benign Overfitting for General Norms and Distributions

Universality of Benign Overfitting in Binary Linear Classification

When Does ℓ2\ell_2ℓ2​-Boosting Overfit Benignly? High-Dimensional Risk Asymptotics and the ℓ1\ell_1ℓ1​ Implicit Bias

When Does $\ell_2$ -Boosting Overfit Benignly? High-Dimensional Risk Asymptotics and the $\ell_1$ Implicit Bias