New Multiple Descent Phenomena Induced by Heterogeneous Data

Mark-Anthony Moisescu-Pareja; Elvis Dohmatob

stat.MLOct 27, 2026

New Multiple Descent Phenomena Induced by Heterogeneous Data

Authors: Mark-Anthony Moisescu-Pareja, Elvis Dohmatob

Organizations: Concordia University · Mila – Quebec AI Institute

Abstract

Modern machine learning models are often trained on large datasets assembled from many different sources, whose statistical properties can differ substantially. Understanding how this heterogeneity affects generalization performance is therefore important for explaining how model performance changes as model size increases. In this work, we study this question in linear random-feature models and show that mixtures of datasets with different covariance structures can give rise to a range of multiple-descent phenomena. We derive exact asymptotic expressions for the test error of each data source in the high-dimensional regime, and show how heterogeneity in covariance scale and spectral structure can introduce additional, group-dependent interpolation thresholds as model size increases. These phenomena arise even with noiseless labels and linear activations, showing that neither label noise nor activation nonlinearity is necessary for multiple-descent. Our results therefore provide a different perspective from classical explanations of multiple descent. More broadly, our results have potential implications for algorithmic fairness by showing that model scaling can affect different data groups unevenly, amplifying disparities in their generalization errors. We validate our theoretical results on synthetic Gaussian mixtures and real-world datasets, including MNIST and CIFAR-10.

Figures & tables

Figure 1: Triple descent in a linear RF model. Cartoon bias curves for a heterogeneous mixture of

n_{1}

samples with power-law covariance

\Sigma_{1}=\Sigma=\operatorname{diag}(1,1/2,\ldots,1/d)

and

n_{2}

samples with rescaled covariance

\Sigma_{2}=\varepsilon\Sigma

, where

\varepsilon\ll 1

. The group-wise bias terms

B_{1}

( left ) and

B_{2}

( right ) are plotted against

\psi=m/n

, with network width

m

and total sample size

n=n_{1}+n_{2}

. For small

\varepsilon

, the peaks near

\psi=p_{1}:=n_{1}/n

(

m=n_{1}

) have respective heights asymptotic to

B(\varepsilon):=\frac{\eta(p_{1}/\phi)}{2}\sqrt{\frac{p_{1}}{p_{2}\varepsilon}}

and

\varepsilon B(\varepsilon)

, as predicted by Corollary 7 . Here,

\phi=d/n>1

d

is the input dimension, and

\eta

is the explicit function defined in that corollary. Both curves diverge at the classical interpolation threshold

\psi=1

(

m=n

). Horizontal dotted lines mark the large-width limits

\eta(1/\phi)

and

\varepsilon\eta(1/\phi)

Figure 2: Triple-descent in linear RF on synthetic data mixture with different spectral exponents. We train a linear RF model ( 3 ) on a mixture of two Gaussian groups ( 1 )–( 2 ) with total sample size

n=1000

. Group 1 has covariance matrix

\Sigma_{1}=\operatorname{diag}(\lambda_{1}^{(1)},\ldots,\lambda_{d}^{(1)})

, where

\lambda_{i}^{(1)}\propto i^{-\alpha}

and

\alpha=1

. Group 2 is defined analogously with

\lambda_{i}^{(2)}\propto i^{-\beta}

and

\beta=2.2

. Thus, Group 1 has the flatter spectrum and corresponds to the harder-to-learn population. We plot the test error of Group 1 ( left ) and Group 2 ( right ) versus the model’s width

m

for different mixture proportions

p_{1}\in\{1/4,1/2,3/4\}

, over 20 runs (error bars). In addition to the classical interpolation threshold at

m=1000

, each curve exhibits an additional peak near the group-specific threshold

m=n_{1}=p_{1}n

as predicted by Corollary 7 .

Figure 3: Multiple-descent in linear RF on synthetic data mixture with different scales. We train a linear RF model ( 3 ) on a mixture of four Gaussian groups with total sample size

n=600

. For Group k, the inputs are sampled as

x^{(k)}\sim\mathcal{N}(0,\Sigma_{k})

, with isotropic covariance

\Sigma_{k}=\varepsilon_{k}I_{d}

. The four groups have sizes

(n_{1},n_{2},n_{3},n_{4})=(50,100,200,250)

and scales

(\varepsilon_{1},\varepsilon_{2},\varepsilon_{3},\varepsilon_{4})=(1,10^{-3},10^{-5},10^{-7})

. We plot the test error of Group 2 versus the model’s width

m

over 20 runs (error bars). As predicted by Theorem 6 , the test error of Group 2 exhibits multiple interpolation peaks near the cumulative sample sizes

m=n_{1}=50

m=n_{1}+n_{2}=150

m=n_{1}+n_{2}+n_{3}=350

, and

m=n=600

. An extended figure with test error curves for all 4 groups is presented in Appendix A.1 .

Figure 4: Triple-descent in linear RF model on MNIST & CIFAR-10 mixtures with different scales. We train linear RF models ( 3 ) on two-group mixtures constructed from MNIST and CIFAR-10. For each dataset, Group 1 consists of the original inputs, while Group 2 is obtained by re-scaling the inputs by

\sqrt{\varepsilon}

, so that

\Sigma_{2}=\varepsilon\Sigma_{1}

. We use

\varepsilon=10^{-3}

for MNIST and

\varepsilon=10^{-5}

for CIFAR-10. In all experiments, the total training sample size is fixed at

n=500

, and we vary the mixture proportion

p_{1}\in\{1/4,1/2,3/4\}

. We plot the test error of Group 2 derived from MNIST ( left ) and derived from CIFAR-10 ( right ) as a function of model size

m

, over 10 runs (error bars). For both groups, an additional interpolation peak appears near the group-specific threshold

m=n_{1}=p_{1}n

, shifting with the mixture proportion, while the classic interpolation peak remains fixed at

m=n=500

. An extended figure (with Group 1 for both datasets) can be found in Appendix A.2 .

Figure 5: Triple-descent in linear RF model on MNIST & CIFAR-10 mixtures with different exponents. We train linear RF models ( 3 ) on two-group mixtures constructed from MNIST and CIFAR-10. For each dataset, Group 1 consists of the original inputs, while Group 2 is obtained through a nonlinear transformation that produces a flatter empirical covariance spectrum. All experiments fix

n=300

and vary

p_{1}\in\{1/4,1/2,3/4\}

. We plot the test error of Group 2 derived from MNIST ( left ) and derived from CIFAR-10 ( right ) as a function of model size

m

, over 50 runs for MNIST and 30 runs for CIFAR-10 (error bars). In both groups, an additional peak appears near the group-specific threshold

m=n_{1}=p_{1}n

predicted by our theory (Corollary 7 ), while the classical interpolation peak remains fixed at

m=n=300

. Additional results are provided in Appendix A.2 .

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Multiple-descent in linear RF on synthetic data mixture with different scales. For this experiment, we train a linear random-features model on a mixture of four Gaussian groups with total sample size $n=600$ . For Group k, the inputs are sampled as $x^{(k)}\sim\mathcal{N}(0,\Sigma_{k})$ , with isotropic covariance $\Sigma_{k}=\varepsilon_{k}I_{d}$ . The four groups have sizes $(n_{1},n_{2},n_{3},n_{4})=(50,100,200,250)$ and scales $(\varepsilon_{1},\varepsilon_{2},\varepsilon_{3},\varepsilon_{4})=(1,10^{-3},10^{-5},10^{-7})$ , so that the groups differ only in the overall scale of their input covariances. The labels in each group are generated noiselessly by the same ground-truth weights (i.e. $w_{*}^{(1)}=w_{*}^{(2)}=w_{*}^{(3)}=w_{*}^{(4)}=w_{*}$ ). We plot the test error of Group 1 ( upper left ), Group 2 ( upper right ), Group 3 ( bottom left ), and Group 4 ( bottom right ) versus the model’s width $m$ over 20 runs (error bars). As predicted by Theorem 6 , the test error of each group exhibits multiple interpolation peaks near the cumulative sample sizes $m=n_{1}=50$ , $m=n_{1}+n_{2}=150$ , $m=n_{1}+n_{2}+n_{3}=350$ , and $m=n=600$ , demonstrating that strong heterogeneity in covariance scale can generate arbitrarily many interpolation peaks (that is dependent on the number of mixtures in the data source).

Figure 7: Power-law spectra for MNIST. We plot the trace-normalized eigenvalues of the empirical covariance matrices against eigenvalue rank $k$ for Group 1 (original MNIST inputs) and Group 2 (nonlinearly transformed inputs). Solid lines denote the empirical spectra, while dashed lines denote the corresponding power-law fits.

Figure 8: Power-law spectra for CIFAR-10. We plot the trace-normalized eigenvalues of the empirical covariance matrices against eigenvalue rank $k$ for Group 1 (original CIFAR-10 inputs) and Group 2 (nonlinearly transformed inputs). Solid lines denote the empirical spectra, while dashed lines denote the corresponding power-law fits.

Panel of Figure 9 — Figure 9: Triple-descent in linear RF on MNIST & CIFAR-10 mixtures with different scales. We train linear random-features models on two-group mixtures constructed from MNIST ( top row ) and CIFAR-10 ( bottom row ). For each dataset, Group 1 consists of the original inputs, while Group 2 is obtained by re-scaling the inputs by $\sqrt{\epsilon}$ , so that $\Sigma_{2}=\epsilon\Sigma_{1}$ . We use $\epsilon=10^{-3}$ for MNIST and $\epsilon=10^{-5}$ for CIFAR-10. In both experiments, the total training sample size is fixed at $n=500$ , and we vary the mixture proportion $p_{1}\in\{1/4,1/2,3/4\}$ , corresponding to $n_{1}\in\{125,250,375\}$ . We plot the test error of Group 1 ( left column ) and Group 2 ( right column ) as a function of model size $m$ , over 10 runs (error bars). For both MNIST and CIFAR-10, an additional interpolation peak appears near the group-specific threshold $m=n_{1}=p_{1}n$ , shifting with the mixture proportion, while the classic interpolation peak remains fixed at $m=n=500$ .

Panel of Figure 10 — Figure 10: Triple-descent in linear RF on MNIST & CIFAR-10 mixtures with different exponents. We train linear random-features models on two-group mixtures derived from MNIST ( top row ) and CIFAR-10 ( bottom row ). For each dataset, Group 1 consists of the original inputs, while Group 2 is obtained through a nonlinear transformation that produces a flatter empirical covariance spectrum. In both experiments, the total training sample size is fixed at $n=300$ , and we vary the mixture proportion $p_{1}\in\{1/4,1/2,3/4\}$ , corresponding to $n_{1}\in\{75,150,225\}$ . We plot the test error of Group 1 ( left column ) and Group 2 ( right column ) as a function of model size $m$ , over 50 runs for MNIST and 30 runs for CIFAR-10 (error bars). For both MNIST and CIFAR-10, an additional interpolation peak appears near the group-specific threshold $m=n_{1}=p_{1}n$ predicted by our theory (Corollary 7 ), while the classic interpolation peak remains fixed at $m=300$ .

Double Descent

Double descent describes non-monotonic test error that can peak near interpolation and fall again as model capacity increases.

See all 21 papers

When do data mixtures improve scaling laws? Insights from high-dimensional regression
Sep 29, 2026Diyuan Wu, Lehan Chen, Theodor Misiakiewicz +1High-Dimensional Linear Regression Data Mixture Optimization
Institute of Science and Technology Austria (ISTA). · Department of Applied and Computational Mathematics, Yale University. · Department of Statistics and Data Science, Yale University.
Decoupled Descent: Exact Test Error Tracking Via Approximate Message Passing
Apr 30, 2026Max LovigNeural Network Generalization Gradient Descent
Statistics and Data Science, Yale University
On the Dynamics of Delayed Generalization in Linear Models
Feb 9, 2026Atreya Vedantam, Nataraj Das, Chandrashekar LakshminarayananLinear Classification Neural Network Generalization
Indian Institute of Technology Madras, Chennai.

Sep 29, 2026J/K move · Enter open · S save

When do data mixtures improve scaling laws? Insights from high-dimensional regression

Diyuan Wu, Lehan Chen, Theodor Misiakiewicz, Marco Mondelli

Institute of Science and Technology Austria (ISTA). · Department of Applied and Computational Mathematics, Yale University. · Department of Statistics and Data Science, Yale University.

High-Dimensional Linear Regression Data Mixture Optimization Scaling Laws Statistical Learning Theory

Open paper AskPDF arXiv

Modern machine learning systems are trained on mixtures of data from different domains, and choosing the right mixture can substantially improve downstream performance. Despite an extensive literature on data mixing and reweighting, existing work is largely empirical and it remains unclear when auxiliary data genuinely improves scaling laws rather than merely providing more samples. To gain insight into this question, we study a high-dimensional mixed-data regression model with a shared regression function, heterogeneous covariances and noise levels, and dataset sizes that may grow at different rates. We establish the minimax risk under an ellipsoidal parameter constraint for the general covariance structure and derive deterministic equivalents for the test error of ridge regression under commutative covariances. We then specialize to a target domain and an auxiliary domain with aligned power-law covariance spectra, where the theory yields explicit scaling laws in terms of spectral decay, target regularity, and the relative growth of the two datasets. These laws identify regimes in which combining data mixtures provably yields a faster scaling rate than using either dataset alone. In particular, improving the scaling law requires a specific interplay between spectra and relative sample sizes of the domains. Our numerical experiments on language models exhibit the same qualitative phenomenon: appropriate data mixtures yield a faster decrease in target-domain test loss than training on either domain alone.

Abstract

Double Descent

Explore similar work

When do data mixtures improve scaling laws? Insights from high-dimensional regression

Decoupled Descent: Exact Test Error Tracking Via Approximate Message Passing

On the Dynamics of Delayed Generalization in Linear Models