Quadratic Weak-to-Strong Generalization in Random Feature Networks via Random Matrix Theory

Deborah Oliveira; Elliot Paquette

stat.MLOct 6, 2026

Quadratic Weak-to-Strong Generalization in Random Feature Networks via Random Matrix Theory

Authors: Deborah Oliveira, Elliot Paquette

Organizations: McGill University Mila – Quebec AI Institute

Abstract

Weak-to-strong generalization is the phenomenon where a strong student model trained with labels produced by a weak teacher model is able to generalize better than the teacher. In this paper, we study this phenomenon in two-layer random feature networks where the model strength is determined by its width. Using tools from random matrix theory, we derive deterministic equivalents for the population errors of an optimally trained teacher and a student trained with gradient flow. For ReLU activation and a pure spherical harmonic target, we obtain sharp asymptotics under a Gaussian universality assumption, showing a quadratic improvement: the student error scales as the square of the teacher error. These results attain the general lower bound of Medvedev at al (2025). We also analyze how the student behaves under more general stopping times and targets supported on multiple harmonic degrees, characterizing the regimes in which weak-to-strong generalization occurs and identifying the transition between quadratic, non-quadratic, and no improvement.

Figures & tables

Figure 1: Quadratic improvement and the stopping window. Unit degree-two target,

m=d^{3}

, with time divided by

T_{2}=4\log(d)/\sigma_{2}^{2}

, the stopping time

T

of Theorem 2 for

k_{0}=2

and

m=d^{3}

. Left: student loss; right: loss divided by squared teacher loss. Lines are the deterministic equivalent; open markers (population-flow samples) are population spherical-teacher observations at

d=8,16,28

(32, 32, and 8 draws); see Appendix G for details. See also uncertainty and numerical approximation quantification in Appendix G.2 . The dotted line marks

T_{2}

Panel of Figure 2 — Figure 2: Stopping-time phases (Theorem 4 ). Left: the five regions in the plane of teacher capacity exponent $\alpha$ and stopping degree $\gamma$ (drawn for $k_{0}=2$ ; admissible degrees form the discrete set $\mathcal{K}$ ). The marked point is the setting of Figure 1 . Right: the phases, with the last four for $\alpha>k_{0}$ , where $\overline{\mathcal{L}}_{\rm TE}\asymp d^{-(\alpha-k_{0})}$ , and $\rho=(3\alpha-k_{0}-2\gamma)/(\alpha-k_{0})\in(1,2)$ . The $\overline{B}/\overline{V}$ indicates which error terms dominates the bias/variance decomposition. Details are in Appendix D.1 .

Panel of Figure 3 — Figure 3: Weak-to-strong generalization regimes as a function of the stopping time exponent $\gamma$ , for (a) $\alpha\leq k_{0}$ and (b) $\alpha>k_{0}$ . guaranteed WTS, for every target; conditional WTS, governed by ( 6.3 ); no WTS. The interval includes $\gamma=k_{0}$ and excludes $\gamma=\alpha$ .

Figure 4: A timing conflict that prevents quadratic improvement. Deterministic-equivalent curves for the degree-two/degree-four mixture in ( 6.7 ), with

B=C^{2}/24

and

C=(12\pi-37)/24

(Remark 3 ). Left: student-to-teacher loss ratio; right: student loss divided by

\overline{\mathcal{L}}_{\rm TE}^{5/3}

. Dots are numerically attained minima. The right-hand curves converge to

C^{1/3}/48

, the leading constant of the normalized optimum in ( 6.7 ) (Remark 3 ).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

term	at a free $s$	at the best $s$ , namely $s=\gamma$
(A) alignment	$d^{\,2s-\alpha}\log^{2}d\cdot\mathcal{L}_{\rm TE}$	$d^{\,2\gamma-\alpha}\log^{2}d\cdot\mathcal{L}_{\rm TE}$
(B) learned high degrees	$d^{-2(s^{+}-\gamma)}\log^{2}d\cdot\mathcal{L}_{\rm TE}$	$d^{-2(\gamma^{+}-\gamma)}\log^{2}d\cdot\mathcal{L}_{\rm TE}$
(C) unlearned target	$\leq\mathcal{L}_{\rm TE}/m=d^{-\alpha}\,\mathcal{L}_{\rm TE}$	$\leq\mathcal{L}_{\rm TE}/m=d^{-\alpha}\,\mathcal{L}_{\rm TE}$

Appendix

Table 1: The three terms of ( A.1 ) for a unit target of degree

k_{0}<\alpha

, stopped at

T_{\gamma}

with

\gamma\geq k_{0}

, at a free

s\in\mathcal{K}

with

s\geq\gamma

and at the

s

that minimizes their sum. Term (A) does not depend on the stopping time, and term (C) does not depend on

s

Figure 5: Proposition 2 for the deterministic equivalent. The floor

w_{d}\bigl(\frac{\nu}{\sigma_{4}^{2}+\nu}\bigr)^{2}/\overline{\mathcal{L}}_{\rm TE}

of ( E.5 ) and the attained ratio

\overline{\mathcal{L}}_{\rm ST}(T_{2})/\overline{\mathcal{L}}_{\rm TE}

bracket the all-time optimum. The figure uses

c=(\pi-3)/2

, for which both tend to

1/2

(Remark 3 ). The companion panel shows that the teacher error and the two student bounds all vanish at rate

d^{-1}

. These are deterministic calculations, not random-teacher observations.

Target, width	$\overline{\mathcal{L}}_{\rm TE}$	$\inf_{t\geq 0}\overline{\mathcal{L}}_{\rm ST}(t)$	Shown in the figures
Pure $Y_{2,d}$ , $m=d^{3}$	$\sim c/d$	$\overline{\mathcal{L}}_{\rm ST}(T_{2})\sim c^{2}/d^{2}$	Figure 1 : ratio $\to 1$
Proposition 2 , $m=d^{3}$	$\sim 2c/d$	$\sim c/d$ , attained at $T_{2}$	Figure 5 : both ratios $\to 1/2$
Proposition 3 , $m=d^{5}$	$\sim Cd^{-3}$	$\sim\dfrac{C^{2}}{48}d^{-5}$ , attained at $\sigma_{4}^{2}t=\log 2$	Figures 4 , 7 : $\inf_{t}\overline{\mathcal{L}}_{\rm ST}(t)/\overline{\mathcal{L}}_{\rm TE}\sim C/(48d^{2})$ , $\inf_{t}\overline{\mathcal{L}}_{\rm ST}(t)/\overline{\mathcal{L}}_{\rm TE}^{\,2}\sim d/48$ , $\inf_{t}\overline{\mathcal{L}}_{\rm ST}(t)/\overline{\mathcal{L}}_{\rm TE}^{5/3}\to C^{1/3}/48$

Appendix

Table 3: Claimed leading constants behind the figures, with

c=(\pi-3)/2

C=(12\pi-37)/24

and

B=C^{2}/24

. The ratio in the last column is

\overline{\mathcal{L}}_{\rm ST}(T_{2})/\overline{\mathcal{L}}_{\rm TE}^{\,2}

in the first row, the two student-to-teacher ratios of Figure 5 in the second, and the three ratios named in the third.

Figure 6: The timing conflict of Proposition 3 at

d=4096

, with

B=C^{2}/24

as in Remark 3 and loss divided by

Bd^{-5}

. The degree-four signal contribution decreases while the error inherited from the dominant degree-two mode increases. Their sum lies above the dashed envelope of ( E.8 ). Dotted lines mark its all-time lower bound and the asymptotically optimal time

\sigma_{4}^{2}t=\log 2

. The plot focuses on the degree-four learning window; all curves are theoretical.

Figure 7: Bounds on the all-time minimum in Proposition 3 , with

B=C^{2}/24

as in Remark 3 . The shaded band lies between the analytic lower bound ( E.8 ) and the best numerically attained loss. The explicit stop

\sigma_{4}^{2}t=\log 2

is another upper bound. The leading ratios

C/(48d^{2})

and

d/48

of Remark 3 are shown separately: WTS occurs, but no stopping sequence achieves quadratic WTS. This is a conditional DE calculation.

Figure 8: Averaging over teachers and the harmonic cutoff. Unit degree-two target,

m=d^{3}

, the three dimensions with population-flow samples in Figure 1 . Top: individual teachers, their mean, a pointwise 95% interval for the mean, and the deterministic equivalent. Bottom, on the same time axis: the relative change of the loss when the harmonic sum ( G.1 ) is cut at

K

instead of

K=128

(worst teacher), the bound on the omitted tail above

K=128

, and the relative half-width of the 95% interval. Cutting at

K\geq 16

is indistinguishable from

K=128

on this window; the cutoff error is far below the statistical uncertainty everywhere.

Explore similar work

Sep 27, 2026stat.ML

An Active-Bottleneck Mechanism for Weak-to-Strong Generalization

Weak-to-strong generalization (W2SG) occurs when a student trained on a teacher's predictions outperforms that teacher. We study when this happens under fully converged, ridgeless two-stage learning, with no early stopping, no explicit regularization, and no assumption that the student is more expressive than the teacher. In two-stage linear regression, a teacher is fit from

n

labeled examples and a student is trained solely on the teacher's predictions on

m

fresh, unlabeled inputs. Although both stages share the same hypothesis class and the same training rule, we show that the student outperforms the teacher exactly when

m

lies in an explicit intermediate range: too few pseudo-labels leave the student without enough signal, too many let it inherit the teacher's noise. Under power-law covariance, we derive this range in closed form as a function of the spectral decay and noise level, including regimes where the improving region splits into two disjoint intervals of

m

. We then study a random-feature model in which the student has strictly more features than the teacher, and identify two regimes, again given by explicit thresholds: one where improvement occurs only for

m

in a bounded interval, and one where it occurs only once the student width

N_S

exceeds an explicit threshold. Both regimes are governed by a single "active-bottleneck" principle: whichever of

m

N_S

is scarcer controls how much teacher error is filtered out, while increasing the other resource only reduces estimation noise. Together, these results show that finite data and finite width can themselves regularize a two-stage learner, with no explicit mechanism doing so.

Mohammad Zeinalpour, Amir Najafi

Department of Computer Engineering, Sharif University of Technology

Multi-Stage Training Bottlenecks Decision Thresholds

Ask PDF

May 7, 2026cs.LG

Weak-to-Strong Generalization is Nearly Inevitable (in Linear Models)

Weak-to-strong generalization is a phenomenon in post-training whereby a strong student model, when finetuned solely with feedback from a weaker teacher, can not only surpass the teacher, but can improve upon its own capabilities. Recent work of Burns et al. (2023) demonstrated that this can occur in the setting of frontier language models, and subsequently there has been a flurry of both empirical work trying to exploit this phenomenon, as well as theoretical work attempting to understand it. In this work, we demonstrate that weak-to-strong generalization occurs in standard linear logistic regression, under mild distributional assumptions on the data. In fact, we show that this happens for most student-teacher pairs, suggesting that weak-to-strong generalization is in fact \emph{almost inevitable}, even in this basic setting. Notably, our setting does not require the student to be more expressive or have more model capacity in any way compared to the teacher, which runs contrary to the prevailing theoretical belief that a mismatch in model capacity is a central mechanism to weak-to-strong generalization.

Scott Geng, Dutch Hansen, Jerry Li

University of Washington, Seattle.

Logistic Regression

Ask PDF

Jun 26, 2026cs.LG

How Width and Data Shape Generalization Scaling Laws in Quadratic Neural Networks

Understanding how performance scales jointly with model size and data is a central problem in modern machine learning. Existing theoretical works on scaling laws typically describe generalization as a function of data or compute, often in fixed-feature or infinite-width regimes and for online SGD. Here, we instead study how generalization scales with the number of trainable parameters and the number of samples in a feature-learning model. We analyze

\ell_2

-regularized empirical test error minimization in a quadratic two-layer network in a finite-sample setting with structured data. This setting allows for an explicit characterization of the generalization error as a function of the number of samples, model width, and regularization. Our results reveal a phase diagram with distinct scaling regimes as the number of parameters varies. In particular, the generalization error follows data-dependent power laws controlled by the spectral structure of the target. We further characterize the transitions between regimes, including the onset of interpolation, and their impact on generalization.

Julius Girardin, Emanuele Troiani, Yizhou Xu +3

1Statistical Physics of Computation Laboratory, École Polytechnique Fédérale de Lausanne (EPFL) · 2Information, Learning and Physics Laboratory, École Polytechnique Fédérale de Lausanne (EPFL)

Two-Layer Neural Networks Scaling Laws Regularization Neural Network Scaling

Ask PDF