Quadratic Weak-to-Strong Generalization in Random Feature Networks via Random Matrix Theory

Deborah Oliveira; Elliot Paquette

stat.MLOct 6, 2026

Quadratic Weak-to-Strong Generalization in Random Feature Networks via Random Matrix Theory

Authors: Deborah Oliveira, Elliot Paquette

Organizations: McGill University Mila – Quebec AI Institute

Abstract

Weak-to-strong generalization is the phenomenon where a strong student model trained with labels produced by a weak teacher model is able to generalize better than the teacher. In this paper, we study this phenomenon in two-layer random feature networks where the model strength is determined by its width. Using tools from random matrix theory, we derive deterministic equivalents for the population errors of an optimally trained teacher and a student trained with gradient flow. For ReLU activation and a pure spherical harmonic target, we obtain sharp asymptotics under a Gaussian universality assumption, showing a quadratic improvement: the student error scales as the square of the teacher error. These results attain the general lower bound of Medvedev at al (2025). We also analyze how the student behaves under more general stopping times and targets supported on multiple harmonic degrees, characterizing the regimes in which weak-to-strong generalization occurs and identifying the transition between quadratic, non-quadratic, and no improvement.

Figures & tables

Figure 1: Quadratic improvement and the stopping window. Unit degree-two target,

m=d^{3}

, with time divided by

T_{2}=4\log(d)/\sigma_{2}^{2}

, the stopping time

T

of Theorem 2 for

k_{0}=2

and

m=d^{3}

. Left: student loss; right: loss divided by squared teacher loss. Lines are the deterministic equivalent; open markers (population-flow samples) are population spherical-teacher observations at

d=8,16,28

(32, 32, and 8 draws); see Appendix G for details. See also uncertainty and numerical approximation quantification in Appendix G.2 . The dotted line marks

T_{2}

Panel of Figure 2 — Figure 2: Stopping-time phases (Theorem 4 ). Left: the five regions in the plane of teacher capacity exponent $\alpha$ and stopping degree $\gamma$ (drawn for $k_{0}=2$ ; admissible degrees form the discrete set $\mathcal{K}$ ). The marked point is the setting of Figure 1 . Right: the phases, with the last four for $\alpha>k_{0}$ , where $\overline{\mathcal{L}}_{\rm TE}\asymp d^{-(\alpha-k_{0})}$ , and $\rho=(3\alpha-k_{0}-2\gamma)/(\alpha-k_{0})\in(1,2)$ . The $\overline{B}/\overline{V}$ indicates which error terms dominates the bias/variance decomposition. Details are in Appendix D.1 .

Panel of Figure 3 — Figure 3: Weak-to-strong generalization regimes as a function of the stopping time exponent $\gamma$ , for (a) $\alpha\leq k_{0}$ and (b) $\alpha>k_{0}$ . guaranteed WTS, for every target; conditional WTS, governed by ( 6.3 ); no WTS. The interval includes $\gamma=k_{0}$ and excludes $\gamma=\alpha$ .

Figure 4: A timing conflict that prevents quadratic improvement. Deterministic-equivalent curves for the degree-two/degree-four mixture in ( 6.7 ), with

B=C^{2}/24

and

C=(12\pi-37)/24

(Remark 3 ). Left: student-to-teacher loss ratio; right: student loss divided by

\overline{\mathcal{L}}_{\rm TE}^{5/3}

. Dots are numerically attained minima. The right-hand curves converge to

C^{1/3}/48

, the leading constant of the normalized optimum in ( 6.7 ) (Remark 3 ).

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

term	at a free $s$	at the best $s$ , namely $s=\gamma$
(A) alignment	$d^{\,2s-\alpha}\log^{2}d\cdot\mathcal{L}_{\rm TE}$	$d^{\,2\gamma-\alpha}\log^{2}d\cdot\mathcal{L}_{\rm TE}$
(B) learned high degrees	$d^{-2(s^{+}-\gamma)}\log^{2}d\cdot\mathcal{L}_{\rm TE}$	$d^{-2(\gamma^{+}-\gamma)}\log^{2}d\cdot\mathcal{L}_{\rm TE}$
(C) unlearned target	$\leq\mathcal{L}_{\rm TE}/m=d^{-\alpha}\,\mathcal{L}_{\rm TE}$	$\leq\mathcal{L}_{\rm TE}/m=d^{-\alpha}\,\mathcal{L}_{\rm TE}$

Appendix

Table 1: The three terms of ( A.1 ) for a unit target of degree

k_{0}<\alpha

, stopped at

T_{\gamma}

with

\gamma\geq k_{0}

, at a free

s\in\mathcal{K}

with

s\geq\gamma

and at the

s

that minimizes their sum. Term (A) does not depend on the stopping time, and term (C) does not depend on

s

Figure 5: Proposition 2 for the deterministic equivalent. The floor

w_{d}\bigl(\frac{\nu}{\sigma_{4}^{2}+\nu}\bigr)^{2}/\overline{\mathcal{L}}_{\rm TE}

of ( E.5 ) and the attained ratio

\overline{\mathcal{L}}_{\rm ST}(T_{2})/\overline{\mathcal{L}}_{\rm TE}

bracket the all-time optimum. The figure uses

c=(\pi-3)/2

, for which both tend to

1/2

(Remark 3 ). The companion panel shows that the teacher error and the two student bounds all vanish at rate

d^{-1}

. These are deterministic calculations, not random-teacher observations.

Target, width	$\overline{\mathcal{L}}_{\rm TE}$	$\inf_{t\geq 0}\overline{\mathcal{L}}_{\rm ST}(t)$	Shown in the figures
Pure $Y_{2,d}$ , $m=d^{3}$	$\sim c/d$	$\overline{\mathcal{L}}_{\rm ST}(T_{2})\sim c^{2}/d^{2}$	Figure 1 : ratio $\to 1$
Proposition 2 , $m=d^{3}$	$\sim 2c/d$	$\sim c/d$ , attained at $T_{2}$	Figure 5 : both ratios $\to 1/2$
Proposition 3 , $m=d^{5}$	$\sim Cd^{-3}$	$\sim\dfrac{C^{2}}{48}d^{-5}$ , attained at $\sigma_{4}^{2}t=\log 2$	Figures 4 , 7 : $\inf_{t}\overline{\mathcal{L}}_{\rm ST}(t)/\overline{\mathcal{L}}_{\rm TE}\sim C/(48d^{2})$ , $\inf_{t}\overline{\mathcal{L}}_{\rm ST}(t)/\overline{\mathcal{L}}_{\rm TE}^{\,2}\sim d/48$ , $\inf_{t}\overline{\mathcal{L}}_{\rm ST}(t)/\overline{\mathcal{L}}_{\rm TE}^{5/3}\to C^{1/3}/48$

Appendix

Table 3: Claimed leading constants behind the figures, with

c=(\pi-3)/2

C=(12\pi-37)/24

and

B=C^{2}/24

. The ratio in the last column is

\overline{\mathcal{L}}_{\rm ST}(T_{2})/\overline{\mathcal{L}}_{\rm TE}^{\,2}

in the first row, the two student-to-teacher ratios of Figure 5 in the second, and the three ratios named in the third.

Figure 6: The timing conflict of Proposition 3 at

d=4096

, with

B=C^{2}/24

as in Remark 3 and loss divided by

Bd^{-5}

. The degree-four signal contribution decreases while the error inherited from the dominant degree-two mode increases. Their sum lies above the dashed envelope of ( E.8 ). Dotted lines mark its all-time lower bound and the asymptotically optimal time

\sigma_{4}^{2}t=\log 2

. The plot focuses on the degree-four learning window; all curves are theoretical.

Figure 7: Bounds on the all-time minimum in Proposition 3 , with

B=C^{2}/24

as in Remark 3 . The shaded band lies between the analytic lower bound ( E.8 ) and the best numerically attained loss. The explicit stop

\sigma_{4}^{2}t=\log 2

is another upper bound. The leading ratios

C/(48d^{2})

and

d/48

of Remark 3 are shown separately: WTS occurs, but no stopping sequence achieves quadratic WTS. This is a conditional DE calculation.

Figure 8: Averaging over teachers and the harmonic cutoff. Unit degree-two target,

m=d^{3}

, the three dimensions with population-flow samples in Figure 1 . Top: individual teachers, their mean, a pointwise 95% interval for the mean, and the deterministic equivalent. Bottom, on the same time axis: the relative change of the loss when the harmonic sum ( G.1 ) is cut at

K

instead of

K=128

(worst teacher), the bound on the omitted tail above

K=128

, and the relative half-width of the 95% interval. Cutting at

K\geq 16

is indistinguishable from

K=128

on this window; the cutoff error is far below the statistical uncertainty everywhere.

Explore similar work

An Active-Bottleneck Mechanism for Weak-to-Strong Generalization
Sep 27, 2026Mohammad Zeinalpour, Amir NajafiMulti-Stage Training Bottlenecks
Department of Computer Engineering, Sharif University of Technology
Weak-to-Strong Generalization is Nearly Inevitable (in Linear Models)
May 7, 2026Scott Geng, Dutch Hansen, Jerry LiLogistic Regression
University of Washington, Seattle.
How Width and Data Shape Generalization Scaling Laws in Quadratic Neural Networks
Jun 26, 2026Julius Girardin, Emanuele Troiani, Yizhou Xu +3Two-Layer Neural Networks Scaling Laws
1Statistical Physics of Computation Laboratory, École Polytechnique Fédérale de Lausanne (EPFL) · 2Information, Learning and Physics Laboratory, École Polytechnique Fédérale de Lausanne (EPFL)

Sep 27, 2026, stat.MLJ/K move · Enter open · S save

An Active-Bottleneck Mechanism for Weak-to-Strong Generalization

Mohammad Zeinalpour, Amir Najafi

Department of Computer Engineering, Sharif University of Technology

Multi-Stage Training Bottlenecks Decision Thresholds

Open paper AskPDF arXiv

Weak-to-strong generalization (W2SG) occurs when a student trained on a teacher's predictions outperforms that teacher. We study when this happens under fully converged, ridgeless two-stage learning, with no early stopping, no explicit regularization, and no assumption that the student is more expressive than the teacher. In two-stage linear regression, a teacher is fit from

n

labeled examples and a student is trained solely on the teacher's predictions on

m

fresh, unlabeled inputs. Although both stages share the same hypothesis class and the same training rule, we show that the student outperforms the teacher exactly when

m

lies in an explicit intermediate range: too few pseudo-labels leave the student without enough signal, too many let it inherit the teacher's noise. Under power-law covariance, we derive this range in closed form as a function of the spectral decay and noise level, including regimes where the improving region splits into two disjoint intervals of

m

. We then study a random-feature model in which the student has strictly more features than the teacher, and identify two regimes, again given by explicit thresholds: one where improvement occurs only for

m

in a bounded interval, and one where it occurs only once the student width

N_S

exceeds an explicit threshold. Both regimes are governed by a single "active-bottleneck" principle: whichever of

m

N_S

is scarcer controls how much teacher error is filtered out, while increasing the other resource only reduces estimation noise. Together, these results show that finite data and finite width can themselves regularize a two-stage learner, with no explicit mechanism doing so.

Abstract

Explore similar work

An Active-Bottleneck Mechanism for Weak-to-Strong Generalization

Weak-to-Strong Generalization is Nearly Inevitable (in Linear Models)

How Width and Data Shape Generalization Scaling Laws in Quadratic Neural Networks