Optimal Unconstrained Self-Distillation in Ridge Regression: Strict Improvements, Precise Asymptotics, and One-Shot Tuning

Hien Dang; Pratik Patil; Alessandro Rinaldo

math.STFeb 19, 2026

Optimal Unconstrained Self-Distillation in Ridge Regression: Strict Improvements, Precise Asymptotics, and One-Shot Tuning

Authors: Hien Dang, Pratik Patil, Alessandro Rinaldo

Organizations: Department of Statistics and Data Sciences, University of Texas, Austin, TX 78712, USA.

Abstract

Self-distillation (SD) is the process of retraining a student on a mixture of ground-truth labels and the teacher's own predictions using the same architecture and training data. Although SD has been empirically shown to often improve generalization, its formal guarantees remain limited. We study SD for ridge regression in unconstrained setting in which the mixing weight $ξ$ may be outside the unit interval. Conditioned on the training data and without any distributional assumptions, we prove that for any squared prediction risk (including out-of-distribution), the optimally mixed student strictly improves upon the ridge teacher for every regularization level $λ> 0$ at which the teacher ridge risk $R(λ)$ is nonstationary (i.e., $R'(λ) \neq 0$ ). We obtain a closed-form expression for the optimal mixing weight $ξ^\star(λ)$ for any value of $λ$ and show that it obeys the sign rule: $\operatorname{sign}(ξ^\star(λ))=-\operatorname{sign}(R'(λ))$ . In particular, $ξ^\star(λ)$ can be negative, which is the case in over-regularized regimes. To quantify the risk improvement due to SD, we derive exact deterministic equivalents for the optimal SD risk in the proportional asymptotics regime (where the sample and feature sizes $n$ and $p$ both diverge but their aspect ratio $p/n$ converges) under general anisotropic covariance and deterministic signals. Our asymptotic analysis extends standard second-order ridge deterministic equivalents to their fourth-order analogs using block linearization, which may be of independent interest. From a practical standpoint, we propose a consistent one-shot tuning method to estimate $ξ^\star$ without grid search, sample splitting, or refitting. Experiments on real-world datasets and pretrained neural network features support our theory and the one-shot tuning method.

Figures & tables

Figure 1: Visual illustration of the self-distillation process.

Panel of Figure 2 — Figure 2: Strict improvement of SD risk with unconstrained mixing. Test squared prediction risk of ridge regression ( $R$ , in blue ), pure-distilled ridge ( $R_{\mathsf{pd}}$ , in light blue ) and optimal self-distilled ridge ( $R^{\star}_{\mathsf{sd}}$ , in green ) as functions of the ridge penalty $\lambda$ . Results are shown on raw features from real-world datasets: BlogFeedback and Communities and Crime datasets, and on pretrained ResNet-18 features. The optimal mixing parameter $\xi^{\star}(\lambda)$ is in red and the one-shot risk estimate $\widehat{R}_{\mathsf{sd}}^{\star}(\lambda)$ computed from the training data is shown in green dashed line. Note that $\xi^{\star}(\lambda)$ lies in $[0,1]$ only for a narrow range of $\lambda$ and can be strongly negative for large $\lambda$ . We also observe that: (i) $R_{\mathsf{sd}}^{\star}(\lambda)$ is strictly smaller than $R(\lambda)$ at every $\lambda$ that is not the stationary point of $R(\lambda)$ , (ii) the sign of $\xi^{\star}(\lambda)$ is opposite to the sign of $R^{\prime}(\lambda)$ , and (iii) the sign change of $\xi^{\star}$ happens at the stationary point of $R(\lambda)$ . (Experiments with $\xi$ restricted to $[0,1]$ appear in Figure 16 .)

Paper	Focus		Nonasymptotic setting			Proportional asymptotics			Weights		Multi-rounds
	Type	Family	Design	Response	Risk	Features	Signal	Model	Range	Tuning
14	SD	Ridge	Fixed	Linear	Estimation				$\mathbb{R}$
43	SD	Ridge	Fixed	Linear	Fixed-X prediction				$\mathbb{R}$	Split CV (three refits)	Yes
21	PD	Ridgeless	Random	Linear	In-distribution prediction				$\{1\}$
42	PD	Ridge (generalized)				Isotropic	Isotropic	Linear	$\{1\}$
23	SD	Ridge (infinite rounds)				Anisotropic	Deterministic	Linear	$[0,1]$		Infinite rounds only

Table 1: Settings overview in the self-distillation ridge/ridgeless literature . Unless stated otherwise, risks in the proportional asymptotics setting are in-distribution squared prediction risks.

Figure 3: Out-of-distribution SD risk improvement Test prediction risk of ridge and optimal SD ridge on Air Quality dataset (see Appendix F for more details). SD yields strict improvements across

\lambda

and achieves a substantially smaller global minimum.

Dataset	Illustration	( 10 ) holds?	Global gain?
BlogFeedback	Figure 2(a)	✓	✓
Communities and Crime	Figure 2(b)	✗	✗
CIFAR10	Figure 2(c)	✗	✗
Air Quality	Figure 3	✓	✓

Table 2: Curvature test at the ridge-optimal

\lambda^{\star}

Figure 4: Theoretical versus empirical risks. Asymptotic SD risks and optimal mixing weights versus

\lambda

across multiple snr values. Empirical curves are averaged over

30

simulations. Estimated curves are obtained using the proposed one-shot tuning method ( Section 4 ), and the theoretical curves are obtained from Theorem 3.1 . Setting:

n=400

p=200

\sigma^{2}=1

r^{2}=\sigma^{2}\textsc{snr}

;

\Sigma

is AR1,

\beta

is a deterministic signal aligned with the top

10\%

eigenvectors of

\Sigma

(alignment factor 0.9). (See Appendix F for more details.)

Figure 5: Asymptotic gain over the teacher. Deterministic limits of the risks and the gain

\mathcal{R}(\lambda)-\mathcal{R}_{\mathsf{sd}}^{\star}(\lambda)

. Same setting as Figure 4 with

r^{2}=\sigma^{2}=1

Figure 6: Approximating the best predictor. Percentage difference between $\mathcal{R}_{\text{sd}}^{\star}(\textsc{snr},\gamma)$ and the Bayes-optimal ridge risk $\mathcal{R}^{\star}(\textsc{snr},\gamma)$ in the isotropic design and signal setting.

Figure 7: Visual illustration of the recursive multi-round self-distillation process for ridge regression.

Figure 8: Recursive versus anchored multi-round self-distillation. Test risks of the teacher ridge (

R

, in blue , trained using

y^{(0)}

), one-round (

k=1

) optimal self-distilled ridge (

R^{\star}_{\mathsf{sd}}

, in green , trained using

y^{(0)}

and

\widehat{y}^{(0)}

), two-round (

k=2

) recursive (

R^{\star}_{\text{recursive}}

, in olive , trained using

y^{(1)}

and

\widehat{y}^{(1)}

) and anchored (

R^{\star}_{\text{anchored}}

, in brown , trained using

y^{(0)}

and

\widehat{y}^{(1)}

) self-distillation. Recursive mixing is monotone when

\xi_{k}

is optimized each round ( Proposition 5.1 ); the two-round risk curve

R^{\star}_{\text{recursive}}(\lambda)

uniformly dominates the optimal one-round

R^{\star}_{\mathsf{sd}}(\lambda)

. Anchored mixing can be nonmonotone because the round-

k

family need not contain the round-

(k-1)

predictor; the two-round risk curve

R^{\star}_{\text{anchored}}(\lambda)

may be larger than optimal one-round

R^{\star}_{\mathsf{sd}}(\lambda)

Figure 9: Visual illustration of self-distillation with fresh unlabeled features.

Figure 10: Same-

X

versus fresh-

X

self-distillation risks. Empirical test risks and optimal mixing weights (averaged over 30 simulations) in an isotropic setting with

n=200

p=100

r^{2}=1

, and

\sigma^{2}=1

. For each

\lambda

, we choose the fresh-

X

mixing weight

\xi

by grid search over

3{,}000

values in

[-100,100]

, restricting to values for which the mixed-loss objective ( 28 ) is strictly convex, and picking the one with the lowest empirical test risk.

Figure 11: Kernel ridge and kernel SD ridge regression. Test risks of Gaussian kernel ridge and kernel SD ridge on two datasets.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Section	Subsection	Purpose
Proofs in Section 2
Appendix A	Section A.1	Preliminaries
	Section A.2	Proof of Equation 4
	Section A.3	Proof of Proposition 2.1
	Section A.4	Proof of Theorem 2.2
	Section A.5	Proof of Proposition 2.3

Appendix

Table 3: Outline of the supplement.

Notation	Description
Typography
Lowercase (e.g., $x$ )	Scalars or vectors
Uppercase (e.g., $X$ )	Matrices or linear operators
Calligraphic (e.g., $\mathcal{D}$ , $\mathcal{R}$ )	Sets, $\sigma$ -fields, events or certain limiting functions
Blackboard bold (e.g., $\mathbb{R}$ , $\mathbb{N}$ )	Standard number systems
Analysis

Appendix

Table 4: Summary of general notation used throughout the paper.

Panel of Figure 12 — Figure 12: Same- $X$ versus fresh- $X$ self-distillation risks. Theoretical asymptotic ( Theorem 5.2 ) and empirical risks (averaged over 20 simulations) and optimal mixing weights for an isotropic setting with $n=400$ , $p=200$ , $r^{2}=5$ , and $\sigma^{2}=1$ .

Figure 13: Squared prediction risks and optimal mixing parameter for ridge, pure-distilled and self-distilled ridge on pretrained ResNet-34 features on CIFAR100 dataset.

Panel of Figure 14 — Figure 14: Test classification accuracy of the ridge and self-distilled ridges on CIFAR10 and CIFAR100 experiments. Although the optimal mixing is chosen to minimize the squared prediction risk, the strict improvement property still somewhat holds for test accuracy as well.

Figure 15: One-round unconstrained versus multi-round constrained mixing weight. Squared prediction risk empirical curves averaged over 20 simulations, $n=400,p=200$ with isotropic design and isotropic signal. We choose the optimal $\xi$ over a grid of 200 values in $[0,1]$ and pick the one with the lowest risk at the $20$ -th round.

Panel of Figure 16 — Figure 16: Test-set squared prediction risk and optimal mixing parameter, being constrained in $[0,1]$ . We choose the optimal $\xi$ over a grid of 100 values in $[0,1]$ and pick the one with the lowest SD risk. The linestyle for the SD risk is changed only for this figure since it mostly overlaps with either the original ridge or the pure-distilled ridge curves. When $\xi=1$ , the SD risk exactly matches the PD risk and when $\xi=0$ , it exactly matches the teacher ridge risk.

Panel of Figure 17 — Figure 17: Squared prediction risks and gain curve on BlogFeedback dataset with different split ratios.

Panel of Figure 18 — Figure 18: Squared prediction risks and gain curve on Communities and Crime dataset with different split ratios.

Panel of Figure 19 — Figure 19: Squared prediction risks and gain curve on Air Quality dataset with different split ratios.

Panel of Figure 20 — Figure 20: Asymptotic SD risk and optimal mixing parameter over various SNR ratios. Empirical curves are averaged over 30 numerical simulations, and the shaded band represents one standard deviation around the mean. The estimated are obtained using proposed tuning method ( Section 4 , averaged over 30 runs) and theoretical curves are from Theorem 3.1 . $n=400$ , $p=200$ , $\sigma^{2}=1$ and $r^{2}=\sigma^{2}\textsc{snr}$ . Top row: Data covariance $\Sigma$ is AR1, deterministic ground-truth signal is aligned with the bottom $10\%$ eigenvalues of $\Sigma$ , with alignment factor 0.9. Bottom row: Data covariance $\Sigma$ is a spiked covariance matrix, deterministic ground-truth signal is aligned with the top $10\%$ eigenvalues of $\Sigma$ , with alignment factor 0.9.

Panel of Figure 21 — Figure 21: Asymptotic SD risk and optimal mixing parameter over different aspect ratios $\gamma=p/n$ . Empirical curves are averaged over 30 numerical simulations, and the shaded band represents one standard deviation around the mean. The estimated are obtained using proposed tuning method ( Section 4 , averaged over 30 runs) and theoretical curves are from Theorem 3.1 . $n=300$ , $p=n\gamma$ , $\sigma^{2}=1$ and $r^{2}=5$ . Top row: Data covariance $\Sigma$ is AR1, deterministic ground-truth signal is aligned with the top $10\%$ eigenvalues of $\Sigma$ , with alignment factor 0.9. Bottom row: Data covariance $\Sigma$ is spiked covariance matrix with $\Sigma=I+5vv^{\top}$ with $v$ is a random isotropic Gaussian vector, deterministic ground-truth signal is aligned with the top $10\%$ eigenvalues of $\Sigma$ , with alignment factor 0.9.

Panel of Figure 22 — Figure 22: Asymptotic risk curves, asymptotic optimal mixing parameter and gain curves over the penalty $\lambda$ . Top row: Data covariance $\Sigma$ is isotropic, isotropic signal, $n=400,p=200,r^{2}=\sigma^{2}=1$ . Bottom row: Data covariance $\Sigma$ is AR-1(0.25), deterministic ground-truth signal is aligned with the bottom $10\%$ eigenvalues of $\Sigma$ , with alignment factor 0.9, $n=400,p=200,r^{2}=\sigma^{2}=1$ .

Panel of Figure 23 — Figure 23: Percentage difference between $\mathcal{R}(\text{SNR},\gamma)$ and $\mathcal{R}_{sd}^{\star}(\text{SNR},\gamma)$ to the optimal ridge $\mathcal{R}^{\star}(\text{SNR},\gamma)$ (also the best predictor in this setting), and with isotropic design $\Sigma=I_{p}$ , random signal follow an isotropic Gaussian distribution. The values plotted are calculated from Proposition 3.3 .

Panel of Figure 24 — Figure 24: Percentage difference between $\mathcal{R}(\text{SNR},\gamma)$ and $\mathcal{R}_{sd}^{\star}(\text{SNR},\gamma)$ to the optimal ridge $\mathcal{R}^{\star}(\text{SNR},\gamma)$ (also the best predictor in this setting), with AR1 covariance design and random signal follows an isotropic Gaussian distribution. The ratios plotted are calculated using risk formulations at Theorem 3.1 at $\lambda=10^{-3}$ and $\lambda=10^{6}$ .

Panel of Figure 25 — Figure 25: Percentage difference between $\mathcal{R}(\text{SNR},\gamma)$ and $\mathcal{R}_{sd}^{\star}(\text{SNR},\gamma)$ to the optimal ridge $\mathcal{R}^{\star}(\text{SNR},\gamma)$ (also the best predictor in this setting), with spiked covariance design and random signal follow an isotropic Gaussian distribution. The ratios plotted are calculated using risk formulations at Theorem 3.1 at $\lambda=10^{-3}$ and $\lambda=10^{6}$ .

Self-Distillation

Self-distillation trains a model from its own predictions, representations, or experience to improve capability, efficiency, and robustness.

See all 131 papers

Jul 16, 2026

Optimal Self-Distillation for Rectified Flow via Linear Probing

Modern generative models are increasingly trained using model-generated signals, creating both opportunities for self-improvement and risks of collapse. We study optimal self-distillation (SD) for rectified flow (RF): given a suboptimal teacher velocity field, can a student trained on a mixture of true RF velocities and teacher velocities provably improve the teacher? For linear RF with ridge regularization on fixed interpolation pairs, we prove an exact affine path identity, derive the optimal mixing coefficient in closed form, and show strict improvement in integrated velocity risk whenever the teacher risk is nonstationary along the regularization path. The optimal coefficient obeys a sign rule: positive mixing corrects under-regularized teachers, while negative mixing corrects over-regularized teachers. We also give one-shot generalized cross-validation (GCV) and validation tuning procedure that avoids grid search over mixing weights and repeated refitting. Combining this theorem with RF Wasserstein convergence bounds, we show that optimal self-distillation improves the velocity estimation terms controlling continuous-time and finite-step generation error. Experiments with Gaussian models, Gaussian mixtures, and image data show that optimal self-distillation improves velocity risk, mode recovery, and finite-step generation relative to both the teacher and pure distillation.

Saptarshi Roy, Debepsita Mukherjee, Pratik Patil

University of Texas, Austin

Rectified Flow Self-Distillation Flow-Based Generative Modeling

Ask PDF

May 18, 2026

Self-Distillation is Optimal Among Spectral Shrinkage Estimators in Spiked Covariance Models

Self-distillation has emerged as a promising technique for improving model performance in modern machine learning systems. We develop the statistical foundations of self-distillation in spiked covariance models, by introducing and analyzing a broad class of estimators, namely spectral shrinkage estimators. We establish that for spiked covariance matrices with

s

spikes,

s

-step self-distillation achieves optimal performance among spectral shrinkage estimators, outperforming well-known estimators in statistics and machine learning. Moreover, we show that

s

steps are necessary for optimality: any

(s-k)

-step distilled estimator is strictly suboptimal for

1 \leq k \leq s

. For the special subclass of isotropic covariances, we show that optimally tuned Ridge regression performs best among spectral shrinkage estimators. We also study a federated approach where multiple data centers share spectral shrinkage estimators and a common server seeks to aggregate them to achieve optimal performance. In this case, we find that the best local rule again takes the form of self-distillation, though it differs from the optimal rule when data are hosted centrally on a single server. Together, our results elucidate why self-distillation improves predictive performance and provide a broader statistical framework connecting it with classical shrinkage-based methods.

Radu Lecoiu, Debarghya Mukherjee, Pragya Sur

Department of Statistics, Harvard University · Department of Mathematics & Statistics, Boston University

Self-Distillation Spectral Methods Covariance Estimation Statistical Learning Theory

Ask PDF

May 27, 2026

Optimal ridge regularization revisited

We consider

L^2

-regularized linear (ridge) regression over a finite data sample

X

with bounded covariance and linear prediction targets

y

with additive isotropic noise of finite variance. We present an iterative procedure to compute the optimal regularization strength numerically from the generative parameters in the fixed-

X

setting and prove its convergence at limited noise levels. Our experimental evaluation over synthetic data shows that the proposed procedure combined with sample-based parameter estimates attains near-optimal random-

X

generalization across a wide range of sample sizes, aspect ratios, and noise levels, at an added computational cost equivalent to one preliminary ridge regression in the underparameterized regime and two in the overparameterized case.

Jack Timmermans, Sergio A. Alvarez

Department of Computer Science, Boston College, Chestnut Hill, MA 02467 USA

Linear Regression Statistical Learning Theory Ridge Regression Hyperparameter Optimization

Ask PDF

Abstract

Self-Distillation

Explore similar work

Optimal Self-Distillation for Rectified Flow via Linear Probing

Self-Distillation is Optimal Among Spectral Shrinkage Estimators in Spiked Covariance Models

Optimal ridge regularization revisited