Residual-as-Teacher: Mitigating Bias Propagation in Student--Teacher Estimation
Authors: Kakei Yamamoto, Martin J. Wainwright
Organizations: Lab for Information and Decision Systems; Statistics and Data Science Center; EECS; Massachusetts Institute of Technology · Lab for Information and Decision Systems; Statistics and Data Science Center; EECS and Mathematics; Massachusetts Institute of Technology
We study statistical estimation in a student--teacher setting, where predictions from a pre-trained teacher are used to guide a student model. A standard approach is to train the student to directly match the teacher's outputs, which we refer to as student soft matching (SM). This approach directly propagates any systematic bias or mis-specification present in the teacher, thereby degrading the student's predictions. We propose and analyze an alternative scheme, known as residual-as-teacher (RaT), in which the teacher is used to estimate residuals in the student's predictions. Our analysis shows how the student can thereby emulate a proximal gradient scheme for solving an oracle optimization problem, and this provably reduces the effect of teacher bias. For general student--teacher pairs, we establish non-asymptotic excess risk bounds for any RaT fixed point, along with convergence guarantees for the student-teacher iterative scheme. For kernel-based student--teacher pairs, we prove a sharp separation: the RaT method achieves the minimax-optimal rate, while the SM method incurs constant prediction error for any sample size. Experiments on both synthetic data and ImageNette classification under covariate shift corroborate our theoretical findings.
Figures & tables
Figure 1: Comparison of the RaT and SM estimates when the student class F is a two-layer neural network with 128 hidden units. Shown are results for four different covariate shifts (rows 1–4), and three different teacher models (columns 2–4). Each row corresponds to a different source–target distribution pair, shown in the leftmost column via their marginal densities. The remaining columns report results for three teacher classes: Boosting, Kernel ridge regression (KRR), and ReLU neural network fits. In each panel, the true regression function is shown as a dotted orange line, while the estimates obtained by SM and RaT are shown in red and blue, respectively. Gray points indicate source samples. Intermediate RaT iterates are shown as faint curves to illustrate the refinement process.
(a) Stepsize η=0.10
(b) Stepsize η=0.20
Figure 2: Convergence behavior of the Picard iteration ( 31 ) when using a neural network teacher to construct the gradient estimate G , and for multinomial classifier over K=10 classes (see Section 4.2 for more details.) Each plot shows the operator defect norm ∥Dη(fk)∥ versus the iteration number k . The three different curves correspond to three-layer neural-net teachers with three different architectures: hidden units (h1,h2) are marked in the labels. Solid lines correspond to the mean defect over T=100 random trials, with the shaded areas showing 95% confidence intervals. Stepsizes η are marked below each plot.
Figure 3: Mean-squared errors of RaT and SM under Gaussian covariate shift using a Hermite feature model, target distribution Q=N(0,1) , and source distributions P=N(0,σP2) with σP∈{0.9,1.0,1.1} . Solid curves show the median MSE with interquartile bands over repetitions, and dashed lines indicate the predicted scaling laws. Consistent with Theorem 2 , RaT achieves the rate n−2β/(2α+1) , whereas SM exhibits a non-vanishing error floor due to distribution shift.
Figure 4: Mean-squared errors of the RaT and SM estimators for Laplace kernel ridge regression, using target distribution Q=Beta(1,1) and source distributions P=Beta(α,1) with α∈0.2,1.0,2.0 . The teacher and student use Laplace kernels with differing bandwidths, so that the teacher is mis-specified relative to the student. Solid curves show the median MSE with interquartile bands over repeated trials. Each curve is normalized by its value at the smallest sample size. The dashed line represents a least-squares linear fit on a log–log scale to the RaT curve.
Figure 5: Mean-squared errors (MSE) of RaT and SM estimators for neural network students with a regression-stump teacher, and Beta covariate shift. (a) Target test MSE versus source sample size on a log-log scale. Solid lines show the median over repeated runs, and shaded regions indicate the interquartile range. The dashed line is a least-squares linear fit in log–log coordinates to the RaT median curve. (b) Function estimates for a single run, with ground truth function (black dashed), source samples (gray points), SM teacher (dark red dotted), and final function estimates (solid blue for RaT , solid red for SM ). Faint blue curves correspond to intermediate RaT iterates. The reported MSE values are computed on held-out target data.
Figure 6: Top row: a sample image from the “French horn” class, and its corrupted versions using the “pixelate” corruption. (a) Plots of the difference in cross-entropy (CE) loss between the SM and RaT procedures versus the corruption level. Two types of neural nets are shown: three-layer architectures with (h1,h2)=(4,32) hidden units, or (h1,h2)=(16,32) hidden units. Solid lines correspond to the mean CE difference over T=100 random splits into train and validation, and the shaded areas correspond to 95% confidence intervals based on these trials. (b) Similar plots for the raw CE scores of the two methods, shown here for the (4,32) -teacher.
Figure 7: Top row: a sample image from the “Gas pump” class, and its corrupted versions using the “elastic-blur” corruption. (a) Plots of the difference in cross-entropy (CE) loss between the SM and RaT procedures versus the corruption level. Two types of neural nets are shown: three-layer architectures with (h1,h2)=(4,32) hidden units, or (h1,h2)=(16,32) hidden units. Solid lines correspond to the mean CE difference over T=100 random splits into train and validation, and the shaded areas correspond to 95% confidence intervals based on these trials. (b) Similar plots for the raw CE scores of the two methods, shown here for the (4,32) -teacher.
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Plots of the operator norm ∣∣∣C∣∣∣ from equation ( 67a ) for two different classes of response-linear teachers, target distribution Q uniform on [−1,1] , and source distributions P from a Beta distribution (shifted to [−1,1] ) with parameter α∈[0.25,2.0] . The setting α=1 corresponds to no covariate shift. (a) Nadaraya–Watson smoother with bandwidth h . Plots of ∣∣∣C∣∣∣ versus covariate shift α and inverse bandwidth 1/h . (b) Kernel ridge regression (KRR) teacher using a Gaussian kernel and regularization parameter λ . Plots of ∣∣∣C∣∣∣ versus covariate shift α and inverse regularization 1/λ .
Statistical learning theory studies when data-driven algorithms generalize, how many samples they need, and what statistical or computational limits apply.