Teacher Geometry Shapes Learnability in Teacher-Student Networks

Kai J. Sandbrink; Flavio Martinelli; Alexander van Meegen; Wulfram Gerstner; Johanni Brea

cs.LGSep 10, 2026

Teacher Geometry Shapes Learnability in Teacher-Student Networks

Authors: Kai J. Sandbrink, Flavio Martinelli, Alexander van Meegen, Wulfram Gerstner, Johanni Brea

Abstract

Teacher-student systems, in which a teacher neural network generates training labels so that a student neural network can learn to implement the same function, are widely used as an abstract setting to study learning. However, the structure of the teachers is often overlooked by assuming randomly-generated, normally-distributed parameters. This hides substantial variation in how learnable different teachers are. We formalize learnability as the success rate of converging to the global minimum, as a function of overparameterization, learning algorithm, student initialization distribution, and teacher geometry. We both identify an easy distribution that maximizes node dissimilarity and a hard distribution that minimizes it, and show that these two distributions induce markedly different success rates across a large range of settings and for different activation functions. To explain the gap, we study the loss landscape of small neural networks that contain two distinct kinds of suboptimal local minima, out-of-bounds (OOB) minima at the edge of the data distribution and interior minima within. Assuming infinite data and a fast readout layer, we analytically reduce the loss landscape of small networks to two dimensions, showing that the region of attraction of interior minima changes as a function of teacher structure. In larger networks, maximally dissimilar teachers induce more interior minima, while minimally dissimilar teachers induce more OOB minima. Motivated by these analyses, we show that differentially increasing the learning rate of the readout layer and decreasing the learning rate of the inner biases increases success rates. These findings provide an important step in narrowing the gap between the study of teacher-student networks and more structured functions that arise in practice.

Explore similar work

May 11, 2026stat.ML

Sharp feature-learning transitions and Bayes-optimal neural scaling laws in extensive-width networks

We study the information-theoretic limits of learning a one-hidden-layer teacher network with hierarchical features from noisy queries, in the context of knowledge transfer to a smaller student model. We work in the high-dimensional regime where the teacher width

k

scales linearly with the input dimension

d

-- a setting that captures large-but-finite-width networks and has only recently become analytically tractable. Using a heuristic leave-one-out decoupling argument, validated numerically throughout, we derive asymptotically sharp characterizations of the Bayes-optimal generalization error and individual feature overlaps via a system of closed fixed-point equations. These equations reveal that feature learnability is governed by a sequence of sharp phase transitions: as data grows, teacher features become recoverable sequentially, each through a discontinuous jump in overlap. This sequential acquisition underlies a precise notion of \textit{effective width}

k_c

-- the number of learnable features at a given data budget

n

-- which unifies two distinct scaling regimes: a feature-learning regime in which the Bayes-optimal generalization error

\varepsilon^{\rm BO}

scales as

n^{1/(2β)-1}

, and a refinement regime in which it scales as

n^{-1}

, where

β>1/2

is the exponent of the power-law feature hierarchy. Both laws collapse to the single relation

\varepsilon^{\rm BO}=Θ(k_c d/n)

. We further show empirically that a student trained with \textsc{Adam} near the effective width

k_c

achieves these optimal scaling laws (up to a small algorithmic gap), and provide an information-theoretic account of the associated scaling in model size.

Minh-Toan Nguyen, Jean Barbier

Feature Learning Two-Layer Neural Networks Scaling Laws Bayesian

Ask PDF

Sep 16, 2026cs.LG

Trajectory Learnability for Offline On-Policy Distillation with Imperfect Teachers

Offline on-policy distillation gains efficiency by collecting student trajectories and teacher supervision once and reusing them throughout optimization. The same reuse makes imperfect supervision persistent. Since even strong teachers can fail, we ask \emph{what remains learnable from imperfect teacher supervision?} Teacher failure is only a coarse problem-level signal and does not imply that all supervision along the associated student trajectory is unhelpful. A natural alternative is to estimate teacher recoverability along the trajectory, but repeated continuations largely erase the efficiency advantage of offline distillation. We instead use teacher-successful problems to define a cheap reference for what the student can learn. We train on teacher-successful problems and measure how the likelihood of each observed token in trajectories from teacher-failed problems changes. We use these signed likelihood changes as an operational \emph{learnability signal}: larger increases indicate behavior more strongly promoted by successful-only learning. We aggregate this signal into trajectory-level weights for the original distillation loss. Unlike continuation-based estimates, our learnability requires no additional generation and can be computed once from stored trajectories and model checkpoints. Across mathematical reasoning and code generation, our method improves an offline OPD baseline by up to 2.7 percentage points and matches or outperforms online OPD variants on multiple benchmarks. Despite the additional successful-only distillation stage, it uses 2 GPUs and about 22 GPU hours, compared with 3 GPUs and 36--48 GPU hours for representative online OPD methods.

Yihao Ai, Weilong Yan

On-Policy Distillation

Ask PDF

May 21, 2026cs.LG

A Boundary-Layer Mechanism for One-Third Scaling in Online Softmax Classification

Hard-label classification is usually trained with smooth surrogate losses, most prominently softmax cross-entropy. We isolate an asymptotic mechanism by which this mismatch between smooth surrogate and discrete labels produces power-law learning curves in an online teacher-student model. After subtracting the mean logit, the thermodynamic-limit dynamics close in centered variables: a growing centered student-teacher alignment

D

and the residual student variance

Δ

. At late times, examples away from teacher decision boundaries are already classified confidently and contribute exponentially little. Only boundary layers of width

O(D^{-1})

remain active, while the noise of fixed-learning-rate online gradient descent maintains a nonzero

Δ

. As a function of the training time

α

the late-time solution yields a

α^{-1/3}

power law not only for the test loss but also for the generalization error

ε_g

, i.e., one minus test accuracy. This is much slower than the

α^{-1}

Bayes-optimal reference for the same model. We further show that learning-rate schedules can improve the generalization error towards a

ε_g \sim α^{-1/2}

power law. Simulations support the predicted order parameter dynamics and learning curves. Controlled experiments with correlated Gaussian inputs and whitened pretrained features show that data structure can dominate transients. Therefore, our result is an asymptotic, complementary mechanism rather than an alternative to spectral explanations of neural scaling laws.

Marcel Kühn, Yoon Thelge, Bernd Rosenow

Learning Rates Scaling Laws Two-Layer Neural Networks Gumbel-Softmax Relaxation Asymptotics Teacher-Student Log-Probability Gap+2

Ask PDF