cs.LGSep 30, 2026
SaveAwakening of the Buddha: Subspace Learning During Population-Loss Plateaus
Organizations: Department of Computer Science & Engineering University of California-San Diego
Abstract
Population loss can remain nearly constant while a neural network learns a substantially more predictive representation. We establish this separation for two-layer ReLU and leaky-ReLU networks trained on Gaussian inputs by simultaneous fixed-step population gradient descent on all parameters. For structured additive teachers whose links are positive mixtures of Gaussian-damped cubics in , we give explicit conditions under which small IID Gaussian initialization yields a high-probability guarantee: at a checkpoint during a high-loss plateau, minimum alignment between the rank- teacher subspace and the leading -dimensional eigenspace of the predictor's average gradient outer product (AGOP) increases by at least , and the minimum refit MSE under unchanged coefficient budgets decreases by more than , both relative to initialization. The same trajectory subsequently attains a trained loss below every value in the plateau window. A complementary result treats unequal-weight cubic teachers and small additive Sobolev perturbations using projected-feature refits. For SwiGLU networks with an exactly fitted intercept, we prove leading-AGOP alignment during a loss plateau at fixed width and dimension as Gaussian initialization vanishes, for square-integrable teachers with nonzero Hermite content of degree one, two, or three. A rank-one cubic specialization also gives simultaneous unrestricted-refit gains at a prescribed width. An approximation lower bound further shows that certain interaction targets retain nonzero error when ridge neurons are restricted to shared orthogonal axes within the teacher subspace. Population-moment experiments with ReLU students across 21 teachers and 50 initializations per teacher complement the analysis.
Figures & tables
Figure 1: Features improve before loss falls; gated teachers retain mixed neurons. Fifty initializations of one ReLU-student configuration per normalized teacher ( , ). Read down each column: training loss stays near one, AGOP subspace alignment rises, and a same-budget readout refit improves before loss release. Shading marks the loss-only plateau shared by all fifty seeds; the labeled boundaries are the earliest individual plateau endpoints. Horizontal position is : early time is expanded, ticks show actual updates, and initialization and all 20,000 updates remain visible. For SiLU, mean rises and refit MSE falls by update 200 while loss remains near one; refit improvement is not monotone. Row four projects each weight’s teacher-plane component onto its nearest axis in the indicated frame, retaining outside-plane components and biases, then refits. Bottom-row color shows mean importance share of projected neuron directions within the teacher plane: are its two axes; interior angles mix their coordinates. This is not the angle to the plane and does not measure the outside-plane component. Curves use all fifty seeds; light bands are pointwise 10th–90th percentiles, not confidence intervals. Dotted alignment bounds retain unresolved spectra; red crosses mark snapped-refit solver tolerance misses. Diagnostics are recorded every 100 updates (early dots); connecting lines add no observations. Appendix A gives numerical brackets, the plateau rule and eighteen further teachers.
Figure 2: Subspace recovery before loss release at increasing teacher rank. ReLU students ( , ), ten initializations per column, learn at . Rows show training MSE, AGOP alignment, bounded-head refit MSE, neuron angle to the teacher subspace, and projected neuron angle to its nearest axis . The last two rows distinguish subspace entry from individual-axis alignment. Unlike the bottom row of Figure 1 , the bottom row here uses a projected nearest-axis angle at arbitrary rank. Means and 10th–90th percentile bands retain all ten seeds; heatmaps average separately normalized importance-weighted histograms. The dotted line is an isotropic reference for . Shading marks the common 5% loss-ratio prefix, with a 1% comparison boundary; the loss also remains in . Grey heatmap regions have no shared saved state; hatching marks geometrically impossible angles. The quadratic teacher is invariant to rotations inside its subspace, so its axes are not identifiable: this is a symmetry control for coordinate mixing. Appendix B defines the angles, protocol, numerical checks and additional teacher comparisons.
Figure 3: SwiGLU students: twenty initializations per teacher. All seeds 0–19; , , . Loss and numerical unrestricted-refit MSE are divided by teacher variance; rank-one AGOP alignment is . Navy, teal and purple consistently denote loss, AGOP alignment and refit MSE. Solid means and light 10th–90th percentile bands use all twenty seeds on their shared support; later individual tails remain visible. Shading marks the shared loss-only prefix . The axis counts adaptive Euler updates, with early updates expanded. Diagnostic interpolation is for display only; bands show seed variation, not confidence intervals or integration-error bounds. Crosses, if present, mark the last finite point of censored or failed trajectories.
Figure 4: SwiGLU subspace learning and gate geometry across ranks and links. Ten initializations per setting, , ; additive quadratic, absolute-value and Gaussian-bump teachers vary rank and link. Loss and numerical refit risk are divided by teacher variance. The first three rows retain the loss, AGOP and refit conventions; the dotted line references isotropic . The fourth row shows effective gate angles obtained from weighted and median squared-cosine scores; these are saved scalar summaries, not per-neuron angle distributions. The last row reports the fraction of gates within of a teacher axis and the fraction of axes covered by such gates. Quadratic-teacher axes remain unidentifiable. Shading uses a common 5% loss-ratio prefix, with a 1% comparison boundary; this is broader than the prefix retained in Figure 3 . All ten seeds contribute to means and 10th–90th percentile bands; individual tails and diagnostic gaps remain visible. Gate entry and axis specialization are distinct observations; neither alone establishes a prediction gain. Appendix B gives definitions, numerical diagnostics, threshold sensitivity and rank-16 limitations.
Appendix figures & tables34 assets
Supplementary material from the paper’s appendix.
Appendix
| Teacher | Mean | Mean refit decrease | Final MSE |
|---|---|---|---|
Table 1: Baseline results for all 21 teachers and all 50 declared seeds per teacher. Endpoint gains use each seed’s exact loss-prefix endpoint. Brackets are mean lower and upper bounds, not confidence intervals; unresolved AGOPs contribute before taking gain differences. Refits retain numerical lower and feasible upper risks. Final MSE is mean seed standard deviation.
| Teacher | [range] | ||
|---|---|---|---|
| Raw additive SiLU | 128 [122,137] | 0.222 | 0.576 |
| Centered additive SiLU | 203 [190,227] | 0.000411 | 0.476 |
| Additive ReLU | 113 [108,120] | 0.115 | 0.534 |
| Additive sine | 203.5 [191,219] | 0.0231 | 0.487 |
Table 2: Separate four-teacher followup, fifty seeds each. Endpoint columns report medians of individual-run endpoint changes, not changes at an averaged endpoint. is the plateau endpoint update (median and range); for each alignment score; is the conservative numerical refit gain. Later loss is the original model at update 20,000.
| Teacher | ||
|---|---|---|
| Raw additive SiLU | 0.43 | 0.054 |
| Centered additive SiLU | 0.589 | 0.00056 |
| Additive ReLU | 0.313 | 1.2e-06 |
| Additive sine | 0.607 | 0.062 |
Figure 5: Additive damped and mixture links, and the ReLU gate. The two additive-family targets and the ReLU product use five diagnostic rows. Each teacher uses fifty fixed initialization seeds with identical optimization parameters. Means, numerical bounds, percentile bands and loss-prefix shading follow Figure 1 .
Figure 6: Other gated teachers. Exact GELU and tanh products, together with the bilinear rotation control, use the same five rows. Each teacher uses fifty fixed initialization seeds with identical optimization parameters. Means, numerical bounds, percentile bands and loss-prefix shading follow Figure 1 .
Figure 7: Raw and centered additive SiLU: denser followups. Each teacher has paired linear early (0–500 updates) and full (0–20,000 updates) views, with fifty seeds at , , . Rows show original loss, minimum and mean full-AGOP alignment, and constrained refit MSE. Bands are empirical 10th–90th percentiles; the gray trace reports the resolved fraction. Refit lower/upper numerical bounds remain separate. Shading marks the loss-prefix intersection across all fifty seeds. The common configuration was chosen after the pilot did not establish reliable full-subspace learning, as disclosed in Section A.2 . The enlarged early view exposes partial alignment and refit changes; the averages do not establish full-subspace recovery for every seed.
Figure 8: Quadratic additive teacher. The three rows display original loss, full-current-head AGOP alignment and same-budget refit MSE. Each teacher uses fifty fixed initialization seeds with identical optimization parameters. Means, numerical bounds, percentile bands and loss-prefix shading follow Figure 1 .
Figure 9: Additional positive-mixture links. Damped links and the three-scale mixture use the same three-row format. Each teacher uses fifty fixed initialization seeds with identical optimization parameters. Means, numerical bounds, percentile bands and loss-prefix shading follow Figure 1 .
Figure 10: Perturbed cubic and fourth-Hermite teachers. These standalone perturbation magnitudes and the fourth Hermite are empirical cases, not claims of the small-neighborhood theorem. Each teacher uses fifty fixed initialization seeds with identical optimization parameters. Means, numerical bounds, percentile bands and loss-prefix shading follow Figure 1 .
Figure 11: Additive ReLU and sine: denser followups. The paired early and full views use the same fifty-seed configuration, three rows, linear update axes, numerical uncertainty conventions and loss-prefix shading as Figure 7 . Both teacher means and all Hermite components are retained. Partial geometric learning and refit gains are distinct from recovery of every teacher direction; Table 2 reports continuous endpoint changes.
Figure 12: Absolute-value teacher: original baseline. This panel retains the original fifty seeds at , , , and initial force step ; it was not rerun. The three rows show original loss, full AGOP alignment and constrained refit MSE over 20,000 updates. Means, percentiles, unresolved spectral intervals and shared loss-prefix shading follow the original baseline protocol. It is separated here so the four new followups do not displace the absolute-value control.
Figure 13: ReLU teacher-link comparison at rank eight. Ten seeds each for quadratic, absolute-value and Gaussian-bump teachers, . Loss and bounded refit are in units of ; target means are retained and fitted by the intercept. The 5% shared plateaus and 1% boundaries use ( 26 ). Means, bands and support conventions are those defined above.
Figure 14: ReLU subspace entry and teacher-axis specialization. The same rank-eight runs as Figure 13 . Importance-weighted angle histograms distinguish entry into the teacher subspace from alignment with a single projected teacher axis. For the nonquadratic links, axis concentration develops later than subspace entry; the quadratic column is a rotation-invariant control. Five-degree histograms average the ten separately normalized seed distributions. Grey means no shared saved state, and hatching marks angles above the nearest-axis geometric limit.
Figure 15: SwiGLU rank sweep, including the weak rank-sixteen case. Quadratic teachers at ranks , with , head-rate multiplier and ten seeds each. The rank-sixteen column does not establish robust all-direction recovery during the plateau. All seeds and later diagnostic gaps remain visible; large mean alignment does not replace the weakest-direction and same-checkpoint requirements.
Figure 16: SwiGLU teacher-link comparison at rank eight. Quadratic, absolute-value and Gaussian-bump links use the same ten-seed protocol with head-rate multiplier . Loss and numerical unrestricted-refit risks are divided by . These are empirical trajectories under declared loss tolerances, not guarantees for every link or initialization.
Figure 17: Slower SwiGLU heads help some runs without resolving the rank limit. Quadratic teachers at ranks eight and sixteen compare head-rate multipliers and , with ten paired seeds per setting. The curves compare how head-update speed affects loss and alignment; the rank-sixteen setting remains a limitation. The settings remain distinct cohorts; outcomes are not pooled or selected by performance.
Figure 18: SwiGLU gate geometry across ranks. The same standard-head rank sweep as Figure 15 . Gate-summary angles from ( 29 ), together with the dotted value-weight angle in ( 30 ), can improve even where the weakest AGOP direction is not recovered during the plateau. They describe different quantities, and weighted gate alignment alone is not an all-direction recovery result. The quadratic teacher has no identifiable internal axes.
Figure 19: SwiGLU gate entry and axis specialization for different links. The same rank-eight teacher comparison as Figure 16 . Effective gate and value angles and thresholded gate-axis counts use saved scalar summaries; no per-neuron angle distributions are inferred. For the nonquadratic teachers, the fraction of near-axis gates rises later than the decline in the effective subspace angle. The weighted angle and unweighted gate counts must be interpreted together and relative to the stated loss window; they do not classify every neuron or establish that coordinate mixing is necessary.
Figure 20: Fixed representative seed 0 for the four rank-one teacher links. Rows show original normalized loss, leading AGOP alignment and numerical frozen-feature refit risk. These are adaptive Euler flow approximations, with . The display includes the subsequent measured loss decrease.
Figure 21: All three predeclared seeds for every rank-one link, using the same scales and diagnostics as Figure 20 . Each curve is an individual run; no seed is selected by its outcome.
Figure 22: Fixed seed 0 for balanced and unbalanced rank-two cubic teachers. In the alignment row, solid curves are and dashed curves are . Strong leading-direction alignment can coexist with weak recovery of the least aligned teacher direction.
Figure 23: All three seeds for each rank-two coefficient vector. The weakest-direction statistic is retained for every seed, including cases with poor simultaneous recovery.
Figure 24: Width-256 rank-one reference reproduction ( , seed 0). The refit panel reports attained prediction risk from the unregularized numerical head solve. The triangle is the separately validated reference checkpoint 66, whose corrected risk is ; it does not change the primary prefix. The width exceeds . Quadrature and spectral-cutoff checks accompany these finite-initialization values; the display does not assert an exact zero-risk fit.
Figure 25: Fixed mean-field steps versus the adaptive flow approximation, all with the same rank-one cubic seed 0 initialization. Parameter-space learning rates are . The x-axis is accumulated mean-field time, not optimizer step count. The right column retains numerically divergent controls on their full loss scale, while omitting alignment/refit after the reference safeguard activates; the left column displays finite controls and the adaptive reference. The nominal time horizon is 250, checked on the accumulated update clock with at most one-step overshoot (actual endpoint times are retained); any trajectory reaching that horizon is censored, and no long-time convergence is claimed. These controls do not convert measured release into a fixed-step release theorem.
Figure 26: The two predeclared reference smoke checks ( , seed 0), retained for completeness. The rank-one smoke uses and the rank-two smoke .
| Case (seed) | |||
|---|---|---|---|
| Smoke (0) | 445.6 | 0.8994 | |
| Smoke (0) | 235.4 | 0.8977 | |
| (0) | 226.4 | 0.4999 | |
| (1) | 266.2 | 0.4997 | |
| (2) | 311.6 | 0.5000 | |
| , (0) | 55.5 | 0.8994 |
Table 3: Every declared SwiGLU trajectory at the loss-selected prefix with . is the last diagnostic inside the prefix, so may precede the exact last admissible update (both clocks are retained in the audit). is the attained prediction risk of the equilibrated eigencutoff solve, not a certified unrestricted infimum. -- means the weakest-direction statistic duplicates at rank one. The reference smokes use ; the scientific runs use , except . The three GD controls share seed 0; marks the stated time/step analysis horizon reached before the loss stopping target, rather than convergence; denotes the retained numerical divergence of the control. Loss values are divided by the teacher variance.
| Case (seed) | |||
|---|---|---|---|
| Smoke (0) | |||
| Smoke (0) | – | ||
| (0) | – | ||
| (1) | – | ||
| (2) | – | ||
| , (0) | – |
| Case (seed) | |||
|---|---|---|---|
| Smoke (0) | 432.5 | 0.8994 | |
| Smoke (0) | 224.9 | 0.8977 | |
| (0) | 221.5 | 0.4999 | |
| (1) | 262.4 | 0.4997 | |
| (2) | 307.6 | 0.5000 | |
| , (0) | 52.7 | 0.8994 |
Table 4: Every declared SwiGLU trajectory at the loss-selected prefix with . is the last diagnostic inside the prefix, so may precede the exact last admissible update (both clocks are retained in the audit). is the attained prediction risk of the equilibrated eigencutoff solve, not a certified unrestricted infimum. -- means the weakest-direction statistic duplicates at rank one. The reference smokes use ; the scientific runs use , except . The three GD controls share seed 0; marks the stated time/step analysis horizon reached before the loss stopping target, rather than convergence; denotes the retained numerical divergence of the control. Loss values are divided by the teacher variance.
| Case (seed) | |||
|---|---|---|---|
| Smoke (0) | |||
| Smoke (0) | – | ||
| (0) | – | ||
| (1) | – | ||
| (2) | – | ||
| , (0) | – |
| Case (seed) | Status | ||
|---|---|---|---|
| Smoke (0) | 455.2 | 0.8994 | loss stop |
| Smoke (0) | 242.4 | 0.8977 | loss stop |
| (0) | 238.0 | 0.4999 | loss stop |
| (1) | 277.7 | 0.4997 | loss stop |
| (2) | 323.8 | 0.5000 | loss stop |
| , (0) | 57.3 | 0.8994 | loss stop |
Table 5: Terminal observations under the stopping rules in Section U.6 , for all 24 conditions. “Loss stop” means the prescribed was reached; differences below the target reflect update-grid undershoot, not comparable unconstrained final losses. use the complete update history inside the stated analysis budget; is the latest saved diagnostic within that budget. For balanced rank-two seed 2, is update 3000, whereas the last diagnostic is update 2928 at ; the raw 60-update tail remains archived and dotted in the plot. Failed-control post-safeguard metrics are omitted. Refit risks are numerical attained risks, subject to the quadrature limitations in the text; in particular the fixed endpoint risk changes by on doubling quadrature.
| Case (seed) | ||||
|---|---|---|---|---|
| Smoke (0) | 455.2 | 1.000 | 0.463 | 0.856 |
| Smoke (0) | 242.4 | 1.000 | – | 0.845 |
| (0) | 238.0 | 1.000 | – | 0.445 |
| (1) | 277.7 | 1.000 | – | 0.291 |
| (2) | 323.8 | 1.000 | – | 0.332 |
| , (0) | 57.3 | 1.000 | – | 0.077 |
| Teacher | ||
|---|---|---|
| 0.962 [0.745, 0.996] | 0.222 [0.146, 0.491] | |
| 0.956 [0.742, 0.996] | 0.475 [0.320, 0.672] | |
| ReLU | 0.927 [0.719, 0.997] | 0.519 [0.414, 0.683] |
| 0.942 [0.709, 0.983] | 0.476 [0.299, 0.676] |
Table 6: Twenty-seed rank-one outcomes. Alignment and refit gains are median [minimum, maximum] at the final saved in-prefix diagnostic, using all twenty values unless a parenthesized resolved count is shown.
Figure 27: All twenty recorded trajectories per rank-one teacher. Individual curves retain their own saved grids and terminal times. The three rows and numerical meanings are those of Figure 3 ; all outcomes remain visible.
| ReLU | ||||||||
|---|---|---|---|---|---|---|---|---|
| Seed | ||||||||
| 0 | 0.870 | 0.164 | 0.869 | 0.426 | 0.862 | 0.635 | 0.852 | 0.518 |
| 1 | 0.972 | 0.371 | 0.969 | 0.397 | 0.957 | 0.647 | 0.961 | 0.676 |
| 2 | 0.994 | 0.358 | 0.993 | 0.558 | 0.997 | 0.428 | 0.956 | 0.460 |
| 3 | 0.981 | 0.235 | 0.979 | 0.483 | 0.969 | 0.551 | 0.955 | 0.459 |
| 4 | 0.745 | 0.239 | 0.742 | 0.360 | 0.719 | 0.486 | 0.709 | 0.438 |
Table 7: Every seed’s endpoint changes in leading AGOP alignment and normalized numerical refit MSE. Positive means lower error. Values are rounded numerical diagnostics, not error certificates; -- indicates an unresolved value.