cs.LGAug 16, 2026
SaveMeasuring Structured Predictability in Neural Training Dynamics: A Cross-Regime Study
Organizations: Department of Electrical Engineering and Computer Science, University of Tennessee, Knoxville
Abstract
Modern deep networks are trained through long update trajectories, yet their temporal organization remains less systematically characterized than architectures, losses, or optimizers. We study short-horizon predictability as a measure of temporal redundancy: where, when, and under which training conditions recent updates contain information about near-future parameter motion. We combine three complementary probe families, displacement-direction, subspace-residual, and predictor-based probes, with convention-aware, null-calibrated group-level readouts, and apply them to multi-pass vision training on CIFAR and public Pythia pretraining checkpoints. Across both regimes, vector-like tensors such as normalization parameters and biases (auxiliary parameters) exhibit simpler short-horizon dynamics than matrix-like feature-transforming weights (bulk parameters), whose predictable behavior concentrates in localized, time-varying pockets. Agreement within and across probe families, and with independent trajectory diagnostics, indicates that these measurements capture intrinsic trajectory structure, while probe differences distinguish complementary forms of temporal organization. Controlled CIFAR comparisons further show that architecture and training recipe systematically modulate the measured structure. A Pythia-70M case study further exposes a sequence of role-, depth-, and scale-dependent events, including bulk ESA falling below the random sign-agreement level and the emergence and redistribution of predictable qkv pockets across layers. These results position short-horizon predictability as a retrospective, parameter-resolved diagnostic of training dynamics.
Figures & tables
Figure 1: Within-family agreement on the controlled vision panel. (a) ESA versus group-mean VCS; (b) out-of-window versus in-window FRR; (c) SESA under DMD and linear-trend selectors. Circled points mark cells with near-zero-displacement coordinates.
Figure 2: Cross-family relations on the controlled vision panel. (a) ESA versus out-of-window FRR; (b) out-of-window FRR versus DMD direction cosine; (c) ESA versus DMD lift. Circled points mark cells flagged by the near-zero-displacement diagnostic.
Figure 3: Auxiliary–bulk separation across training regimes. Each line connects the auxiliary and bulk summaries within a matched CIFAR run or Pythia trajectory; circles denote CIFAR runs and stars denote Pythia trajectories. Auxiliary tensors exceed bulk tensors in comparisons under ESA and under the selector-free DMD direction cosine. Because the two regimes use different grouping protocols, we compare the within-line ordering rather than absolute levels across regimes.
Figure 4: Distribution of FRR excess above the approximate isotropic reference over structurally grouped bulk parameters, with . The highlighted tail ( ) contains approximately of pooled group–anchor observations.
Figure 5: Pythia-70M case study. (a) ESA by parameter role. (b) Released training loss and its smoothed trajectory. (c) Fraction of row groups satisfying . (d) Head-aligned excess. Vertical lines mark the early fading of head-aligned excess, the loss minimum, and the later ESA rise.
Figure 6: Optimizer-family profiles for ViT-mid along training loss (horizontal axis reversed), with canonical bulk ESA on the vertical axis. All-family comparisons use the common support – ; matched loss does not imply matched optimizer state. Dashed line: sign-agreement reference.
Appendix figures & tables35 assets
Supplementary material from the paper’s appendix.
Appendix
| Item | Setting |
|---|---|
| Dataset | CIFAR-10 (full 50k train / 10k test); CIFAR-100 only in the transfer family |
| Loss / batch size | cross-entropy; batch size 128 everywhere |
| Epochs / schedule | 50 epochs; schedules stepped once per epoch |
| Cosine schedule | CosineAnnealingLR to with = epochs |
| Augmentation | RandomCrop(32, padding=4) RandomHorizontalFlip |
| AlexNet exception | Resize(128) , Normalize(0.5, 0.5) , no augmentation |
Table 1: Vision panel: shared recipe, optimizer definitions, and per-cell learning rates. (a) the recipe held fixed across every run in this appendix; (b) the six training configurations of the panel; (c) the five architectures with their parameter counts and the learning rate of each cell.
| Family | Runs | Recipe deltas with respect to the panel |
|---|---|---|
| Panel | 30 | The five architectures of Table 1 (c) crossed with the six configurations of Table 1 (b); from scratch, seed . |
| Muon arm | 5 | The same five architectures under Muon (momentum , ns_steps ), learning rates (MLP), (AlexNet), (ResNet), (ViT), (ViT-mid). One additional stalled diagnostic run at the ViT model-default learning rate is kept on disk and excluded from readouts. |
| Weight-decay and momentum dose | 26 | Plain SGD at the model-default learning rate, cosine, epochs, seed . Weight-decay ladder over the five architectures, with the override applied to plain SGD only; momentum ladder over the five architectures; one further MLP run at momentum . Two runs collapsed and are kept on disk but excluded from every readout: MLP at momentum (final test accuracy ) and AlexNet at momentum (final test accuracy , chance level). |
| Seed replicates | 10 | Five panel cells re-trained at seeds and , recipe otherwise identical: ResNet–SAM, ResNet–Muon, ViT–Adam, ViT–SAM, ViT-mid–SAM. The two SAM re-runs are not learning-rate-matched (Table 1 note ). |
| Transfer dynamics | 22 | All ViT-mid under plain SGD (momentum , weight decay ), CIFAR-10 statistics pinned, plus one frozen source checkpoint. CIFAR-10 to CIFAR-100 and CIFAR-100 to CIFAR-10 at constant learning rate, two arms ( and ), seeds with weight seed and data-order seed ; scratch controls on both datasets; a stationary continuation ( epochs at ) and step learning-rate switches at epoch . |
| Probe calibration | 45 | Plain SGD, seed , CIFAR-10 statistics pinned. A -run schedule zoo (ResNet and ViT-mid over linear, two step schedules, exponential, warmup-cosine and cosine-with-floor, two ResNet constant-rate runs, a -epoch cosine run and a basin anneal); basin slices (early and late basins crossed with five constant rates, epochs); a -run learning-rate dose ladder ( of base ); two stationary continuations; and a -run blind audit whose configurations were withheld from the analysis side. |
Table 2: Run families of the vision corpus. The panel is the set analyzed throughout Section 4.1 to Section 4.3; the Muon arm and the dose ladders supply the controlled comparisons of Section 4.4; the remaining families support the robustness and calibration checks. The two collapsed momentum runs are kept on disk and excluded from every readout.
| Parameters | ||||
| Model | total | non-embedding | Coverage | |
| Main panel | ||||
| Pythia-70M | 70,426,624 | 18,915,328 | 154 revisions; 123 anchors | |
| Pythia-160M | 162,322,944 | 85,056,000 | 154 revisions; 123 anchors | |
| Pythia-410M | 405,334,016 | 302,311,424 | 123 anchors (246 shards); spectrum 154 rev. | |
| Extended set, reported in Appendix F | ||||
Table 3: Pretraining checkpoint coverage. (a) parameter counts, architecture, and on-disk coverage, split into the main panel and the extended set reported in Appendix F ; (b) public telemetry and evaluation provenance, including the splice checks for the two series whose released telemetry is assembled from more than one run. The 1B rows carry the step116000 checkpoint defect and its masking windows.
Figure 1: Checkpoint and anchor coverage across the pretraining scales. Each row is one model scale on a shared training-step axis, with tokens on the top axis at per step. Ticks give the public checkpoint revisions, released on the schedule and every k steps thereafter up to step k. Bars give the anchor domain actually measured, anchors on a k grid spanning k to k for every scale. The domain is trimmed at both ends for protocol reasons rather than for data availability: the head drops the warmup phase and leaves room for the backward history window, and the tail reserves a complete future window for the longest horizon ( at ). Lines give the span of public training-loss telemetry, and the two smallest scales carry no line because no public run exists for them. Triangles mark splices where the released telemetry is assembled from more than one run. Hatching on the 1B row marks the step- k checkpoint defect and the anchors masked because of it, with short ticks at the horizon-specific future endpoints; the telemetry line and the official evaluations are unaffected by the masking.
| Cut | One group is | What it reads | Chance status at | |
| Flat protocol, the main measurement convention | ||||
| flat | contiguous scalars of the row-major flattened tensor | Between-group ESA dispersion sits at the binomial sampling floor ( , median over cells), so blocks are statistically homogeneous. Bulk group-level FRR hugs the chance floor (median excess , against for the auxiliary class). Cross-architecture ordering claims do not survive recutting. | Not immune: of the bulk pass rate, about pp is chance ( median to ), so the calibrated readout is required. | |
| Structural cuts, physical and functional axes | ||||
| filter | one ResNet output channel’s full kernel stack | – | Names the unit behind ResNet’s flat readout: to of ResNet flat is intra-kernel coherence, destroyed by within-tensor permutation ( shortcut ). : . ESA dispersion floor, not coupled to FRR. | Immune. |
| row | one -convolution row | – | Essentially no pockets: . | Immune. |
| head | per-head q/k/v row block of the fused tensor | / | Strongest structural unit: ESA dispersion floor; of (ViT-mid) and (ViT); restores group-level ESA–FRR coupling (cell-wide median against for flat). | Immune (pass rate for ). |
Table 4: Grouping conventions on the vision panel. is the number of scalars per group. “Immune” means the membership-resample null passes at a rate near zero, so the raw pass fractions carry a negligible chance contribution. The flat protocol is the one cut where a large share of raw passes is chance, which is what motivated the calibrated readout . Levels are not comparable across cuts, since group sizes and chance floors differ, so every level claim in this paper names its cut. Structural and magnitude readouts are pooled over the recipe cells of the group-level lanes and anchors to .
| Cut | One group is | What it reads | Chance status at | |
|---|---|---|---|---|
| positional (reference) | : per-head block; MLP: neuron pair | – | Reference convention of the Pythia protocol. Within-tensor permutation raises the flat reading ( , structural against permuted ), the signature of comoving mass concentrated in a few strong coordinates. | Bare blind at large ; calibrated readout used. |
| attn_out head | per-head output block | Excess at step k while the bare- column reads , the sharpest case of large- blindness. | Calibrated only. | |
| row | one row, the natural coherence scale | (70M) | Absolute pockets revive at row granularity: stable at to , against at head level. Row-length invariance across to (410M peaks at ; 160M scan; 1B below after masking). | legal (analytic floor ). |
| magnitude | within (layer, segment), coordinates sorted by pre-anchor and chunked to the same | as reference | Loses to head grouping on at step k ( against at 70M and 160M); MLP sensitivity is local (70M layer 5 mid-training, about ) with of grid cells negative; at most on absolute across five sizes. | Null shared with the positional cut (grouping-independent). |
| MLP input half | neuron input row ( ) | Reads near zero (mean ). | legal. | |
| MLP output half | neuron output column ( ) | Real emergence carried by the deepest layer: the 70M layer-5 wave rises from k to a peak of at k. Across five model sizes the output half is the only MLP cut with signal (peaks to ), with the input half and the magnitude cut near zero. | legal. |
Table 5: Grouping conventions on the Pythia side. All Pythia readings are registered as correspondences and are not used to certify one another. Column conventions follow Table 4 . The last two rows are the recut used for the granularity ladder in Appendix B.6 .
Figure 2: Probes that use no grouping convention, against the flat readout (bulk scope; one line per run, matched ViT-mid and ResNet pairs; the long-cosine ViT-mid run extends to epoch ). On all three cut-free probes, (a) element-level ESA, (b) tensor-level VCS, and (c) tensor-level FRR, where lower means more predictable, ViT-mid reads as more predictable than ResNet at every anchor, while (d) the flat reference orders the same runs and anchors the other way ( to of pairs). Canonical protocol ( , , ); dashed grey lines mark chance (ESA , VCS ).
Figure 3: Permutation-null decomposition of the flat (bulk scope; one line per run): (a) structural blocking, (b) the membership-permutation null (mean of draws, shading gives min to max), and (c) their difference , which is the intra-block coherence contribution. Permutation destroys to of ResNet’s flat readout and the destroyed part localizes on the kernels ( shortcut convolutions: ), whereas to of ViT-mid’s readout survives and the architecture ordering flips to ViT-mid at or above ResNet on the null column ( pairs). A flat- level is therefore a statement about a grouping convention as well as about an architecture.
Figure 4: Structural-axis cuts under the calibrated readout: excess ( , so zero is chance), median over runs. (a) ViT-mid: physically defined groups surface above-chance structure that the flat axis does not resolve, with whole patch filters reaching and head-level groups at anchor , while slab and projection cuts sit near chance. (b) ResNet: whole filters reach to and rows . Each cut carries a size-matched membership-permutation null that preserves the group-size multiset, and groups use only pre-anchor information. Group size differs across cuts, so levels are comparable only through the calibrated excess.
Figure 5: Pocket density by grouping cut and architecture on the vision panel: share of groups with , pooled over the recipe cells and anchors to of the group-level lanes (bulk tensors, out-of-window target). The structural cuts and the magnitude cut (top three of causal bins) are chance-immune at , so the chance contribution to their bars is negligible. The flat bars are pooled over the same cells weighted by group count and are stacked: the hatched portion is the measured chance-pass fraction under the membership-resample null ( , , and pp of the totals for ResNet, ViT, and ViT-mid). Most of the ViT-family flat readout is chance passing, and most of the ResNet flat readout is intra-kernel coherence. Group size differs across cuts, so levels are comparable within a bar group only in the chance-immune sense.
Figure 6: Magnitude-stratified cut at anchor : groups are equal-width bins of the per-tensor pre-anchor rank (bin weakest, bin strongest), reported as excess over chance, median over runs; the floor at means no group passes. (a) ViT-mid feature-dimension tensors show a monotone concentration signature (top bins to ), while the patch embedding stays at the floor because its structure lies on the spatial axis (Figure 4 ). (b) ResNet reads near zero under the same cut. This is the strongest single-axis sorter on the ViT side, and its architecture contrast runs opposite to the flat readout.
Figure 7: Spectrum diagnostic at the late anchors (anchor on the vision panel, step k on Pythia): top-five eigenvalue mass of the coordinate-trajectory correlation spectrum per tensor family, against a simulated independent null of matching shape (grey band gives min to max over draws), median over runs. Every family of all three architectures exceeds the null by two to four times, so co-moving structure is present in every family; which grouping axis can access it is what differs by architecture.
| Tensors | Parameters | |||||
|---|---|---|---|---|---|---|
| Model | aux | bulk | aux | bulk | aux share | aux composition (tensors) |
| MLP | 3 | 4 | 1,792 | 3,803,648 | 0.05% | bias 3 |
| ViT | 37 | 18 | 14,464 | 531,712 | 2.65% | ln 18, bias 17, tok 2 |
| ViT-mid | 61 | 30 | 32,704 | 2,822,848 | 1.15% | ln 30, bias 29, tok 2 |
| ResNet | 31 | 15 | 7,040 | 2,770,624 | 0.25% | bn 26, bias 4, linear 1 |
| AlexNet | 7 | 8 | 9,344 | 57,035,456 | 0.02% | bias 7 |
Table 6: Auxiliary–bulk attribution and dual-definition audit on the vision panel. (a) Per-model tensor attribution under the operational split, with the composition of the auxiliary class by coarse role (ln/bn/bias/tok; the single defector appears as linear ). (b) Agreement between the morphological and operational definitions at three audit levels; the complete defector list is the single tensor discussed in Appendix C.2 . Definitions are given in Appendix C.1 . Data: the tensor-level table of the canonical group-level panel, with attribution fields constant across anchors and optimizer cells. Parameter counts cover the parameter vector only, with no batch-normalization running statistics.
| Segment | Class | Tensor patterns | Shape (70M) | Params (70M) | Share | |
| bulk | attention.query_key_value.weight | 6 | 4,718,592 | 6.70% | ||
| attn_out | bulk | attention.dense.weight | 6 | 1,572,864 | 2.23% | |
| mlp | bulk | mlp.dense_h_to_4h.weight | 12 | 12,582,912 | 17.87% | |
| mlp.dense_4h_to_h.weight | ||||||
| vec | aux | input_layernorm.{weight,bias} | 50 | 40,960 | 0.058% | |
| post_attention_layernorm.{weight,bias} |
Table 7: Pythia segment scheme and its mapping to the auxiliary–bulk partition, with shapes and parameter counts at 70M.
Figure 8: Where the auxiliary–bulk partition sits in the architecture. One box per parameter tensor (LayerNorm boxes carry weight and bias jointly; linear biases are drawn as chips on their matrix box), with fill colour giving the partition class: orange for auxiliary, which is the vector segment and covers all vector-like parameters including the final LayerNorm; blue for bulk, the weight matrices ( , attn_out , MLP); hatched grey for tensors excluded from the aggregation. (a) Pythia-70M: one parallel-residual block, repeated six times, plus the top level; the two untied embedding matrices are their own segments and are excluded. The marked bias pair ( ) coincides under the parallel residual, so both rows are kept in the panel while aggregates count the pair once. (b) ViT-mid: sequential residuals, repeated seven times; here the token and positional embedding vectors are vector-like and therefore auxiliary, which is the one cross-regime difference from (a), and the ten-parameter head bias ( ) is below the grouping minimum and is the only tensor excluded on the vision side. Shapes are the 70M reference.
| Pythia (one-pass) | CIFAR (multi-pass) | ||
|---|---|---|---|
| Segment | 70M | 160M | counterpart |
| qkv | (ViT-mid qkv) | ||
| vec | – (ViT-mid / ResNet vec) | ||
| mlp | (ViT-mid mlp) | ||
| attn_out | — | ||
Table 8: Pocket stability across training regimes. Top: Jaccard overlap of pocket membership between consecutive anchors ( , median over anchors), at the group-level unit defined at the start of Appendix D . Bottom: residence run lengths on the pretraining side, counting consecutive anchors a group spends inside the pocket set, with the number of runs. Stability is stratified by segment and the stratification pattern differs between the two regimes, so a blanket ordering between them does not hold. At one pretraining anchor spans about of training against about for one vision epoch, so the sparser vision sampling mechanically depresses its values; the transient MLP contrast runs against that bias and is the most robust entry. These overlaps are registered as descriptions of pocket dynamics under the two regimes and are not treated as cross-regime confirmation of a shared mechanism.
| qkv, row level ( = one row) | attn_out | mlp (out-half) | ||||
| Model | Null-line | Main peak | Later wave | Peak | Peak | |
| ( / ) | onset | onset | (layer @ step) | (layer @ step) | (layer @ step) | (layer @ step) |
| 70M ( / ) | k | k | (L4 @ k) | (L5 @ k) | (L4 @ k) | (L5 @ k) |
| 160M ( / ) | k † | k | (L8 @ k) | (L7 @ k) | (L8 @ k) | (L11 @ k) |
| 410M ( / ) | k | k | (L21 @ k) | – (L18–L20 @ k) | not scanned | (L23 @ k) |
| 1B ( / ) | none in window | none | (L1 @ k) | — | not scanned | (L15 @ k) |
Table 9: Row-level emergence readings across five pretraining scales, with the onset criteria and the per-segment conventions given in the table footer.
Figure 9: Row-level pocket emergence across four pretraining scales ( ). Every panel shows the fraction of rows with resolved by depth and training step, where one row is one row ( ); at these group sizes the membership-resample pass rate at is near zero, so the fraction is read as an absolute population rather than as excess over a null. Panels (a) to (d) share one colour scale and one colour bar, so the four sizes are directly comparable; the pooled maximum is (410M layer 21 at k). Dashed vertical lines mark the row-level pocket onset, the earliest anchor at which some layer stays above for three consecutive anchors of the k scan grid: k at 70M and 160M, k at 410M, and none inside the window at 1.4B, whose layer-15 population is still rising at the edge. Rings mark the maximum over layer and anchor. Panel (e) shows the carrier layer of each size on the same step axis with the criterion as a reference line. The 1B series is excluded from this figure on the grounds registered in Appendix F , and its row-level fraction stays at or below throughout in any case. All entries are single-run readings on the public checkpoint series, registered as descriptions of those runs; no ordering between the row-level onset and the pool-coherence onset is asserted here.
Figure 10: The same row-level population on three segments. (a) peak fraction of rows with , maximized over layer and anchor, for , the attention output projection, and the MLP output half at four sizes, with the bar label giving the carrier layer; row scans of the attention output projection exist for 70M and 160M only, and the missing cells are drawn as explicit markers rather than as gaps. (b) and (c) trajectories of the carrier layer of each segment at 70M and 160M. Two conventions travel with the figure: the attention-output granularity is less clean than ’s, since groups are flattened on with as the unit and 70M reads at against at ; and group size differs between segments, so levels are compared within a segment across sizes rather than across segments. The 1B series is excluded on the grounds registered in Appendix F .
Figure 11: Where the k event sits, in the measurement or in its null. (a) and (c) layer-level calibrated readout for ( , raw) at 70M and 160M on a common scale. (b) and (d) the same lane read as two curves: the measured median FRR of those layers (solid) and the pooled membership-resample (dashed), with the FRR axis inverted so that upward means lower FRR and the null pool merged across layers per segment and group size. At 70M the pocket set collapses from groups to between anchors k and k and the layer-level median calibrated readout falls from to , while the measured median FRR of the same groups is unchanged ( to ). What moves is the null, whose falls from to and stays at to through k. The calibrated readout is a difference between a measurement and its null, so the event is located on the null side of that difference and is not described here as a change in the predictability of the groups. The 160M null line descends in two stages, a moderate descent at k and a deep one over about to k, the latter in the neighbourhood of that run’s smoothed loss minimum ( k); the time co-location with the loss landmark is registered as a correspondence only.
Figure 12: Turnover of the pocket set itself, at group level, for the one-pass 70M run (top row, dashed vertical at the to k anchor pair) and the two multi-pass vision runs (bottom row). Columns: set size on a log scale, Jaccard against the previous anchor, and Jaccard against the final anchor. On the set collapses from groups to and then holds to groups, with Jaccard against the final set at or below before k, rising to to over to k, while the adjacent-anchor Jaccard after the collapse has median . The attention output projection never stabilizes. The multi-pass runs show no event of this kind, with counts growing monotonically and the Jaccard against the final set rising steadily, which is cumulative expansion rather than replacement. This panel sets the reading conditions for the two-anchor movement overlaps: because the pocket set changes both size and membership between anchors, an anchor-to-anchor difference in overlap contains both a change in overlap and turnover of the set, and the two are separated only by reporting them side by side.
| Indicator pair | Paradigm side | Direction | Key readings | Null convention | Caveat registered |
| Movement top- pocket coordinates | Pythia qkv (70M) | above the null band | precision at a k ( null ) and at a k ( null ); time-resolved ratio steps across k k, then stays high (cumulative median , recent-window median , max at k); pocket coordinates sit at family movement quantile at a k | hypergeometric, equal cardinality ( = number of pocket scalars), with interval | from checkpoint differences only; AdamW preconditioning not removed. Pocket sets change both membership and size between anchors, so a two-anchor comparison mixes overlap change with set turnover |
| Pythia attn_out (70M), late anchors | below the null band on signed-top | precision vs. null at a k; for anchors k the cumulative top ratio has median and the cumulative bottom ratio , while both recent-window ratios sit at | same | small sets ( – groups) with heavy turnover (adjacent-anchor Jaccard median , Jaccard vs. final set median ); the two time conventions disagree, and both are reported | |
| CIFAR (ViT-mid, ResNet) | above the upper bound, small margin | ViT-mid magnitude-bin families – vs. null (five families, both anchors); ResNet conv / vs. null / ( / ); patch-filter inside the band at both anchors | same | ViT-mid patch at a : the cut is near saturation ( ), which narrows the null interval mechanically; ResNet conv falls below the lower bound at a | |
| LN outlier coordinates per-coordinate dynamics | Pythia 70M ( LN gains); ViT-mid control ( ) | co-located | outlier vs. non-outlier median ESA higher in tensors (e.g. post5 vs. ); median coherence higher in (e.g. in2 vs. ); positive in ( – ), negative for in0 ( ) and post0 ( ); cross-layer outlier Jaccard concentrates on adjacent LN pairs (median , , max for post3 post4) | robust on at the late anchor; a top- control cut reproduces every direction | the final LN is saturated (ESA and coherence on all coordinates, no resolution). ViT-mid gain range is narrow ( ) and only cross-layer pairs are non-zero. Outlier-set formation times are non-monotone (final LN set size at k, at k) |
| M8 low-participation heads pocket membership | Pythia 70M qkv | co-located | heads fall below the pooled-bulk Tukey fence ( ); of them are layer-4 q-heads; the seven lowest ( – ) are exactly the a k pocket set L4:q_h0,1,2,4,5,6,7 | Tukey fence (pooled bulk ) on dimension-normalised participation ratio | participation ratio and pocket flag are computed from the same update history, so this is registered as co-location of two readouts of one record, not as two independent measurements |
| qkv column-space effective rank calibrated-readout extinction | Pythia 70M, 160M | co-located in depth | floor window: 70M qkv L3/L4 participation ratio per dimension / against – in shallow qkv layers and – in mlp; 160M hump-rise window L6–L10 – against – in mlp. The same depths carry the persistent band after the k step | none (descriptive ratio; no resampling null defined for this quantity) | window boundaries are operational (ballistic plus phase-diagram floor windows), the two readouts share the same sequence, and fp16 quantisation noise on per-coordinate increments is registered for this lane |
Table 10: Registry of measured overlaps between pocket membership and indicators defined outside the subspace-residual probe family. Each row records the direction, the key readings, the null convention, and the caveats that travel with it. The rows are not treated as evidence for one another, and none of them supports a statement about what pocket membership represents.
Figure 13: Overlap between pocket coordinates and the top- set of the signed norm-change proxy, labelled movement in the panels, registered against an equal-cardinality hypergeometric null. The proxy is built from checkpoint differences alone; preconditioning is not removed and is not a curvature-based importance measure. (a) observed precision of the top- movement set restricted to pocket coordinates, against the null and its interval, with the pretraining rows annotated by the ratio. (b) quantile position of the pocket coordinates inside their family’s movement distribution, with the diagonal as the no-association reference; the set at k sits at median quantile against about for the vision reference, so the whole set is displaced rather than a tail of it. Pocket sets are the group-level flag at , raw, head granularity. The two anchors carry different pocket sets in both size and membership, so a two-anchor comparison mixes a change in overlap with turnover of the set itself. Each reading is a registration of a measured overlap and is not used to support any statement about what pocket membership represents.
Figure 14: The same overlap resolved anchor by anchor, with the pocket set re-matched at every anchor. The readout is the ratio of the observed overlap to its hypergeometric expectation, with the grey band giving the null interval. Two time conventions for the movement proxy are reported side by side: one accumulates from the start of training, the other over the last five checkpoint or epoch pairs, matching the probe’s history window. (a) : the ratio hugs the random line over to k, steps to at k in one anchor, and stays high for the rest of training. (b) the attention output projection, on small sets with heavy turnover, where the two time conventions disagree and both are reported. (c) ViT-mid patch filters sit slightly above the band early and inside it from epoch apart from single-epoch excursions; the late approach to one has a mechanical component, since the cut approaches saturation. (d) ResNet sits below the band over epochs to and above it from epoch . Every curve is a registration of a measured overlap over time.
Figure 15: Normalization gain outliers and per-coordinate dynamics on the same gain trajectories. A coordinate is an outlier when the robust of its absolute gain at the late anchor exceeds . (a) and (b) per-tensor medians for outlier against non-outlier coordinates across the pretraining gain vectors: median ESA is higher for the outliers in of tensors and median drift coherence in of , with connectors marking the tensors where it is not. The final normalization tensor is saturated, with agreement and coherence equal to on all coordinates, so it carries no resolution and is drawn only for completeness. (c) rank correlation between the outlier score and coherence per tensor, positive in of pretraining tensors and negative in two. On the ViT-mid control one chain reads with the same sign and the others with the opposite sign; its gain range is narrow and its outlier sets hold at most four coordinates. A top- control cut reproduces every direction in both regimes.
Figure 16: Probe readings against two externally defined summaries of realized optimization, over the eight conditions and five anchors of the transfer family, on the bulk backbone scope at . (a) ESA against movement directedness, Eq. ( 5 ), the net displacement of the future window divided by the path length travelled inside it. (b) ESA against the loss-side gradient-coherence estimate, Eq. ( 6 ), which is built from the loss curve, the per-step learning rates, and the per-step gradient norms, and carries no parameter-direction information. Horizontal lines mark the random sign-agreement level of . Conditions differ in dataset, in whether the run is fine-tuned or trained from scratch, in learning rate, and in schedule. Both relations are retrospective correspondences between summaries of the same stored trajectories.
| Parameters | S2 | Train-loss minimum | Bulk ESA landmark | |||
|---|---|---|---|---|---|---|
| Model | total | non-embed. | anchors | in | ( , group ) | |
| Pythia-14M | 14,067,712 | 1,189,888 | 123 | no public telemetry | — | |
| Pythia-31M | 30,494,720 | 4,739,072 | 123 | no public telemetry | — | |
| Pythia-70M | 70,426,624 | 18,915,328 | 123 | step 50,345 ( ) | min @ k; local rise peak @ k | |
| Pythia-160M | 162,322,944 | 85,056,000 | 123 | step 76,409 ( ) | min @ k; sustained rise from k to – | |
| Pythia-410M | 405,334,016 | 302,311,424 | 123 | none in window a | late rise from k, peak @ – k | |
Table 11: Cross-scale readings. (a) sizes, coverage, and the external landmarks available per size. (b) the internal transition readings, all under persistence criteria rather than visual reading; the two onset instruments are different lanes and are not merged. (c) the one model excluded from every cross-scale reading, with its three independent grounds.
Figure 17: Auxiliary and bulk coordinates stay separated at every size and every anchor. Rows: (a) ESA, (b) , (c) DMD direction cosine; columns: the six sizes. The solid line is the group median over the pooled group set and the shaded band the interquartile range; orange is auxiliary, the vector segment, and blue is bulk. Grey dashed references mark in (a) and in (c). All panels use one readout protocol at over the shared anchors, and because the suite trains every size with the same M-token batch, a matched step grid is also a matched token grid. Horizontal comparisons between sizes are therefore at matched steps and tokens, and not at matched loss, compute, or training progress. The median separation holds at every anchor at all six sizes under all three probes; the smallest median gaps are , , and for the three rows, all at 1.4B late in training. Interquartile ranges are disjoint at every anchor for (b) and (c) at all sizes, and for (a) at every size except 1.4B, where they overlap on of anchors. Bulk is the strict matrix set, with both embedding segments dropped; under the looser alternative the bulk means move by at most for (a) and (b) but by up to for (c), which is why the strict rule is used throughout. Pythia-1B is excluded here and in every other cross-scale panel, on the three independent grounds of Table 11 (c).
Figure 18: Segment division of labour across sizes: ESA barely separates the matrix segments while the calibrated readout does. Columns are the six sizes. (a) ESA group median per segment. (b) pooled calibrated excess per segment, drawn as a five-anchor centred rolling median, with the onset markers computed on the unsmoothed series. The three matrix segments share a blue system ( solid, attention output dashed, MLP dotted) and the vector segment is orange; grey dashed references mark in (a) and the null level in (b), where the floor at is the zero-fraction case. The dark dotted line is the sustained collapse onset of the series, and where the series dips and recovers the first crossing is given in grey parentheses. Onsets are k (70M), k (410M), k (31M), k (14M), and k (160M), with no sustained collapse at 1.4B, so the ordering is not monotone in parameter count under either crossing. In (a) the three matrix segments stay within a median of , , , , , and of one another in ascending size order, so the spread between matrix segments shrinks monotonically with size, while the vector segment stays above all of them by at least at every anchor and size.
Figure 19: Where the calibrated signal survives: layer against step at six sizes on one colour scale. Each panel is one model, rows are transformer blocks with layer at the bottom, and columns are the shared anchors. The probe is reported as excess over its null level, so is exactly the null and the raw pocket fraction is the excess plus ; the colour ramp is anchored with white at , every cell at or below the null renders white, and the ramp is clipped at , which affects of cells. Ticks on the right edge mark the layers whose tail-phase mean stays above the null. Early in training the signal covers , , , , , and layers in ascending size order, and then contracts onto a persistent band whose absolute layer index moves deeper as the model deepens. 1.4B does not contract inside the window: of layers are still above the null in the tail phase, and it shows instead an extinction window near to k followed by a recovery from about k. The calibrated readout is identical under both bulk rules, since the vector segment does not enter it.
Figure 20: Internal predictor-family diagnostics on the bulk set across six sizes. (a) median predicted-over-realized step amplitude of the fit; (b) median largest eigenvalue modulus of the fitted operator; (c) the onset step of the amplitude deficit against non-embedding parameter count on a log axis. Line shade encodes size on a single ramp and every curve is additionally labelled at its right end, so colour is never the only encoding. Onset is the first anchor opening five consecutive anchors whose median amplitude ratio is below , marked by the grey dashed reference in (a). The deficit sets in earlier the larger the model, at k, k, k, k, k, and k in ascending size order, which is the only cross-scale ordering in this appendix that is monotone in parameter count. By the last anchor the bulk median amplitude ratio has fallen to to at every size while the eigenvalue modulus settles into to , with 14M the exception at .
Figure 21: Size-controlled architecture comparison under SAM. Curves report canonical bulk ESA across ten anchors for ViT-tiny, ViT-mid, and the parameter-matched ResNet; ViT-mid and the ResNet have nearly equal parameter counts. The dashed line marks the random sign-agreement level of . Each model uses its own standard learning rate, so the comparison controls scale but not every hyperparameter.
Figure 22: Controlled weight-decay and momentum dose ladders for ViT-mid. Panels report (a) bulk ESA, (b) bulk VCS, and (c) bulk FRR against reversed training loss under the canonical vision grouping. Each arm changes only the indicated weight-decay or momentum value relative to plain SGD, and curves are compared over their overlapping loss ranges. The two ladders order the three readouts in opposite directions, and the subspace-residual readout separates from plain SGD at a dose where the two displacement-direction readouts do not.
| Architecture | Arm | Test accuracy | Gain (pp) | Accepted |
|---|---|---|---|---|
| ViT-tiny | baseline | — | — | |
| full scope | ||||
| auxiliary only | ||||
| ViT-mid | baseline | — | — | |
| full scope | ||||
| auxiliary only |
Table 12: Online comparison on CIFAR-10 over three seeds under pure SGD, with momentum and weight decay both zero. All three arms share the same predictor and the same coordinate-wise acceptance mask; the arms differ only in which parameters are eligible for write-back. “Accepted” is the fraction of parameters that actually received a written-back forecast. Values are means over seeds with one standard deviation, and each arm is read at the epoch chosen on a held-out split within its own run.
Figure 23: Multi-seed online comparison under pure SGD for ViT-tiny and ViT-mid. Curves show seed means with standard-deviation bands for ordinary training, full-scope eligibility, and auxiliary-only eligibility.
More in this topic
Neural Network Training Dynamics
Neural network training dynamics studies how parameters and representations change during optimization and how those changes shape learning, stability, and generalization.
See all 499 papers