Measuring Structured Predictability in Neural Training Dynamics: A Cross-Regime Study

Fanqi Wang; Weisheng Tang; Hairong Qi

cs.LGAug 16, 2026

Measuring Structured Predictability in Neural Training Dynamics: A Cross-Regime Study

Authors: Fanqi Wang, Weisheng Tang, Hairong Qi

Organizations: Department of Electrical Engineering and Computer Science, University of Tennessee, Knoxville

Abstract

Modern deep networks are trained through long update trajectories, yet their temporal organization remains less systematically characterized than architectures, losses, or optimizers. We study short-horizon predictability as a measure of temporal redundancy: where, when, and under which training conditions recent updates contain information about near-future parameter motion. We combine three complementary probe families, displacement-direction, subspace-residual, and predictor-based probes, with convention-aware, null-calibrated group-level readouts, and apply them to multi-pass vision training on CIFAR and public Pythia pretraining checkpoints. Across both regimes, vector-like tensors such as normalization parameters and biases (auxiliary parameters) exhibit simpler short-horizon dynamics than matrix-like feature-transforming weights (bulk parameters), whose predictable behavior concentrates in localized, time-varying pockets. Agreement within and across probe families, and with independent trajectory diagnostics, indicates that these measurements capture intrinsic trajectory structure, while probe differences distinguish complementary forms of temporal organization. Controlled CIFAR comparisons further show that architecture and training recipe systematically modulate the measured structure. A Pythia-70M case study further exposes a sequence of role-, depth-, and scale-dependent events, including bulk ESA falling below the random sign-agreement level and the emergence and redistribution of predictable qkv pockets across layers. These results position short-horizon predictability as a retrospective, parameter-resolved diagnostic of training dynamics.

Figures & tables

Figure 1: Within-family agreement on the controlled vision panel. (a) ESA versus group-mean VCS; (b) out-of-window versus in-window FRR; (c) SESA under DMD and linear-trend selectors. Circled points mark cells with near-zero-displacement coordinates.

Figure 2: Cross-family relations on the controlled vision panel. (a) ESA versus out-of-window FRR; (b) out-of-window FRR versus DMD direction cosine; (c) ESA versus DMD lift. Circled points mark cells flagged by the near-zero-displacement diagnostic.

Figure 3: Auxiliary–bulk separation across training regimes. Each line connects the auxiliary and bulk summaries within a matched CIFAR run or Pythia trajectory; circles denote CIFAR runs and stars denote Pythia trajectories. Auxiliary tensors exceed bulk tensors in

29/32

comparisons under ESA and

32/32

under the selector-free DMD direction cosine. Because the two regimes use different grouping protocols, we compare the within-line ordering rather than absolute levels across regimes.

Figure 4: Distribution of FRR excess above the approximate isotropic reference $\sqrt{(D-r)/D}-\mathrm{FRR}$ over structurally grouped bulk parameters, with $r=5$ . The highlighted tail ( $\mathrm{excess}>0.3$ ) contains approximately $6\%$ of pooled group–anchor observations.

Figure 5: Pythia-70M case study. (a) ESA by parameter role. (b) Released training loss and its smoothed trajectory. (c) Fraction of $qkv$ row groups satisfying $\mathrm{FRR}<0.7$ . (d) Head-aligned $qkv$ $\mathrm{LRGF}_{\mathrm{cal}}$ excess. Vertical lines mark the early fading of head-aligned excess, the loss minimum, and the later $qkv$ ESA rise.

Figure 6: Optimizer-family profiles for ViT-mid along training loss (horizontal axis reversed), with canonical bulk ESA on the vertical axis. All-family comparisons use the common support $\mathrm{loss}\approx 0.9$ – $1.4$ ; matched loss does not imply matched optimizer state. Dashed line: $0.5$ sign-agreement reference.

Appendix figures & tables35 assets

Supplementary material from the paper’s appendix.

Appendix

Item	Setting
Dataset	CIFAR-10 (full 50k train / 10k test); CIFAR-100 only in the transfer family
Loss / batch size	cross-entropy; batch size 128 everywhere
Epochs / schedule	50 epochs; schedules stepped once per epoch
Cosine schedule	CosineAnnealingLR to $0$ with $T_{\max}$ = epochs
Augmentation	RandomCrop(32, padding=4) $+$ RandomHorizontalFlip
AlexNet exception	Resize(128) , Normalize(0.5, 0.5) , no augmentation

Appendix

Table 1: Vision panel: shared recipe, optimizer definitions, and per-cell learning rates. (a) the recipe held fixed across every run in this appendix; (b) the six training configurations of the panel; (c) the five architectures with their parameter counts and the learning rate of each cell.

Family	Runs	Recipe deltas with respect to the panel
Panel	30	The five architectures of Table 1 (c) crossed with the six configurations of Table 1 (b); from scratch, seed $42$ .
Muon arm	5	The same five architectures under Muon (momentum $0.95$ , ns_steps ${=}5$ ), learning rates $0.02$ (MLP), $0.01$ (AlexNet), $0.02$ (ResNet), $0.005$ (ViT), $0.01$ (ViT-mid). One additional stalled diagnostic run at the ViT model-default learning rate $0.02$ is kept on disk and excluded from readouts.
Weight-decay and momentum dose	26	Plain SGD at the model-default learning rate, cosine, $50$ epochs, seed $42$ . Weight-decay ladder $\{10^{-4},10^{-3},2\times 10^{-3}\}$ over the five architectures, with the override applied to plain SGD only; momentum ladder $\{0.5,0.95\}$ over the five architectures; one further MLP run at momentum $0.7$ . Two runs collapsed and are kept on disk but excluded from every readout: MLP at momentum $0.95$ (final test accuracy $0.170$ ) and AlexNet at momentum $0.95$ (final test accuracy $0.100$ , chance level).
Seed replicates	10	Five panel cells re-trained at seeds $43$ and $44$ , recipe otherwise identical: ResNet–SAM, ResNet–Muon, ViT–Adam, ViT–SAM, ViT-mid–SAM. The two SAM re-runs are not learning-rate-matched (Table 1 note $\dagger$ ).
Transfer dynamics	22	All ViT-mid under plain SGD (momentum $0$ , weight decay $0$ ), CIFAR-10 statistics pinned, plus one frozen source checkpoint. CIFAR-10 to CIFAR-100 and CIFAR-100 to CIFAR-10 at constant learning rate, two arms ( $0.005$ and $0.025$ ), seeds $42/43/44$ with weight seed $k$ and data-order seed $1000{+}k$ ; scratch controls on both datasets; a stationary continuation ( $+50$ epochs at $0.025$ ) and step learning-rate switches at epoch $25$ .
Probe calibration	45	Plain SGD, seed $42$ , CIFAR-10 statistics pinned. A $16$ -run schedule zoo (ResNet and ViT-mid over linear, two step schedules, exponential, warmup-cosine and cosine-with-floor, two ResNet constant-rate runs, a $100$ -epoch cosine run and a basin anneal); $10$ basin slices (early and late basins crossed with five constant rates, $12$ epochs); a $5$ -run learning-rate dose ladder ( $\times\{\tfrac{1}{25},\tfrac{1}{10},\tfrac{1}{2},2,4\}$ of base $0.025$ ); two stationary continuations; and a $12$ -run blind audit whose configurations were withheld from the analysis side.

Appendix

Table 2: Run families of the vision corpus. The panel is the set analyzed throughout Section 4.1 to Section 4.3; the Muon arm and the dose ladders supply the controlled comparisons of Section 4.4; the remaining families support the robustness and calibration checks. The two collapsed momentum runs are kept on disk and excluded from every readout.

	Parameters
Model	total	non-embedding	$L\times d_{\mathrm{model}}$	Coverage
Main panel
Pythia-70M	70,426,624	18,915,328	$6\times 512$	154 revisions; 123 anchors
Pythia-160M	162,322,944	85,056,000	$12\times 768$	154 revisions; 123 anchors
Pythia-410M	405,334,016	302,311,424	$24\times 1024$	123 anchors (246 shards); spectrum 154 rev.
Extended set, reported in Appendix F

Appendix

Table 3: Pretraining checkpoint coverage. (a) parameter counts, architecture, and on-disk coverage, split into the main panel and the extended set reported in Appendix F ; (b) public telemetry and evaluation provenance, including the splice checks for the two series whose released telemetry is assembled from more than one run. The 1B rows carry the step116000 checkpoint defect and its masking windows.

Figure 1: Checkpoint and anchor coverage across the pretraining scales. Each row is one model scale on a shared training-step axis, with tokens on the top axis at

2{,}097{,}152

per step. Ticks give the

154

public checkpoint revisions, released on the schedule

0,1,2,4,\ldots,512

and every

1

k steps thereafter up to step

143

k. Bars give the anchor domain actually measured,

123

anchors on a

1

k grid spanning

7

k to

129

k for every scale. The domain is trimmed at both ends for protocol reasons rather than for data availability: the head drops the warmup phase and leaves room for the backward history window, and the tail reserves a complete future window for the longest horizon (

143\mathrm{k}-14\mathrm{k}=129\mathrm{k}

\tau{=}14

). Lines give the span of public training-loss telemetry, and the two smallest scales carry no line because no public run exists for them. Triangles mark splices where the released telemetry is assembled from more than one run. Hatching on the 1B row marks the step-

116

k checkpoint defect and the anchors masked because of it, with short ticks at the horizon-specific future endpoints; the telemetry line and the official evaluations are unaffected by the masking.

Cut	One group is	$D$	What it reads	Chance status at $\theta{=}0.7$
Flat protocol, the main measurement convention
flat $K{=}16$	$16$ contiguous scalars of the row-major flattened tensor	$16$	Between-group ESA dispersion sits at the binomial sampling floor ( $1.1\times$ , median over $30$ cells), so blocks are statistically homogeneous. Bulk group-level FRR hugs the chance floor (median excess $.014$ , against $.148$ for the auxiliary class). Cross-architecture ordering claims do not survive recutting.	Not immune: of the $31.2\%$ bulk pass rate, about $25$ pp is chance ( $\mathrm{null}_{\theta}$ median $.23$ to $.25$ ), so the calibrated readout is required.
Structural cuts, physical and functional axes
filter $3{\times}3$	one ResNet output channel’s full $3{\times}3$ kernel stack	$27$ – $2304$	Names the unit behind ResNet’s flat readout: $66$ to $87\%$ of ResNet flat $\mathrm{LRGF}$ is intra-kernel coherence, destroyed by within-tensor permutation ( $1{\times}1$ shortcut $\Delta{=}.0025$ ). $\mathrm{FRR}{<}0.7$ : $1.5\%$ . ESA dispersion $1.9\times$ floor, not coupled to FRR.	Immune.
row $1{\times}1$	one $1{\times}1$ -convolution row	$64$ – $128$	Essentially no pockets: $\mathrm{FRR}{<}0.7=0.1\%$ .	Immune.
head $qkv$	per-head q/k/v row block of the fused $qkv$ tensor	$4096$ / $12544$	Strongest structural unit: ESA dispersion $4.4\times$ floor; $\mathrm{FRR}{<}0.7$ of $19.8\%$ (ViT-mid) and $5.2\%$ (ViT); restores group-level ESA–FRR coupling (cell-wide median $-.42$ against $-.16$ for flat).	Immune (pass rate ${\approx}0$ for $D{\geq}448$ ).

Appendix

Table 4: Grouping conventions on the vision panel.

D

is the number of scalars per group. “Immune” means the membership-resample null passes

\theta{=}0.7

at a rate near zero, so the raw pass fractions carry a negligible chance contribution. The flat

K{=}16

protocol is the one cut where a large share of raw passes is chance, which is what motivated the calibrated readout

\mathrm{LRGF}_{\mathrm{cal}}

. Levels are not comparable across cuts, since group sizes and chance floors differ, so every level claim in this paper names its cut. Structural and magnitude readouts are pooled over the

18

recipe cells of the group-level lanes and anchors

5

40

Cut	One group is	$D$	What it reads	Chance status at $\theta{=}0.7$
positional (reference)	$qkv$ : per-head block; MLP: neuron pair	$10^{3}$ – $10^{5}$	Reference convention of the Pythia protocol. Within-tensor permutation raises the flat $qkv$ reading ( $\Delta{=}-.140$ , structural $.099$ against permuted $.239$ ), the signature of comoving mass concentrated in a few strong coordinates.	Bare $\theta$ blind at large $D$ ; calibrated readout used.
attn_out head	per-head output block	$32768$	Excess $+.263$ at step $70$ k while the bare- $\theta$ column reads $.000$ , the sharpest case of large- $D$ blindness.	Calibrated only.
$W_{qkv}$ row	one $qkv$ row, the natural coherence scale	$512$ (70M)	Absolute pockets revive at row granularity: $\mathrm{FRR}{<}0.7$ stable at $4$ to $5\%$ , against $\approx 0$ at head level. Row-length invariance across $D{=}128$ to $3072$ (410M peaks $.432/.434/.438$ at $D{=}1024/512/256$ ; 160M scan; 1B below $0.5\%$ after masking).	$\theta$ legal (analytic floor ${\approx}0$ ).
magnitude	within (layer, segment), coordinates sorted by pre-anchor $L_{2}$ and chunked to the same $(n,D)$	as reference	Loses to head grouping on $qkv$ at step $10$ k ( $\mathrm{LRGF}_{\mathrm{cal}}$ $.333/.389$ against $.104/.069$ at 70M and 160M); MLP sensitivity is local (70M layer 5 mid-training, about $3\times$ ) with $7$ of $10$ grid cells negative; at most $0.12\%$ on absolute $\theta$ across five sizes.	Null shared with the positional cut (grouping-independent).
MLP input half	neuron input row ( $w_{\mathrm{in}}$ )	$d_{\mathrm{model}}$	Reads near zero (mean $0.04\%$ ).	$\theta$ legal.
MLP output half	neuron output column ( $w_{\mathrm{out}}$ )	$d_{\mathrm{model}}$	Real emergence carried by the deepest layer: the 70M layer-5 wave rises from $64$ k to a peak of $5.4\%$ at $96$ k. Across five model sizes the output half is the only MLP cut with signal (peaks $1.8$ to $15.7\%$ ), with the input half and the magnitude cut near zero.	$\theta$ legal.

Appendix

Table 5: Grouping conventions on the Pythia side. All Pythia readings are registered as correspondences and are not used to certify one another. Column conventions follow Table 4 . The last two rows are the recut used for the granularity ladder in Appendix B.6 .

Figure 2: Probes that use no grouping convention, against the flat readout (bulk scope; one line per run,

8

matched ViT-mid and ResNet pairs; the long-cosine ViT-mid run extends to epoch

90

). On all three cut-free probes, (a) element-level ESA, (b) tensor-level VCS, and (c) tensor-level FRR, where lower means more predictable, ViT-mid reads as more predictable than ResNet at every anchor, while (d) the flat

K{=}16

\mathrm{LRGF}

reference orders the same runs and anchors the other way (

7

8

8

pairs). Canonical protocol (

n_{\mathrm{past}}{=}5

\tau{=}5

\theta{=}0.7

); dashed grey lines mark chance (ESA

0.5

, VCS

0

Figure 3: Permutation-null decomposition of the flat

K{=}16

\mathrm{LRGF}

(bulk scope; one line per run): (a) structural blocking, (b) the membership-permutation null (mean of

5

draws, shading gives min to max), and (c) their difference

\Delta

, which is the intra-block coherence contribution. Permutation destroys

66

87\%

of ResNet’s flat readout and the destroyed part localizes on the

3{\times}3

kernels (

1{\times}1

shortcut convolutions:

\Delta\approx.003

), whereas

82

94\%

of ViT-mid’s readout survives and the architecture ordering flips to ViT-mid at or above ResNet on the null column (

24/24

pairs). A flat-

\mathrm{LRGF}

level is therefore a statement about a grouping convention as well as about an architecture.

Figure 4: Structural-axis cuts under the calibrated readout: excess

=\mathrm{LRGF}_{\mathrm{cal}}-\alpha

(

\alpha{=}0.05

, so zero is chance), median over runs. (a) ViT-mid: physically defined groups surface above-chance structure that the flat axis does not resolve, with whole patch filters reaching

.61

and head-level

qkv

groups

.21

at anchor

45

, while slab and projection cuts sit near chance. (b) ResNet: whole

3{\times}3

filters reach

.55

.62

and

1{\times}1

rows

.18

. Each cut carries a size-matched membership-permutation null that preserves the group-size multiset, and groups use only pre-anchor information. Group size differs across cuts, so levels are comparable only through the calibrated excess.

Figure 5: Pocket density by grouping cut and architecture on the vision panel: share of groups with

\mathrm{FRR}<0.7

, pooled over the

18

recipe cells and anchors

5

40

of the group-level lanes (bulk tensors, out-of-window target). The structural cuts and the magnitude cut (top three of

16

causal bins) are chance-immune at

\theta{=}0.7

, so the chance contribution to their bars is negligible. The flat

K{=}16

bars are pooled over the same

18

cells weighted by group count and are stacked: the hatched portion is the measured chance-pass fraction under the membership-resample null (

6.0

15.5

, and

26.6

pp of the

23.5/18.6/29.8\%

totals for ResNet, ViT, and ViT-mid). Most of the ViT-family flat readout is chance passing, and most of the ResNet flat readout is intra-kernel coherence. Group size differs across cuts, so levels are comparable within a bar group only in the chance-immune sense.

Figure 6: Magnitude-stratified cut at anchor

45

: groups are

16

equal-width bins of the per-tensor pre-anchor

\lVert\Delta w\rVert_{2}

rank (bin

0

weakest, bin

15

strongest), reported as excess over chance, median over runs; the floor at

-0.05

means no group passes. (a) ViT-mid feature-dimension tensors show a monotone concentration signature (top bins

.38

.66

), while the patch embedding stays at the floor because its structure lies on the spatial axis (Figure 4 ). (b) ResNet reads near zero under the same cut. This is the strongest single-axis sorter on the ViT side, and its architecture contrast runs opposite to the flat readout.

Figure 7: Spectrum diagnostic at the late anchors (anchor

45

on the vision panel, step

70

k on Pythia): top-five eigenvalue mass of the coordinate-trajectory correlation spectrum per tensor family, against a simulated independent null of matching shape (grey band gives min to max over

5

draws), median over runs. Every family of all three architectures exceeds the null by two to four times, so co-moving structure is present in every family; which grouping axis can access it is what differs by architecture.

Model	aux	bulk	aux	bulk	aux share	aux composition (tensors)
	Tensors		Parameters
MLP	3	4	1,792	3,803,648	0.05%	bias 3
ViT	37	18	14,464	531,712	2.65%	ln 18, bias 17, tok 2
ViT-mid	61	30	32,704	2,822,848	1.15%	ln 30, bias 29, tok 2
ResNet	31	15	7,040	2,770,624	0.25%	bn 26, bias 4, linear 1
AlexNet	7	8	9,344	57,035,456	0.02%	bias 7

Appendix

Table 6: Auxiliary–bulk attribution and dual-definition audit on the vision panel. (a) Per-model tensor attribution under the operational split, with the composition of the auxiliary class by coarse role (ln/bn/bias/tok; the single defector appears as linear ). (b) Agreement between the morphological and operational definitions at three audit levels; the complete defector list is the single tensor discussed in Appendix C.2 . Definitions are given in Appendix C.1 . Data: the tensor-level table of the canonical group-level panel, with attribution fields constant across anchors and optimizer cells. Parameter counts cover the parameter vector only, with no batch-normalization running statistics.

Segment	Class	Tensor patterns	Shape (70M)	$n$	Params (70M)	Share
$qkv$	bulk	attention.query_key_value.weight	$1536\times 512$	6	4,718,592	6.70%
attn_out	bulk	attention.dense.weight	$512\times 512$	6	1,572,864	2.23%
mlp	bulk	mlp.dense_h_to_4h.weight	$2048\times 512$	12	12,582,912	17.87%
		mlp.dense_4h_to_h.weight	$512\times 2048$
vec	aux	input_layernorm.{weight,bias}	$512$	50	40,960	0.058%
		post_attention_layernorm.{weight,bias}	$512$

Appendix

Table 7: Pythia segment scheme and its mapping to the auxiliary–bulk partition, with shapes and parameter counts at 70M.

Figure 8: Where the auxiliary–bulk partition sits in the architecture. One box per parameter tensor (LayerNorm boxes carry weight and bias jointly; linear biases are drawn as chips on their matrix box), with fill colour giving the partition class: orange for auxiliary, which is the vector segment and covers all vector-like parameters including the final LayerNorm; blue for bulk, the weight matrices (

qkv

, attn_out , MLP); hatched grey for tensors excluded from the aggregation. (a) Pythia-70M: one parallel-residual block, repeated six times, plus the top level; the two untied embedding matrices are their own segments and are excluded. The marked bias pair (

\equiv

) coincides under the parallel residual, so both rows are kept in the panel while aggregates count the pair once. (b) ViT-mid: sequential residuals, repeated seven times; here the token and positional embedding vectors are vector-like and therefore auxiliary, which is the one cross-regime difference from (a), and the ten-parameter head bias (

\dagger

) is below the

K{=}16

grouping minimum and is the only tensor excluded on the vision side. Shapes are the 70M reference.

Segment	70M	160M	counterpart
	Pythia (one-pass)		CIFAR (multi-pass)
qkv	$0.875$	$0.842$	$0.658$ (ViT-mid qkv)
vec	$1.000$	$1.000$	$0.733$ – $0.750$ (ViT-mid / ResNet vec)
mlp	$\mathbf{0.141}$	$\mathbf{0.141}$	$\mathbf{0.480}$ (ViT-mid mlp)
attn_out	$0.333$	$0.476$	—

Appendix

Table 8: Pocket stability across training regimes. Top: Jaccard overlap of pocket membership between consecutive anchors (

k{=}1

, median over anchors), at the group-level unit defined at the start of Appendix D . Bottom: residence run lengths on the pretraining side, counting consecutive anchors a group spends inside the pocket set, with

n

the number of runs. Stability is stratified by segment and the stratification pattern differs between the two regimes, so a blanket ordering between them does not hold. At

k{=}1

one pretraining anchor spans about

0.8\%

of training against about

2\%

for one vision epoch, so the sparser vision sampling mechanically depresses its values; the transient MLP contrast runs against that bias and is the most robust entry. These overlaps are registered as descriptions of pocket dynamics under the two regimes and are not treated as cross-regime confirmation of a shared mechanism.

	qkv, row level ( $D$ = one $W_{\mathrm{qkv}}$ row)				attn_out	mlp (out-half)
Model	Null-line	Pocket	Main peak	Later wave	Peak	Peak
( $L$ / $d_{\mathrm{model}}$ )	onset	onset	(layer @ step)	(layer @ step)	(layer @ step)	(layer @ step)
70M ( $6$ / $512$ )	$20$ k	$20$ k	$\mathbf{42.1\%}$ (L4 @ $84$ k)	$12.3\%$ (L5 @ $116$ k)	$19.3\%$ (L4 @ $80$ k)	$6.9\%$ (L5 @ $84$ k)
160M ( $12$ / $768$ )	$13$ k †	$20$ k	$\mathbf{32.8\%}$ (L8 @ $92$ k)	$30.3\%$ (L7 @ $116$ k)	$15.7\%$ (L8 @ $104$ k)	$15.7\%$ (L11 @ $108$ k)
410M ( $24$ / $1024$ )	$92$ k	$48$ k	$\mathbf{43.2\%}$ (L21 @ $116$ k)	$32$ – $38\%$ (L18–L20 @ $116$ k)	not scanned	$5.2\%$ (L23 @ $116$ k)
1B ( $16$ / $2048$ )	none in window	none	$0.3\%$ (L1 @ $8$ k)	—	not scanned	$1.8\%$ (L15 @ $8$ k)

Appendix

Table 9: Row-level emergence readings across five pretraining scales, with the onset criteria and the per-segment conventions given in the table footer.

Figure 9: Row-level pocket emergence across four pretraining scales ( $qkv$ ). Every panel shows the fraction of rows with $\mathrm{FRR}<0.7$ resolved by depth and training step, where one row is one $W_{qkv}$ row ( $D=d_{\mathrm{model}}$ ); at these group sizes the membership-resample pass rate at $\theta{=}0.7$ is near zero, so the fraction is read as an absolute population rather than as excess over a null. Panels (a) to (d) share one colour scale and one colour bar, so the four sizes are directly comparable; the pooled maximum is $0.4323$ (410M layer 21 at $116$ k). Dashed vertical lines mark the row-level pocket onset, the earliest anchor at which some layer stays above $5\%$ for three consecutive anchors of the $4$ k scan grid: $20$ k at 70M and 160M, $48$ k at 410M, and none inside the window at 1.4B, whose layer-15 population is still rising at the edge. Rings mark the maximum over layer and anchor. Panel (e) shows the carrier layer of each size on the same step axis with the $5\%$ criterion as a reference line. The 1B series is excluded from this figure on the grounds registered in Appendix F , and its row-level fraction stays at or below $0.33\%$ throughout in any case. All entries are single-run readings on the public checkpoint series, registered as descriptions of those runs; no ordering between the row-level onset and the pool-coherence onset is asserted here.

Figure 10: The same row-level population on three segments. (a) peak fraction of rows with

\mathrm{FRR}<0.7

, maximized over layer and anchor, for

qkv

, the attention output projection, and the MLP output half at four sizes, with the bar label giving the carrier layer; row scans of the attention output projection exist for 70M and 160M only, and the missing cells are drawn as explicit markers rather than as gaps. (b) and (c) trajectories of the carrier layer of each segment at 70M and 160M. Two conventions travel with the figure: the attention-output granularity is less clean than

qkv

’s, since groups are flattened on

(d_{\mathrm{model}},d_{\mathrm{head}})

with

d_{\mathrm{head}}

as the unit and 70M reads

19.3\%

D{=}64

against

13.1\%

D{=}512

; and group size differs between segments, so levels are compared within a segment across sizes rather than across segments. The 1B series is excluded on the grounds registered in Appendix F .

Figure 11: Where the ${\sim}20$ k $qkv$ event sits, in the measurement or in its null. (a) and (c) layer-level calibrated readout for $qkv$ ( $\tau{=}5$ , raw) at 70M and 160M on a common scale. (b) and (d) the same lane read as two curves: the measured median FRR of those layers (solid) and the pooled membership-resample $q_{05}$ (dashed), with the FRR axis inverted so that upward means lower FRR and the null pool merged across layers per segment and group size. At 70M the pocket set collapses from $51$ groups to $3$ between anchors $19$ k and $20$ k and the layer-level median calibrated readout falls from $.354$ to $.000$ , while the measured median FRR of the same groups is unchanged ( $.9959$ to $.9952$ ). What moves is the null, whose $q_{05}$ falls from $.9937$ to $.7201$ and stays at $.59$ to $.72$ through $25$ k. The calibrated readout is a difference between a measurement and its null, so the event is located on the null side of that difference and is not described here as a change in the predictability of the groups. The 160M null line descends in two stages, a moderate descent at $13$ k and a deep one over about $72$ to $76$ k, the latter in the neighbourhood of that run’s smoothed loss minimum ( $77.3$ k); the time co-location with the loss landmark is registered as a correspondence only.

Figure 12: Turnover of the pocket set itself, at group level, for the one-pass 70M run (top row, dashed vertical at the

19

20

k anchor pair) and the two multi-pass vision runs (bottom row). Columns: set size on a log scale, Jaccard against the previous anchor, and Jaccard against the final anchor. On

qkv

the set collapses from

51

groups to

3

and then holds

2

11

groups, with Jaccard against the final set at or below

.06

before

48

k, rising to

.5

.8

over

52

70

k, while the adjacent-anchor Jaccard after the collapse has median

.88

. The attention output projection never stabilizes. The multi-pass runs show no event of this kind, with counts growing monotonically and the Jaccard against the final set rising steadily, which is cumulative expansion rather than replacement. This panel sets the reading conditions for the two-anchor movement overlaps: because the pocket set changes both size and membership between anchors, an anchor-to-anchor difference in overlap contains both a change in overlap and turnover of the set, and the two are separated only by reporting them side by side.

Indicator pair	Paradigm side	Direction	Key readings	Null convention	Caveat registered
Movement top- $K$ $\times$ pocket coordinates	Pythia qkv (70M)	above the null band	precision $.195$ at a $70$ k ( $4.0\times$ null $.0486$ ) and $.312$ at a $129$ k ( $9.0\times$ null $.0347$ ); time-resolved ratio steps $0.98\to 3.77$ across $19$ k $\to$ $20$ k, then stays high (cumulative median $7.5\times$ , recent-window median $9.6\times$ , max $16.2\times$ at $23$ k); pocket coordinates sit at family movement quantile $p_{50}{=}.93$ at a $129$ k	hypergeometric, equal cardinality ( $K$ = number of pocket scalars), with $95\%$ interval	$S_{i}=\sum_{t}w_{i}(t)\Delta w_{i}(t)$ from checkpoint differences only; AdamW preconditioning not removed. Pocket sets change both membership and size between anchors, so a two-anchor comparison mixes overlap change with set turnover
	Pythia attn_out (70M), late anchors	below the null band on signed-top	precision $.093$ vs. null $.167$ at a $129$ k; for anchors ${\geq}100$ k the cumulative top ratio has median $0.34$ and the cumulative bottom ratio $1.90$ , while both recent-window ratios sit at ${\approx}0.6$	same	small sets ( $1$ – $15$ groups) with heavy turnover (adjacent-anchor Jaccard median $.33$ , Jaccard vs. final set median $.09$ ); the two time conventions disagree, and both are reported
	CIFAR (ViT-mid, ResNet)	above the upper bound, small margin	ViT-mid magnitude-bin families $.142$ – $.217$ vs. null $.125$ (five families, both anchors); ResNet conv $3{\times}3$ $.304$ / $.382$ vs. null $.282$ / $.367$ ( $1.08\times$ / $1.04\times$ ); patch-filter inside the band at both anchors	same	ViT-mid patch at a $45$ : the cut is near saturation ( $K/N{=}86\%$ ), which narrows the null interval mechanically; ResNet conv $1{\times}1$ falls below the lower bound at a $25$
LN outlier coordinates $\times$ per-coordinate dynamics	Pythia 70M ( $13$ LN gains); ViT-mid control ( $15$ )	co-located	outlier vs. non-outlier median ESA higher in $12/13$ tensors (e.g. post5 $.992$ vs. $.854$ ); median coherence higher in $11/13$ (e.g. in2 $.898$ vs. $.114$ ); $\rho_{S}(z,\text{coherence})$ positive in $11/13$ ( $.78$ – $.94$ ), negative for in0 ( $-.37$ ) and post0 ( $-.48$ ); cross-layer outlier Jaccard concentrates on adjacent LN pairs (median $.023$ , $p_{90}$ $.32$ , max $.56$ for post3 $\times$ post4)	robust $z>3$ on $\|\text{gain}\|$ at the late anchor; a top- $1\%$ control cut reproduces every direction	the final LN is saturated (ESA and coherence $=1.0$ on all $512$ coordinates, no resolution). ViT-mid gain range is narrow ( $\|\text{gain}\|\leq 1.05$ ) and only $4/105$ cross-layer pairs are non-zero. Outlier-set formation times are non-monotone (final LN set size $49$ at $88$ k, $9$ at $129$ k)
M8 low-participation heads $\times$ pocket membership	Pythia 70M qkv	co-located	$18$ heads fall below the pooled-bulk Tukey fence ( $\mathrm{pr}_{\mathrm{norm}}<.9047$ ); $8$ of them are layer-4 q-heads; the seven lowest ( $\mathrm{pr}_{\mathrm{norm}}$ $.262$ – $.622$ ) are exactly the a $70$ k pocket set $\{$ L4:q_h0,1,2,4,5,6,7 $\}$	Tukey fence (pooled bulk $Q_{1}-1.5\,\mathrm{IQR}$ ) on dimension-normalised participation ratio	participation ratio and pocket flag are computed from the same update history, so this is registered as co-location of two readouts of one record, not as two independent measurements
qkv column-space effective rank $\times$ calibrated-readout extinction	Pythia 70M, 160M	co-located in depth	floor window: 70M qkv L3/L4 participation ratio per dimension $.011$ / $.007$ against $.59$ – $.66$ in shallow qkv layers and $.75$ – $.77$ in mlp; 160M hump-rise window L6–L10 $.007$ – $.030$ against $.65$ – $.85$ in mlp. The same depths carry the persistent $\mathrm{LRGF_{cal}}$ band after the $20$ k step	none (descriptive ratio; no resampling null defined for this quantity)	window boundaries are operational (ballistic $[1\mathrm{k},10\mathrm{k})$ plus phase-diagram floor windows), the two readouts share the same $\Delta W$ sequence, and fp16 quantisation noise on per-coordinate increments is registered for this lane

Appendix

Table 10: Registry of measured overlaps between pocket membership and indicators defined outside the subspace-residual probe family. Each row records the direction, the key readings, the null convention, and the caveats that travel with it. The rows are not treated as evidence for one another, and none of them supports a statement about what pocket membership represents.

Figure 13: Overlap between pocket coordinates and the top-

K

set of the signed norm-change proxy, labelled movement in the panels, registered against an equal-cardinality hypergeometric null. The proxy

S_{i}=\sum_{t}w_{i}(t)\Delta w_{i}(t)

is built from checkpoint differences alone; preconditioning is not removed and

S

is not a curvature-based importance measure. (a) observed precision of the top-

K

movement set restricted to pocket coordinates, against the null

K/N

and its

95\%

interval, with the pretraining rows annotated by the ratio. (b) quantile position of the pocket coordinates inside their family’s movement distribution, with the diagonal as the no-association reference; the

qkv

set at

129

k sits at median quantile

.93

against about

.50

for the vision reference, so the whole set is displaced rather than a tail of it. Pocket sets are the group-level flag at

\tau{=}5

, raw, head granularity. The two anchors carry different pocket sets in both size and membership, so a two-anchor comparison mixes a change in overlap with turnover of the set itself. Each reading is a registration of a measured overlap and is not used to support any statement about what pocket membership represents.

Figure 14: The same overlap resolved anchor by anchor, with the pocket set re-matched at every anchor. The readout is the ratio of the observed overlap to its hypergeometric expectation, with the grey band giving the null

95\%

interval. Two time conventions for the movement proxy are reported side by side: one accumulates from the start of training, the other over the last five checkpoint or epoch pairs, matching the probe’s history window. (a)

qkv

: the ratio hugs the random line over

7

19

k, steps to

3.77

20

k in one anchor, and stays high for the rest of training. (b) the attention output projection, on small sets with heavy turnover, where the two time conventions disagree and both are reported. (c) ViT-mid patch filters sit slightly above the band early and inside it from epoch

23

apart from single-epoch excursions; the late approach to one has a mechanical component, since the cut approaches saturation. (d) ResNet

3{\times}3

sits below the band over epochs

9

16

and above it from epoch

17

. Every curve is a registration of a measured overlap over time.

Figure 15: Normalization gain outliers and per-coordinate dynamics on the same gain trajectories. A coordinate is an outlier when the robust

z

of its absolute gain at the late anchor exceeds

3

. (a) and (b) per-tensor medians for outlier against non-outlier coordinates across the

13

pretraining gain vectors: median ESA is higher for the outliers in

12

13

tensors and median drift coherence in

11

13

, with connectors marking the tensors where it is not. The final normalization tensor is saturated, with agreement and coherence equal to

1.0

on all

512

coordinates, so it carries no resolution and is drawn only for completeness. (c) rank correlation between the outlier score and coherence per tensor, positive in

11

13

pretraining tensors and negative in two. On the ViT-mid control one chain reads with the same sign and the others with the opposite sign; its gain range is narrow and its outlier sets hold at most four coordinates. A top-

1\%

control cut reproduces every direction in both regimes.

Figure 16: Probe readings against two externally defined summaries of realized optimization, over the eight conditions and five anchors of the transfer family, on the bulk backbone scope at $\tau{=}5$ . (a) ESA against movement directedness, Eq. ( 5 ), the net displacement of the future window divided by the path length travelled inside it. (b) ESA against the loss-side gradient-coherence estimate, Eq. ( 6 ), which is built from the loss curve, the per-step learning rates, and the per-step gradient norms, and carries no parameter-direction information. Horizontal lines mark the random sign-agreement level of $0.5$ . Conditions differ in dataset, in whether the run is fine-tuned or trained from scratch, in learning rate, and in schedule. Both relations are retrospective correspondences between summaries of the same stored trajectories.

Model	total	non-embed.	$L\times d_{\mathrm{model}}$	anchors	in $[7\mathrm{k},129\mathrm{k}]$	( $\tau{=}5$ , group $p_{50}$ )
	Parameters			S2	Train-loss minimum	Bulk ESA landmark
Pythia-14M	14,067,712	1,189,888	$6\times 128$	123	no public telemetry	—
Pythia-31M	30,494,720	4,739,072	$6\times 256$	123	no public telemetry	—
Pythia-70M	70,426,624	18,915,328	$6\times 512$	123	step 50,345 ( $2.828$ )	min $0.426$ @ $98$ k; local rise peak $0.467$ @ $77$ k
Pythia-160M	162,322,944	85,056,000	$12\times 768$	123	step 76,409 ( $2.465$ )	min $0.424$ @ $68$ k; sustained rise from $78$ k to $0.45$ – $0.47$
Pythia-410M	405,334,016	302,311,424	$24\times 1024$	123	none in window a	late rise from ${\approx}85$ k, peak ${\approx}0.49$ @ $110$ – $118$ k

Appendix

Table 11: Cross-scale readings. (a) sizes, coverage, and the external landmarks available per size. (b) the internal transition readings, all under persistence criteria rather than visual reading; the two onset instruments are different lanes and are not merged. (c) the one model excluded from every cross-scale reading, with its three independent grounds.

Figure 17: Auxiliary and bulk coordinates stay separated at every size and every anchor. Rows: (a) ESA, (b)

1-\mathrm{FRR}

, (c) DMD direction cosine; columns: the six sizes. The solid line is the group median over the pooled group set and the shaded band the interquartile range; orange is auxiliary, the vector segment, and blue is bulk. Grey dashed references mark

0.5

in (a) and

0

in (c). All panels use one readout protocol at

\tau{=}5

over the

123

shared anchors, and because the suite trains every size with the same

2

M-token batch, a matched step grid is also a matched token grid. Horizontal comparisons between sizes are therefore at matched steps and tokens, and not at matched loss, compute, or training progress. The median separation holds at every anchor at all six sizes under all three probes; the smallest median gaps are

0.036

0.351

, and

0.115

for the three rows, all at 1.4B late in training. Interquartile ranges are disjoint at every anchor for (b) and (c) at all sizes, and for (a) at every size except 1.4B, where they overlap on

30\%

of anchors. Bulk is the strict matrix set, with both embedding segments dropped; under the looser alternative the bulk means move by at most

1.2\times 10^{-4}

for (a) and (b) but by up to

6.3\times 10^{-3}

for (c), which is why the strict rule is used throughout. Pythia-1B is excluded here and in every other cross-scale panel, on the three independent grounds of Table 11 (c).

Figure 18: Segment division of labour across sizes: ESA barely separates the matrix segments while the calibrated readout does. Columns are the six sizes. (a) ESA group median per segment. (b) pooled calibrated excess per segment, drawn as a five-anchor centred rolling median, with the onset markers computed on the unsmoothed series. The three matrix segments share a blue system (

qkv

solid, attention output dashed, MLP dotted) and the vector segment is orange; grey dashed references mark

0.5

in (a) and the null level

0

in (b), where the floor at

-0.05

is the zero-fraction case. The dark dotted line is the sustained collapse onset of the

qkv

series, and where the series dips and recovers the first crossing is given in grey parentheses. Onsets are

20

k (70M),

25

k (410M),

28

k (31M),

43

k (14M), and

71

k (160M), with no sustained collapse at 1.4B, so the ordering is not monotone in parameter count under either crossing. In (a) the three matrix segments stay within a median of

0.048

0.038

0.027

0.021

0.014

, and

0.011

of one another in ascending size order, so the spread between matrix segments shrinks monotonically with size, while the vector segment stays above all of them by at least

0.034

at every anchor and size.

Figure 19: Where the $qkv$ calibrated signal survives: layer against step at six sizes on one colour scale. Each panel is one model, rows are transformer blocks with layer $0$ at the bottom, and columns are the $123$ shared anchors. The probe is reported as excess over its $\alpha{=}0.05$ null level, so $0$ is exactly the null and the raw pocket fraction is the excess plus $0.05$ ; the colour ramp is anchored with white at $0$ , every cell at or below the null renders white, and the ramp is clipped at $0.60$ , which affects $0.21\%$ of cells. Ticks on the right edge mark the layers whose tail-phase mean stays above the null. Early in training the signal covers $5/6$ , $6/6$ , $6/6$ , $10/12$ , $23/24$ , and $23/24$ layers in ascending size order, and then contracts onto a persistent band whose absolute layer index moves deeper as the model deepens. 1.4B does not contract inside the window: $22$ of $24$ layers are still above the null in the tail phase, and it shows instead an extinction window near $96$ to $118$ k followed by a recovery from about $120$ k. The calibrated readout is identical under both bulk rules, since the vector segment does not enter it.

Figure 20: Internal predictor-family diagnostics on the bulk set across six sizes. (a) median predicted-over-realized step amplitude of the fit; (b) median largest eigenvalue modulus of the fitted operator; (c) the onset step of the amplitude deficit against non-embedding parameter count on a log axis. Line shade encodes size on a single ramp and every curve is additionally labelled at its right end, so colour is never the only encoding. Onset is the first anchor opening five consecutive anchors whose median amplitude ratio is below

0.5

, marked by the grey dashed reference in (a). The deficit sets in earlier the larger the model, at

58

31

21

15

14

k, and

11

k in ascending size order, which is the only cross-scale ordering in this appendix that is monotone in parameter count. By the last anchor the bulk median amplitude ratio has fallen to

0.13

0.38

at every size while the eigenvalue modulus settles into

0.59

0.68

, with 14M the exception at

0.78

Figure 21: Size-controlled architecture comparison under SAM. Curves report canonical bulk ESA across ten anchors for ViT-tiny, ViT-mid, and the parameter-matched ResNet; ViT-mid and the ResNet have nearly equal parameter counts. The dashed line marks the random sign-agreement level of $0.5$ . Each model uses its own standard learning rate, so the comparison controls scale but not every hyperparameter.

Figure 22: Controlled weight-decay and momentum dose ladders for ViT-mid. Panels report (a) bulk ESA, (b) bulk VCS, and (c) bulk FRR against reversed training loss under the canonical $K{=}16$ vision grouping. Each arm changes only the indicated weight-decay or momentum value relative to plain SGD, and curves are compared over their overlapping loss ranges. The two ladders order the three readouts in opposite directions, and the subspace-residual readout separates from plain SGD at a dose where the two displacement-direction readouts do not.

Architecture	Arm	Test accuracy	Gain (pp)	Accepted
ViT-tiny	baseline	$0.7422\pm 0.0057$	—	—
	full scope	$0.7484\pm 0.0018$	$+0.62\pm 0.43$	$23.9\%$
	auxiliary only	$\mathbf{0.7516\pm 0.0058}$	$\mathbf{+0.94\pm 0.09}$	$\mathbf{0.80\%}$
ViT-mid	baseline	$0.7425\pm 0.0032$	—	—
	full scope	$0.7498\pm 0.0032$	$+0.73\pm 0.18$	$23.2\%$
	auxiliary only	$\mathbf{0.7511\pm 0.0029}$	$\mathbf{+0.87\pm 0.09}$	$\mathbf{0.32\%}$

Appendix

Table 12: Online comparison on CIFAR-10 over three seeds under pure SGD, with momentum and weight decay both zero. All three arms share the same predictor and the same coordinate-wise acceptance mask; the arms differ only in which parameters are eligible for write-back. “Accepted” is the fraction of parameters that actually received a written-back forecast. Values are means over seeds with one standard deviation, and each arm is read at the epoch chosen on a held-out split within its own run.

Figure 23: Multi-seed online comparison under pure SGD for ViT-tiny and ViT-mid. Curves show seed means with $\pm 1$ standard-deviation bands for ordinary training, full-scope eligibility, and auxiliary-only eligibility.

Neural Network Training Dynamics

Neural network training dynamics studies how parameters and representations change during optimization and how those changes shape learning, stability, and generalization.

See all 499 papers

Sep 28, 2026

LLN: Learnable Lens Networks for Parameter-Efficient Long-Horizon Dynamical Prediction

Explicit residual connections of the form (x+f(x)), often combined with normalization layers, have become a standard strategy for training very deep neural networks. However, residual addition primarily provides an algebraic shortcut for gradient propagation, while leaving the evolution of feature geometry across layers largely unconstrained. We introduce Learnable Lens Networks (LLN), a physics-inspired architecture that replaces direct feature-space residual accumulation with learnable optical transport in an augmented position-angle phase space. Each layer alternates between free propagation, which provides an implicit transport path, and a learnable lens field that performs nonlinear trajectory transformation and focusing. Theoretically, we establish that LLN transport is globally invertible and volume-preserving for any differentiable lens field, with the implemented coordinate-wise Gaussian transport further satisfying symplecticity. Importantly, these structural constraints do not limit expressivity: with unrestricted embeddings and readouts, LLN retain universal approximation of continuous end-to-end maps. Experiments across diverse dynamical systems demonstrate that LLN improves long-horizon prediction while using substantially fewer parameters than same-depth comparators. Further analysis reveals stable depth-wise gradient transport and interpretable learned dynamics under the coupled propagation and refraction design.

Binbin Yong, Zhao Su, Lan Guo +3

Lanzhou University · Monash University · University of Wollongong

Invertible Neural Networks Residual Learning Neural Network Training Dynamics Structure-Preserving Neural Networks

Ask PDF

May 9, 2026

Predicting Plasticity in Deep Continual Learning: A Theoretical Perspective

Deep continual learning requires models to adapt to new tasks without retraining from scratch. However, neural networks can lose their ability to adapt to new tasks after training on previous ones, a phenomenon known as loss of plasticity. There have been several explanations and diagnostics proposed for plasticity loss. Motivated by the philosophy that "all models are wrong, but some are useful", we ask: can existing diagnostics predict a neural network's plasticity? In this work, we take a practical view to interpret plasticity as trainability, i.e., a neural network's future optimization gain on a target task. We first take a theoretical approach, showing, by constructing a few counterexamples, that some widely adopted diagnostics of plasticity, including representation rank and neural tangent kernel rank, can fail to predict the loss of trainability in both regression and classification settings. We instead propose a novel metric, called optimization readiness, which combines gradient strength and gradient reliability. We prove that optimization readiness lower bounds one-step optimization gain under standard smoothness assumptions, providing a theoretical guarantee for its predictive power. Empirically, we show that across commonly used deep continual learning settings, such as Slowly-Changing Regression and Permuted MNIST, optimization readiness more reliably ranks checkpoints by trainability than prior diagnostics, even with substantially fewer samples.

Jiuqi Wang, Jayanth Srinivasa, Claire Chen +3

University of Virginia · Cisco Research · California Institute of Technology +1

Deep Learning Optimization Loss of Plasticity Continual Learning Neural Network Training Dynamics

Ask PDF

Oct 7, 2026

When does a network's training history predict its future learning better than its current state? Evidence from a response probe and a forecasting screen

Networks that behave alike now can still learn differently when training continues. Work on loss of plasticity and critical periods shows that the path to a state shapes what follows; it does not show whether the path carries information that a measurement of the state itself misses. We ask when the training history of a network predicts its future learning better than its current state. In a main study, small multilayer perceptrons were trained under three history regimes (42 histories), and future learning was measured at four checkpoints by a short probe: a copy of the network trained for 100 updates on a new task. Before the prediction result was read, the protocol checked the probe. It responded monotonically to a function-preserving rescaling of hidden units, repeated measurements agreed (intraclass correlation 0.940, [0.903, 0.997], in the least reliable class, mean of three repeats), and a re-initialisation of units was visible directly after it but not 100 to 200 updates later. A history state of at most four dimensions did not improve on a calibrated model of the current state (gain -21.4%, 90% interval [-91.9, 8.1]; required in advance: 10%). A companion screen on 1,560 synthetic regression runs asked the same question for a target further away, the final error of the run. There, history models forecast better than the current validation error after 12 of up to 240 epochs (compact state 30.3%, [15.8, 39.4], a contextual comparison) and were not distinguishable from it after 48. In both studies the history was informative only while the current state was not yet informative about the target; this reading was formed after the results.

Martin Hofmann, Patrick Mäder

Data-intensive Systems and Visualization Group (dAI.SY), Technische Universität Ilmenau, Max-Planck-Ring 14, 98693 Ilmenau, Thuringia, Germany · German Centre for Integrative Biodiversity Research (iDiv) Halle–Jena–Leipzig, Deutscher Platz 5e, 04103 Leipzig, Saxony, Germany · Faculty of Biological Sciences, Friedrich Schiller University, Fürstengraben 1, 07745 Jena, Thuringia, Germany

Neural Network Training Dynamics

Ask PDF

Abstract

Neural Network Training Dynamics

Explore similar work

LLN: Learnable Lens Networks for Parameter-Efficient Long-Horizon Dynamical Prediction

Predicting Plasticity in Deep Continual Learning: A Theoretical Perspective

When does a network's training history predict its future learning better than its current state? Evidence from a response probe and a forecasting screen