What a Weight Update Buys: Leverage Across Training Stages

Shikhar Shiromani

cs.LGOct 27, 2026

What a Weight Update Buys: Leverage Across Training Stages

Authors: Shikhar Shiromani

Abstract

Post-training moves a language model's weights by about a thousandth of their norm and changes its behavior sharply; pretraining moves them by a third and buys a slow drift. We introduce leverage, an exact measure of what a unit of weight change buys: the Fisher quadratic form of an update, computed with one forward-mode pass per direction, divided by the same form for random directions of the same size at the same checkpoint. Leverage orders released Olmo-3, Tülu-3, Zephyr, and OLMo-2 checkpoints over four orders of magnitude: pretraining windows sit at the random level on prose, SFT five times above it, RL-Zero and RLVR at 400-1,500, and DPO at 400-23,000. With this instrument we find that pretraining accumulates into noise: windows of 30k to 1M steps are no more consequential on prose than random directions of their size, as a noisy quadratic model of SGD predicts. We find that training precision changes what an update looks like: training bf16 parameters without an fp32 master copy freezes 36-99% of coordinates, lowers leverage up to sevenfold, and creates an alignment between update magnitudes and the diagonal Fisher that fp32 training does not show. We derive and verify a law for reuse: the sign-flip null predicts the relative damage of random drop-and-rescale (DARE) at keep rate $p$ as $(1-p)/(p\,ρ_{\mathrm{sign}})$ , within a factor of two of measurement wherever DARE succeeds. Two controls make these readings reliable: every controlled SFT and DPO trajectory with fp32 master weights loses leverage as it moves, and two re-runs of a released DPO stage from its anchor with its published recipe land within 1% of the released displacement and 4% of its leverage, where a plain recipe reads five to six times lower; leaving one component out traces the level to the length-normalized loss and its batch size of 128 together.

Figures & tables

Figure 1: Leverage against displacement, measured exactly, for three model families. Leverage

\rho

(isotropic null; mixed corpus) against relative displacement, on the same axes in every panel; colors give the stage. (a) Controlled SFT, DPO, and GRPO runs on Qwen2.5-0.5B with fp32 master weights, three learning rates each, one seed per curve (repeats: Table 5 ). (b) OLMo-2-1B: the released DPO delta (square), the released RLVR checkpoints every 200 steps (line; its square is the released final), and the released DPO recipe re-run from the released SFT checkpoint (thick line), ending on the released delta. (c) Released Olmo-3-7B (circles), Tülu-3-8B, and Zephyr-7B (diamonds) deltas, and controlled SFT and DPO on Qwen2.5-7B (dashed, two learning rates).

stage	$n$	$\mathrm{rel}\Delta W$	$\rho_{\text{iso}}$	$\rho_{\text{profile}}$	$\rho_{\text{sign}}$	$c_{\text{sign}}/c_{\text{profile}}$
pretraining windows	2	3.7e-1–3.8e-1	0.86 (0.78–0.95)	0.87 (0.77–0.99)	0.87 (0.82–0.92)	1.0 (0.93–1.1)
SFT	2	1.5e-1–1.6e-1	5.5 (5.4–5.7)	4.1 (4.0–4.3)	3.2 (3.0–3.5)	1.3 (1.2–1.3)
RL-Zero	5	6.4e-4–1.5e-3	708 (388–1,508)	837 (442–1,630)	616 (354–1,226)	1.4 (1.1–1.6)
RLVR after DPO	3	7.1e-4–1.1e-3	553 (410–665)	607 (519–725)	408 (264–533)	1.5 (1.2–2.0)
DPO	4	3.8e-4–1.4e-3	3,632 (378–22,570)	4,088 (392–21,557)	1,702 (489–9,844)	2.4 (0.8–5.3)
OLMo-2-1B DPO	1	8.5e-4	1,452	1,287	1,012	1.3

Table 1: The released spectrum, measured exactly. One row per training stage;

n

is the number of released directions in it, and

\mathrm{rel}\Delta W=\|\Delta\|/\|\theta_{0}\|

is how far the stage moved the weights as a fraction of their norm. The three leverage columns divide a direction’s Fisher curvature by that of a random direction of the same size drawn at the same anchor: an isotropic Gaussian (

\rho_{\text{iso}}

), a Gaussian with the delta’s own per-tensor norms (

\rho_{\text{profile}}

), or the delta with its signs randomized (

\rho_{\text{sign}}

); 1 means no more consequential than random. The last column, the magnitude alignment, is the sign-flip null’s curvature over the profile null’s: how much the delta’s squared magnitudes sit on high-curvature coordinates. Mixed corpus; geometric mean over the stage’s directions, range in parentheses. Stages above the rule are Olmo-3-7B, Tülu-3-8B, and Zephyr-7B; per-direction values, the small corpora, and the null uncertainty are in Table 2 and Appendix A .

Figure 2: A released DPO stage, reproduced, and the recipe knobs under plain optimization settings. (a) The released OLMo-2-0425-1B DPO recipe re-run from the released SFT checkpoint with fp32 master weights, two data-order seeds, measured every 296 steps. Open diamond: the final checkpoint rounded to bf16. Star: the released DPO delta. Below: a plain DPO recipe from the same anchor and its swaps to the released data or loss (mean of three seeds, range shaded). (b) Leverage at a matched displacement for every DPO variant with seed repeats: filled, mean; open, seeds. TRL ( von Werra et al., 2020 ) rows vary batch size and schedule at

5\times 10^{-6}

; the rest use our trainer with one component swapped, plus the loss decompositions at the OLMo-2-1B anchor (two seeds each).

Figure 3: Leaving one component out of the released recipe. Leverage against displacement at the OLMo-2-1B SFT anchor for 600 steps of the released optimization settings (batch 128,

2.5\times 10^{-6}

with warmup and linear decay over 2,955 steps, context 2,048). Arms: the plain loss and data, or the released recipe minus its loss, data, or context length (three seeds; line the mean, band the range). The released re-run (two seeds) is shown for comparison. Table 6 gives matched-displacement ratios.

Figure 4: bf16 parameters without a master copy distort the update. Qwen2.5-0.5B, SFT and DPO at learning rates

5\times 10^{-7}

5\times 10^{-6}

, and

5\times 10^{-5}

(end markers), 600 steps, identical data and seed. Blue solid: fp32 master weights. Red dashed: bf16 parameters. (a, b) Leverage and (c, d) magnitude alignment (dotted at 1) against displacement. One seed per curve; at

5\times 10^{-6}

three seeds differ by at most 8% (Table 5 ).

Figure 5: The sign-flip null predicts DARE damage until DARE breaks. Measured damage, the KL between the DARE’d and the unpruned model relative to the delta’s own KL, both at full scale (mixed corpus, mean of two masks), against

(1-p)/(p\,\rho_{\text{sign}})

for six released deltas at keep rates

p=0.5

(circles), 0.1 (squares), and 0.01 (triangles). Dotted: equality.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

direction	$\mathrm{rel}\Delta W$	unchanged	$\rho_{\text{iso}}$ math	wiki	mixed	FD mixed	$\rho_{\text{profile}}$	$\rho_{\text{sign}}$	$c_{\text{sign}}/c_{\text{profile}}$
pre_late_back	3.83e-1	0.00	1.0	1.1	0.95	0.46	0.99	0.92	1.08
pre_mid_back	3.71e-1	0.00	0.84	0.81	0.78	0.41	0.77	0.82	0.93
sft_instruct	1.62e-1	0.01	5.1	10	5.7	2.0	4.3	3.5	1.22
sft_think	1.50e-1	0.01	6.7	13	5.4	2.8	4.0	3.0	1.34
rlzero_math	1.27e-3	0.58	4,356	235	710	707	865	544	1.59
rlzero_code	1.51e-3	0.53	937	186	388	384	442	354	1.25

Appendix

Table 2: Every released direction, exact form. Columns as in Table 1 , one row per direction. Leverage against isotropic nulls on the three corpora, and against profile and sign-flip nulls on the mixed corpus (6 isotropic draws per anchor and 3 profile and 3 sign-flip draws per direction; 24, 12, and 12 at the Think anchors; 8, 2, and 2 for the OLMo-2-1B rows); “unchanged” is the fraction of coordinates bit-identical to the anchor (bf16 storage). “FD mixed” is the earlier finite-difference read at the certified common scale (Appendix B ); dpo_think has no common certified scale there and the OLMo-2-1B deltas were not in that campaign.

recipe	arm	lr	$\mathrm{rel}\Delta W$	$\rho_{\text{iso}}$ (mixed)	slope mixed	math	wiki	$c_{\text{sign}}/c_{\text{profile}}$
fp32, b8, const.	SFT	5e-7	2.4e-4–1.3e-3	267 $\to$ 35	-1.29	-0.94	-1.23	0.37–1.38
fp32, b8, const.	SFT	5e-6	1.4e-3–6.0e-3	59 $\to$ 8.0	-1.38	-1.29	-1.20	0.37–0.64
fp32, b8, const.	SFT	5e-5	1.2e-2–3.7e-2	21 $\to$ 6.7	-0.99	-1.23	-0.93	0.60–0.80
fp32, b8, const.	DPO	5e-7	9.4e-5–3.2e-4	728 $\to$ 509	-0.36	+0.61	+0.17	1.30–1.37
fp32, b8, const.	DPO	5e-6	9.1e-4–2.9e-3	198 $\to$ 92	-0.70	-1.34	-0.46	0.97–1.11
fp32, b8, const.	DPO	5e-5	9.1e-3–3.0e-2	65 $\to$ 25	-0.83	-0.66	-0.51	0.73–0.97

Appendix

Table 3: Every controlled run on Qwen2.5-0.5B (UltraFeedback, 600 steps for batch 8 and 300 for batch 128, seed 0; the GRPO rows train on GSM8K for 150 steps and the last two rows for 10,000 steps). “unchanged” is the fraction of coordinates bit-identical to the base at the last checkpoint (seed 0; the other seeds of the

5\times 10^{-5}

pure-bf16 runs reach 0.46). Slopes are ordinary least squares of

\log c

\log\mathrm{rel}\Delta W

over all checkpoints of a run (12; 6 for GRPO, 40 for the 10,000-step runs); the last column ranges over those with nulls (6; 3 and 10).

anchor	recipe	$\mathrm{rel}\Delta W$	$\rho_{\text{iso}}$ (mixed)	slope	$c_{\text{sign}}/c_{\text{profile}}$
OLMo-2-1B-SFT	released recipe (dpo_norm, $\beta$ 5, OLMo mix, b128, linear, $2.5\times 10^{-6}$ )	2.9e-4–8.6e-4	3,271 $\to$ 1,447	-0.77	1.26–1.32
OLMo-2-1B-SFT	plain: sigmoid, $\beta$ 0.1, UltraFeedback, b8, const., $5\times 10^{-6}$	4.1e-4–1.4e-3	415 $\to$ 205	-0.52	0.96–0.99
OLMo-2-1B-SFT	plain, OLMo mix	4.2e-4–1.4e-3	430 $\to$ 191	-0.62	0.93–1.01
OLMo-2-1B-SFT	plain, dpo_norm $\beta$ 5	4.1e-4–1.2e-3	206 $\to$ 282	+0.25	1.00–1.02
Qwen2.5-0.5B	plain	9.5e-4–3.0e-3	185 $\to$ 78	-0.77	0.84–1.01
Qwen2.5-0.5B	plain, dpo_norm $\beta$ 5	9.8e-4–2.7e-3	90 $\to$ 132	+0.42	1.00–1.02

Appendix

Table 4: Runs of the reproduction trainer. The released OLMo-2-1B recipe and single-knob variants of a plain DPO recipe at three anchors; leverage at the first and last checkpoint, slope of

\log c

\log\mathrm{rel}\Delta W

, and the range of the magnitude alignment over checkpoints with nulls. One seed each; Table 5 gives the repeats.

condition	seeds	ref. $\mathrm{rel}\Delta W$	$\rho_{\text{iso}}$ at ref. (range)	slope (mean $\pm$ s.d.)
SFT, bf16 params, b8 const.	3	2.0e-3	6.8 (6.7–6.8)	-1.50 $\pm$ 0.01
SFT, fp32, b8 const.	3	2.0e-3	32 (31–32)	-1.37 $\pm$ 0.00
DPO, bf16 params, b8 const.	3	5.0e-4	70 (69–72)	-0.24 $\pm$ 0.06
DPO, fp32, b8 const.	3	1.0e-3	177 (171–184)	-0.64 $\pm$ 0.05
DPO, fp32, b8 linear	3	1.0e-3	179 (172–188)	-0.73 $\pm$ 0.02
DPO, fp32, b128 const.	3	1.0e-3	165 (161–167)	-1.02 $\pm$ 0.04

Appendix

Table 5: Seed repeats. Leverage at a reference displacement (mixed corpus; mean and range over seeds) and the slope of

\log c

\log\mathrm{rel}\Delta W

(mean

\pm

s.d.), for every controlled comparison in the body. Seeds change the data order; the released-recipe re-run uses data-order seeds 111 and 222. The pure-bf16 DPO run at

5\times 10^{-6}

stops at

\mathrm{rel}\Delta W=5.9\times 10^{-4}

, so its reference displacement is

5\times 10^{-4}

run (600 steps at the released settings)	seeds	$\mathrm{rel}\Delta W$	$\rho$ at $4\times 10^{-4}$ : run / released (released $\div$ run)	$\rho$ at $5\times 10^{-4}$	slope	$c_{\text{sign}}/c_{\text{profile}}$	$\rho_{\text{sign}}$
plain loss + plain data	3	5.8e-5–5.1e-4	707 (685–737) / 2,711 (3.8 $\times$ )	511 (502–520) / 2,353 (4.6 $\times$ )	-0.95 $\pm$ 0.01	1.01–1.02	462–482
released minus its loss	3	5.7e-5–5.2e-4	495 (446–549) / 2,711 (5.5 $\times$ )	323 (311–335) / 2,353 (7.3 $\times$ )	-1.55 $\pm$ 0.04	0.92–1.02	293–524
released minus its data	3	5.4e-5–5.1e-4	1,528 (1,426–1,646) / 2,711 (1.8 $\times$ )	1,282 (1,276–1,289) / 2,353 (1.8 $\times$ )	-0.27 $\pm$ 0.06	1.23–1.25	977–1,006
released minus its context length	3	5.6e-5–5.6e-4	2,804 (2,722–2,870) / 2,711 (1.0 $\times$ )	2,307 (2,227–2,350) / 2,353 (1.0 $\times$ )	-0.48 $\pm$ 0.02	1.29–1.31	1,472–1,552

Appendix

Table 6: Leave-one-out at the released settings , OLMo-2-1B SFT anchor, three data-order seeds per arm: leverage at two matched displacements (seed mean and range) against the released re-run (mean of two seeds, interpolated), the slope of

\log c

\log\mathrm{rel}\Delta W

over the six checkpoints (mean

\pm

s.d.), and the range over seeds of the last checkpoint’s magnitude alignment and

\rho_{\text{sign}}

. One seed of the minus-loss arm met a gradient spike at step 558 (gradient norm 727 against a median of 32 and no other step above 163; the recipe does not clip). Its step-600 checkpoint then carries 93% of its form in 1% of positions. We exclude that checkpoint; its median-based read, 228, continues the arm’s decline. The

5\times 10^{-4}

cell of that arm therefore averages the two seeds whose clean trajectories reach it. The upper end of its

\rho_{\text{sign}}

range (524) is the spiking seed’s step-400 checkpoint, its last with nulls.

run (released loss and data)	seeds	$\mathrm{rel}\Delta W$	$\rho$ at $4\times 10^{-4}$ : run / released (released $\div$ run)	$\rho$ at $5\times 10^{-4}$	slope	$c_{\text{sign}}/c_{\text{profile}}$	$\rho_{\text{sign}}$
constant $2.5\times 10^{-6}$ , batch 128, 600 steps	3	3.0e-4–6.9e-4	2,236 (2,031–2,363) / 2,711 (1.2 $\times$ )	2,113 (1,944–2,321) / 2,353 (1.1 $\times$ )	-0.21 $\pm$ 0.04	1.25–1.30	1,282–1,385
constant $2.5\times 10^{-6}$ , batch 8, 800 steps	3	2.7e-4–7.6e-4	500 (363–584) / 2,711 (5.4 $\times$ )	573 (442–716) / 2,353 (4.1 $\times$ )	+0.08 $\pm$ 0.36	1.04–1.11	583–704

Appendix

Table 7: Constant-rate control for the leave-one-out. The released loss and data at the OLMo-2-1B SFT anchor with a constant learning rate of

2.5\times 10^{-6}

, no warmup or decay, at batch 128 (600 steps) and batch 8 (800 steps), three data-order seeds each; columns as in Table 6 .

arm	lr	$\mathrm{rel}\Delta W$	$\rho$ at $5\times 10^{-4}$	$10^{-3}$	$3\times 10^{-3}$	slope	$c_{\text{sign}}/c_{\text{profile}}$	unchanged
SFT	5e-7	3.0e-4–1.3e-3	804	235	–	-1.46	0.17–1.91	0.045
SFT	5e-6	1.9e-3–6.0e-3	–	–	79	-1.19	0.64–0.66	0.045
DPO	5e-7	1.7e-4–5.8e-4	647	–	–	-0.26	1.57–1.79	0.043
DPO	5e-6	1.5e-3–5.5e-3	–	–	227	-0.73	1.27–1.66	0.043

Appendix

Table 8: The 0.5B recipe at 7B. SFT and DPO on Qwen2.5-7B with fp32 master weights, batch 8, constant learning rate, 600 steps, one seed each: displacement range, leverage at three displacements where the run covers them, slope of

\log c

\log\mathrm{rel}\Delta W

, the range of the magnitude alignment over checkpoints with nulls, and the fraction of coordinates unchanged at the last checkpoint (unused vocabulary rows).

delta	threshold (ulp)	unchanged	$\mathrm{rel}\Delta W$	$\rho_{\text{iso}}$	$\rho_{\text{profile}}$	$\rho_{\text{sign}}$	$c_{\text{sign}}/c_{\text{profile}}$
OLMo-2-1B DPO	as stored	0.48	8.52e-4	1,449	1,283	1,015	1.26
	1	0.48	8.51e-4	1,450	1,283	1,015	1.26
	2	0.75	6.04e-4	1,589	1,367	827	1.65
	4	0.88	4.17e-4	1,483	1,277	547	2.33
	8	0.94	2.93e-4	1,192	1,038	327	3.17
Tülu-3-8B DPO	as stored	0.76	8.40e-4	22,570	21,557	9,844	2.19

Appendix

Table 9: Truncation ladder. Four released deltas as stored and with every coordinate below

t

bf16 units in the last place of the anchor value zeroed; each row is a full measurement with 6 isotropic, 3 profile, and 3 sign-flip draws on the mixed corpus; Table 2 uses 8, 2, and 2 draws for the OLMo-2-1B delta, so its values differ from the as-stored row here in the third digit.

width (steps)	$\mathrm{rel}\Delta W$	exact	FD	exact	FD	web	chat	math	code	exact	FD	median	$\rho_{\text{sign}}$	$c_{\text{sign}}/c_{\text{profile}}$
		plain-math		WikiText		mixed, by domain				mixed, pooled
814	0.027	11	5.9	5.8	5.2	15	3.9	750	37	322	4.7	5.2	616	0.54
1,814	0.037	18	3.4	4.3	3.3	4.0	2.7	186	21	82	3.1	3.2	164	0.54
2,814	0.045	3.2	2.9	2.7	2.5	3.6	2.1	118	8.5	52	2.4	2.4	105	0.53
3,814	0.050	4.4	2.5	2.9	2.3	3.6	1.8	140	12	62	2.1	2.1	127	0.53
4,814	0.055	4.4	2.0	2.5	2.0	2.9	1.6	62	11	29	2.0	1.9	61	0.52

Appendix

Table 10: Olmo-3-7B stage-1 windows, exact form against finite differences. Backward windows anchored at stage1-step1413814 ; isotropic leverage by the exact form and by the earlier finite-difference read (FD) on the two small corpora, per domain and pooled on the mixed corpus (mean and median), and the exact sign-flip read and magnitude alignment on the mixed corpus.

			plain-math			WikiText			mixed			mixed, median
width (steps)	rel $\Delta W$	read	iso	prof.	sign	iso	prof.	sign	iso	prof.	sign	iso
814	0.027	$\alpha=0.3$	5.94	6.06	10.5	5.17	5.52	8.50	4.65	4.96	7.92	4.01
1,814	0.037	$\alpha=0.3$	3.41	3.58	6.28	3.26	3.58	5.67	3.09	3.36	5.54	2.58
2,814	0.045	$\alpha=0.3$	2.94	3.14	5.51	2.46	2.73	4.55	2.41	2.65	4.42	2.02
3,814	0.050	$\alpha=0.3$	2.54	2.75	4.99	2.28	2.54	4.25	2.14	2.36	3.99	1.75
4,814	0.055	$\alpha=0.3$	2.04	2.23	3.79	2.04	2.27	3.81	2.00	2.22	3.80	1.58

Appendix

Table 11: Olmo-3-7B backward windows at the last stage-1 checkpoint , every width from 814 to 1,000,814 steps, leverage by finite differences against the three per-delta nulls on the three corpora. “read”: both sides at

\alpha=0.3

(narrow windows) or at the certified common scale.

width (steps)	$\mathrm{rel}\Delta W$	exact	FD	exact	FD	exact	FD	$\rho_{\text{sign}}$	$c_{\text{sign}}/c_{\text{profile}}$	top-1% share
		plain-math		WikiText		mixed
1,000	0.012	4.0	4.5	5.4	6.3	5.0	5.7	14	0.37	0.11
2,000	0.017	2.7	3.2	3.2	3.7	3.0	3.5	8.7	0.35	0.11
4,000	0.023	1.8	2.1	1.9	2.2	1.9	2.2	5.5	0.32	0.11
8,000	0.032	0.94	1.2	1.1	1.4	1.1	1.4	3.2	0.29	0.11
16,000	0.047	0.54	0.79	0.62	0.84	0.58	0.87	1.6	0.27	0.11

Appendix

Table 12: Pythia-1.4B windows at step 143,000, exact form against finite differences (FD: the end-anchored certified-scale read of Table 13 ).

width	end	start	end	start	ratio	end	start	ratio	end	start	ratio	prof.	sign
	rel $\Delta W$		plain-math			WikiText			mixed			end, mixed
1,000	0.012	0.027	4.54	2.94	1.55	6.30	3.51	1.80	5.74	3.16	1.82	5.82	15.3
2,000	0.017	0.035	3.20	1.94	1.65	3.75	2.24	1.68	3.54	2.03	1.75	3.51	9.83
4,000	0.023	0.045	2.10	1.15	1.82	2.19	1.38	1.59	2.23	1.26	1.77	2.11	6.31
8,000	0.032	0.058	1.19	0.76	1.57	1.45	0.84	1.72	1.42	0.75	1.90	1.23	3.89
16,000	0.047	0.073	0.79	0.43	1.84	0.84	0.51	1.65	0.87	0.44	1.97	0.65	2.23

Appendix

Table 13: Pythia-1.4B windows measured at their end (step 143,000) and at their start (step 100,000) , isotropic leverage per corpus with the end/start ratio, and the profile and sign-flip reads of the end-anchored windows on the mixed corpus. The 43k row is one delta at two anchors.

$a$	$\rho$ at $\eta w=1$	inv. participation ratio	$\rho$ at $\eta w=10^{5}$	$d\log\rho/d\log w$	$d\log\\|\Delta\\|/d\log w$
0.60	13.4	15.6	1.00	$-0.29$	0.38
0.67	41.7	50.7	1.00	$-0.41$	0.42
0.75	191	242	1.04	$-0.54$	0.44
0.90	2,580	3,410	2.03	$-0.70$	0.45

Appendix

Table 14: The noisy quadratic model for power-law spectra

\lambda_{k}=k^{-a}

: leverage at one unit of

\eta w

, its

w\to 0

limit (the inverse participation ratio), its value at

\eta w=10^{5}

, and the fitted exponents of

\rho

and

\|\Delta\|

in width.

		plain-math		WikiText
	$\mathrm{rel}\Delta W$ bin	short runs	long runs	short runs	long runs
SFT	$[1e-04,3e-04)$	82.1 (9)	65.3 (6)	34.2 (1)	22.3 (6)
	$[3e-04,1e-03)$	44.1 (14)	48.5 (34)	17.3 (14)	17.0 (34)
	$[1e-03,3e-03)$	12.6 (12)	11.0 (1)	5.70 (12)	4.73 (1)
	$[3e-03,1e-02)$	4.71 (34)	3.97 (39)	2.58 (34)	1.40 (39)
	$[1e-02,3e-02)$	6.11 (22)	–	7.67 (22)	–

Appendix

Table 15: Training length at matched displacement in the earlier bf16 runs : geometric-mean

\rho=c/95

per half-decade bin for the short runs (600 steps, two of them 2,400; 150 for GRPO) and the long runs (10,000 steps; 3,000 and 1,000 for GRPO),

n

checkpoints in parentheses.

family	delta	$\mathrm{rel}\Delta W$	$\alpha^{*}$ wiki	$\alpha^{*}$ math	$\alpha^{*}$	prediction	$a_{0}$	$a_{1}$	$a_{1.5}$	kept	outcome
DeepScaleR	deepscaler_rlvr	8.10e-04	1.00	1.00	1.00	tolerant	0.740	0.693	0.553	–	unscorable
Llama/Tülu-3	tulu_rlvr	7.07e-04	1.00	1.00	1.00	tolerant	0.827	0.820	0.820	–	unscorable
Llama/Tülu-3	tulu_dpo	8.40e-04	0.30	0.30	0.30	fragile	0.740	0.827	0.860	1.38	miss
Mistral/Zephyr	zephyr_dpo	1.40e-03	1.00	1.00	1.00	tolerant	0.373	0.367	0.233	–	unscorable
Olmo-3-7B	olmo_dpo_think	3.81e-04	1.00	1.00	1.00	tolerant	0.833	0.873	0.873	1.00	hit
Olmo-3-7B	olmo_rlzero_gen	6.39e-04	1.00	1.00	1.00	tolerant	0.687	0.713	0.720	1.25	hit

Appendix

Table 16: The first prospective test (

\alpha^{*}

rule, GSM8K, 150 problems, 17 released deltas): quadratic radius per corpus, prediction, accuracy at

0\times

1\times

, and

1.5\times

, the fraction of the endpoint gain kept, and the outcome.

delta	stage	rel $\Delta W$	prediction	acc 0/1/1.5	verdict	acc 0/1/1.5/2	kept 1.5	verdict 1.5	verdict 2.0	pooled	stop 1.0/1.5	$\alpha^{*}$
				GSM8K, 200 problems		MATH-500, 200 problems
olmo2_7b_rlvr	RLVR	$3.27\times 10^{-4}$	tolerant	0.75/0.76/0.74 (new)	unscorable	0.31/0.37/0.36/0.38	–	unscorable	unscorable	unscorable	0.92/0.91	1.00
tulu_rlvr	RLVR	$7.07\times 10^{-4}$	tolerant	0.83/0.82/0.82 (earlier)	unscorable	0.41/0.38/0.41/0.38	–	unscorable	unscorable	unscorable	0.86/0.78	1.00
olmo_rlvr_think	RLVR (think)	$8.06\times 10^{-4}$	tolerant	0.87/0.87/0.85 (earlier)	unscorable	0.62/0.62/0.61/0.62	–	unscorable	unscorable	unscorable	0.58/0.57	1.00
deepscaler_rlvr	RLVR (GRPO)	$8.10\times 10^{-4}$	tolerant	0.74/0.69/0.55 (earlier)	unscorable	0.32/0.29/0.24/0.14	–	unscorable	unscorable	unscorable	0.56/0.49	1.00
olmo_dpo_instruct	DPO	$8.84\times 10^{-4}$	tolerant	0.76/0.71/0.63 (earlier)	unscorable	0.41/0.39/0.45/0.45	–	unscorable	unscorable	unscorable	0.92/0.81	1.00

Appendix

Table 17: The preregistered displacement rule on 21 released deltas. GSM8K columns from the displacement-rule test (new, 200 problems) or the

\alpha^{*}

test (earlier, 150 problems); MATH-500 columns from the second test. “stop” is the fraction of MATH-500 completions that stop at the next few-shot block at

1.0\times

and

1.5\times

; “

\alpha^{*}

” is the post hoc quadratic radius. The GSM8K verdict column applies to the twelve deltas of the second test; earlier rows show the first test’s accuracies and are marked unscorable in that column. “kept” is computed from exact counts before the accuracies are rounded. Deltas are anchor

\rightarrow

endpoint pairs: OLMo-2-7B SFT/DPO/Instruct, Olmo-3-7B pipelines and RL-Zero, Tülu-3 and 3.1, Zephyr, Mistral-7B-sft-beta, Qwen2.5-0.5B/3B/7B-Instruct, SmolLM2-1.7B-Instruct, Llama-3.1-Tülu-3-SFT, R1-Distill-Qwen-1.5B ( DeepSeek-AI, 2025 ) , DeepScaleR ( Luo et al., 2025 ) , ProRL ( Liu et al., 2025 ) , and Qwen2.5-Math-1.5B-Instruct ( Yang et al., 2024 ) .

Transformer Training Dynamics

See all 37 papers

Oct 15, 2026

Rank Measures of Fine-Tuning Updates Need a Noise Baseline

Studies of what fine-tuning does to a language model often summarise the weight update

ΔW

by its rank. We compare the rank measures used for this against the same measures on a random matrix of the same shape, and against two harder references: the pretrained weights, and matched fine-tunes without alignment content. Measures that count directions put our fine-tuning updates only a little below noise; stable rank, computed from the same singular values, puts them far below. Both are right: an update has a few strong directions on top of a long tail. The pattern holds for thirteen released pairs up to 8B parameters, and it is not a mark of alignment: with training held fixed, instruction data does not stand out from Wikipedia or code, DPO gives the same update whether its preference labels are real or half swapped at random (one model, modest preference strength), and scrambled text gives the most concentrated update of all. Applied to a recent claim that post-training is superficial because the effective rank of the weights barely changes, the baselines show that a random update of the same size leaves that rank almost unchanged while wrecking the model, and that an edit built on the effective rank of the update has no measurable effect in our tests. Rank measures of weight updates need a noise baseline, and claims about alignment need a control without alignment content.

Muhammad Usman Altaf

Independent Researcher

Fine-Tuning Language Models Neural Network Training Dynamics LLM Alignment

Ask PDF

May 7, 2026

Optimizer-Model Consistency: Full Finetuning with the Same Optimizer as Pretraining Forgets Less

Optimizers play an important role in both pretraining and finetuning stages when training large language models (LLMs). In this paper, we present an observation that full finetuning with the same optimizer as in pretraining achieves a better learning-forgetting tradeoff, i.e., forgetting less while achieving the same or better performance on the new task, than other optimizers and, possibly surprisingly, LoRA, during the supervised finetuning (SFT) stage. We term this phenomenon optimizer-model consistency. To better understand it, through controlled experiments and theoretical analysis, we show that: 1) optimizers can shape the models by having regularization effects on the activations, leading to different landscapes around the pretrained checkpoints; 2) in response to this regularization effect, the weight update in SFT should follow some specific structures to lower forgetting of the knowledge learned in pretraining, which can be obtained by using the same optimizer. Moreover, we specifically compare Muon and AdamW when they are employed throughout the pretraining and SFT stages and find that Muon performs worse when finetuned for reasoning tasks. With a synthetic language modeling experiment, we demonstrate that this can come from Muon's strong tendency towards rote memorization, which may hurt pattern acquisition with a small amount of data, as for SFT.

Yuxing Liu, Jianyu Wang, Tong Zhang

1UIUC∗ · 2Apple

Supervised Fine-Tuning LLM Fine-Tuning Catastrophic Forgetting

Ask PDF

Aug 25, 2026

Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining

We uncover ELR collapse in language model pretraining: learning rate (LR) and parameter norm govern loss dynamics primarily through their ratio, the effective learning rate (ELR). When ELR is matched across runs, their loss trajectories collapse throughout training despite substantially different LRs and parameter norms. Across optimizers, architectures, datasets, and model scales, mean collapse errors are typically a few x 10^-3, below the seed-to-seed variation measured in a representative configuration. Systematic ablations identify normalization design and the timescale of LR-norm variation as key determinants of collapse precision. Controlled interventions further show that weight decay and Hyperball shape loss dynamics primarily through the ELR schedules they induce. Replacing LR with ELR enables a fitted functional scaling law (FSL) to transfer across norm-control methods. The resulting ELR-based FSL also explains delayed acceleration, a recurring effect of norm control. Together, these results establish ELR as a common coordinate linking LR scheduling, norm control, and loss dynamics.

Zihan Liu, Ruiheng Zheng, Shaobo Zhang +4

Peking University · Ant Group

Neural Network Training Dynamics Language Model Pretraining Learning Rate Scheduling Language Model Scaling Laws

Ask PDF

Abstract

Transformer Training Dynamics

Explore similar work

Rank Measures of Fine-Tuning Updates Need a Noise Baseline

Optimizer-Model Consistency: Full Finetuning with the Same Optimizer as Pretraining Forgets Less

Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining