A Refresh Interval Does Not Fully Specify a Refresh Schedule: Learning-Rate-Dependent Placement in Kronecker-Preconditioned Language Model Training

Xinyu Wang; Zhiwei Zhai; Yu Gu; Zijun Yu; Sicheng Lyu; Jingrui Tian; Lei Ding; Ziyang Song

cs.LGOct 13, 2026

A Refresh Interval Does Not Fully Specify a Refresh Schedule: Learning-Rate-Dependent Placement in Kronecker-Preconditioned Language Model Training

Authors: Xinyu Wang, Zhiwei Zhai, Yu Gu, Zijun Yu, Sicheng Lyu, Jingrui Tian, Lei Ding, Ziyang Song,

Organizations: McGill University · The Chinese University of Hong Kong · University of Manitoba · Ohio University

Abstract

Kronecker-preconditioned optimizers recompute a cached curvature eigendecomposition only periodically, and implementations and adaptive methods control this through the refresh frequency. We show that frequency leaves part of a schedule free: at a fixed number of refreshes, when they occur changes held-out loss at the same compute. Training GPT-style models with the Mousse optimizer across model sizes, we move the middle refreshes of a four-refresh schedule while fixing the first and last. On FineWeb the two placements exchanged rank as the learning rate rose (later placement was better at low rates, earlier placement at high ones), so a single-learning-rate comparison can report either ranking. The reversal held under controls including the published optimizer internals, and placement still mattered at the published refresh density. The learning rate at which the preference flips fell with model size, consistent with a held-out smaller model, and shifted with training length. A spectral-width staleness score predicted the same ranking at every learning rate and missed the reversal. Refresh schedules should be reported in full and compared across the learning rates at which they will be used.

Figures & tables

Figure 1: (a) The two schedules. Both perform four refreshes and share their first and last refresh step, and only the two middle refreshes differ. (b) Seed-mean difference

\tau

in endpoint held-out negative log-likelihood (NLL), later schedule minus earlier, against the matrix learning rate (FineWeb, 350M). Positive values mean earlier placement is better.

\lambda_{\times}

marks the interpolated sign change, and the grey band is the tie tolerance.

Figure 2: The placement contrast

\tau(\mathrm{corpus},\eta)

at GPT-350M (positive

=

early better). FineWeb is drawn heavier, the grey band is the

\pm 0.002

tie tolerance, and the marker is the interpolated crossing. Crossing corpora are solid, non-crossing dashed.

corpus	tuned LR	$\|\tau\|$ tuned	surface max
FineWeb	$.012$	$.009$	$.033$ (at $.004$ )
Pile-CC	$.012$	$.011$	$.031$ (at $.024$ )
GitHub	$.012$	$.084$	$.084$ (at $.012$ )
PubMed	$.012$	$.068$	$.082$ (at $.024$ )
arXiv	$.016$	$.039$	$.095$ (at $.024$ )
Books3	$.012$	$.043$	$.063$ (at $.016$ )

Table 1: Placement effect at each corpus’s tuned learning rate beside the largest effect on the surface (350M, 591 steps).

Figure 3: Endpoint NLL relative to the earliest centre

c{=}251

for five placements of the middle pair at the same budget and endpoints (FineWeb, 350M). Faint points are individual seeds, and the grey band is the tie tolerance.

condition	reversal	$\lambda_{\times}$
primary pair (reference)	✓	$.0166$
schedule construction
shared refresh added at warmup end	✓	$.0164$
budget 8, first and largest gaps kept	✓	$.0166$
budget 16, first and largest gaps kept	✓	$.0171$
budget 59, every gap at most 15 steps	— a	—

Table 2: Controls on the FineWeb reversal (350M, primary pair unless stated). A check mark means a seed-consistent sign change, a cross means none, and

\lambda_{\times}

is the interpolated crossing. Details are in App. C .

Figure 4: (a) The budget-59 schedules as tick marks (grey bands are WSD warmup and warmdown). (b)

\tau

against the matrix learning rate on FineWeb at 350M for the budget-59 pair (median and individual seeds) and the primary budget-4 pair (seed mean). Filled seed points mark seed-consistent cells, open points the others, and the grey band is the tie tolerance. Per-seed values are in Table 5 .

Figure 5: (a)

\lambda_{\times}

against parameter count, one experiment per scale. The dashed line is the fit to the four larger scales, and the square is the held-out 64M measurement. Filled markers are the main estimates, and open markers are independent fresh-seed cohorts (App. D ). (b) The held-out test: the preselected tolerance band, the zero-slope benchmark and the 64M measurement. (c)

\lambda_{\times}

against the uniform-schedule optimal learning rate

\lambda_{\mathrm{ref\text{-}opt}}

Figure 6: (a) Normalised spectral-width profiles

w(t)

of the 20-refresh reference runs (FineWeb, five learning rates). The dashed line marks the first step at which the two schedules differ. (b) Top: decomposition of the score contrast

\Delta I_{w}

into

D_{1}

and

R_{w}

(Eq. 5 ), in width

\times

steps. Bottom: the measured

\tau

of the same cells, in NLL, with the tie band. (c)

\Delta I_{w}

(Eq. 4 ) against measured

\tau

over all 30 cells, with per-corpus fits.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Optimizer (matrix parameters)
momentum $\mu$	$0.95$ (heavy ball, Nesterov off)
curvature EMA $\beta$	$0.95$ , bias-corrected at refresh
curvature exponent $\alpha$	0.125, before and after Newton–Schulz
eigenvalue cleanup $\varepsilon$	$10^{-10}$ (nonnegative correction after trace normalization)
orthogonalization	Newton–Schulz, 5 iterations
norm grafting	update rescaled to $\\|O\\|_{F}$

Appendix

Table 3: Hyperparameters used in all experiments unless stated otherwise. Matrix learning-rate grids are given with each experiment.

Figure 7: Per-cell

\tau

from the two seeds of the 350M surface, one point per (corpus, LR) cell. The line is

y=x

corpus	.004	.008	.012	.016	.024
FineWeb	$-.0332$	$-.0177$	$-.0091$	$-.0013$	$+.0189$
Pile-CC	$+.0016$	$-.0043$	$+.0105$	$+.0164$	$+.0306$
GitHub	$+.0253$	$+.0635$	$+.0844$	$+.0825$	$+.0793$
PubMed	$+.0219$	$+.0321$	$+.0675$	$+.0676$	$+.0821$
arXiv	$+.0236$	$+.0477$	$+.0452$	$+.0393$	$+.0947$
Books3	$+.0182$	$+.0505$	$+.0432$	$+.0630$	$+.0023$

Appendix

Table 4: Seed-mean

\tau

at 350M and 591 steps (positive

=

early better).

LR	seed 1	seed 2	seed 3	median	consistent
$.008$	$-.0080$	$-.0067$	$-.0077$	$-.0077$	yes
$.012$	$-.0048$	$-.0054$	$-.0042$	$-.0048$	yes
$.016$	$-.0065$	$-.0153$	$-.0037$	$-.0065$	yes
$.020$	$-.0007$	$-.0079$	$+.0018$	$-.0007$	no
$.024$	$+.0069$	$+.0144$	$-.0028$	$+.0069$	no

Appendix

Table 5:

\tau

at budget 59 (FineWeb, 350M, three seeds). A cell is consistent when all seeds receive the same label (early better, late better, or tie).

Figure 8: The schedules as tick marks (grey bands are WSD warmup and warmdown). Top: the primary mirrored pair. Bottom: the budget-59 pair, whose interval lengths run in opposite directions while every gap stays between 5 and 15 steps. The labels early_mid and late_mid denote

S_{E}

and

S_{L}

, and early_b59 and late_b59 the budget-59 pair.

cond.	LR	AdamW	seed 1	seed 2	seed 3	median
fixed	$.012$	$3.0$ e $-4$	$-.0060$	$-.0049$	$-.0046$	$-.0049$
fixed	$.016$	$3.0$ e $-4$	$+.0005$	$-.0007$	$+.0033$	$+.0005$
fixed	$.024$	$3.0$ e $-4$	$+.0157$	$+.0141$	$+.0163$	$+.0157$
prop.	$.012$	$2.25$ e $-4$	$-.0056$	$-.0045$	$-.0044$	$-.0045$
prop.	$.016$	$3.0$ e $-4$	$+.0005$	$-.0007$	$+.0033$	$+.0005$
prop.	$.024$	$4.5$ e $-4$	$+.0204$	$+.0197$	$+.0181$	$+.0197$

Appendix

Table 6:

\tau

with the AdamW rate fixed at

3\times 10^{-4}

versus scaled with the matrix rate (FineWeb, 350M, primary pair, three seeds). The two conditions coincide at

.016

by construction.

scale	params	$\lambda_{\times}$	resampling range	seeds
64M	64.1M	$.029393$	$[.029072,.029780]$	6
124M	124.5M	$.024049$	$[.023101,.024160]$	2
350M	354.9M	$.016560$	—	3
350M †		$.016433$	$[.016107,.016713]$	2
695M	695.4M	$.014327$	$[.013943,.014757]$	2
929M	928.8M	$.013196$	$[.013053,.013357]$	2

Appendix

Table 7: Per-scale crossings. Ranges resample seeds within each cohort. † A second 350M cohort. ‡ Independent cohorts with four fresh seeds on local three-point grids, kept separate from the main estimates.

prediction	outcome
Single-interval drift tracks $\tau$ across the surface	runs opposite within the crossing corpora (FineWeb $-0.80$ , Pile-CC $-0.60$ )
Refresh content (data freshness) acts as a gain	refuted by an erasure test
$\tau$ reduces to $A\,x\ln x$ in factor age	fails on held-out cells
$\tau$ is a generalization effect	refuted: training and evaluation contrasts agree (corr $0.88$ )

Appendix

Table 8: Earlier attempts to predict the boundary from curvature staleness.

system	what is refreshed	exposed timing control
K-FAC ( Martens and Grosse, 2015 )	factored curvature inverses	periodic recomputation interval
Shampoo ( Gupta et al., 2018 ; Anil et al., 2020 )	preconditioner roots	preconditioner update interval
SOAP ( Vyas et al., 2025 )	eigenbasis of the preconditioner	preconditioning frequency
EKFAC ( George et al., 2018 )	diagonal scalings inside a fixed eigenbasis	rebuild interval for the eigenbasis
adaptive-frequency methods ( Eschenhagen et al., 2025 ; Nam and Ahn, 2026 )	eigenbasis or roots, on a staleness trigger	adaptive update frequency (budget varies)
Muon ( Liu et al., 2025 )	(no curvature factorization)	—

Appendix

Table 9: Refresh control across Kronecker-preconditioned systems.

Deep Learning Optimization

Deep learning optimization develops algorithms and training strategies that help neural networks converge, generalize, and use compute efficiently.

See all 206 papers

Optimizer-Model Consistency: Full Finetuning with the Same Optimizer as Pretraining Forgets Less
May 7, 2026Yuxing Liu, Jianyu Wang, Tong ZhangSupervised Fine-Tuning LLM Fine-Tuning
1UIUC∗ · 2Apple
ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Models
May 18, 2026Aaron DefazioLanguage Model Pretraining Learning Rate Scheduling
FAIR at Meta Super-Intelligence Labs
MuPACE: Learning to Pace Muon for LoRA Fine-Tuning
Oct 20, 2026Jan Kobiolka, Christian Frey, Josif GrabockaLearning Rate Scheduling Adaptive Learning Rate
Department of Computer Science, University of Technology Nuremberg, Nuremberg, Germany

May 7, 2026J/K move · Enter open · S save

Optimizer-Model Consistency: Full Finetuning with the Same Optimizer as Pretraining Forgets Less

Yuxing Liu, Jianyu Wang, Tong Zhang

1UIUC∗ · 2Apple

Supervised Fine-Tuning LLM Fine-Tuning Catastrophic Forgetting

Open paper AskPDF arXiv

Optimizers play an important role in both pretraining and finetuning stages when training large language models (LLMs). In this paper, we present an observation that full finetuning with the same optimizer as in pretraining achieves a better learning-forgetting tradeoff, i.e., forgetting less while achieving the same or better performance on the new task, than other optimizers and, possibly surprisingly, LoRA, during the supervised finetuning (SFT) stage. We term this phenomenon optimizer-model consistency. To better understand it, through controlled experiments and theoretical analysis, we show that: 1) optimizers can shape the models by having regularization effects on the activations, leading to different landscapes around the pretrained checkpoints; 2) in response to this regularization effect, the weight update in SFT should follow some specific structures to lower forgetting of the knowledge learned in pretraining, which can be obtained by using the same optimizer. Moreover, we specifically compare Muon and AdamW when they are employed throughout the pretraining and SFT stages and find that Muon performs worse when finetuned for reasoning tasks. With a synthetic language modeling experiment, we demonstrate that this can come from Muon's strong tendency towards rote memorization, which may hurt pattern acquisition with a small amount of data, as for SFT.

Abstract

Deep Learning Optimization

Explore similar work

Optimizer-Model Consistency: Full Finetuning with the Same Optimizer as Pretraining Forgets Less

ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Models

MuPACE: Learning to Pace Muon for LoRA Fine-Tuning