Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining

Zihan Liu; Ruiheng Zheng; Shaobo Zhang; Changxin Tian; Kunlong Chen; Zhiqiang Zhang; Lei Wu

cs.LGAug 25, 2026

Effective Learning Rate Governs Loss Dynamics in Language Model Pretraining

Authors: Zihan Liu, Ruiheng Zheng, Shaobo Zhang, Changxin Tian, Kunlong Chen, Zhiqiang Zhang, Lei Wu

Organizations: Peking University · Ant Group

Abstract

We uncover ELR collapse in language model pretraining: learning rate (LR) and parameter norm govern loss dynamics primarily through their ratio, the effective learning rate (ELR). When ELR is matched across runs, their loss trajectories collapse throughout training despite substantially different LRs and parameter norms. Across optimizers, architectures, datasets, and model scales, mean collapse errors are typically a few x 10^-3, below the seed-to-seed variation measured in a representative configuration. Systematic ablations identify normalization design and the timescale of LR-norm variation as key determinants of collapse precision. Controlled interventions further show that weight decay and Hyperball shape loss dynamics primarily through the ELR schedules they induce. Replacing LR with ELR enables a fitted functional scaling law (FSL) to transfer across norm-control methods. The resulting ELR-based FSL also explains delayed acceleration, a recurring effect of norm control. Together, these results establish ELR as a common coordinate linking LR scheduling, norm control, and loss dynamics.

Figures & tables

Figure 1: ELR matching collapses loss trajectories despite different LR and norm schedules. In each panel, three nonconstant LR schedules are paired with prescribed norm schedules to match the ELR of a reference run. Legend values report the post-warmup mean absolute loss discrepancy from the reference in units of

10^{-3}

. Loss panels are clipped above 4.2 in (a) and 4.0 in (b) for display. See Section 4 and Appendices B and C for experimental details and additional results.

Figure 2: ELR-matched residuals remain centered near zero without systematic drift. Signed loss residuals

r_{k}=L_{k}-L_{k}^{\mathrm{ref}}

for the linear-up, linear-down, and sinusoidal LR schedules over the post-warmup evaluation window. Both panels use the prescribed-ELR protocol of Figure 1 ; the dashed line marks zero residual.

Figure 3: QK-Norm and learnable RMSNorm gains sharpen ELR collapse. Experiments use Llama-124M trained on FineWeb with AdamW. Starting from the default configuration, the ablations sequentially remove QK-Norm and then fix RMSNorm gains. Left: mean absolute loss residual at each step, averaged over the three ELR-matched schedules. Right: Mean collapse error; bars show the mean across schedules, and error bars show their range. Fixing the RMSNorm gains produces the largest error despite making the parameterization more scale invariant. In the plot labels, RMS denotes RMSNorm.

Figure 4: Rapid LR–norm variation degrades ELR collapse. Experiments use Llama-124M trained on FineWeb with AdamW. Left: The collapse error

\Delta_{\mathrm{coll}}

increases monotonically with the number of modulation cycles. Right: Loss residuals for representative 2- and 32-cycle modulations. The right insets show the corresponding LR schedules together with the WSD reference. In each run, the target-norm schedule is modulated by the same factor as the LR, preserving the prescribed ELR schedule. Faster modulation produces larger oscillatory residuals at the imposed frequency.

Figure 5: Weight decay and Hyperball affect loss dynamics through ELR. Experiments use Llama-124M trained on FineWeb. Dashed gray curves denote the norm-controlled targets, blue curves use the original LR, and red curves adapt the LR to match the target ELR. (a) An AdamW run with weight decay

\lambda=0.1

serves as the target. Removing weight decay changes the loss dynamics under the original LR, whereas ELR matching recovers the target loss with a collapse error of

4.8\times 10^{-3}

. (b) A MuonH run serves as the target. Replacing Hyperball with weight decay changes the loss dynamics under the original LR, whereas ELR matching reduces the collapse error (steps

\geqslant 7.5\mathrm{k}

) to

1.2\times 10^{-3}

Panel of Figure 6 — Figure 6: ELR enables FSL to transfer to unseen Hyperball runs. Experiments use Llama-124M trained on FineWeb with Adam under three norm-control regimes: no weight decay, decoupled weight decay, and Hyperball. (a) Mean RMSE on four fitting runs, four held-out ID runs, and two OOD-Hyperball runs. Both FSL variants are fitted on the same four non-Hyperball curves and evaluated without refitting. The lr-FSL/elr-FSL RMSE ratio increases from $1.8\times$ on the held-out ID runs to $12.9\times$ on the OOD-Hyperball runs. (b) A representative held-out ID using weight-decay coefficient $\lambda=0.1$ . (c) A representative OOD-Hyperball run; no Hyperball data are used for fitting. Without refitting, elr-FSL continues to track the observed loss, whereas lr-FSL breaks down. Table entries are averages across runs; legend values report RMSE for the individual runs shown.

Figure 7: ELR explains and controls delayed acceleration. Experiments use Llama-124M trained on FineWeb with AdamW. Left: Weight decay initially yields higher loss than the unregularized baseline, but overtakes it during the terminal phase. ELR-guided norm control further lowers the final loss. Middle: Corresponding norm evolution. Explicit norm control approximately follows the weight-decay run early, then increases the norm more rapidly after the vertical dashed line. Right: Corresponding ELR trajectories, with the shared LR shown in the inset. The accelerated late-stage norm growth produces a faster ELR decay.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A.1 : Run-to-run loss variation substantially exceeds ELR-collapse error. (a,b) Training-loss trajectories from repeated runs differing only in initialization seed (a) or data-order seed (b). (c) The step-wise mean pairwise loss discrepancy over training. Legend values report the mean and standard deviation of the pairwise time-averaged discrepancies across unordered run pairs, in units of

10^{-2}

. The resulting time-averaged variations are

\Delta_{\mathrm{stoch}}(\mathrm{init})=1.11\times 10^{-2}

and

\Delta_{\mathrm{stoch}}(\mathrm{data\text{-}order})=1.62\times 10^{-2}

. By comparison, the three ELR-matched Llama-124M runs in Figures 1 and 2 , with both seeds fixed, have collapse errors of

1.8\times 10^{-3}

2.5\times 10^{-3}

, and

2.6\times 10^{-3}

Model	size	$d_{\mathrm{model}}$	$d_{\mathrm{FF}}$	Q/KV	depth	peak lr	final lr
Llama-124M	124M	768	3072	6	12	$1.8\times 10^{-3}$	0
Llama-135M	135M	576	1536	9/3	30	$6\times 10^{-4}$	$6\times 10^{-5}$
Llama-1B	1.07B	2560	10240	20	12	$3\times 10^{-4}$	0
Qwen3-145M	145M	768	2304	12/6	12	$1.8\times 10^{-3}$	0
Kimi Delta Attention-127M	127M	768	3072	6	12	$1.8\times 10^{-3}$	0

Appendix

Table B.1 : Dense model configurations and peak learning rates used in the ELR-collapse experiments. Q/KV lists the numbers of query and key/value heads.

Model	size	$d_{\mathrm{model}}$	$d_{\mathrm{FF}}$	Q/KV	depth	$n_{\mathrm{experts}}$	top- $k$	peak lr	final lr
Qwen3-MoE	586M	768	576	12/4	12	32	4	$1.8\times 10^{-3}$	0

Appendix

Table B.2 : Qwen3-MoE configuration and peak learning rate used in the ELR-collapse experiment. Q/KV lists the numbers of query and key/value heads; top-

k

gives the number of active experts per token.

No.	Validation setting	Model	Dataset	$\Delta_{\mathrm{coll}}\;(\times 10^{-3})$			Figure
No.	Validation setting	Model	Dataset	Up	Down	Sine	Figure
Experiments reported in the main text
1	Dense Llama	Llama-124M	FineWeb	$1.8$	$2.5$	$2.6$	Fig. 1
2	MoE	Qwen3-MoE 586M	FineWeb	$3.0$	$4.1$	$3.3$	Fig. 1
Additional experiments reported in the appendix
3	1B scale	Llama-1B	FineWeb	$2.0$	$2.0$	—	Fig. C.1

Appendix

Table C.1 : Collapse errors across prescribed-ELR validation settings. Each row compares runs with different LR and parameter-norm schedules that realize the same prescribed ELR schedule. Entries report the mean collapse error

\Delta_{\mathrm{coll}}

, in units of

10^{-3}

, for linear-up, linear-down, and sinusoidal LR schedules. An em dash denotes a schedule that was not evaluated.

Figure C.1 : ELR collapse in Llama-1B on FineWeb. The compact panels show LR and ELR. The loss panel compares the two varied schedules with the constant reference and reports their mean absolute collapse errors.

Figure C.2 : Additional validation of ELR collapse across training configurations. (a) Replicates the prescribed-ELR experiment with Qwen3-145M on FineWeb. (b) Applies the same protocol to Kimi Delta Attention (KDA). (c,d) Repeat the Qwen3-145M experiment on C4 and OpenWebText, respectively. (e,f) Evaluate Signum and Muon with Llama-124M, respectively. Together, these experiments test ELR collapse across model architectures, pretraining datasets, and optimizers. In each subfigure, the loss-panel legend reports the mean absolute collapse error relative to the constant-schedule reference run.

Figure D.1 : LR-only ELR matching recovers loss dynamics across AdamW weight-decay settings. Experiments use Llama-124M trained on FineWeb with AdamW. In each subfigure, only the LR of the comparison run is adapted to match the target ELR, while its parameter norm evolves freely. The compact panels show the LR and ELR trajectories; the large panel compares the losses and reports the mean collapse error. (a) The run with weight decay

\lambda=0.1

serves as the target, and the LR of the

\lambda=0

run is adapted. (b) The matching direction is reversed: the

\lambda=0

run serves as the target, and the LR of the

\lambda=0.1

run is adapted. The loss trajectories remain closely aligned in both directions.

Figure D.2 : LR-only ELR matching aligns MuonW and MuonH loss dynamics. Experiments use Llama-124M trained on FineWeb. MuonW denotes Muon with weight decay, whereas MuonH uses Hyperball. In each subfigure, only the LR of the comparison run is adapted to match the target ELR, while its norm-control method remains unchanged. The compact panels show the LR and ELR trajectories, and the large panel compares the resulting losses. (a) MuonH serves as the target, and the LR of MuonW is adapted. (b) The matching direction is reversed: MuonW serves as the target, and the LR of MuonH is adapted. For MuonH, we use the optimizer-specific ELR defined in Appendix B .

Figure D.3 : Fixing RMSNorm gains degrades LR-only ELR matching. Dashed black curves show the target losses, red curves show the comparison runs whose LRs are adapted to match the target ELR schedules, and blue curves show the collapse residuals

r_{k}

on the right axis. (a) The run with

\lambda=0.1

serves as the target, and the LR of the

\lambda=0

run is adapted. (b) The matching direction is reversed. In both panels, all RMSNorm gains are fixed and no norm trajectory is prescribed. Compared with the learnable-gain results in Figure D.1 , fixing the gains produces sustained

10^{-2}

-scale residuals in both directions.

Figure E.1 : LR schedule, ELR-denominator evolution, and ELR schedule under distinct norm-control regimes. All curves use WSD-10%. Standard training uses weight decay with

\lambda\in\{0,0.1\}

. For standard runs, the ELR denominator is the parameter Frobenius norm; for Hyperball, it is the average update norm over the post-warmup steps.

Split	Trajectory	lr-FSL $\downarrow$	elr-FSL $\downarrow$	Improvement $\uparrow$
Fit	$\mathrm{wd}=0$ , WSD-30%	0.0251	0.0150	$1.68\times$
	$\mathrm{wd}=0$ , cosine	0.0069	0.0138	$0.50\times$
	$\mathrm{wd}=0.1$ , cosine	0.0180	0.0122	$1.48\times$
	$\mathrm{wd}=0.1$ , WSD-10%	0.0184	0.0112	$1.65\times$
ID	$\mathrm{wd}=0$ , WSD-10%	0.0327	0.0160	$2.05\times$
	$\mathrm{wd}=0$ , 7111	0.0233	0.0143	$1.63\times$

Appendix

Table E.1 : Per-trajectory prediction error. Improvement =

\text{RMSE}_{lr}/\text{RMSE}_{elr}

; values above one favor elr-FSL. Bold marks the lower error.

Figure E.2 : Predictions for all fitted and held-out trajectories. Rows show fit, ID schedule transfer, and OOD Hyperball trajectories. Black denotes ground truth; blue and red denote lr-FSL and elr-FSL. We see that elr-FSL is substantially more accurate on both held-out ID and OOD schedules.

Figure F.1 : A longer WSD decay phase reveals the signal-learning gain acquired earlier. Using Llama-135M trained on FineWeb, we explicitly vary the duration of the terminal WSD decay phase. Each panel compares

\lambda\in\{0,0.1\}

at a fixed WSD decay ratio; black and red denote

\lambda=0

and

0.1

, respectively. With a decay ratio of

0.1

(left), the decay phase is too short for the

\lambda=0.1

run to overtake the unregularized baseline. Increasing the decay ratio to

0.3

(right) extends the low-ELR phase, allowing the accumulated noise to be forgotten and revealing the earlier signal-learning advantage: the

\lambda=0.1

run overtakes

\lambda=0

and attains a lower final loss.

Figure F.2 : Delayed acceleration under Hyperball. On Llama-124M trained on FineWeb, MuonH initially has higher loss than MuonW, but later overtakes it and reaches a lower late-stage loss. Both runs use linear schedules.

Figure G.1 : ELR collapse for a 9.8M-parameter ViT trained on ImageNet-1K. At step 6,250, training is branched into four runs with distinct LR and parameter-norm schedules but a matched ELR schedule. Their smoothed loss trajectories remain collapsed throughout the post-branch phase. The legend reports the collapse error

\Delta_{\mathrm{coll}}

of each run relative to the constant-LR reference.

Figure G.2 : Normalization improves the precision of ELR collapse in ViTs. Left: Mean absolute loss residual across the linear-up, linear-down, and sine-wave ELR-matched schedules relative to the constant-LR reference during training. Right: Corresponding trajectory-averaged collapse error. Removing QK-Norm weakens the collapse, while additionally fixing the remaining gains produces the largest discrepancy.

Neural Network Training Dynamics

Neural network training dynamics studies how parameters and representations change during optimization and how those changes shape learning, stability, and generalization.

See all 507 papers

Deriving Scaling Laws for OpenEuroLLM Models: Learning Rate, Batch Size and Loss
Aug 28, 2026Niccolò Ajroldi, Diana Alexandra Onutu, Haider Al-Tahan +4Language Model Pretraining Learning Rate Scheduling
ELLIS Institute Tübingen · OpenEuroLLM · Eindhoven University of Technology +6
On the Nonlinearity of Learning Rate Scaling for LLM Training
Jun 28, 2026Zaiwen Yang, Huaqing Zhang, Jing Xu +1Language Model Scaling Laws Hyperparameter Transfer
Tsinghua University · Tsinghua University, Shanghai Qi Zhi Institute
Pre-training LLM without Learning Rate Decay Enhances Supervised Fine-Tuning
Mar 17, 2026Kazuki Yano, Shun Kiyono, Sosuke Kobayashi +2Learning Rate Scheduling Language Model Pretraining
Tohoku University · SB Intuitions

Aug 28, 2026J/K move · Enter open · S save

Deriving Scaling Laws for OpenEuroLLM Models: Learning Rate, Batch Size and Loss

Niccolò Ajroldi, Diana Alexandra Onutu, Haider Al-Tahan, Jörg Franke+3

ELLIS Institute Tübingen · OpenEuroLLM · Eindhoven University of Technology+6

Language Model Pretraining Learning Rate Scheduling Language Model Scaling Laws Scaling Laws

Open paper AskPDF arXiv

We study the scaling behavior of learning rate and batch size in pretraining dense large language models on English-prevalent corpora. Beyond scaling jointly optimal learning rates and batch sizes, we investigate their marginal evolution with model capacity and data scale and develop a model that captures these relationships. As we employ a Warmup-Stable-Decay learning rate schedule, we further investigate the gains from learning rate annealing over a broad range of hyperparameters settings, models and data budgets, and whether the optimal learning rate and batch size transfer between the stable and decay phases. Finally, we characterize the dependence of loss on model capacity and dataset size, evaluating recently proposed scaling forms that explicitly model their interaction. We find these approaches particularly effective at capturing both undertraining and overtraining regimes across our experiments. This study establishes a first baseline and scaling procedure for the development of future OpenEuroLLM models. We open-source the complete collection of pretraining runs used in this study.

Abstract

Neural Network Training Dynamics

Explore similar work

Deriving Scaling Laws for OpenEuroLLM Models: Learning Rate, Batch Size and Loss

On the Nonlinearity of Learning Rate Scaling for LLM Training

Pre-training LLM without Learning Rate Decay Enhances Supervised Fine-Tuning