We uncover ELR collapse in language model pretraining: learning rate (LR) and parameter norm govern loss dynamics primarily through their ratio, the effective learning rate (ELR). When ELR is matched across runs, their loss trajectories collapse throughout training despite substantially different LRs and parameter norms. Across optimizers, architectures, datasets, and model scales, mean collapse errors are typically a few x 10^-3, below the seed-to-seed variation measured in a representative configuration. Systematic ablations identify normalization design and the timescale of LR-norm variation as key determinants of collapse precision. Controlled interventions further show that weight decay and Hyperball shape loss dynamics primarily through the ELR schedules they induce. Replacing LR with ELR enables a fitted functional scaling law (FSL) to transfer across norm-control methods. The resulting ELR-based FSL also explains delayed acceleration, a recurring effect of norm control. Together, these results establish ELR as a common coordinate linking LR scheduling, norm control, and loss dynamics.
Figures & tables
Figure 1: ELR matching collapses loss trajectories despite different LR and norm schedules. In each panel, three nonconstant LR schedules are paired with prescribed norm schedules to match the ELR of a reference run. Legend values report the post-warmup mean absolute loss discrepancy from the reference in units of 10−3 . Loss panels are clipped above 4.2 in (a) and 4.0 in (b) for display. See Section 4 and Appendices B and C for experimental details and additional results.
Figure 2: ELR-matched residuals remain centered near zero without systematic drift. Signed loss residuals rk=Lk−Lkref for the linear-up, linear-down, and sinusoidal LR schedules over the post-warmup evaluation window. Both panels use the prescribed-ELR protocol of Figure 1 ; the dashed line marks zero residual.
Figure 3: QK-Norm and learnable RMSNorm gains sharpen ELR collapse. Experiments use Llama-124M trained on FineWeb with AdamW. Starting from the default configuration, the ablations sequentially remove QK-Norm and then fix RMSNorm gains. Left: mean absolute loss residual at each step, averaged over the three ELR-matched schedules. Right: Mean collapse error; bars show the mean across schedules, and error bars show their range. Fixing the RMSNorm gains produces the largest error despite making the parameterization more scale invariant. In the plot labels, RMS denotes RMSNorm.
Figure 4: Rapid LR–norm variation degrades ELR collapse. Experiments use Llama-124M trained on FineWeb with AdamW. Left: The collapse error Δcoll increases monotonically with the number of modulation cycles. Right: Loss residuals for representative 2- and 32-cycle modulations. The right insets show the corresponding LR schedules together with the WSD reference. In each run, the target-norm schedule is modulated by the same factor as the LR, preserving the prescribed ELR schedule. Faster modulation produces larger oscillatory residuals at the imposed frequency.
Figure 5: Weight decay and Hyperball affect loss dynamics through ELR. Experiments use Llama-124M trained on FineWeb. Dashed gray curves denote the norm-controlled targets, blue curves use the original LR, and red curves adapt the LR to match the target ELR. (a) An AdamW run with weight decay λ=0.1 serves as the target. Removing weight decay changes the loss dynamics under the original LR, whereas ELR matching recovers the target loss with a collapse error of 4.8×10−3 . (b) A MuonH run serves as the target. Replacing Hyperball with weight decay changes the loss dynamics under the original LR, whereas ELR matching reduces the collapse error (steps ⩾7.5k ) to 1.2×10−3 .
Figure 6: ELR enables FSL to transfer to unseen Hyperball runs. Experiments use Llama-124M trained on FineWeb with Adam under three norm-control regimes: no weight decay, decoupled weight decay, and Hyperball. (a) Mean RMSE on four fitting runs, four held-out ID runs, and two OOD-Hyperball runs. Both FSL variants are fitted on the same four non-Hyperball curves and evaluated without refitting. The lr-FSL/elr-FSL RMSE ratio increases from 1.8× on the held-out ID runs to 12.9× on the OOD-Hyperball runs. (b) A representative held-out ID using weight-decay coefficient λ=0.1 . (c) A representative OOD-Hyperball run; no Hyperball data are used for fitting. Without refitting, elr-FSL continues to track the observed loss, whereas lr-FSL breaks down. Table entries are averages across runs; legend values report RMSE for the individual runs shown.
Figure 7: ELR explains and controls delayed acceleration. Experiments use Llama-124M trained on FineWeb with AdamW. Left: Weight decay initially yields higher loss than the unregularized baseline, but overtakes it during the terminal phase. ELR-guided norm control further lowers the final loss. Middle: Corresponding norm evolution. Explicit norm control approximately follows the weight-decay run early, then increases the norm more rapidly after the vertical dashed line. Right: Corresponding ELR trajectories, with the shared LR shown in the inset. The accelerated late-stage norm growth produces a faster ELR decay.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A.1 : Run-to-run loss variation substantially exceeds ELR-collapse error. (a,b) Training-loss trajectories from repeated runs differing only in initialization seed (a) or data-order seed (b). (c) The step-wise mean pairwise loss discrepancy over training. Legend values report the mean and standard deviation of the pairwise time-averaged discrepancies across unordered run pairs, in units of 10−2 . The resulting time-averaged variations are Δstoch(init)=1.11×10−2 and Δstoch(data-order)=1.62×10−2 . By comparison, the three ELR-matched Llama-124M runs in Figures 1 and 2 , with both seeds fixed, have collapse errors of 1.8×10−3 , 2.5×10−3 , and 2.6×10−3 .
Model
size
dmodel
dFF
Q/KV
depth
peak lr
final lr
Llama-124M
124M
768
3072
6
12
1.8×10−3
0
Llama-135M
135M
576
1536
9/3
30
6×10−4
6×10−5
Llama-1B
1.07B
2560
10240
20
12
3×10−4
0
Qwen3-145M
145M
768
2304
12/6
12
1.8×10−3
0
Kimi Delta Attention-127M
127M
768
3072
6
12
1.8×10−3
0
Appendix
Table B.1 : Dense model configurations and peak learning rates used in the ELR-collapse experiments. Q/KV lists the numbers of query and key/value heads.
Model
size
dmodel
dFF
Q/KV
depth
nexperts
top- k
peak lr
final lr
Qwen3-MoE
586M
768
576
12/4
12
32
4
1.8×10−3
0
Appendix
Table B.2 : Qwen3-MoE configuration and peak learning rate used in the ELR-collapse experiment. Q/KV lists the numbers of query and key/value heads; top- k gives the number of active experts per token.
No.
Validation setting
Model
Dataset
Δcoll(×10−3)
Figure
Up
Down
Sine
Experiments reported in the main text
1
Dense Llama
Llama-124M
FineWeb
1.8
2.5
2.6
Fig. 1
2
MoE
Qwen3-MoE 586M
FineWeb
3.0
4.1
3.3
Fig. 1
Additional experiments reported in the appendix
3
1B scale
Llama-1B
FineWeb
2.0
2.0
—
Fig. C.1
Appendix
Table C.1 : Collapse errors across prescribed-ELR validation settings. Each row compares runs with different LR and parameter-norm schedules that realize the same prescribed ELR schedule. Entries report the mean collapse error Δcoll , in units of 10−3 , for linear-up, linear-down, and sinusoidal LR schedules. An em dash denotes a schedule that was not evaluated.
Figure C.1 : ELR collapse in Llama-1B on FineWeb. The compact panels show LR and ELR. The loss panel compares the two varied schedules with the constant reference and reports their mean absolute collapse errors.
Figure C.2 : Additional validation of ELR collapse across training configurations. (a) Replicates the prescribed-ELR experiment with Qwen3-145M on FineWeb. (b) Applies the same protocol to Kimi Delta Attention (KDA). (c,d) Repeat the Qwen3-145M experiment on C4 and OpenWebText, respectively. (e,f) Evaluate Signum and Muon with Llama-124M, respectively. Together, these experiments test ELR collapse across model architectures, pretraining datasets, and optimizers. In each subfigure, the loss-panel legend reports the mean absolute collapse error relative to the constant-schedule reference run.
Figure D.1 : LR-only ELR matching recovers loss dynamics across AdamW weight-decay settings. Experiments use Llama-124M trained on FineWeb with AdamW. In each subfigure, only the LR of the comparison run is adapted to match the target ELR, while its parameter norm evolves freely. The compact panels show the LR and ELR trajectories; the large panel compares the losses and reports the mean collapse error. (a) The run with weight decay λ=0.1 serves as the target, and the LR of the λ=0 run is adapted. (b) The matching direction is reversed: the λ=0 run serves as the target, and the LR of the λ=0.1 run is adapted. The loss trajectories remain closely aligned in both directions.
Figure D.2 : LR-only ELR matching aligns MuonW and MuonH loss dynamics. Experiments use Llama-124M trained on FineWeb. MuonW denotes Muon with weight decay, whereas MuonH uses Hyperball. In each subfigure, only the LR of the comparison run is adapted to match the target ELR, while its norm-control method remains unchanged. The compact panels show the LR and ELR trajectories, and the large panel compares the resulting losses. (a) MuonH serves as the target, and the LR of MuonW is adapted. (b) The matching direction is reversed: MuonW serves as the target, and the LR of MuonH is adapted. For MuonH, we use the optimizer-specific ELR defined in Appendix B .
Figure D.3 : Fixing RMSNorm gains degrades LR-only ELR matching. Dashed black curves show the target losses, red curves show the comparison runs whose LRs are adapted to match the target ELR schedules, and blue curves show the collapse residuals rk on the right axis. (a) The run with λ=0.1 serves as the target, and the LR of the λ=0 run is adapted. (b) The matching direction is reversed. In both panels, all RMSNorm gains are fixed and no norm trajectory is prescribed. Compared with the learnable-gain results in Figure D.1 , fixing the gains produces sustained 10−2 -scale residuals in both directions.
Figure E.1 : LR schedule, ELR-denominator evolution, and ELR schedule under distinct norm-control regimes. All curves use WSD-10%. Standard training uses weight decay with λ∈{0,0.1} . For standard runs, the ELR denominator is the parameter Frobenius norm; for Hyperball, it is the average update norm over the post-warmup steps.
Split
Trajectory
lr-FSL ↓
elr-FSL ↓
Improvement ↑
Fit
wd=0 , WSD-30%
0.0251
0.0150
1.68×
wd=0 , cosine
0.0069
0.0138
0.50×
wd=0.1 , cosine
0.0180
0.0122
1.48×
wd=0.1 , WSD-10%
0.0184
0.0112
1.65×
ID
wd=0 , WSD-10%
0.0327
0.0160
2.05×
wd=0 , 7111
0.0233
0.0143
1.63×
Appendix
Table E.1 : Per-trajectory prediction error. Improvement = RMSElr/RMSEelr ; values above one favor elr-FSL. Bold marks the lower error.
Figure E.2 : Predictions for all fitted and held-out trajectories. Rows show fit, ID schedule transfer, and OOD Hyperball trajectories. Black denotes ground truth; blue and red denote lr-FSL and elr-FSL. We see that elr-FSL is substantially more accurate on both held-out ID and OOD schedules.
Figure F.1 : A longer WSD decay phase reveals the signal-learning gain acquired earlier. Using Llama-135M trained on FineWeb, we explicitly vary the duration of the terminal WSD decay phase. Each panel compares λ∈{0,0.1} at a fixed WSD decay ratio; black and red denote λ=0 and 0.1 , respectively. With a decay ratio of 0.1 (left), the decay phase is too short for the λ=0.1 run to overtake the unregularized baseline. Increasing the decay ratio to 0.3 (right) extends the low-ELR phase, allowing the accumulated noise to be forgotten and revealing the earlier signal-learning advantage: the λ=0.1 run overtakes λ=0 and attains a lower final loss.
Figure F.2 : Delayed acceleration under Hyperball. On Llama-124M trained on FineWeb, MuonH initially has higher loss than MuonW, but later overtakes it and reaches a lower late-stage loss. Both runs use linear schedules.
Figure G.1 : ELR collapse for a 9.8M-parameter ViT trained on ImageNet-1K. At step 6,250, training is branched into four runs with distinct LR and parameter-norm schedules but a matched ELR schedule. Their smoothed loss trajectories remain collapsed throughout the post-branch phase. The legend reports the collapse error Δcoll of each run relative to the constant-LR reference.
Figure G.2 : Normalization improves the precision of ELR collapse in ViTs. Left: Mean absolute loss residual across the linear-up, linear-down, and sine-wave ELR-matched schedules relative to the constant-LR reference during training. Right: Corresponding trajectory-averaged collapse error. Removing QK-Norm weakens the collapse, while additionally fixing the remaining gains produces the largest discrepancy.
Neural network training dynamics studies how parameters and representations change during optimization and how those changes shape learning, stability, and generalization.
We study the scaling behavior of learning rate and batch size in pretraining dense large language models on English-prevalent corpora. Beyond scaling jointly optimal learning rates and batch sizes, we investigate their marginal evolution with model capacity and data scale and develop a model that captures these relationships. As we employ a Warmup-Stable-Decay learning rate schedule, we further investigate the gains from learning rate annealing over a broad range of hyperparameters settings, models and data budgets, and whether the optimal learning rate and batch size transfer between the stable and decay phases. Finally, we characterize the dependence of loss on model capacity and dataset size, evaluating recently proposed scaling forms that explicitly model their interaction. We find these approaches particularly effective at capturing both undertraining and overtraining regimes across our experiments. This study establishes a first baseline and scaling procedure for the development of future OpenEuroLLM models. We open-source the complete collection of pretraining runs used in this study.
Niccolò Ajroldi, Diana Alexandra Onutu, Haider Al-Tahan +4
ELLIS Institute Tübingen · OpenEuroLLM · Eindhoven University of Technology +6
Learning-rate transfer can reduce the cost of training large language models: instead of sweeping learning rates at target scale, practitioners extrapolate from smaller runs. Existing approaches often assume that the optimal learning rate follows a log-linear scaling law in data scale and model size. We carefully examine and evaluate this scaling law. In our empirical study of GPT-2--style models from 22M to 707M parameters trained on 5B to 100B tokens, the optimal learning rate develops upward curvature at larger scales, leading to inaccurate extrapolation. We find that this curvature largely disappears when learning rates are replaced by effective learning rate (the step size in normalized weight space), and when data D extrapolation is used instead of model size N extrapolation. Next, we explain nonlinearity in scaling: weight-norm converges to equilibrium slower when optimal learning is small, requiring a larger step size to reduce the transient phase. Experiments with AdamH, which directly controls the effective learning rate, further support this explanation.
Zaiwen Yang, Huaqing Zhang, Jing Xu +1
Tsinghua University · Tsinghua University, Shanghai Qi Zhi Institute
We investigate the role of learning rate scheduling in the large-scale pre-training of large language models, focusing on its influence on downstream performance after supervised fine-tuning (SFT). Decay-based learning rate schedulers are widely used to minimize pre-training loss. However, despite their widespread use, how these schedulers affect performance after SFT remains underexplored. In this paper, we examine Warmup-Stable-Only (WSO), which maintains a constant learning rate after warmup without any decay. Through experiments with 1B and 8B parameter models, we show that WSO consistently outperforms decay-based schedulers in terms of performance after SFT, even though decay-based schedulers may exhibit better performance after pre-training. The result also holds across different regimes with mid-training and over-training. Loss landscape analysis further reveals that decay-based schedulers lead models into sharper minima, whereas WSO preserves flatter minima that support adaptability. These findings indicate that applying LR decay to improve pre-training metrics may compromise downstream adaptability. Our work also provides practical guidance for training and model release strategies, highlighting that pre-training models with WSO enhances their adaptability for downstream tasks.