Delayed Effects of Temporary Weight Decay in Muon Pretraining
Organizations: Meta · Mila-Quebec AI Institute · Université de Montréal · Samsung SAIL Montreal
Abstract
A temporary optimization intervention can increase validation loss when it ends yet reduce it at the final training budget. We demonstrate this with temporary weight decay in Muon-trained decoder-only Transformers with 162M, 405M, 1.4B, and 3.4B parameters. For models up to 1.4B parameters, moving a fixed-duration decay window from early to late training reverses its final effect. At 162M, changing window duration alters final loss even at approximately matched total shrinkage. Most strikingly, a schedule selected at 162M and transferred without retuning to 1.4B is 0.26 nats worse than no decay when removed, overtakes it only after 73% of training, and ultimately finishes 0.028 nats better while also outperforming the best sampled constant-decay baseline. Early decay leaves a persistent weight-norm deficit, which keeps relative steps larger after the intervention ends. An approximate norm-dynamics recurrence captures the slow convergence of these relative steps, and fixed-norm replays of the measured trajectories reproduce the performance ordering and a similar loss difference. At 162M and 405M, norm-reset experiments show that smaller weight norms at the end of the decay window explain most of the final loss improvement. A separate Gram intervention exhibits related post-removal norm and relative-step dynamics. These results show that temporary optimization schedules can have delayed effects long after removal and should therefore be evaluated at the intended final training budget, not only at removal.
Figures & tables
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
| Purpose | Data | Scale | Reference |
|---|---|---|---|
| Replicated comparison | FineWeb-Edu | 162M | No decay; best constant |
| Timing and scale | FineWeb100B | 162M–1.4B | No decay; best constant |
| Norm and replay controls | FineWeb-Edu | 162M | No decay |
| Norm reset | FineWeb-Edu; FineWeb100B | 162M; 405M | No decay; early decay |
| Gradual decay; rate tuning | FineWeb100B | 162M; 405M | No decay; best constant |
| Larger-scale comparison | FineWeb100B | 3.4B | No decay; best constant |
| Parameters | Width | Layers | Heads | Updates |
|---|---|---|---|---|
| 162M | 768 | 12 | 6 | 3,200 |
| 405M | 1,024 | 24 | 32 | 8,100 |
| 1.4B | 2,048 | 24 | 32 | 28,000 |
| 3.4B | 3,072 | 27 | 48 | 66,000 |
| Data | Scale | Reference | Gram | Difference | |
|---|---|---|---|---|---|
| FineWeb-Edu | 162M | 3.1829 | 3.1699 | ||
| FineWeb-Edu | 405M | 2.8551 | 2.8421 | ||
| FineWeb100B | 162M | 3.3820 | 3.3673 | ||
| FineWeb100B | 405M | 3.0480 | 3.0332 | ||
| FineWeb100B | 1.4B | 2.6796 | 2.6813 | ||
| FineWeb100B | 1.4B | 2.6796 | 2.6742 |
| Intervention | Final loss | Difference from none |
|---|---|---|
| No regularization | 0 | |
| Constant decay ( ) | ||
| Gram penalty | ||
| Early weight decay ( ) | ||
| Both |
| Scale | 5% | 10% | 20% | 50% | 75% | 100% |
|---|---|---|---|---|---|---|
| 162M | – | |||||
| 405M |
| Schedule | 162M | 405M | 1.4B |
|---|---|---|---|
| No decay | 3.3860 | 3.0537 | 2.6754 |
| Constant | 3.3852 | 3.0529 | 2.6744 |
| Constant | – | – | 2.6743 |
| Constant | 3.3836 | 3.0502 | – |
| Constant | 3.3820 | 3.0480 | 2.6802 |
| Constant | 3.3796 | 3.0462 | – |
| Optimizer | MLP | Scale | No decay | Early | Difference |
|---|---|---|---|---|---|
| Muon | 162M | 3.1864 | 0.3 | ||
| Muon | 405M | 2.8601 | 0.3 | ||
| Muon | 1.4B | 2.8811 | 0.2 | ||
| Muon | SwiGLU | 162M | 3.2019 | 0.3 | |
| Muon | SwiGLU | 405M | 2.8790 | 0.3 | |
| AdamW | SwiGLU | 162M | 3.2401 | 3.0 |
| Domain | |||
|---|---|---|---|
| Web | 5.982 | ||
| Mathematics | 5.513 | ||
| Multilingual | 4.245 | ||
| Code | 3.815 |
| Contribution | Final loss | |
| Before Muon | Full: | 3.1718 |
| Norm only: | 3.1887–3.1981 | |
| Direction only: | 3.1868 | |
| Full plus early weight decay ( ) | 3.1663 | |
| Direction plus early weight decay ( ) | 3.1710 | |
| After Muon | Full penalty gradient | 3.1750 |
| Scale / updates | Const. | Early ( , window) | vs. none | vs. const. |
|---|---|---|---|---|
| 162M / 3,200 | 3.1944 (0.05) | 3.1940 (0.6, 320) | ||
| 405M / 8,100 | 2.8724 (0.02) | 2.8648 (0.3, 810) | ||
| 1.4B / 6,000 | 2.8965 (0.01) | 2.8847 (0.4, 600) | ||
| 162M / 3,200 | 3.1944 (0.05) | 3.1730 (0.3, 960) | ||
| 405M / 8,100 | 2.8724 (0.02) | 2.8473 (0.3, 1,620) |
| Scale | Schedule | HS | ARC-E | PIQA | LAMBADA | WG | Mean |
|---|---|---|---|---|---|---|---|
| 405M | Constant | 38.21 | 52.40 | 65.81 | 30.19 | 52.80 | 47.88 |
| 405M | Gram | 38.67 | 52.83 | 65.83 | 30.10 | 52.46 | 47.98 |
| 405M | Early | 38.72 | 52.30 | 66.21 | 30.65 | 51.85 | 47.95 |
| 1.4B | No decay | 54.97 | 54.59 | 73.67 | 52.59 | 56.20 | 58.40 |
| 1.4B | Constant | 55.78 | 54.12 | 72.52 | 53.31 | 58.96 | 58.94 |
| 1.4B | Early | 56.92 | 52.65 | 73.83 | 54.55 | 57.30 | 59.05 |
| Scale | A unchanged | B unchanged | A with B norms | B with A norms |
|---|---|---|---|---|
| 162M | 3.1866 | 3.1694 | 3.1701 | 3.1861 |
| 405M | 3.0539 | 3.0302 | 3.0323 | 3.0471 |
| Scale | No decay | Best constant | First 10% | Linear 20% | Linear full run |
|---|---|---|---|---|---|
| 162M | 3.3869 | 3.3779 | 3.3708 | 3.3689 | 3.3668 |
| 405M | 3.0537 | 3.0462 | 3.0287 | 3.0257 | 3.0340 |
| Scale | No decay | Constant decay | Early decay |
|---|---|---|---|
| 162M | 3.3089 (0.04, 0) | 3.3066 (0.04, 0.01) | 3.2999 (0.04, 0.075) |
| 405M | 3.0118 (0.04, 0) | 3.0170 (0.02, 0.005) | 3.0019 (0.02, 0.15) |
Deep Learning Optimization
Deep learning optimization develops algorithms and training strategies that help neural networks converge, generalize, and use compute efficiently.
See all 217 papers