cs.CLMar 17, 2026

Pre-training LLM without Learning Rate Decay Enhances Supervised Fine-Tuning

Authors: Kazuki Yano, Shun Kiyono, Sosuke Kobayashi, Sho Takase, Jun Suzuki

Organizations: Tohoku University · SB Intuitions

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We investigate the role of learning rate scheduling in the large-scale pre-training of large language models, focusing on its influence on downstream performance after supervised fine-tuning (SFT). Decay-based learning rate schedulers are widely used to minimize pre-training loss. However, despite their widespread use, how these schedulers affect performance after SFT remains underexplored. In this paper, we examine Warmup-Stable-Only (WSO), which maintains a constant learning rate after warmup without any decay. Through experiments with 1B and 8B parameter models, we show that WSO consistently outperforms decay-based schedulers in terms of performance after SFT, even though decay-based schedulers may exhibit better performance after pre-training. The result also holds across different regimes with mid-training and over-training. Loss landscape analysis further reveals that decay-based schedulers lead models into sharper minima, whereas WSO preserves flatter minima that support adaptability. These findings indicate that applying LR decay to improve pre-training metrics may compromise downstream adaptability. Our work also provides practical guidance for training and model release strategies, highlighting that pre-training models with WSO enhances their adaptability for downstream tasks.

Figures & tables

Figure 1: Learning rate schedulers used in pre-training and their impact on performance after supervised fine-tuning (SFT). Warmup-Stable-Only (WSO), which removes the decay phase, achieves the highest performance after SFT.
Figure 2: Mid-training LR schedulers with different αpre\alpha_{\text{pre}} and αmid\alpha_{\text{mid}} values.
ModelSchedulerαpre\alpha_{\text{pre}}PT Valid Loss ↓\downarrow Δ\DeltaPT Task Avg Δ\DeltaSFT Task Avg Δ\Delta
1BWarmup-Stable-Only (WSO)1.0+0.071-1.7+0.3
WSD0.1+0.004-1.5+0.0
0.0+0.000-1.2-1.0
Linear0.1+0.021-2.0-0.7
0.0+0.016+0.0-0.9
Cosine0.1+0.019-0.1-0.7
Table 1: Relative performance across pre-training (PT) and supervised fine-tuning (SFT). For each model size and each metric, values are differences ( Δ\Delta ) from the best-performing decay-based scheduler for that metric. Note that WSO could perform poorly after PT but best after SFT . Bold indicates the best performance.
Model(Pre-training) Schedulerαpre\alpha_{\text{pre}}αmid\alpha_{\text{mid}}MT Valid Loss ↓\downarrow Δ\DeltaMT Task Avg Δ\DeltaSFT Task Avg Δ\Delta
1BWarmup-Stable-Only (WSO)1.01.0+0.062-0.1+0.8
WSD1.00.0+0.000+ 0.0+0.0
0.11.0+0.038-1.5-0.5
0.10.0+0.047-1.7-1.3
Linear0.11.0+0.053-2.1-2.5
0.10.0+0.058-3.3-3.8
Table 2: Relative performance across mid-training (MT) and SFT stages. Values are differences from the best decay-based schedule. WSO throughout both stages yields the best SFT performance.
ModelSchedulerαpre\alpha_{\text{pre}}PT Valid Loss ↓\downarrow Δ\DeltaPT Task Avg Δ\DeltaSFT Task Avg Δ\Delta
1BWarmup-Stable-Only (WSO)1.0+ 0.048- 1.5+0.7
WSD0.1+0.004+0.0+0.0
0.0+0.000+0.0-0.3
Linear0.1+0.021-0.9-0.5
0.0+0.017-0.4-0.6
Cosine0.1+0.017+0.0-0.4
Table 3: Relative performance after over-training (2T tokens). Values are differences ( Δ\Delta ) from the best-performing decay-based scheduler for each metric. Similar to Section 3 , WSO achieves the best SFT performance.
Model(Pre-training) Schedulerαpre\alpha_{\text{pre}}αmid\alpha_{\text{mid}}MT Valid Loss ↓\downarrow Δ\DeltaMT Task Avg Δ\DeltaSFT Task Avg Δ\Delta
1BWarmup-Stable-Only (WSO)1.01.0+0.055-0.3+1.4
WSD1.00.0+0.000-1.6-0.5
0.11.0+0.033+ 0.0-1.0
0.10.0+0.038-1.7-1.2
Linear0.11.0+0.068-2.2+0.0
0.10.0+0.051-2.8-0.6
Table 4: Relative performance after over-training with mid-training (2T + 500B tokens). Values are differences ( Δ\Delta ) from the best-performing decay-based scheduler for each metric. Similar to Section 4 , WSO yields the best SFT performance.
Figure 3: Sharpness(θt)\text{Sharpness}(\theta_{t}) during pre-training of the 1B model. Vertical line at step TstableT_{\text{stable}} indicating where WSD decays LR. Decay-based schedulers ( αpre=0\alpha_{\text{pre}}=0 or αpre=0.1\alpha_{\text{pre}}=0.1 ) lead to sharper minima, while WSO ( αpre=1.0\alpha_{\text{pre}}=1.0 ) maintains flatter landscapes.
Figure 4: Pre-training sharpness negatively correlates with downstream SFT performance.
Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Configuration1B8B
Hidden dimension20484096
FFN dimension819214336
Number of layers1632
Number of heads3232
Number of KV heads88
Head dimension64128
Appendix
Table 5: Model configurations for the 1B and 8B models.
Hyperparameter1B8B
Training Configuration
Total training steps80,00080,000
Total tokens350B500B
Batch size (tokens)4,194,30412,582,912
Sequence length2,0482,048
Optimizer (AdamW)
Appendix
Table 6: Pre-training hyperparameters for 1B and 8B models. The WSD stable ratio ρ=0.75\rho=0.75 means the LR remains stable for 75% of training after warmup, with decay occurring in the final 25% when αpre<1\alpha_{\text{pre}}<1 .
HyperparameterValue
Training Configuration
Total training steps120,000
Total tokens2T
Batch size (tokens)16,777,216
Appendix
Table 7: Over-training configuration for the 1B model trained on 2T tokens. All other hyperparameters are identical to those in Table 6 .
HyperparameterValue
LR5.0×10−75.0\times 10^{-7} , 1.0×10−61.0\times 10^{-6} , 5.0×10−65.0\times 10^{-6} , 1.0×10−51.0\times 10^{-5} , 5.0×10−55.0\times 10^{-5} , 1.0×10−41.0\times 10^{-4} , 5.0×10−45.0\times 10^{-4} , 1.0×10−31.0\times 10^{-3}
Global Batch size128
LR schedulerCosine with warmup
Minimum LR0
OptimizerAdamW
Weight decay0.0
Appendix
Table 8: SFT hyperparameters used in our experiments. We perform a sweep over the specified LRs and select the best value based on AlpacaEval performance.
ModelSchedulerαpre\alpha_{\text{pre}}Valid Loss ↓\downarrowARC-eARC-cBoolQHellaOBQAPIQAWinoAvg.
1BWarmup-Stable-Only (WSO)1.02.43170.842.262.056.345.470.858.558.0
WSD0.12.36472.040.062.157.446.472.557.158.2
0.02.36072.239.763.757.645.672.258.658.5
Linear0.12.38070.342.663.255.645.271.655.757.7
0.02.37674.443.465.758.447.470.957.559.7
Cosine0.12.37971.143.666.559.947.871.756.359.6
Appendix
Table 9: Pre-training evaluation results. Models with more decay ( αpre=0\alpha_{\text{pre}}=0 ) generally achieve lower validation loss, but not always better zero-shot task performance.
ModelSchedulerαpre\alpha_{\text{pre}}Valid Loss ↓\downarrowARC-eARC-cBoolQHellaOBQAPIQAWinoAvg.
1BWarmup-Stable-Only (WSO)1.02.62574.443.359.763.548.673.262.060.7
WSD0.12.58275.445.860.066.650.074.762.762.2
0.02.57875.346.859.266.250.474.463.062.2
Linear0.12.59973.445.664.765.448.073.258.861.3
0.02.59573.944.466.665.249.273.659.861.8
Cosine0.12.59572.944.165.764.952.074.061.562.2
Appendix
Table 10: Pre-training evaluation results for over-trained 1B models (2T tokens).
ModelSchedulerαpre\alpha_{\text{pre}}Selected SFT LR
1BWarmup-Stable-Only (WSO)1.03×10−43\times 10^{-4}
WSD0.11×10−41\times 10^{-4}
0.01×10−41\times 10^{-4}
Linear0.11×10−41\times 10^{-4}
0.01×10−41\times 10^{-4}
Cosine0.11×10−41\times 10^{-4}
Appendix
Table 11: SFT learning rates selected for each pre-trained model based on AlpacaEval performance.
ModelSchedulerαpre\alpha_{\text{pre}}AlpacaEvalTruthfulQAMMLUAvg.
1BWarmup-Stable-Only (WSO)1.084.043.435.954.4
WSD0.183.941.936.654.1
0.082.340.236.753.1
Linear0.182.042.036.353.4
0.082.441.735.653.2
Cosine0.183.641.035.553.4
Appendix
Table 12: SFT evaluation results. Models pre-trained with WSO achieve the best downstream performance.
ModelSchedulerαpre\alpha_{\text{pre}}Selected SFT LR
1BWarmup-Stable-Only (WSO)1.01×10−41\times 10^{-4}
WSD0.13×10−53\times 10^{-5}
0.03×10−53\times 10^{-5}
Linear0.13×10−53\times 10^{-5}
0.03×10−53\times 10^{-5}
Cosine0.11×10−51\times 10^{-5}
Appendix
Table 13: SFT learning rates selected for each over-trained 1B model based on AlpacaEval performance.
ModelSchedulerαpre\alpha_{\text{pre}}AlpacaEvalTruthfulQAMMLUAvg.
1BWarmup-Stable-Only (WSO)1.078.138.734.550.4
WSD0.177.238.333.649.7
0.076.038.433.749.4
Linear0.175.637.834.249.2
0.075.537.933.949.1
Cosine0.176.037.933.949.3
Appendix
Table 14: SFT evaluation results for over-trained 1B models (pre-trained on 2T tokens).
ModelPre-training Schedulerαpre\alpha_{\text{pre}}αmid\alpha_{\text{mid}}Valid Loss ↓\downarrowARC-CHellaSwagWinoGrandeDROPGSM8KAvg.
1BWarmup-Stable-Only (WSO)1.01.02.33547.060.558.623.920.442.1
WSD1.00.02.27345.061.160.423.321.142.2
0.10.02.32045.062.060.723.811.040.5
0.11.02.31045.160.759.824.513.040.6
Cosine0.10.02.33243.861.459.520.210.739.1
0.11.02.32644.360.759.721.412.839.8
Appendix
Table 15: Mid-training evaluation results in Section 4
ModelPre-training Schedulerαpre\alpha_{\text{pre}}αmid\alpha_{\text{mid}}Valid Loss ↓\downarrowARC-CHellaSwagWinoGrandeDROPGSM8KAvg.
1BWarmup-Stable-Only (WSO)1.01.02.25446.761.360.427.023.143.7
WSD1.00.02.19947.165.262.223.713.742.4
0.10.02.23747.165.262.223.413.742.3
0.11.02.23147.465.762.625.319.144.0
Cosine0.10.02.25346.065.162.323.811.441.7
0.11.02.24543.564.762.125.914.842.2
Appendix
Table 16: Mid-training evaluation results for over-trained 1B models (pre-trained on 2T tokens, mid-trained on 500B tokens).
ModelSchedulerαpre\alpha_{\text{pre}}αmid\alpha_{\text{mid}}Selected SFT LR
1BWarmup-Stable-Only (WSO)1.01.03×10−43\times 10^{-4}
WSD1.00.03×10−53\times 10^{-5}
0.11.01×10−41\times 10^{-4}
0.10.03×10−53\times 10^{-5}
Linear0.11.03×10−53\times 10^{-5}
0.10.03×10−53\times 10^{-5}
Appendix
Table 17: SFT learning rates selected for each model configuration based on AlpacaEval performance.
ModelPre-training Schedulerαpre\alpha_{\text{pre}}αmid\alpha_{\text{mid}}AlpacaEvalTruthfulQAGSM8KDROPAGI EvalBBHMMLUAvg.
1BWarmup-Stable-Only (WSO)1.01.079.441.829.022.721.823.135.736.2
WSD1.00.079.439.927.222.021.522.735.435.4
0.10.076.841.018.922.022.423.834.234.2
0.11.078.740.021.223.723.123.834.435.0
Cosine0.10.072.938.119.917.622.117.933.931.8
0.11.074.337.922.217.122.619.634.032.5
Appendix
Table 18: SFT evaluation results after mid-training. WSO throughout pre- and mid-training generally achieves better SFT performance.
ModelSchedulerαpre\alpha_{\text{pre}}αmid\alpha_{\text{mid}}Selected SFT LR
1BWarmup-Stable-Only (WSO)1.01.01×10−51\times 10^{-5}
WSD1.00.01×10−51\times 10^{-5}
0.11.03×10−53\times 10^{-5}
0.10.01×10−51\times 10^{-5}
Linear0.11.01×10−51\times 10^{-5}
0.10.01×10−51\times 10^{-5}
Appendix
Table 19: SFT learning rates selected for each over-trained and mid-trained 1B model based on AlpacaEval performance.
ModelPre-training Schedulerαpre\alpha_{\text{pre}}αmid\alpha_{\text{mid}}AlpacaEvalTruthfulQAGSM8KDROPAGI EvalBBHMMLUAvg.
1BWarmup-Stable-Only (WSO)1.01.066.238.130.319.424.124.836.634.2
WSD1.00.064.040.420.418.421.326.135.532.3
0.10.064.839.815.619.521.423.736.031.5
0.11.062.139.721.916.821.125.035.931.8
Cosine0.10.062.541.118.720.523.218.835.931.5
0.11.064.642.021.018.723.020.035.432.1
Appendix
Table 20: SFT evaluation results for over-trained 1B models after mid-training (pre-trained on 2T tokens, mid-trained on 500B tokens, then supervised fine-tuned).
Hyperparameter1B8B
Training Configuration
Total training steps36,00036,000
Total tokens150B225B
Batch size (tokens)4,194,30412,582,912
Sequence length2,0482,048
Appendix
Table 21: Mid-training configuration for 1B and 8B models.
HyperparameterValue
Training Configuration
Total training steps30,000
Total tokens500BT
Batch size (tokens)16,777,216
Appendix
Table 22: Mid-training configurations in over-training settings for the 1B model trained on 500BT tokens. All other hyperparameters are identical to those in Table 6 .
HyperparameterValue
Sequence length1,024
Batch size1
Number of views2
Hutchinson samples50
Maximum batches4,096
Maximum texts16,192
Appendix
Table 23: Configuration for sharpness (Hessian trace) computation using Hutchinson’s estimator.
More in this topic

Learning Rate Scheduling

Learning rate scheduling studies how to vary optimizer step sizes during training to improve stability, convergence, efficiency, and generalization.

See all 58 papers

Explore similar work

CardsList

  1. ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Models

    May 18, 2026Aaron DefazioLanguage Model PretrainingLearning Rate Scheduling

    FAIR at Meta Super-Intelligence Labs

  2. SOLAR: A State-Driven Online Learning Rate Scheduler for LLM Pretraining

    Sep 28, 2026Qiulin Shang, Binyu Wang, Yongqi Qiao +3Language Model PretrainingLearning Rate Scheduling

    Peking University · Nanjing University

  3. Learning Rate Engineering: From Coarse Single Parameter to Layered Evolution

    Apr 30, 2026Ming-Hong Yao, Di Wang, Jian Cui +5Deep Learning OptimizationFine-Tuning

    School of AI, Jilin International Sciences University, Changchun, China.