cs.CLMar 17, 2026

Pre-training LLM without Learning Rate Decay Enhances Supervised Fine-Tuning

Authors: Kazuki Yano, Shun Kiyono, Sosuke Kobayashi, Sho Takase, Jun Suzuki

Organizations: Tohoku University · SB Intuitions

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We investigate the role of learning rate scheduling in the large-scale pre-training of large language models, focusing on its influence on downstream performance after supervised fine-tuning (SFT). Decay-based learning rate schedulers are widely used to minimize pre-training loss. However, despite their widespread use, how these schedulers affect performance after SFT remains underexplored. In this paper, we examine Warmup-Stable-Only (WSO), which maintains a constant learning rate after warmup without any decay. Through experiments with 1B and 8B parameter models, we show that WSO consistently outperforms decay-based schedulers in terms of performance after SFT, even though decay-based schedulers may exhibit better performance after pre-training. The result also holds across different regimes with mid-training and over-training. Loss landscape analysis further reveals that decay-based schedulers lead models into sharper minima, whereas WSO preserves flatter minima that support adaptability. These findings indicate that applying LR decay to improve pre-training metrics may compromise downstream adaptability. Our work also provides practical guidance for training and model release strategies, highlighting that pre-training models with WSO enhances their adaptability for downstream tasks.

Figures & tables

Figure 1: Learning rate schedulers used in pre-training and their impact on performance after supervised fine-tuning (SFT). Warmup-Stable-Only (WSO), which removes the decay phase, achieves the highest performance after SFT.
Figure 2: Mid-training LR schedulers with different αpre\alpha_{\text{pre}} and αmid\alpha_{\text{mid}} values.
ModelSchedulerαpre\alpha_{\text{pre}}PT Valid Loss ↓\downarrow Δ\DeltaPT Task Avg Δ\DeltaSFT Task Avg Δ\Delta
1BWarmup-Stable-Only (WSO)1.0+0.071-1.7+0.3
WSD0.1+0.004-1.5+0.0
0.0+0.000-1.2-1.0
Linear0.1+0.021-2.0-0.7
0.0+0.016+0.0-0.9
Cosine0.1+0.019-0.1-0.7
Table 1: Relative performance across pre-training (PT) and supervised fine-tuning (SFT). For each model size and each metric, values are differences ( Δ\Delta ) from the best-performing decay-based scheduler for that metric. Note that WSO could perform poorly after PT but best after SFT . Bold indicates the best performance.
Model(Pre-training) Schedulerαpre\alpha_{\text{pre}}αmid\alpha_{\text{mid}}MT Valid Loss ↓\downarrow Δ\DeltaMT Task Avg Δ\DeltaSFT Task Avg Δ\Delta
1BWarmup-Stable-Only (WSO)1.01.0+0.062-0.1+0.8
WSD1.00.0+0.000+ 0.0+0.0
0.11.0+0.038-1.5-0.5
0.10.0+0.047-1.7-1.3
Linear0.11.0+0.053-2.1-2.5
0.10.0+0.058-3.3-3.8
Table 2: Relative performance across mid-training (MT) and SFT stages. Values are differences from the best decay-based schedule. WSO throughout both stages yields the best SFT performance.
ModelSchedulerαpre\alpha_{\text{pre}}PT Valid Loss ↓\downarrow Δ\DeltaPT Task Avg Δ\DeltaSFT Task Avg Δ\Delta
1BWarmup-Stable-Only (WSO)1.0+ 0.048- 1.5+0.7
WSD0.1+0.004+0.0+0.0
0.0+0.000+0.0-0.3
Linear0.1+0.021-0.9-0.5
0.0+0.017-0.4-0.6
Cosine0.1+0.017+0.0-0.4
Table 3: Relative performance after over-training (2T tokens). Values are differences ( Δ\Delta ) from the best-performing decay-based scheduler for each metric. Similar to Section 3 , WSO achieves the best SFT performance.
Model(Pre-training) Schedulerαpre\alpha_{\text{pre}}αmid\alpha_{\text{mid}}MT Valid Loss ↓\downarrow Δ\DeltaMT Task Avg Δ\DeltaSFT Task Avg Δ\Delta
1BWarmup-Stable-Only (WSO)1.01.0+0.055-0.3+1.4
WSD1.00.0+0.000-1.6-0.5
0.11.0+0.033+ 0.0-1.0
0.10.0+0.038-1.7-1.2
Linear0.11.0+0.068-2.2+0.0
0.10.0+0.051-2.8-0.6
Table 4: Relative performance after over-training with mid-training (2T + 500B tokens). Values are differences ( Δ\Delta ) from the best-performing decay-based scheduler for each metric. Similar to Section 4 , WSO yields the best SFT performance.
Figure 3: Sharpness(θt)\text{Sharpness}(\theta_{t}) during pre-training of the 1B model. Vertical line at step TstableT_{\text{stable}} indicating where WSD decays LR. Decay-based schedulers ( αpre=0\alpha_{\text{pre}}=0 or αpre=0.1\alpha_{\text{pre}}=0.1 ) lead to sharper minima, while WSO ( αpre=1.0\alpha_{\text{pre}}=1.0 ) maintains flatter landscapes.
Figure 4: Pre-training sharpness negatively correlates with downstream SFT performance.
Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Configuration1B8B
Hidden dimension20484096
FFN dimension819214336
Number of layers1632
Number of heads3232
Number of KV heads88
Head dimension64128
Appendix
Table 5: Model configurations for the 1B and 8B models.
Hyperparameter1B8B
Training Configuration
Total training steps80,00080,000
Total tokens350B500B
Batch size (tokens)4,194,30412,582,912
Sequence length2,0482,048
Optimizer (AdamW)
Appendix
Table 6: Pre-training hyperparameters for 1B and 8B models. The WSD stable ratio ρ=0.75\rho=0.75 means the LR remains stable for 75% of training after warmup, with decay occurring in the final 25% when αpre<1\alpha_{\text{pre}}<1 .
HyperparameterValue
Training Configuration
Total training steps120,000
Total tokens2T
Batch size (tokens)16,777,216
Appendix
Table 7: Over-training configuration for the 1B model trained on 2T tokens. All other hyperparameters are identical to those in Table 6 .
HyperparameterValue
LR5.0×10−75.0\times 10^{-7} , 1.0×10−61.0\times 10^{-6} , 5.0×10−65.0\times 10^{-6} , 1.0×10−51.0\times 10^{-5} , 5.0×10−55.0\times 10^{-5} , 1.0×10−41.0\times 10^{-4} , 5.0×10−45.0\times 10^{-4} , 1.0×10−31.0\times 10^{-3}
Global Batch size128
LR schedulerCosine with warmup
Minimum LR0
OptimizerAdamW
Weight decay0.0
Appendix
Table 8: SFT hyperparameters used in our experiments. We perform a sweep over the specified LRs and select the best value based on AlpacaEval performance.
ModelSchedulerαpre\alpha_{\text{pre}}Valid Loss ↓\downarrowARC-eARC-cBoolQHellaOBQAPIQAWinoAvg.
1BWarmup-Stable-Only (WSO)1.02.43170.842.262.056.345.470.858.558.0
WSD0.12.36472.040.062.157.446.472.557.158.2
0.02.36072.239.763.757.645.672.258.658.5
Linear0.12.38070.342.663.255.645.271.655.757.7
0.02.37674.443.465.758.447.470.957.559.7
Cosine0.12.37971.143.666.559.947.871.756.359.6
Appendix
Table 9: Pre-training evaluation results. Models with more decay ( αpre=0\alpha_{\text{pre}}=0 ) generally achieve lower validation loss, but not always better zero-shot task performance.
ModelSchedulerαpre\alpha_{\text{pre}}Valid Loss ↓\downarrowARC-eARC-cBoolQHellaOBQAPIQAWinoAvg.
1BWarmup-Stable-Only (WSO)1.02.62574.443.359.763.548.673.262.060.7
WSD0.12.58275.445.860.066.650.074.762.762.2
0.02.57875.346.859.266.250.474.463.062.2
Linear0.12.59973.445.664.765.448.073.258.861.3
0.02.59573.944.466.665.249.273.659.861.8
Cosine0.12.59572.944.165.764.952.074.061.562.2
Appendix
Table 10: Pre-training evaluation results for over-trained 1B models (2T tokens).
ModelSchedulerαpre\alpha_{\text{pre}}Selected SFT LR
1BWarmup-Stable-Only (WSO)1.03×10−43\times 10^{-4}
WSD0.11×10−41\times 10^{-4}
0.01×10−41\times 10^{-4}
Linear0.11×10−41\times 10^{-4}
0.01×10−41\times 10^{-4}
Cosine0.11×10−41\times 10^{-4}
Appendix
Table 11: SFT learning rates selected for each pre-trained model based on AlpacaEval performance.
ModelSchedulerαpre\alpha_{\text{pre}}AlpacaEvalTruthfulQAMMLUAvg.
1BWarmup-Stable-Only (WSO)1.084.043.435.954.4
WSD0.183.941.936.654.1
0.082.340.236.753.1
Linear0.182.042.036.353.4
0.082.441.735.653.2
Cosine0.183.641.035.553.4
Appendix
Table 12: SFT evaluation results. Models pre-trained with WSO achieve the best downstream performance.
ModelSchedulerαpre\alpha_{\text{pre}}Selected SFT LR
1BWarmup-Stable-Only (WSO)1.01×10−41\times 10^{-4}
WSD0.13×10−53\times 10^{-5}
0.03×10−53\times 10^{-5}
Linear0.13×10−53\times 10^{-5}
0.03×10−53\times 10^{-5}
Cosine0.11×10−51\times 10^{-5}
Appendix
Table 13: SFT learning rates selected for each over-trained 1B model based on AlpacaEval performance.
ModelSchedulerαpre\alpha_{\text{pre}}AlpacaEvalTruthfulQAMMLUAvg.
1BWarmup-Stable-Only (WSO)1.078.138.734.550.4
WSD0.177.238.333.649.7
0.076.038.433.749.4
Linear0.175.637.834.249.2
0.075.537.933.949.1
Cosine0.176.037.933.949.3
Appendix
Table 14: SFT evaluation results for over-trained 1B models (pre-trained on 2T tokens).
ModelPre-training Schedulerαpre\alpha_{\text{pre}}αmid\alpha_{\text{mid}}Valid Loss ↓\downarrowARC-CHellaSwagWinoGrandeDROPGSM8KAvg.
1BWarmup-Stable-Only (WSO)1.01.02.33547.060.558.623.920.442.1
WSD1.00.02.27345.061.160.423.321.142.2
0.10.02.32045.062.060.723.811.040.5
0.11.02.31045.160.759.824.513.040.6
Cosine0.10.02.33243.861.459.520.210.739.1
0.11.02.32644.360.759.721.412.839.8
Appendix
Table 15: Mid-training evaluation results in Section 4
ModelPre-training Schedulerαpre\alpha_{\text{pre}}αmid\alpha_{\text{mid}}Valid Loss ↓\downarrowARC-CHellaSwagWinoGrandeDROPGSM8KAvg.
1BWarmup-Stable-Only (WSO)1.01.02.25446.761.360.427.023.143.7
WSD1.00.02.19947.165.262.223.713.742.4
0.10.02.23747.165.262.223.413.742.3
0.11.02.23147.465.762.625.319.144.0
Cosine0.10.02.25346.065.162.323.811.441.7
0.11.02.24543.564.762.125.914.842.2
Appendix
Table 16: Mid-training evaluation results for over-trained 1B models (pre-trained on 2T tokens, mid-trained on 500B tokens).
ModelSchedulerαpre\alpha_{\text{pre}}αmid\alpha_{\text{mid}}Selected SFT LR
1BWarmup-Stable-Only (WSO)1.01.03×10−43\times 10^{-4}
WSD1.00.03×10−53\times 10^{-5}
0.11.01×10−41\times 10^{-4}
0.10.03×10−53\times 10^{-5}
Linear0.11.03×10−53\times 10^{-5}
0.10.03×10−53\times 10^{-5}
Appendix
Table 17: SFT learning rates selected for each model configuration based on AlpacaEval performance.
ModelPre-training Schedulerαpre\alpha_{\text{pre}}αmid\alpha_{\text{mid}}AlpacaEvalTruthfulQAGSM8KDROPAGI EvalBBHMMLUAvg.
1BWarmup-Stable-Only (WSO)1.01.079.441.829.022.721.823.135.736.2
WSD1.00.079.439.927.222.021.522.735.435.4
0.10.076.841.018.922.022.423.834.234.2
0.11.078.740.021.223.723.123.834.435.0
Cosine0.10.072.938.119.917.622.117.933.931.8
0.11.074.337.922.217.122.619.634.032.5
Appendix
Table 18: SFT evaluation results after mid-training. WSO throughout pre- and mid-training generally achieves better SFT performance.
ModelSchedulerαpre\alpha_{\text{pre}}αmid\alpha_{\text{mid}}Selected SFT LR
1BWarmup-Stable-Only (WSO)1.01.01×10−51\times 10^{-5}
WSD1.00.01×10−51\times 10^{-5}
0.11.03×10−53\times 10^{-5}
0.10.01×10−51\times 10^{-5}
Linear0.11.01×10−51\times 10^{-5}
0.10.01×10−51\times 10^{-5}
Appendix
Table 19: SFT learning rates selected for each over-trained and mid-trained 1B model based on AlpacaEval performance.
ModelPre-training Schedulerαpre\alpha_{\text{pre}}αmid\alpha_{\text{mid}}AlpacaEvalTruthfulQAGSM8KDROPAGI EvalBBHMMLUAvg.
1BWarmup-Stable-Only (WSO)1.01.066.238.130.319.424.124.836.634.2
WSD1.00.064.040.420.418.421.326.135.532.3
0.10.064.839.815.619.521.423.736.031.5
0.11.062.139.721.916.821.125.035.931.8
Cosine0.10.062.541.118.720.523.218.835.931.5
0.11.064.642.021.018.723.020.035.432.1
Appendix
Table 20: SFT evaluation results for over-trained 1B models after mid-training (pre-trained on 2T tokens, mid-trained on 500B tokens, then supervised fine-tuned).
Hyperparameter1B8B
Training Configuration
Total training steps36,00036,000
Total tokens150B225B
Batch size (tokens)4,194,30412,582,912
Sequence length2,0482,048
Appendix
Table 21: Mid-training configuration for 1B and 8B models.
HyperparameterValue
Training Configuration
Total training steps30,000
Total tokens500BT
Batch size (tokens)16,777,216
Appendix
Table 22: Mid-training configurations in over-training settings for the 1B model trained on 500BT tokens. All other hyperparameters are identical to those in Table 6 .
HyperparameterValue
Sequence length1,024
Batch size1
Number of views2
Hutchinson samples50
Maximum batches4,096
Maximum texts16,192
Appendix
Table 23: Configuration for sharpness (Hessian trace) computation using Hutchinson’s estimator.
More in this topic

Learning Rate Scheduling

Learning rate scheduling studies how to vary optimizer step sizes during training to improve stability, convergence, efficiency, and generalization.

See all 58 papers

Explore similar work

CardsList
May 18, 2026

ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Models

Schedule-Free Learning has shown promise as a practical anytime training method for machine learning, showing success across dozens of standard benchmark problems. However, strong performance for LLM training has only been demonstrated at small scales. We identify a number of fixes necessary to scale up Schedule-Free Learning to larger batch sizes and model sizes, and present a learning-rate-free and schedule-free method (ScheduleFree+) for training large language models which greatly outperforms Warmup-Stable-Decay (WSD) schedules. We also demonstrate that Schedule-Free Learning is most effective for long duration training, and at 1000 tokens per parameter, it outperforms SOTA schedules by 31%. Schedule-Free Learning provides a theoretical foundation for the use of model averaging and checkpoint merging during pretraining.

Aaron Defazio

FAIR at Meta Super-Intelligence Labs

Language Model PretrainingLearning Rate SchedulingLLM Training
AskPDF
Sep 28, 2026

SOLAR: A State-Driven Online Learning Rate Scheduler for LLM Pretraining

Learning-rate (LR) scheduling plays a central role in large language model (LLM) pretraining, yet current practice still relies heavily on hand-crafted heuristics such as Warmup-Cosine-Decay and Warmup-Stable-Decay. Because these schedules are fixed in advance, they cannot adapt to evolving optimization dynamics. Online learned scheduling within the Learning to Optimize (L2O) framework offers a dynamic alternative, but remains brittle at LLM scale due to noisy signals, delayed feedback, and the risk of catastrophic divergence. We propose SOLAR (State-driven Online Learning rAte scheduleR), a stabilized framework for reliable online LR adaptation. SOLAR uses a base schedule as a reference and learns bounded, state-dependent residual corrections for individual parameter groups. Each correction re-anchors to the base at every step, allowing the policy to adapt the LR without relearning the warmup-decay profile. A lightweight state representation and progress-aware reward guide online learning, while a Circuit-Breaker restores training after rare unsafe actions. Across autoregressive language-model pretraining, SOLAR improves final perplexity over tuned static schedules and automatic LR tuners for dense models from 60M to 1B, AdamW and Muon, and two MoE settings up to 3B. Matched 130M controls show that adding base anchoring and action bounds improves a global PPO controller from 27.09 to 23.74 final PPL, while group-wise control reaches 22.87 on the same two seeds. A residual policy trained on a 60M proxy can also be frozen and reused at larger dense scales without target PPO updates, remaining effective across a fourfold base-LR range. These results establish SOLAR as a practical learned LR controller for LLM pretraining.

Qiulin Shang, Binyu Wang, Yongqi Qiao +3

Peking University · Nanjing University

Language Model PretrainingLearning Rate SchedulingLearning to OptimizePolicy LearningAdaptive Learning Rate
AskPDF
Apr 30, 2026

Learning Rate Engineering: From Coarse Single Parameter to Layered Evolution

Learning rate scheduling has evolved from the single global fixed rate of early SGD to sophisticated layer-wise adaptive strategies. We systematize this evolution into five generations: (Gen1) global fixed learning rates, (Gen2) global scheduling, (Gen3) parameter-level adaptation, (Gen4) layer-level differentiation, and (Gen5) joint layer-time scheduling. We trace the fundamental motivation behind each transition, showing how the shift from one-size-fits-all to tailoring by layer and time addresses the impossible trinity of transfer learning: lower layers require small updates to preserve general knowledge while higher layers need large updates to adapt to new tasks. Building on this taxonomy, we propose Discriminative Adaptive Layer Scaling (DALS), a unified framework that integrates phase-adaptive cosine scheduling, depth-aware Grokfast gradient filtering, and LARS-style trust ratios into a single coherent optimizer. We benchmark 18 strategies including three DALS variants across all five generations on five datasets: synthetic, CIFAR-10 (from scratch), RTE, TREC-6, and IMDb (fine-tuning). On synthetic, DALS achieves the best accuracy at 98.0%, while DALS-Fast reaches 90% in just 3 epochs. The cross-dataset analysis reveals striking regime-dependent patterns -- no single strategy wins across all regimes. Critically, STLR+Discriminative, the ULMFiT champion, catastrophically fails on from-scratch tasks (43.6% on TREC-6 from scratch vs. 96.8% with RAdam), confirming that directional decay biases are harmful without pretrained features. DALS avoids either extreme, achieving the best synthetic result while maintaining competitive fine-tuning performance.

Ming-Hong Yao, Di Wang, Jian Cui +5

School of AI, Jilin International Sciences University, Changchun, China.

Deep Learning OptimizationFine-TuningLearning Rate SchedulingAdaptive Gradient MethodsLayer-Wise Learning Rate AdaptationAdaptive Learning Rate
AskPDF