From Spectra to Joint Schedules in LLM Pre-training: 3+3(+2) Scaling-Law Regimes

Yichen Wang; Fanghui Liu; Yudong Chen

cs.LGSep 30, 2026

From Spectra to Joint Schedules in LLM Pre-training: 3+3(+2) Scaling-Law Regimes

Authors: Yichen Wang, Fanghui Liu, Yudong Chen

Organizations: Department of Computer Sciences, University of Wisconsin–Madison, USA. · School of Mathematical Sciences, Institute of Natural Sciences and MOE-LSC, Shanghai Jiao Tong University, China.

Abstract

Power-law learning curves are often treated as fixed properties of a model and its data, although learning-rate and batch-size schedules can change the observed loss. We study this dependence in noisy online SGD with linear random features. Conditional on the representation, an exact Volterra equation separates two response components: a forcing term that propagates unresolved target error and a memory kernel that propagates stochastic-error injections. We prove that either component follows a power law if and only if its cumulative weighted spectral mass has the corresponding low-spectrum scaling; individual eigenvalues and target coefficients need not obey coordinatewise power laws. Under a joint schedule, intrinsic time $T_t=\sum_{s<t}η_s$ controls optimization progress, while $r_t=B_t/η_t$ controls noise injection. Their interaction yields sharp conditions under which a schedule preserves, changes, or destroys the clean power law, together with a memory ceiling on noise reduction. The power-law random-feature model realizes this mechanism in $3+3(+2)$ propagation regimes with phase-dependent compute rates. Controlled nanoGPT experiments show that (1) learning-rate and batch-size schedules with matched $B/η$ paths are nearly equivalent in intrinsic time, (2) a forcing-memory surrogate accurately predicts loss across schedules, and (3) its fitted exponents across real-world datasets identify the regime of LLMs in $3+3(+2)$ map.

Figures & tables

Panel of Figure 1 — Figure 1 : Proxy-derived response coordinates in 300M nanoGPT training. (a) We control the same ratio path $r_{t}=B_{t}/\eta_{t}$ by changing batch-size and learning rate schedules equivalently, e.g., the WSD ( Hu et al., 2024 ) and 8-1-1 ( Bi et al., 2024 ) , leading to different trajectories against optimizer step. (b) After reparameterization by intrinsic time, each trajectory coincides. A surrogate fitted only to the 8-1-1 trajectory also predicts the held-out WSD trajectory without refitting. (c) The fitted exponent satisfies $q_{\mathcal{K}}\approx 1$ and $q_{\mathcal{F}}<1$ , placing the LLM response near the $\mathrm{IM}_{1}/\mathrm{LM}_{1}$ boundary of our $3+3(+2)$ scaling law regimes.

Panel of Figure 2 — Figure 2 : Finite-width rapid-target construction with $\lambda_{j}=j^{-0.8}$ , $\lvert\theta_{j}^{\star}\rvert^{2}=e^{-j}$ , and $d=524{,}288$ . (a) Normalized forcing and cumulative forcing mass both decay faster than every inverse power of $T$ . (b) Normalized one-injection memory and cumulative memory mass both track $T^{-3/4}$ . (c) The minibatch-SGD loss curve averaged over 20 runs follows the $T^{-3/4}$ memory tail.

Panel of Figure 3 — Figure 3 : PLRF regimes in $(\alpha,\beta)$ . Colors match the response-coordinate map in Figure 1 ; the red band marks the fitted LLM regime near $\alpha=1/2$ .

Panel of Figure 4 — Figure 4 : LM/IM schedule response. The red line marks the destroy boundary, where the noisy–clean gap has no power decay; the gray region to its left is unstable. The black curve separates changed from preserved scaling and plateaus at the memory ceiling.

Regime and branch	$\mathcal{R}_{\sigma}^{\star}(D)$	$\mathcal{R}_{\sigma}^{\star}(\mathfrak{f})$
$\mathrm{IM}_{1},\ \beta<0$	$D^{-p/(2\alpha)}$	$\mathfrak{f}^{-p/(1+2\alpha)}$
$\mathrm{LM}_{1,2};\ \mathrm{IM}_{1},\ \beta>0$	$D^{-p/(1+p)}$	$\mathfrak{f}^{-p/(2+p)}$
$\mathrm{LM}_{3}$	$D^{-p/(1+p)}$	$\mathfrak{f}^{-2\alpha p/[p+2\alpha(1+p)]}$
$\mathrm{IM}_{2,3},\ 1/2<\alpha\leq 1$	$D^{-p/(1+p)}$	$\mathfrak{f}^{-p/(1+p+2\beta)}$
$\mathrm{IM}_{2,3},\ \alpha>1$	$D^{-p/(1+p)}$	$\mathfrak{f}^{-\alpha/(1+\alpha)}$
$\mathrm{FB}_{1}$	$D^{-p/(1+p)}$	$\mathfrak{f}^{-p/(2+p)}$

Table 1 : Optimal risk rates across the PLRF propagation regimes. Each entry is an

\asymp

-order, with

p=2\alpha+2\beta-1

, data budget

D

, and feature-compute budget

\mathfrak{f}\asymp mD

Panel of Figure 5 — Figure 5 : LLM surrogate fit, zero-refit transfer, and effective $q_{\mathcal{K}}$ . (a) For a $300$ M SGD LLM, we fix $q_{\mathcal{K}}=1$ in Eq. 5.3 and fit the remaining 6 parameters to the 8-1-1 learning-rate schedule. (b) With all fitted parameters frozen, it predicts the WSD schedule without refitting. (c) Fixing $q_{\mathcal{K}}$ , fitting the remaining parameters only on 8-1-1, and evaluating the WSD schedules without refitting yields prediction errors, each divided by its best value, that are minimized near $q_{\mathcal{K}}\approx 1$ .

Figure 6 : Different datasets independently recover

q_{\mathcal{K}}\approx 1

. Each surrogate is fitted only to fixed-batch 8-1-1 and transferred without refitting to the alternative factorization and both WSD trajectories. The fitted

q_{\mathcal{K}}

are

0.952

on FineWeb and

0.995

on peS2o V2; together with the independently fitted OpenWebText value

1.017

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Limit regime	Results in this paper
Exact finite width: $d,m,\bm{W}$ fixed, $t<\infty$	Exact finite-bulk kernel and noisy–clean gap ( Lemmas C.8 and C.9 )
Fixed-width terminal: $d,m,\bm{W}$ fixed, $T\to\infty$	Stable terminal-risk transfer and the noise-induced plateau ( Theorems A.9 , E.3 and E.5 )
Fixed-time DE: $d,m\to\infty$ , $d/m\to c$ , $T$ fixed	Positive spectral representation and DE risk recursion for varying schedules ( Lemmas A.1 , C.1 and C.5 )
Fixed infinite-spectrum system: $T\to\infty$	Schedule response, full classification, and gap-to-ratio identification ( Theorems 5.1 , A.6 , D.13 , D.16 , D.19 and D.23 )
Joint width–time: $m,T_{m}\to\infty$ , $1\ll T_{m}\lesssim m^{2\alpha}$	Spectral iff criteria, uniform transfer, and LM/IM/FB risk laws ( Theorems 3.1 , A.10 , B.1 , D.4 , C.14 , C.15 , C.18 and C.21 )
Resource limit: $D\to\infty$ or $\mathfrak{f}\to\infty$ ; $m,T,r$ vary	Optimal ratio control and phasewise data/compute rates ( Theorems C.22 , 5.2 and C.23 )

Appendix

Table 2 : Limit regimes of the principal results.

Analytical level	Formal results
Exact conditional finite-width dynamics	Exact finite-bulk kernel and noisy–clean gap ( Lemmas C.8 and C.9 ).
Deterministic equivalent	Positive DE spectral representation, varying-schedule DE recursion, and conditional transfer to realized SGD ( Lemmas A.1 , C.1 and C.21 ).
Power-law asymptotics	Spectral iff criteria, uniform Tauberian and schedule transfer, and LM/IM/FB risk laws ( Theorems A.4 , B.1 , D.4 , C.14 and C.18 ).

Appendix

Table 3 : Analytical levels and representative formal results.

Symbol	Meaning	Symbol	Meaning
Model and spectrum
$d$	input dimension	$m$	feature width
$\alpha$	spectral-decay exponent	$\beta$	target-regularity exponent
$p$	target-energy tail exponent $2\alpha+2\beta-1$	$\bm{x}$	covariate
$\bm{\Lambda}$	population covariance	$\lambda_{j}$	population eigenvalue
$\bm{\theta}^{\star}$	teacher coefficients	$f_{\star}$	teacher predictor

Appendix

Table 4: Core notation; proof-local symbols are defined at first use.

Figure 7 : Online SGD, the exact finite-width recursion, and its DE reduction for centered risk and the noisy–clean gap in long and integrable memory. Their agreement supports both approximation steps and the predicted boundary, changed, and preserved schedule responses.

Figure 8 : Online SGD, the exact finite-width recursion, and its DE reduction for the three injection responses in

\mathrm{FB}_{1}

and

\mathrm{FB}_{2}

. Their agreement supports the finite-bulk Volterra hierarchy and accumulation laws.

Figure 9 : Finite-width realizations of minibatch-SGD excess risk. Across the four constructions, the risk curves exhibit an overall power-law decay over the displayed finite-width window, including logarithmic corrections and log-periodic modulation.

Figure 10 : Finite-width realizations of minibatch-SGD excess risk. The rapidly decaying target develops a memory-driven

T^{-3/4}

tail, whereas the spectrally gapped construction relaxes exponentially.

Figure 11 : Floor-centered risk, noisy–clean gap, and predicted exponents across four noise-control schedules. The transition from growth through destruction and change to clean-law preservation supports Theorem 5.1 .

Figure 12 : Centered finite-width DE risk across widths in the three integrable-memory subregimes. The stabilization of the minimizing iteration supports the finite-time local optimum in Theorem E.4 .

Figure 13 : The PLRF propagation map and representative centered-risk compute frontiers in $\mathrm{FB}_{2}$ and $\mathrm{IM}_{1}$ . Their continuing forcing–noise balance and finite-time stopping illustrate the phase-dependent shift of compute from iterations to width.

Figure 14 : The minimizing iteration and stopping class across noise levels and widths in the three integrable-memory subregimes. The terminal–finite and finite–no-training transitions support the predicted subregime dependence of early stopping.

Figure 15 : Effect of the ratio path at matched intrinsic time. All runs start from the same checkpoint and stop at the same intrinsic-time horizon, but use different numbers of updates and tokens. Larger $\vartheta$ means faster growth of $B/\eta$ , and darker curves correspond to larger $\vartheta$ . Validation loss decreases with $\vartheta$ , with diminishing improvement at large $\vartheta$ .

Figure 16 : Factorization collapse under matched ratio schedules. Fixed-probe validation cross-entropy for the same

300

M nanoGPT architecture trained for

6.5

B OpenWebText tokens with plain SGD (left pair) and hybrid Muon (right pair), shown against optimizer step and the corresponding intrinsic clock. Within each WSD or 8-1-1 schedule family, the LR schedule varies

\eta_{t}

at fixed

B_{t}

, whereas the batch-size schedule varies

B_{t}

at fixed

\eta_{t}

. For plain SGD, these two realizations follow the same ratio path

r_{t}=B_{t}/\eta_{t}

under

T_{t}=\sum_{s<t}\eta_{s}

; for Muon, they follow the same empirically calibrated path

\widetilde{r}_{t}=B_{t}/\eta_{t}^{2}

under

\widetilde{T}_{t}=\sum_{s<t}\eta_{s}^{2}

. The paired trajectories differ in optimizer step but nearly coincide in the corresponding intrinsic time. Blue denotes WSD, orange denotes 8-1-1, and gray denotes the shared prefix.

Figure 17 : A joint-schedule adaptation of the functional scaling law fits and transfers across LLM schedules. Following the fit-then-transfer protocol of Li et al. (2025) , the ratio-based surrogate is fitted only to the fixed-batch 8-1-1 validation trajectory, after which all parameters are frozen. It tracks the alternative 8-1-1 factorization and predicts both WSD trajectories without refitting, showing that the same forcing–memory parameterization captures long-horizon LLM loss decay across learning-rate and batch-size schedules. The fitted exponents are

q_{\mathcal{K}}=1.017

and

q_{\mathcal{F}}=0.374

, placing the effective memory response close to the

q_{\mathcal{K}}=1

boundary. Risk denotes fixed-probe validation cross-entropy and

T=\sum_{t}\eta_{t}

Figure 18 : The joint-schedule surrogate transfers at

300

M scale. The seven parameters are fitted only to the raw fixed-batch 8-1-1 validation trajectory. With those parameters frozen, the surrogate tracks the alternative 8-1-1 factorization and predicts both WSD trajectories without refitting. The fitted exponents are

q_{\mathcal{K}}=0.989

and

q_{\mathcal{F}}=0.291

, again placing the effective surrogate coordinate close to

q_{\mathcal{K}}=1

. Risk denotes fixed-probe validation cross-entropy and

T=\sum_{t}\eta_{t}

Explore similar work

Feb 6, 2026stat.ML

Optimal Learning Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decay

We study optimal learning rate (LR) schedules under the functional scaling law (FSL) framework (Li et al., 2025), which decomposes training dynamics into signal learning and noise forgetting. In power-law kernel regression, these two components are governed by a source exponent

s>0

and a capacity exponent

q>1

, respectively, with smaller

s

corresponding to harder tasks. For a fixed training horizon

N

, we characterize the schedules that minimize the final-step loss under a stability constraint and reveal a sharp phase transition. In the easy-task regime

s>1-1/q

, the optimal schedule follows power decay from the beginning of training; in the hard-task regime

s<1-1/q

, it becomes warmup-stable-decay (WSD)-like (Hu et al., 2024), staying at the largest admissible LR for most of training before a final decay. In both regimes, the decay exponent is

2q-1

: task difficulty determines when to decay, while model capacity determines how to decay. Beyond the exact optimum, we study fractional schedules, whose shape is defined over relative training progress. We show that precise tuning of the decay shape is often unnecessary: a broad class of profiles attains the optimal convergence rate, while overly slow terminal decay leads to schedule-induced capacity saturation. Finally, for one-pass SGD in kernel regression, FSL-motivated power-decay schedules achieve optimal last-iterate rates. Experiments support the theoretical predictions and the task-dependent transition between early and delayed decay.

Binghui Li, Zilin Wang, Fengling Chen +3

Center for Machine Learning Research, Peking University · School of Mathematical Sciences, Peking University · AI for Science Institute, Beijing

Batch Scaling Laws Warm Starts Kernel Method

Ask PDF

Jul 30, 2026cs.LG

Towards joint scaling laws with optimal batch size schedules

Modern deep learning typically keeps the batch size static throughout training, thus overlooking the joint effect of learning rate and batch size on the training dynamics. In this paper, we study the deep learning dynamics through the lens of convex optimization and derive a joint characterization of loss in terms of both schedules, applicable to general optimizers and model architectures. This characterization yields a closed-form optimal batch size schedule for any prescribed learning rate schedule, and further leads to joint scaling laws that consistently outperform static batch size baselines, highlighting the significance of dynamic batch size schedule in large language model training.

Jiaxiang Li, Zhiqi Bu, Shiyun Xu

Meta · Independent researcher

Batch Large Language Model Training Scaling Laws Learning Dynamics Schedule Convex Optimization

Ask PDF

Jun 28, 2026cs.LG

On the Nonlinearity of Learning Rate Scaling for LLM Training

Learning-rate transfer can reduce the cost of training large language models: instead of sweeping learning rates at target scale, practitioners extrapolate from smaller runs. Existing approaches often assume that the optimal learning rate follows a log-linear scaling law in data scale and model size. We carefully examine and evaluate this scaling law. In our empirical study of GPT-2--style models from 22M to 707M parameters trained on 5B to 100B tokens, the optimal learning rate develops upward curvature at larger scales, leading to inaccurate extrapolation. We find that this curvature largely disappears when learning rates are replaced by effective learning rate (the step size in normalized weight space), and when data

D

extrapolation is used instead of model size

N

extrapolation. Next, we explain nonlinearity in scaling: weight-norm converges to equilibrium slower when optimal learning is small, requiring a larger step size to reduce the transient phase. Experiments with AdamH, which directly controls the effective learning rate, further support this explanation.

Zaiwen Yang, Huaqing Zhang, Jing Xu +1

Tsinghua University · Tsinghua University, Shanghai Qi Zhi Institute

Batch Large Language Model Training Scaling Laws Scaling Transfer Learning Extrapolation+1

Ask PDF