Beyond the Log-Linear Frontier: Pre-Asymptotic Analysis of Finite-Budget Neural Scaling Laws
Organizations: LingQiong AI Lab Shanghai Smart Logic Technology Shanghai, China
Abstract
Classical neural scaling laws model excess risk as a separable power law in model capacity and dataset size . At finite budgets, however, empirical learning curves and compute-optimal allocations depart from this log-linear frontier. This paper analyzes how spectral and domain heterogeneity govern these finite-scale departures. Along the capacity axis, Mellin--Barnes analysis of covariance resolvents yields a three-term expansion where competing spectral and regularization modes produce finite-scale curvature and logarithmic transitions. Because fitting multiple power laws on narrow pilot grids is ill-conditioned, we eliminate the loss floor via first-order Aitken differencing to obtain a well-conditioned closed-form initializer. Along the data axis, KKT duality reduces -domain mixing to scalar water-filling, yielding finite-budget bottleneck bounds, an online sampling curriculum, and a simplex Hessian metric for cross-domain transfer. Along the iso-compute frontier , we prove that the optimal allocation exponent drifts in proportion to capacity and data exponent variances, while Schur-complement reduction separates intrinsic drift from frozen-hyperparameter bias. Across 45 operator spectra, Pythia (70M--12B), Chinchilla (245 runs), and RegMix (17 domains), our estimators achieve holdout MAPE, cut high-compute allocation regret by up to , and reduce cross-scale mixture RMSE by with zero selection regret.
Figures & tables
| Axis & Prior Work | Prior Law | Finite-Budget Gap | Our Pre-Asymptotic Law | Classical Limit |
|---|---|---|---|---|
| Capacity ( Bordelon et al., 2024 ) | Subleading residues | (Thm. 3.1 ) | ||
| Pilot Fit ( Varah, 1985 ) | 3-power fit | Grid collinearity | Rank ; (Prop. 3.3 ) | noiseless |
| Mixing ( Ye et al., 2025 ) | Proxy | Saturation & bound | water-filling (Thm. 4.1 ) | |
| Curriculum ( Ge et al., 2024 ) | Joint | Online quota rate | (Prop. 4.2 ) | |
| Compute ( Kaplan et al., 2020 ) | Scale drift of | Gibbs & mix drift (Thm. 5.1 ) | ||
| Calibration ( Porian et al., 2024 ) | Tuned | Regret & envelope | Thm. 5.2 , Prop. 5.4 | Tuned |
| Fitted | Holdout MAPE | High- Regret | ||||||
|---|---|---|---|---|---|---|---|---|
| Estimator | Fit Sample | Est. | CI | All 24 | Frontier | Surface | ||
| Power law (raw) | (221) | |||||||
| Power law (trimmed) | (205) | |||||||
| Log-quadratic | (205) | |||||||
| Full-sample reference | All (228) | — | — | |||||
| 1M Test RMSE | Spearman ( ) | Selection Regret (nats, ) | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Predictor Family (1M Fit) | Mean | Agg. | |||||||
| 17-domain linear Ridge (baseline) | — | ||||||||
| Single-domain power ( Theorem 4.1 ) | |||||||||
| Monotone PCHIP spline (non-param.) | |||||||||
| 1st-order transfer ( Corollary 4.2 ) | |||||||||
| Diag. log-quad. ( ) | |||||||||
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Sec. | Symbol | Definition / Formula | Theoretical Role | Ref. |
|---|---|---|---|---|
| § 2 | Closed and open -simplices | § 2.1 | ||
| § 2 | , | Covariance, resolvent, effective dimension | ( 2.4 ) | |
| § 2 | , | Euler constant and regularizer cutoffs | ( 2.7 ) | |
| § 2 | , | Softmax Fisher matrix and operator | ( 2.12 ) | |
| § 3 | Cutoff scale and Weyl mode energy | ( 2.16 ) | ||
| § 3 | , | Spectral residual filter and Mellin kernel | ( 3.2 ) |