cs.LGNov 10, 2026

Beyond the Log-Linear Frontier: Pre-Asymptotic Analysis of Finite-Budget Neural Scaling Laws

Authors: Tianyu Zhao, Zhixiang Ren

Organizations: LingQiong AI Lab Shanghai Smart Logic Technology Shanghai, China

Abstract

Classical neural scaling laws model excess risk as a separable power law in model capacity NN and dataset size DD. At finite budgets, however, empirical learning curves and compute-optimal allocations depart from this log-linear frontier. This paper analyzes how spectral and domain heterogeneity govern these finite-scale departures. Along the capacity axis, Mellin--Barnes analysis of covariance resolvents yields a three-term expansion where competing spectral and regularization modes produce finite-scale curvature and logarithmic transitions. Because fitting multiple power laws on narrow pilot grids is ill-conditioned, we eliminate the loss floor via first-order Aitken differencing to obtain a well-conditioned closed-form initializer. Along the data axis, KKT duality reduces KK-domain mixing to scalar water-filling, yielding finite-budget bottleneck bounds, an online sampling curriculum, and a simplex Hessian metric for cross-domain transfer. Along the iso-compute frontier C=6NDC=6ND, we prove that the optimal allocation exponent drifts in proportion to capacity and data exponent variances, while Schur-complement reduction separates intrinsic drift from frozen-hyperparameter bias. Across 45 operator spectra, Pythia (70M--12B), Chinchilla (245 runs), and RegMix (17 domains), our estimators achieve 5.89%5.89\% holdout MAPE, cut high-compute allocation regret by up to 76.4%76.4\%, and reduce cross-scale mixture RMSE by 59.4%59.4\% with zero selection regret.

Figures & tables

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Active Budget Allocation for Efficient Scaling Law Estimation via Surrogate-Guided Pruning

    May 17, 2026Viktoria Schram, Markus Hiller, Daniel Beck +1Multi-Fidelity OptimizationScaling Laws

  2. Prescriptive Scaling Laws for Data Constrained Training

    May 2, 2026Justin Lovelace, Christian Belardi, Srivatsa Kundurthy +2Weight DecayLanguage Model Scaling Laws