Estimating neural scaling laws involves an end-to-end pipeline: designing a scaling recipe, training models across scales, evaluating models on a target metric, and deploying an algorithm to fit a scaling law to the resulting data. This final fitting stage is often treated as routine: standard procedures minimize a non-convex fitting loss using local optimization from a grid of initializations, and return the best solution found. Yet such heuristic searches can fail in two distinct ways: they offer no guarantee of returning a nearly optimal solution, and they obscure whether fundamentally different parameter choices fit the data nearly as well. Indeed, we show that standard scaling law fitting is subject to a Rashomon effect (Breiman, 2001): the sublevel sets of near-minimizers correspond to broad parameter regions whose extrapolations diverge substantially at scale. We resolve these issues with a deterministic branch-and-bound algorithm that guarantees nearly optimal fitting loss and exhaustively maps the near-minimizers to the implied range of scaling curves.
Figures & tables
Figure 1: The Rashomon Effect in Scaling Law Fitting. Synthetic scaling ladder data on which two fits with widely different extrapolations achieve nearly identical fitting losses. The yellow band, computed with our certified prediction-bound algorithm ( Section 4.4 ), contains the predicted loss of every fit whose fitting loss is within 1% of the optimum. Details are in Appendix C .
Figure 2: The Rashomon Effect and Sublevel Prediction Envelopes Across Scaling Ladders. We run Chinchilla Approach 3 on four different scaling ladders taken from the literature (see Section B.1 ) with different initialization grid densities (following the standard heuristic parameter ranges logE∈[−1,1] , logA1,logA2∈[0,25] , and α1,α2∈[0,2] ), where a grid with 55 initialization points is closest to the Hoffmann et al. (2022) default. We record the attained fitting loss Lchin,K at each density K , search for alternative parameter configurations in the sublevel set achieving Lfit(θ)≤Lchin,K , and plot the resulting prediction envelopes along a canonical 20 tokens-per-parameter ( D=20N ) slice. The extreme fits are found by local search ( Appendix C ), so the true envelopes can only be wider. With the standard Chinchilla approach at a 55 grid initialization (gold band), we observe a pronounced Rashomon effect: fits in the sublevel set that match the training data just as well as the returned fit yield a compute efficiency difference between 14× and 146× for a target loss. Horizontal segments indicate this compute efficiency range. While increasing grid density K generally shrinks the sublevel envelope ( Section 3 ), heuristic optimization provides no certificate of how close the returned fit is to global optimality or how wide the true envelope remains.
Figure 3: Visualization of a branch-and-bound step. This figure illustrates a single iteration of the certified search Algorithm 1 in the monotonic coordinate space (E,Axmin−α,e−α) . (Top Row) A parent bounding box with the lowest loss lower bound is selected for splitting. Background heatmaps show the guaranteed loss lower bound at this step. (Top Row Right) The corresponding scaling law predictions for parameters inside the box to be split. (Bottom Row) The algorithm branches the box along the exponent dimension e−α . This produces two new boxes: Child 1 and Child 2. (Bottom Row Right) The corresponding scaling law predictions for parameters inside the two new child boxes. The algorithm can discard Child 1 because its fitting-loss lower bound over the training grid strictly exceeds the best loss found so far, recursively narrowing the search space around a true optimum.
Figure 4: Certified suboptimality of Chinchilla Approach 3 vs. Branch and Bound Algorithm 1 against wall-clock runtime. Comparison of certified branch-and-bound fitting (B&B; red) and Chinchilla Approach 3 (blue) on four 5-parameter scaling-law datasets. For a solution returned by a fitting method with fitting loss L , shaded regions show the guaranteed interval within which the “suboptimality” L−minθLfit(θ;D) must lie (as estimated using the tightest bounds obtained from the branch-and-bound search). Chinchilla points give the best candidate obtained using different K5 initialization-grid densities for K=1,…,8 . Runtimes are wall-clock measurements from sequential implementations: C++ for B&B and SciPy L-BFGS for Chinchilla Approach 3. The runtime comparison is only illustrative as the two implementations have not been optimized to an equal extent.
Figure 5: Sublevel set cover and prediction envelope from certified search. Illustration of the output of the sublevel set cover algorithm ( Algorithm 2 ) on a synthetic single-variable scaling law L(x;θ)=E+Ax−α , fitted with the log-space Huber loss of Section 2.2 ( δ=10−3 ), at inner and outer fitting-loss tolerances ϵ/2=10−5 and ϵ=2×10−5 . The certified minimum fitting loss is approximately 2.3×10−7 , so ϵ is roughly 90 times the optimum. The algorithm returns a collection of boxes B in (E,C,r) coordinates. Its mapped union C=ϕ−1(⋃B∈BB) satisfies Sϵ/2⊆C⊆Sϵ in the original parameters. (Left three panels) Two-dimensional projections of C onto pairs of original coordinates (E,A,α) , shown as light-shaded regions, and cross-sections with the omitted coordinate fixed at its ground-truth value, shown as dark-shaded regions. The diamonds mark the ground-truth parameters θtrue . (Right panel) The corresponding prediction envelope: the shaded band shows the range of scaling curves L(x;θ) achievable by any θ∈C .
Scaling laws guide the design choices for training large foundation models, but deriving them involves training an exhaustive grid over hyperparameters, token budgets, and parameter counts, which is computationally expensive. Fitting a scaling law, however, only requires the best-loss frontier across compute scales, discarding most of the trained configurations. We propose a framework for efficient scaling law construction that formulates data collection as a Bayesian optimization problem, and introduce metrics for comparing scaling law fitting methods under constrained compute budgets. We find that progressively expanding the compute budget during acquisition, mirroring the compute-ordered evaluation of configurations in practice, substantially improves recovery efficiency. Augmenting the observed configurations with surrogate-fantasized evaluations then recovers the broader experimental grid, allowing accurate scaling law fitting without training every configuration. Together, these can closely match scaling law fits over a full dense grid at computational savings of up to 10–100×.
Abhash Kumar Jha, Diana Alexandra Onuţu, Neeratyoy Mallik +6
ELLIS Institute Tübingen · OpenEuroLLM · Eindhoven University of Technology +4
Neural scaling laws establish a predictable relationship between model performance and data or compute, offering crucial guidance for resource allocation in new domains and tasks. Yet such laws are most needed precisely where they are hardest to obtain: fitting one for a new model task pair demands expensive sweeps that typically exhaust the very compute budget the law is meant to economize. This paper poses the research question of how to develop generalizable scaling laws: laws fit once on a well-resourced source domain and reliably transported to new domains where running a full sweep is infeasible, which requires a fundamental understanding of when and why scaling properties change. We address this by identifying the right invariants: scaling laws are preserved under bijective (information-preserving) transformations of the data and modified in predictable, information-theoretically grounded ways under non-bijective transformations that lower its information resolution ρ: a single axis along which a law fit in one domain can be transported to another. We validate this across language, vision, and speech, and demonstrate two cross-domain applications: predicting scaling for language models trained on electronic health records from laws fit on general text, and predicting time-series classification scaling under varying levels of noise injection, recovering the data-scaling exponents to within 3% error.
Xing Han, Liu Ziyin, Suchi Saria +1
Johns Hopkins University Baltimore, MD 21218, USA · NTT Research, MIT Cambridge, MA 02139, USA · Johns Hopkins University, Bayesian Health Baltimore, MD 21218, USA +1
Neural scaling laws approximate a language model's loss as a power-law function of parameter count N and token count D. Following Chinchilla-style compute-optimal training, many studies fit scaling laws from runs performed under a fixed tokens-per-parameter (TPP) ratio k and set D=kN. We show that this collinear design, combined with the empirically common near-equality of the exponents governing N and D, induces an inherent ill-conditioning in the Gauss-Newton least-squares problem: the condition number of the design grows as the inverse square of the gap between the N and D-exponents. The scale coefficients become practically unidentifiable, with confidence intervals inflating by an order of magnitude or more, yielding a ``sloppy'' model whose extrapolations degrade sharply off the training ray. We prove this for four scaling-law formalisms and derive a closed-form TPP-diversity threshold that is necessary and sufficient for well-conditioned estimation. Empirically, non-collinear designs outperform collinear ones on held-out splits with a 97.3% win rate across four laws, five corpora, multiple floating point precision modes. We further show the degeneracy is rooted in Jacobian geometry and is not an artifact of the loss function: any smooth estimation objective whose curvature involves the Jacobian inherits the same ill-conditioning.
Joshua Shay Kricheli, Alexander Lawrence Reid, Soumajyoti Sarkar +2