Ambiguous Fits: The Rashomon Effect and Challenges in Scaling Law Estimation

Bruno Mlodozeniec; Spencer Frei

cs.LGOct 25, 2026

Ambiguous Fits: The Rashomon Effect and Challenges in Scaling Law Estimation

Authors: Bruno Mlodozeniec, Spencer Frei

Organizations: University of Cambridge · Google DeepMind

Abstract

Estimating neural scaling laws involves an end-to-end pipeline: designing a scaling recipe, training models across scales, evaluating models on a target metric, and deploying an algorithm to fit a scaling law to the resulting data. This final fitting stage is often treated as routine: standard procedures minimize a non-convex fitting loss using local optimization from a grid of initializations, and return the best solution found. Yet such heuristic searches can fail in two distinct ways: they offer no guarantee of returning a nearly optimal solution, and they obscure whether fundamentally different parameter choices fit the data nearly as well. Indeed, we show that standard scaling law fitting is subject to a Rashomon effect (Breiman, 2001): the sublevel sets of near-minimizers correspond to broad parameter regions whose extrapolations diverge substantially at scale. We resolve these issues with a deterministic branch-and-bound algorithm that guarantees nearly optimal fitting loss and exhaustively maps the near-minimizers to the implied range of scaling curves.

Figures & tables

Figure 1: The Rashomon Effect in Scaling Law Fitting. Synthetic scaling ladder data on which two fits with widely different extrapolations achieve nearly identical fitting losses. The yellow band, computed with our certified prediction-bound algorithm ( Section 4.4 ), contains the predicted loss of every fit whose fitting loss is within

1\%

of the optimum. Details are in Appendix C .

Figure 2: The Rashomon Effect and Sublevel Prediction Envelopes Across Scaling Ladders. We run Chinchilla Approach 3 on four different scaling ladders taken from the literature (see Section B.1 ) with different initialization grid densities (following the standard heuristic parameter ranges $\log E\in[-1,1]$ , $\log A_{1},\log A_{2}\in[0,25]$ , and $\alpha_{1},\alpha_{2}\in[0,2]$ ), where a grid with $5^{5}$ initialization points is closest to the Hoffmann et al. (2022) default. We record the attained fitting loss $L_{\mathrm{chin},K}$ at each density $K$ , search for alternative parameter configurations in the sublevel set achieving $L_{\texttt{fit}}(\theta)\leq L_{\mathrm{chin},K}$ , and plot the resulting prediction envelopes along a canonical $20$ tokens-per-parameter ( $D=20N$ ) slice. The extreme fits are found by local search ( Appendix C ), so the true envelopes can only be wider. With the standard Chinchilla approach at a $5^{5}$ grid initialization (gold band), we observe a pronounced Rashomon effect: fits in the sublevel set that match the training data just as well as the returned fit yield a compute efficiency difference between $14\times$ and $146\times$ for a target loss. Horizontal segments indicate this compute efficiency range. While increasing grid density $K$ generally shrinks the sublevel envelope ( Section 3 ), heuristic optimization provides no certificate of how close the returned fit is to global optimality or how wide the true envelope remains.

Figure 3: Visualization of a branch-and-bound step. This figure illustrates a single iteration of the certified search Algorithm 1 in the monotonic coordinate space $(E,Ax_{\min}^{-\alpha},e^{-\alpha})$ . (Top Row) A parent bounding box with the lowest loss lower bound is selected for splitting. Background heatmaps show the guaranteed loss lower bound at this step. (Top Row Right) The corresponding scaling law predictions for parameters inside the box to be split. (Bottom Row) The algorithm branches the box along the exponent dimension $e^{-\alpha}$ . This produces two new boxes: Child 1 and Child 2. (Bottom Row Right) The corresponding scaling law predictions for parameters inside the two new child boxes. The algorithm can discard Child 1 because its fitting-loss lower bound over the training grid strictly exceeds the best loss found so far, recursively narrowing the search space around a true optimum.

Figure 4: Certified suboptimality of Chinchilla Approach 3 vs. Branch and Bound Algorithm 1 against wall-clock runtime. Comparison of certified branch-and-bound fitting (B&B; red) and Chinchilla Approach 3 (blue) on four 5-parameter scaling-law datasets. For a solution returned by a fitting method with fitting loss

L

, shaded regions show the guaranteed interval within which the “suboptimality”

L-\min_{\theta}L_{\texttt{fit}}(\theta;{\mathcal{D}})

must lie (as estimated using the tightest bounds obtained from the branch-and-bound search). Chinchilla points give the best candidate obtained using different

K^{5}

initialization-grid densities for

K=1,\dots,8

. Runtimes are wall-clock measurements from sequential implementations: C++ for B&B and SciPy L-BFGS for Chinchilla Approach 3. The runtime comparison is only illustrative as the two implementations have not been optimized to an equal extent.

Figure 5: Sublevel set cover and prediction envelope from certified search. Illustration of the output of the sublevel set cover algorithm ( Algorithm 2 ) on a synthetic single-variable scaling law $L(x;\theta)=E+Ax^{-\alpha}$ , fitted with the log-space Huber loss of Section 2.2 ( $\delta=10^{-3}$ ), at inner and outer fitting-loss tolerances $\epsilon/2=10^{-5}$ and $\epsilon=2\times 10^{-5}$ . The certified minimum fitting loss is approximately $2.3\times 10^{-7}$ , so $\epsilon$ is roughly 90 times the optimum. The algorithm returns a collection of boxes $\mathcal{B}$ in $(E,C,r)$ coordinates. Its mapped union $\mathcal{C}=\phi^{-1}(\bigcup_{B\in\mathcal{B}}B)$ satisfies $\mathcal{S}_{\epsilon/2}\subseteq\mathcal{C}\subseteq\mathcal{S}_{\epsilon}$ in the original parameters. (Left three panels) Two-dimensional projections of $\mathcal{C}$ onto pairs of original coordinates $(E,A,\alpha)$ , shown as light-shaded regions, and cross-sections with the omitted coordinate fixed at its ground-truth value, shown as dark-shaded regions. The diamonds mark the ground-truth parameters $\theta_{\mathrm{true}}$ . (Right panel) The corresponding prediction envelope: the shaded band shows the range of scaling curves $L(x;\theta)$ achievable by any $\theta\in\mathcal{C}$ .

\sup\mathcal{L}_{x^{\ast},\epsilon}\leq\overline{L}_{x^{\ast}}\leq\sup\mathcal{L}_{x^{\ast},\epsilon+\eta}\qquad\text{where }\mathcal{L}_{x^{\ast},\epsilon}\coloneqq\{L(x^{\ast};u):u\in\mathcal{S}^{\phi}_{\epsilon}\}.

Algorithm 3 Certified upper prediction bound with viability slack

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

\sup\mathcal{L}_{x^{\ast},\epsilon}\leq\overline{L}_{x^{\ast}}\leq\sup\mathcal{L}_{x^{\ast},\epsilon+\eta}\qquad\text{where }\mathcal{L}_{x^{\ast},\epsilon}\coloneqq\{L(x^{\ast};u):u\in\mathcal{S}^{\phi}_{\epsilon}\}.

Appendix

Algorithm 4 Certified prediction upper bound at an arbitrary target scale

Neural Scaling Laws

Neural scaling laws describe how model performance changes with training data, compute, model capacity, and other resources.

See all 83 papers

Amortizing Scaling Law Construction Costs
Sep 4, 2026Abhash Kumar Jha, Diana Alexandra Onuţu, Neeratyoy Mallik +6Language Model Scaling Laws Bayesian Optimization
ELLIS Institute Tübingen · OpenEuroLLM · Eindhoven University of Technology +4
On the Invariance and Generality of Neural Scaling Laws
May 8, 2026Xing Han, Liu Ziyin, Suchi Saria +1Language Model Scaling Laws Cross-Domain Generalization
Johns Hopkins University Baltimore, MD 21218, USA · NTT Research, MIT Cambridge, MA 02139, USA · Johns Hopkins University, Bayesian Health Baltimore, MD 21218, USA +1
Tokens-per-Parameter Coverage Is Critical for Robust LLM Scaling Law Extrapolation
May 8, 2026Joshua Shay Kricheli, Alexander Lawrence Reid, Soumajyoti Sarkar +2Parameter Estimation Language Model Scaling Laws
Syracuse University · Amazon AGI Foundations

Sep 4, 2026J/K move · Enter open · S save

Amortizing Scaling Law Construction Costs

Abhash Kumar Jha, Diana Alexandra Onuţu, Neeratyoy Mallik, Swagatam Haldar+5

ELLIS Institute Tübingen · OpenEuroLLM · Eindhoven University of Technology+4

Language Model Scaling Laws Bayesian Optimization Hyperparameter Optimization

Open paper AskPDF arXiv

Scaling laws guide the design choices for training large foundation models, but deriving them involves training an exhaustive grid over hyperparameters, token budgets, and parameter counts, which is computationally expensive. Fitting a scaling law, however, only requires the best-loss frontier across compute scales, discarding most of the trained configurations. We propose a framework for efficient scaling law construction that formulates data collection as a Bayesian optimization problem, and introduce metrics for comparing scaling law fitting methods under constrained compute budgets. We find that progressively expanding the compute budget during acquisition, mirroring the compute-ordered evaluation of configurations in practice, substantially improves recovery efficiency. Augmenting the observed configurations with surrogate-fantasized evaluations then recovers the broader experimental grid, allowing accurate scaling law fitting without training every configuration. Together, these can closely match scaling law fits over a full dense grid at computational savings of up to

10\text{--}100\times

Abstract

Neural Scaling Laws

Explore similar work

Amortizing Scaling Law Construction Costs

On the Invariance and Generality of Neural Scaling Laws

Tokens-per-Parameter Coverage Is Critical for Robust LLM Scaling Law Extrapolation