cs.LGFeb 17, 2026

Prescriptive Scaling Reveals the Evolution of Language Model Capabilities

Authors: Hanlin Zhang, Jikai Jin, Vasilis Syrgkanis, Sham Kakade

Organizations: Harvard University · Stanford University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Machine learning model performance improvements tend to arise from competition and application. For deployment, we consider prescriptive scaling laws: given a pre-training compute budget, what downstream accuracy is attainable with contemporary post-training practice, and how stable is that mapping as the field evolves? Using large-scale observational evaluations with 5k existing and 2k newly evaluated model checkpoints spanning 2022-2026 across six benchmarks, we estimate capability boundaries, high conditional quantiles of benchmark scores as a function of log pre-training FLOPs, via smoothed quantile regression with a monotone, saturating sigmoid parameterization. We validate temporal reliability by fitting on earlier model generations and evaluating on later releases: across four of six tasks, the out-of-distribution coverage error remains below 2%, while math reasoning exhibits a consistently advancing boundary over time. For instance, at a budget of 10^24 FLOPs, the estimated attainable accuracies are 0.83 on IFEval and 0.54 on MATH Lvl 5. We then extend our approach to analyze task-dependent saturation and to probe contamination-related shifts on math reasoning tasks. Finally, we introduce a balanced I-optimal sampling algorithm that recovers near-full-data frontiers using roughly 20% of the parameter-count-weighted evaluation budget, as low as 5% on some tasks, while maintaining comparable calibration. Together, our work releases Proteus-2k, the latest model performance evaluation dataset, and introduces a practical methodology for translating compute budgets into reliable performance expectations and for monitoring when capability boundaries shift across time.

Figures & tables

BenchmarkIFEvalBBHMATH Lvl 5GPQAMUSRMMLU-PRO
Acc. @ 102410^{24} FLOPs0.8280.7000.5390.4240.5350.563
Table 1: Estimated attainable accuracies predicted by the no-split 0.98-quantile sigmoid boundaries at 102410^{24} FLOPs.
Figure 1: Sigmoid capability boundaries across time. In each subfigure, points correspond to post-trained models (x-axis: base-model pre-training compute; y-axis: benchmark score). We compare sigmoid fits across consecutive periods (Pt,Pt+1)(\mathcal{P}_{t},\mathcal{P}_{t+1}) for t=1,2,3t=1,2,3 , visualizing both (i) the boundary fit on Pt\mathcal{P}_{t} and (ii) the boundary fit on Pt+1\mathcal{P}_{t+1} to illustrate boundary shift.
 Pinball lossCalibration error
EstimatorIDOODIDOOD
Constant5.35×10−35.35\times 10^{-3}6.23×10−36.23\times 10^{-3}4.12×10−24.12\times 10^{-2}3.60×10−23.60\times 10^{-2}
Binwise4.01×10−34.01\times 10^{-3}5.00×10−35.00\times 10^{-3}1.66×10−21.66\times 10^{-2}2.81×10−22.81\times 10^{-2}
I-spline4.00×10−34.00\times 10^{-3}4.92×10−34.92\times 10^{-3}1.83×10−21.83\times 10^{-2}2.41×10−22.41\times 10^{-2}
Sigmoid4.08×10−34.08\times 10^{-3}4.93×10−34.93\times 10^{-3}1.84×10−21.84\times 10^{-2}2.21×10−22.21\times 10^{-2}
Table 2: Results averaged over three rolling splits t=1,2,3t=1,2,3 and six tasks. Values are absolute pinball loss and calibration error.
Figure 2: Temporal drift and the stability of knowledge-intensive capabilities. Left: coverage error τ^−τ\hat{\tau}-\tau . Right: pinball loss ρτ\rho_{\tau} . Both fit on Pt\mathcal{P}_{t} and evaluate on Pt+1\mathcal{P}_{t+1} .
Figure 3: Pre-training vs. post-training scaling laws. Panels (a) and (b) compare capability boundaries for pretrained and post-trained models. Panel (c) compares how frequently pretrained accuracies and post-trained capability boundaries violate monotonicity in compute.
Figure 4: MATH Lvl 5 : evaluation on newly released open-weight models. (a) and (b): fitted sigmoid capability boundaries on leaderboard models (red) and newly evaluated models (blue) in periods Pt\mathcal{P}_{t} for t∈{3,4}t\in\{3,4\} . (c) and (d): on Proteus-2k , fitted capability boundary on leaderboard models in period P4\mathcal{P}_{4} (red) and on models released after the retirement of the Open LLM Leaderboard. (c) contains models from old base model families ( i.e., base models that already exist in the leaderboard), while (d) contains new model families.
Figure 5: Performance of balanced I-optimal design as a function of budget parameter α\alpha , averaged over t=1,2,3t=1,2,3 .
Figure 6: Task-dependent saturation on Open LLM Leaderboard v2. “Dominated large” marks large ( >13B>13B ) models whose task score is below the cumulative best score achieved by small models up to that date (the small-model boundary). MMLU-Pro appears less saturated than MATH Lvl 5.
Benchmarkβ^\hat{\beta}β^+θ^\hat{\beta}+\hat{\theta}q^0.98(13B,g=0)\hat{q}_{0.98}(13\mathrm{B},g{=}0)q^0.98(13B,g=1)\hat{q}_{0.98}(13\mathrm{B},g{=}1)
MATH Lvl 50.250.760.030.94
MMLU-Pro0.000.470.150.52
Table 3: Key fitted statistics from the size–time boundary model. Here g=0g{=}0 and g=1g{=}1 denote the earliest and latest times in the pooled dataset, respectively; β^+θ^\hat{\beta}+\hat{\theta} is the late-period marginal size effect on the boundary.
Figure 7: Scaling laws for frontier models.
Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Visualizing the pinball loss and its effect on boundary fitting. (a) The true pinball loss and its smoothed approximation for several quantile levels τ\tau ; smoothing (softplus) affects only a narrow band around r=0r=0 . (b) The corresponding gradients, showing that positive residuals (boundary below data) produce much larger gradients than negative residuals. (c) On synthetic data, the pinball boundary closely tracks the upper envelope, whereas the squared-loss boundary is pulled toward the bulk of underperforming points.
Figure 9: Summary diagnostics connecting pretraining compute to post-training capability. These plots quantify (i) how far pretrained checkpoints lie below the post-trained τ\tau -capability boundary, and (ii) how much post-training can lift performance at fixed compute.
Figure 10: Pretrained vs. post-trained overlays across all tasks. See Figure 3 for the legend description.
Figure 11: Sigmoid capability boundaries over time. Complementary of Figure 1 .
Base model≤\leq 2024-062024-07..2024-092024-10..2024-122025-01..2025-03
gemma-1-2b13–––
gemma-2-27b––14–
gemma-2-2b–1312–
gemma-2-9b–276521
llama-2-13b15–––
llama-2-70b14––10
Appendix
Table 15
Figure 12: Bin-wise coverage across periods (supplement to Figure 2 (Left)). Bin-wise signed OOD coverage error within log-compute bins when fitting on Pt\mathcal{P}_{t} and validating on Pt+1\mathcal{P}_{t+1} , for t=1,2,3t=1,2,3 .
Figure 13: Bin-wise pinball loss across periods (supplement to Figure 2 (Right)). Bin-wise OOD pinball loss within log-compute bins when fitting on Pt\mathcal{P}_{t} and validating on Pt+1\mathcal{P}_{t+1} , shown for MMLU-Pro , MATH Lvl 5 , and IFEval with t=1,2,3t=1,2,3 .
Figure 14: In-distribution and out-of-distribution coverage error across log-compute bins for train (Pt)(\mathcal{P}_{t}) and validation (Pt+1)(\mathcal{P}_{t+1}) , t=1,2,3t=1,2,3 across six Open LLM Leaderboard tasks.
Figure 15: In-distribution and out-of-distribution pinball loss across log-compute bins for train (Pt)(\mathcal{P}_{t}) and validation (Pt+1)(\mathcal{P}_{t+1}) , t=1,2,3t=1,2,3 across six Open LLM Leaderboard tasks.
Figure 16: Sigmoid Capability Boundaries as functions of model size.
Figure 17: Comparison of sigmoid performance boundaries for (Pt)(\mathcal{P}_{t}) and (Pt+1)(\mathcal{P}_{t+1}) , 1≤t≤31\leq t\leq 3 across different tasks.
Figure 18: Comparison of sigmoid performance boundaries within each Pt\mathcal{P}_{t} for old and held-out new models.
Figure 19: Comparison of sigmoid performance boundaries within each Pt\mathcal{P}_{t} for old and held-out new models
Base model family# models
Mistral-7B-v0.3 ( Jiang et al., 2023 )143
Qwen2.5-1.5B ( Team et al., 2024b )124
Llama-3.2-3B ( Grattafiori et al., 2024 )100
gemma-2-9b ( Team et al., 2024a )97
Qwen3-4B-Base ( Yang et al., 2025 )90
Qwen2.5-3B ( Team et al., 2024b )89
Appendix
Table 5: Model counts by base model family among the newly evaluated models.
Figure 20: In-period tuning has negligible effect on cross-temporal generalization. The fixed default (κ,λ)=(50,10−3)(\kappa,\lambda)=(50,10^{-3}) used throughout the paper is competitive with values selected by random in-period splits. Left: κ\kappa and log⁡10λ\log_{10}\lambda selected by in-period tuning (aggregated across random splits). Right: OOD metrics on Pt+1\mathcal{P}_{t+1} : fixed default vs. in-period tuned (κ,λ)(\kappa,\lambda) .
Figure 21: OOD pinball loss under a (κ,λ)(\kappa,\lambda) sweep. Lower is better. Results are shown for two tasks and two representative period splits ( t=1t{=}1 and t=3t{=}3 ).
Figure 22: OOD absolute coverage error ∣τ^−τ∣\lvert\hat{\tau}-\tau\rvert under a (κ,λ)(\kappa,\lambda) sweep. Lower is better. Coverage is most sensitive to overly large λ\lambda , while κ\kappa has a weaker effect once it is in a moderate range.
Figure 23: Sigmoid scaling law for frontier models. Evaluation data is publicly available from Artificial Analysis and lifearchitect.ai .
Figure 24: Comparison of sigmoid performance boundaries across different time periods Pt\mathcal{P}_{t} Open LLM Leaderboard v1.
Figure 25: The scaling of different principal components.
Figure 26: Open LLM Leaderboard v2: saturation diagnostics by task.
Figure 27: Open LLM Leaderboard v1: saturation diagnostics by task.
Figure 28: In-sample and out-of-sample coverage calibration error on period t+1t+1 as a function of budget parameter α\alpha when the boundary is estimated using balanced I-optimal design on period tt . Curves correspond to different evaluation tasks.
Figure 29: In-sample and out-of-sample pinball loss on period t+1t+1 as a function of budget parameter α\alpha when the boundary is estimated using balanced I-optimal design on period tt . Curves correspond to different evaluation tasks.
More in this topic

Language Model Performance Prediction

See all 14 papers

Explore similar work

CardsList

  1. How Inference Compute Shapes Frontier LLM Evaluation

    Jun 16, 2026Jessica McFadyen, Ole Jorgensen, Harry Coppock +2LLM EvaluationLanguage Model Generation Evaluation

    AI Security Institute, London, UK · University of Oxford, Oxford, UK · Harvard University, Cambridge, MA, USA

  2. ScAn-Bench: Evaluating Scaling Analysis Methodology

    Sep 28, 2026Artin Sermaxhaj, Nastaran Alipour, Donat Sinani +4VLM EvaluationLLM Evaluation

    University of Freiburg · Zuse School ELIZA · Juelich Supercomputing Center (JSC), Research Center Juelich (FZJ)

  3. Brevity Constraints Reverse Performance Hierarchies in Language Models

    Mar 11, 2026MD Azizul HakimLLM EvaluationLanguage Model Scaling

    Department of Computer Science Bangladesh Sweden Polytechnic Institute Chattogram, Bangladesh