cs.LGFeb 17, 2026

Prescriptive Scaling Reveals the Evolution of Language Model Capabilities

Authors: Hanlin Zhang, Jikai Jin, Vasilis Syrgkanis, Sham Kakade

Organizations: Harvard University · Stanford University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Machine learning model performance improvements tend to arise from competition and application. For deployment, we consider prescriptive scaling laws: given a pre-training compute budget, what downstream accuracy is attainable with contemporary post-training practice, and how stable is that mapping as the field evolves? Using large-scale observational evaluations with 5k existing and 2k newly evaluated model checkpoints spanning 2022-2026 across six benchmarks, we estimate capability boundaries, high conditional quantiles of benchmark scores as a function of log pre-training FLOPs, via smoothed quantile regression with a monotone, saturating sigmoid parameterization. We validate temporal reliability by fitting on earlier model generations and evaluating on later releases: across four of six tasks, the out-of-distribution coverage error remains below 2%, while math reasoning exhibits a consistently advancing boundary over time. For instance, at a budget of 10^24 FLOPs, the estimated attainable accuracies are 0.83 on IFEval and 0.54 on MATH Lvl 5. We then extend our approach to analyze task-dependent saturation and to probe contamination-related shifts on math reasoning tasks. Finally, we introduce a balanced I-optimal sampling algorithm that recovers near-full-data frontiers using roughly 20% of the parameter-count-weighted evaluation budget, as low as 5% on some tasks, while maintaining comparable calibration. Together, our work releases Proteus-2k, the latest model performance evaluation dataset, and introduces a practical methodology for translating compute budgets into reliable performance expectations and for monitoring when capability boundaries shift across time.

Figures & tables

BenchmarkIFEvalBBHMATH Lvl 5GPQAMUSRMMLU-PRO
Acc. @ 102410^{24} FLOPs0.8280.7000.5390.4240.5350.563
Table 1: Estimated attainable accuracies predicted by the no-split 0.98-quantile sigmoid boundaries at 102410^{24} FLOPs.
Figure 1: Sigmoid capability boundaries across time. In each subfigure, points correspond to post-trained models (x-axis: base-model pre-training compute; y-axis: benchmark score). We compare sigmoid fits across consecutive periods (Pt,Pt+1)(\mathcal{P}_{t},\mathcal{P}_{t+1}) for t=1,2,3t=1,2,3 , visualizing both (i) the boundary fit on Pt\mathcal{P}_{t} and (ii) the boundary fit on Pt+1\mathcal{P}_{t+1} to illustrate boundary shift.
 Pinball lossCalibration error
EstimatorIDOODIDOOD
Constant5.35×10−35.35\times 10^{-3}6.23×10−36.23\times 10^{-3}4.12×10−24.12\times 10^{-2}3.60×10−23.60\times 10^{-2}
Binwise4.01×10−34.01\times 10^{-3}5.00×10−35.00\times 10^{-3}1.66×10−21.66\times 10^{-2}2.81×10−22.81\times 10^{-2}
I-spline4.00×10−34.00\times 10^{-3}4.92×10−34.92\times 10^{-3}1.83×10−21.83\times 10^{-2}2.41×10−22.41\times 10^{-2}
Sigmoid4.08×10−34.08\times 10^{-3}4.93×10−34.93\times 10^{-3}1.84×10−21.84\times 10^{-2}2.21×10−22.21\times 10^{-2}
Table 2: Results averaged over three rolling splits t=1,2,3t=1,2,3 and six tasks. Values are absolute pinball loss and calibration error.
Figure 2: Temporal drift and the stability of knowledge-intensive capabilities. Left: coverage error τ^−τ\hat{\tau}-\tau . Right: pinball loss ρτ\rho_{\tau} . Both fit on Pt\mathcal{P}_{t} and evaluate on Pt+1\mathcal{P}_{t+1} .
Figure 3: Pre-training vs. post-training scaling laws. Panels (a) and (b) compare capability boundaries for pretrained and post-trained models. Panel (c) compares how frequently pretrained accuracies and post-trained capability boundaries violate monotonicity in compute.
Figure 4: MATH Lvl 5 : evaluation on newly released open-weight models. (a) and (b): fitted sigmoid capability boundaries on leaderboard models (red) and newly evaluated models (blue) in periods Pt\mathcal{P}_{t} for t∈{3,4}t\in\{3,4\} . (c) and (d): on Proteus-2k , fitted capability boundary on leaderboard models in period P4\mathcal{P}_{4} (red) and on models released after the retirement of the Open LLM Leaderboard. (c) contains models from old base model families ( i.e., base models that already exist in the leaderboard), while (d) contains new model families.
Figure 5: Performance of balanced I-optimal design as a function of budget parameter α\alpha , averaged over t=1,2,3t=1,2,3 .
Figure 6: Task-dependent saturation on Open LLM Leaderboard v2. “Dominated large” marks large ( >13B>13B ) models whose task score is below the cumulative best score achieved by small models up to that date (the small-model boundary). MMLU-Pro appears less saturated than MATH Lvl 5.
Benchmarkβ^\hat{\beta}β^+θ^\hat{\beta}+\hat{\theta}q^0.98(13B,g=0)\hat{q}_{0.98}(13\mathrm{B},g{=}0)q^0.98(13B,g=1)\hat{q}_{0.98}(13\mathrm{B},g{=}1)
MATH Lvl 50.250.760.030.94
MMLU-Pro0.000.470.150.52
Table 3: Key fitted statistics from the size–time boundary model. Here g=0g{=}0 and g=1g{=}1 denote the earliest and latest times in the pooled dataset, respectively; β^+θ^\hat{\beta}+\hat{\theta} is the late-period marginal size effect on the boundary.
Figure 7: Scaling laws for frontier models.
Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8: Visualizing the pinball loss and its effect on boundary fitting. (a) The true pinball loss and its smoothed approximation for several quantile levels τ\tau ; smoothing (softplus) affects only a narrow band around r=0r=0 . (b) The corresponding gradients, showing that positive residuals (boundary below data) produce much larger gradients than negative residuals. (c) On synthetic data, the pinball boundary closely tracks the upper envelope, whereas the squared-loss boundary is pulled toward the bulk of underperforming points.
Figure 9: Summary diagnostics connecting pretraining compute to post-training capability. These plots quantify (i) how far pretrained checkpoints lie below the post-trained τ\tau -capability boundary, and (ii) how much post-training can lift performance at fixed compute.
Figure 10: Pretrained vs. post-trained overlays across all tasks. See Figure 3 for the legend description.
Figure 11: Sigmoid capability boundaries over time. Complementary of Figure 1 .
Base model≤\leq 2024-062024-07..2024-092024-10..2024-122025-01..2025-03
gemma-1-2b13–––
gemma-2-27b––14–
gemma-2-2b–1312–
gemma-2-9b–276521
llama-2-13b15–––
llama-2-70b14––10
Appendix
Table 15
Figure 12: Bin-wise coverage across periods (supplement to Figure 2 (Left)). Bin-wise signed OOD coverage error within log-compute bins when fitting on Pt\mathcal{P}_{t} and validating on Pt+1\mathcal{P}_{t+1} , for t=1,2,3t=1,2,3 .
Figure 13: Bin-wise pinball loss across periods (supplement to Figure 2 (Right)). Bin-wise OOD pinball loss within log-compute bins when fitting on Pt\mathcal{P}_{t} and validating on Pt+1\mathcal{P}_{t+1} , shown for MMLU-Pro , MATH Lvl 5 , and IFEval with t=1,2,3t=1,2,3 .
Figure 14: In-distribution and out-of-distribution coverage error across log-compute bins for train (Pt)(\mathcal{P}_{t}) and validation (Pt+1)(\mathcal{P}_{t+1}) , t=1,2,3t=1,2,3 across six Open LLM Leaderboard tasks.
Figure 15: In-distribution and out-of-distribution pinball loss across log-compute bins for train (Pt)(\mathcal{P}_{t}) and validation (Pt+1)(\mathcal{P}_{t+1}) , t=1,2,3t=1,2,3 across six Open LLM Leaderboard tasks.
Figure 16: Sigmoid Capability Boundaries as functions of model size.
Figure 17: Comparison of sigmoid performance boundaries for (Pt)(\mathcal{P}_{t}) and (Pt+1)(\mathcal{P}_{t+1}) , 1≤t≤31\leq t\leq 3 across different tasks.
Figure 18: Comparison of sigmoid performance boundaries within each Pt\mathcal{P}_{t} for old and held-out new models.
Figure 19: Comparison of sigmoid performance boundaries within each Pt\mathcal{P}_{t} for old and held-out new models
Base model family# models
Mistral-7B-v0.3 ( Jiang et al., 2023 )143
Qwen2.5-1.5B ( Team et al., 2024b )124
Llama-3.2-3B ( Grattafiori et al., 2024 )100
gemma-2-9b ( Team et al., 2024a )97
Qwen3-4B-Base ( Yang et al., 2025 )90
Qwen2.5-3B ( Team et al., 2024b )89
Appendix
Table 5: Model counts by base model family among the newly evaluated models.
Figure 20: In-period tuning has negligible effect on cross-temporal generalization. The fixed default (κ,λ)=(50,10−3)(\kappa,\lambda)=(50,10^{-3}) used throughout the paper is competitive with values selected by random in-period splits. Left: κ\kappa and log⁡10λ\log_{10}\lambda selected by in-period tuning (aggregated across random splits). Right: OOD metrics on Pt+1\mathcal{P}_{t+1} : fixed default vs. in-period tuned (κ,λ)(\kappa,\lambda) .
Figure 21: OOD pinball loss under a (κ,λ)(\kappa,\lambda) sweep. Lower is better. Results are shown for two tasks and two representative period splits ( t=1t{=}1 and t=3t{=}3 ).
Figure 22: OOD absolute coverage error ∣τ^−τ∣\lvert\hat{\tau}-\tau\rvert under a (κ,λ)(\kappa,\lambda) sweep. Lower is better. Coverage is most sensitive to overly large λ\lambda , while κ\kappa has a weaker effect once it is in a moderate range.
Figure 23: Sigmoid scaling law for frontier models. Evaluation data is publicly available from Artificial Analysis and lifearchitect.ai .
Figure 24: Comparison of sigmoid performance boundaries across different time periods Pt\mathcal{P}_{t} Open LLM Leaderboard v1.
Figure 25: The scaling of different principal components.
Figure 26: Open LLM Leaderboard v2: saturation diagnostics by task.
Figure 27: Open LLM Leaderboard v1: saturation diagnostics by task.
Figure 28: In-sample and out-of-sample coverage calibration error on period t+1t+1 as a function of budget parameter α\alpha when the boundary is estimated using balanced I-optimal design on period tt . Curves correspond to different evaluation tasks.
Figure 29: In-sample and out-of-sample pinball loss on period t+1t+1 as a function of budget parameter α\alpha when the boundary is estimated using balanced I-optimal design on period tt . Curves correspond to different evaluation tasks.
More in this topic

Language Model Performance Prediction

See all 14 papers

Explore similar work

CardsList
Jun 16, 2026

How Inference Compute Shapes Frontier LLM Evaluation

AI evaluations are shifting toward harder tasks that benefit from longer trajectories involving tool use and iterative problem solving. As a result, performance is increasingly sensitive to the amount and allocation of compute available at test time ("inference compute"). Yet many evaluations still report performance at a single restrictive budget, meaning that low scores may reflect the evaluation setup rather than the model's underlying capability. To test this, we evaluate up to 12 frontier language models on seven challenging benchmarks spanning software engineering, mathematics, medicine, and cybersecurity. We use a controlled setup combining three simple inference-scaling interventions: larger token budgets, context compaction, and repeated submission attempts, guided either by the model itself or by minimal correctness feedback. We find three main results. First, larger token budgets substantially improve performance on benchmarks across multiple domains, including cybersecurity, FrontierMath, Humanity's Last Exam, and TerminalBench. Second, fixed-budget evaluations can increasingly understate frontier capability as models advance. Newer models reach higher performance at large budgets, where they unlock harder tasks and solve them more reliably. Third, benchmarks differ in which inference-scaling methods help most: repeated submission broadly improves performance, but the value of larger token budgets, external feedback, and parallel attempts varies by benchmark. Overall, our results show that benchmark scores are protocol-dependent. We therefore argue that evaluations should report capability as a function of inference-time compute, specify protocol choices explicitly, and compare model generations over a large shared compute range at matched budgets, especially in safety- or policy-relevant settings.

Jessica McFadyen, Ole Jorgensen, Harry Coppock +2

AI Security Institute, London, UK · University of Oxford, Oxford, UK · Harvard University, Cambridge, MA, USA

LLM EvaluationLanguage Model Generation EvaluationLLM InferenceInference-Time Scaling
AskPDF
Sep 28, 2026

ScAn-Bench: Evaluating Scaling Analysis Methodology

Recent progress in machine learning is driven by large-scale foundation models, where scaling laws and finding optimal scaling prescriptions for architecture, data, and hyperparameters are key in advancing the state-of-the-art. Therefore, it is surprising that no systematic study evaluates the methodology to obtain scaling laws and prescriptions across different model types. To shed light on this crucial blind spot and facilitate future research, we introduce the surrogate benchmarks ScAn-Bench-LLM and ScAn-Bench-VLM based on 4524 and 8024 checkpoints of language and vision-language model pipelines. On our benchmarks, we perform the first systematic evaluation of both data acquisition and extrapolation methodology for scaling analysis across different data modalities.

Artin Sermaxhaj, Nastaran Alipour, Donat Sinani +4

University of Freiburg · Zuse School ELIZA · Juelich Supercomputing Center (JSC), Research Center Juelich (FZJ)

VLM EvaluationLLM EvaluationLanguage Model Scaling LawsScaling Laws
AskPDF
Mar 11, 2026

Brevity Constraints Reverse Performance Hierarchies in Language Models

Standard evaluation protocols reveal a counterintuitive phenomenon: on 7.7% of benchmark problems spanning five datasets, larger language models underperform smaller ones by 28.4 percentage points despite 10-100x more parameters. Through systematic evaluation of 31 models (0.5B-405B parameters) across 1,485 problems, we identify the mechanism as spontaneous scale-dependent verbosity that introduces errors through overelaboration. Causal intervention experiments demonstrate this reflects correctable prompt design rather than fundamental capability limitations. Constraining large models to produce brief responses improves accuracy by 26 percentage points and reduces performance gaps by up to two-thirds. Most critically, brevity constraints completely reverse performance hierarchies on mathematical reasoning and scientific knowledge benchmarks, with large models achieving 7.7-15.9 percentage point advantages over small models -- direct inversions of the original gaps. These reversals prove large models possess superior latent capabilities that universal prompting masks. We validate findings through three independent contamination tests and demonstrate inverse scaling operates continuously across the full parameter spectrum, with dataset-specific optimal scales ranging from 0.5B to 3.0B parameters. Our results establish that maximizing large model performance requires scale-aware prompt engineering rather than universal evaluation protocols, with immediate implications for deployment: prompt adaptation simultaneously improves accuracy and reduces computational costs.

MD Azizul Hakim

Department of Computer Science Bangladesh Sweden Polytechnic Institute Chattogram, Bangladesh

LLM EvaluationLanguage Model ScalingLanguage Model Scaling Laws
AskPDF