Hinted Scaling Laws

Suze van Adrichem; Luke Bailey; Rohith Kuditipudi; Christine Ye; Neil Band; Ahmed Ahmed; John Duchi; Percy Liang

cs.LGNov 1, 2026

Hinted Scaling Laws

Authors: Suze van Adrichem, Luke Bailey, Rohith Kuditipudi, Christine Ye, Neil Band, Ahmed Ahmed, John Duchi, Percy Liang,

Organizations: Stanford University

Abstract

Scaling laws predict the outcome of large, expensive training runs by extrapolating trends from smaller, cheaper runs. On downstream accuracy evaluations, these predictions fail when the weaker models produced by the cheaper runs score near zero accuracy, because these scores give no trend to extrapolate to larger scales. We introduce hinted scaling laws, which use privileged information to turn a benchmark into a continuum of easier versions so that even the small models score above zero. We find that accuracy rises smoothly and predictably with the hint level, enabling new types of scaling laws that operate below the noise threshold. On the Delphi pretraining ladder, a hinted scaling law fit on models trained with at most $10^{20}$ FLOPs forecasts the GSM8K accuracy of a $10^{23}$ FLOP model (25.9%) to within 3 percentage points; a standard scaling law under-forecasts it by 13 percentage points when fit on the same models and by 12 even with models up to $10^{22}$ FLOPs in the fit. Across both this pretraining ladder and a collection of post-trained model families, hinted scaling laws reduce the median forecasting error on GSM8K, AIME and HLE from 21 percentage points to 4.3. Hinting also reduces the number of test examples needed to tell whether one model is better than another by widening the accuracy gaps between models while approximately preserving their order: a single hinted rollout per problem ranks nearly indistinguishable models as accurately as five unhinted rollouts. Our results suggest that hinting is a general primitive for improving the accuracy and cost of downstream scaling laws and other prediction tasks in model development.

Figures & tables

Figure 1: Hinted scaling laws. Top: Benchmark hinting turns a benchmark into a continuous sequence of easier versions indexed by a hint level $H$ , converting an answer sampling protocol $\omega$ into a hinted protocol $\omega_{H}$ by truncation, masking, or logit mixing. Bottom: results for the Delphi pretraining ladder on GSM8K. (1) Accuracy of the $10^{22}$ FLOPs model rises smoothly with $H$ , where $H=0$ is the original benchmark. (2) A single joint law in model scale and hint level ( Eq. 2 ) is fit to the three smallest models at every $H$ of logit mixing and extrapolates to the held-out larger models. (3) The same law, whose fitted models have 5600x less compute than the largest and score near zero at $H=0$ , is evaluated at $H=0$ to give a hinted prediction that forecasts the held-out larger models . A baseline scaling law fit to the same models’ unhinted accuracy alone overshoots.

Figure 2: Hinted benchmark accuracy varies smoothly in both hint level and scale. Rows are masking and truncation. Columns are the Delphi pretraining ladder ( Held, 2026 ) on GSM8K and the Qwen3.5 family on HLE. Left: accuracy against

H

, one curve per model, darker is larger. Right: accuracy against model scale

C

, one curve per hint level, colored by

H

. Each curve is a sigmoid fit to a single model scale or hint level. As a function of

H

the asymptotes are free:

L+(A-L)\,\sigma(a_{1}H+a_{0})

, with

0\leq L\leq A\leq 1

. As a function of

C

, they are pinned to 0 and 1:

\sigma(a_{1}C+a_{0})

, since accuracy should approach 0 as scale shrinks and 1 as it grows.

Figure 3: Iso-accuracy contours in the scale and hint level plane. Rows are the Delphi ladder on GSM8K and the Qwen3.5 family on HLE, columns masking and truncation. Each dot is one measured cell, colored by accuracy. Lines are contours of constant accuracy at 10% to 90%, interpolated linearly between the cells. The slope of a contour is the exchange rate between scale and hint level at that accuracy.

Figure 4: Benchmark hinting leads to scaling laws with low fit error. Joint laws in

C

and

H

fit to the post-trained model families. Rows are the two benchmark hinting methods, masking and truncation, columns the model family and benchmark. For each family and method we fit a single joint law , Eq. 2 , to the accuracy of every model in the family at all eleven hint levels. Left: the hint slices (fixing model scale and letting hint level vary) of each joint law. Right: the capability slices (fixing hint level and letting model scale vary) of each joint law.

Method	Joint Fit	Individual Fit
Masking	3.1	1.8
Truncation	3.3	1.6
Logit mixing	1.7	1.7

Table 1: Mean absolute error in percentage points of the fitted laws over the families of Figs. 4 , 5 and 7 . The joint law has six parameters per family, the individual fit averages the errors of each cell’s hint- and capability-slice fits (

4|M|+2|H|

parameters). Logit mixing is Delphi only.

Figure 5: Benchmark hinting on the Delphi ladder (GSM8K). Top: for each hinting method (masking and truncation) we fit one joint law ( Eq. 2 ) over all models and hint levels, with

C

the

\log_{10}

pretraining FLOPs. Bottom: the same law is refitted on only the

n

smallest models and its unhinted (

H{=}0

) slice is evaluated at the larger, held-out models. The black curve is the baseline which fits a sigmoid to unhinted accuracies alone. Each individual method’s forecast is shown in faint grey , with the red curve being the hinting ensemble forecast, the average of the masking and truncation forecasts. A ✓ marks panels in which the hinting ensemble forecast beats the baseline forecast on the largest ladder model. The right-most panel reports the absolute error (AE) on the 0–1 accuracy scale against the size of the largest training model.

Figure 6: Benchmark hinting leads to scaling laws with low extrapolation error. Benchmark hinting forecasts of benchmark accuracy on post-trained model families. Columns add train set models from left to right. For truncation and masking, we fit the joint law ( Eq. 2 ) to all hint levels of the train set models, and use the

H=0

slice for the extrapolated curve. Each method is shown in faint grey , with the red curve being the hinting ensemble forecast, the average of masking and truncation forecasts. The black curve is the baseline sigmoid fit to original accuracies. A ✓ marks panels in which our average forecast beats the baseline on the largest model in the family. The right-most column reports absolute error (AE) on the 0–1 accuracy scale against the size of the largest train set model.

Figure 7: Logit mixing with a shared teacher on the Delphi ladder (GSM8K). Top left and middle: Joint law fit over all models and hint levels, with $C$ the $\log_{10}$ pretraining FLOPs. Top right: empirical iso-accuracy contours of the same measurements for $H\leq 1.4$ , as in Fig. 3 . Bottom: the same law is refitted on only the $n$ smallest models and its unhinted ( $H{=}0$ ) slice is evaluated at the larger, held-out models. The black curve is the baseline which fits a sigmoid to unhinted accuracies alone. The red curve shows our prediction.

Figure 8: Benchmark hinting largely preserves the model ranking and recovers it from fewer rollouts (truncation hinting; masking in Fig. 20 ). Left: accuracy versus hint level

H

, one curve per model, colored by ground-truth rank at

H=0

from light (lowest) to dark (highest). Middle: Kendall’s

\tau

between the ground-truth ranking and the ranking from one rollout per problem at each

H

. Bands are

\pm 1

std over simulated single-rollout runs. Right: Kendall’s

\tau

against the budget of

k

rollouts per problem for avg@

k

, the hint-prior ranking ( Eq. 3 ), the single-level rankings of the middle panel ( semi-transparent lines ), and the log-loss baseline .

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Family	Temp.	Top- $p$	Top- $k$	Token limit	Other
Qwen3, Qwen3.5	0.6	0.95	20	32,000 new
Qwen2.5	0.7	0.8	20	32,000 new	rep. penalty 1.05 (1.1 for 0.5B, 1.5B)
Gemma-3, Gemma-4	1.0	0.95	64	32,000 new
Gemma-1.1, Gemma-2	1.0	0.95	64	8,192 context
Llama-3	0.6	0.9	50	32,000 new
Llama-2	0.6	0.9	50	4,096 context

Appendix

Table 2: Sampling settings per model family. Gemma-1.1, Gemma-2 and Llama-2 generate up to their shorter context windows.

Figure 9: Logit mixing on post-trained families on AIME. The top half uses the raw rule ( Eq. 1 ), and the bottom half rescales the teacher’s logits to the evaluated model’s standard deviation at each step ( Eq. 4 ). Each half shows the hint and capability marginals of the joint law fitted to every model except the largest, and its forecast of the largest model at

H=0

Figure 10: Hint and capability slices for the remaining families. As Fig. 2 , for Qwen2.5, Qwen3 and Gemma-3 on AIME (top) and HLE (bottom). Rows are masking and truncation. Left: accuracy against hint level

H

, one curve per model, darker is larger. Right: accuracy against model scale, one curve per hint level. Each curve is a sigmoid fitted to that model or that level alone, with free asymptotes in

H

and asymptotes pinned to 0 and 1 in scale.

Figure 11: Iso-accuracy contours for the remaining families. As Fig. 3 , for Qwen2.5, Qwen3 and Gemma-3 on AIME and HLE. Columns are masking and truncation.

Figure 12: As Fig. 4 , for Qwen3 on AIME and Qwen2.5 and Gemma-3 on HLE.

Figure 13: As Fig. 6 , for Qwen3 on AIME and Qwen2.5 and Gemma-3 on HLE.

Figure 14: Empirical against joint-law iso-accuracy contours. Top two rows: Fig. 3 , the Delphi ladder on GSM8K and the Qwen3.5 family on HLE with contours interpolated from the measured cells. Bottom two rows: the same cells with the contours of the joint law of Eq. 2 fitted to each panel, which above the law’s floor are straight parallel lines of slope $-\beta_{1}/\beta_{2}$ . Columns are masking and truncation. Each dot is one measured cell colored by accuracy.

Figure 15: Bits-per-byte against parameter count on the post-trained families. Each panel holds out the largest model and fits Eq. 5 to the rest (the shaded region), then reads the fit at the held-out model’s parameter count.

\bullet

are fitted models,

\bullet

the held-out model and crosses the forecast. Fitted

L_{\infty}

and

\alpha

are annotated per panel;

L_{\infty}

reaches its lower bound of zero in three of the four families, indicating the floor is not identified at this number of points.

Figure 16: Two-stage scaling law forecasts on the Delphi ladder (GSM8K). Each row is fitted on the smallest models only (the shaded region) and forecasts the held-out larger ones. Columns within a block: the compute-to-

\mathrm{BPB}

power law of Eq. 5 ; the

\mathrm{BPB}

-to-accuracy sigmoid of Eq. 6 , with the

x

axis running high to low so the curve rises left to right; and the two composed, giving accuracy implied at any compute.

\bullet

are fitted models and

\bullet

held-out models; crosses are forecasts, which use only the held-out model’s compute. Left block: the sigmoid is fitted on the fitted models alone. Right block: the eleven external reference models of Held [2026] ( triangles ) are added to its fitting data. Bottom: absolute error for the largest model on the 0–1 accuracy scale against the size of the largest fitted model as a fraction of the target’s, with the unhinted baseline, the hinting ensemble of Fig. 5 and logit mixing ( Section 4.2 ) for reference, all refitted on the same ten models.

Figure 17: Pass@

k

as a benchmark hinting method. Top two rows: the joint law of Eq. 2 fitted to the pass@

k

grid of each (benchmark, family) pair, drawn as its hint slices (accuracy against

H=\log_{2}k

, one line per model, darker with scale) and its capability slices (accuracy against scale, one line per hint level, brighter with

H

). Bottom: the same law refitted on the

n

smallest models only (the shaded region) and evaluated at

H=0

on the held-out larger ones. We compare against the hinting ensemble of masking and truncation ( Section 4.1 ), logit mixing on the Delphi ladder ( Section 4.2 ), and the regular sigmoid fitted to the

H=0

accuracies.

Figure 18: Answer mixing carries no information for forecasting. Top: hint and capability slices of answer mixing, generated from each family’s unhinted accuracies as

\mathrm{Acc}(M,H)=H+(1-H)\,\mathrm{Acc}(M,0)

. Each hint slice is fitted with a straight line and each capability slice with a sigmoid with free asymptotes. Bottom: forecasts of the largest model’s unhinted accuracy from the joint law of Eq. 2 fitted to the answer-mixing cells of the smallest models (red), against the baseline sigmoid fitted to the unhinted scores alone (black). The shaded region spans the fitted models, grey dots are fitted models and orange dots held-out models.

AIME
Model	Acc. (%)	Rollouts	Set
Qwen3 32B	75.091	128
Gemma-4 31B	73.984	128
Qwen3 14B	72.201	128
Qwen3 30B-A3B	71.927	128
Gemma-4 26B-A4B	71.003	128

Appendix

Table 3: Model pools for the ranking experiments. Checkmarks mark the evaluated set. Rollouts counts the ground-truth unhinted rollouts per problem.

Figure 19: Sensitivity of the hint-prior ranking to

\lambda

. Kendall’s

\tau

against the budget

k

for the hint-prior ranking ( Eq. 3 ) with

\lambda\in\{1,2,4,8,16,32,64,128\}

, for both hinting methods (rows) on both benchmarks (columns), with avg@

k

in black.

\lambda

is the number of unhinted rollouts per problem the hinted score is worth: small

\lambda

hands over to the unhinted score early, large

\lambda

late.

Figure 20: Fig. 8 for truncation and masking. Rows are hinting methods, columns benchmarks; left, middle and right panels as in Fig. 8 .

Language Model Performance Prediction

See all 41 papers

Feb 17, 2026

Prescriptive Scaling Reveals the Evolution of Language Model Capabilities

Machine learning model performance improvements tend to arise from competition and application. For deployment, we consider prescriptive scaling laws: given a pre-training compute budget, what downstream accuracy is attainable with contemporary post-training practice, and how stable is that mapping as the field evolves? Using large-scale observational evaluations with 5k existing and 2k newly evaluated model checkpoints spanning 2022-2026 across six benchmarks, we estimate capability boundaries, high conditional quantiles of benchmark scores as a function of log pre-training FLOPs, via smoothed quantile regression with a monotone, saturating sigmoid parameterization. We validate temporal reliability by fitting on earlier model generations and evaluating on later releases: across four of six tasks, the out-of-distribution coverage error remains below 2%, while math reasoning exhibits a consistently advancing boundary over time. For instance, at a budget of 10^24 FLOPs, the estimated attainable accuracies are 0.83 on IFEval and 0.54 on MATH Lvl 5. We then extend our approach to analyze task-dependent saturation and to probe contamination-related shifts on math reasoning tasks. Finally, we introduce a balanced I-optimal sampling algorithm that recovers near-full-data frontiers using roughly 20% of the parameter-count-weighted evaluation budget, as low as 5% on some tasks, while maintaining comparable calibration. Together, our work releases Proteus-2k, the latest model performance evaluation dataset, and introduces a practical methodology for translating compute budgets into reliable performance expectations and for monitoring when capability boundaries shift across time.

Hanlin Zhang, Jikai Jin, Vasilis Syrgkanis +1

Harvard University · Stanford University

Language Model Performance Prediction Language Model Scaling Laws LLM Evaluation

Ask PDF

Oct 16, 2026

Predictable Scaling: Efficient Model Exploration via Scaling-Law Prediction

Evaluating candidate changes (hypotheses) to recommendation models requires full-scale training runs consuming thousands of GPU-hours, severely limiting the hypothesis space engineers can feasibly explore. We introduce Predictable Scaling (PS), a systematic framework that exploits empirically verified regularity conditions to predict full-scale model performance from small-scale proxies. PS constructs scaled-down variants via composable scaling functions along distinct resource axes, and projects metrics to the target scale via curve fitting with calibrated uncertainty estimates. We compare three prediction strategies of increasing expressiveness: even the simplest (raw proxy) achieves prediction error within the empirical noise floor of repeated-run variance; the most expressive (Gaussian Process) provides an indicative 15% relative MAE improvement and calibrated uncertainty. Evaluation on two ranking models of increasing complexity---a task-specific model and a foundation model---demonstrates that PS correctly ranks actionable hypotheses across diverse hypothesis types at 79-88% compute reductions.

Aishwarya Wesanekar, Sheng-Tao Yang, Jiawei Zang +20

Meta Platforms, Inc.

Model Selection Scaling Laws Recommender Systems

Ask PDF

Oct 12, 2026

SLDBench v2: Active Scaling Law Discovery

Scaling laws predict large-model performance from small experiments, but finding a reliable law itself requires costly experiments. SLDBench tests whether language-model agents can discover scaling laws when all experimental results are given upfront. We introduce SLDBench v2 with two extensions. First, we expand the corpus from 8 to 127 scenarios spanning language-model pretraining, downstream performance, and vision-language transfer, and select 30 challenging scenarios on which a reference power-law fit reaches

R^2 \le 0.8

; they represent at least

1.09 \times 10^{23}

FLOPs of source-reported training compute. Second, we introduce active scaling law discovery: an agent chooses which experiments to run, pays each one's recorded cost (e.g., training FLOPs), and revises its law as outcomes arrive, and is scored on the same hidden extrapolation targets as in the passive setting. Across 8 frontier agent systems, median

R^2

drops from 0.72-0.75 (passive) to 0.24-0.57 (active), and prediction error rises 1.5-4.7 times, depending on the system. Agents favor cheap experiments, spending 8-44% of the full-pool cost on average; low-cost acquisition can yield strong predictions, but cheap purchases also leave experimental conditions needed for extrapolation unobserved. Experimental planning thus remains a key open challenge for agents that assist model development in the prevailing AI-for-AI study. Project page: https://lingbei-meng.github.io/SLDBench-v2/.

Lingbei Meng, Rui Huang, Zhirong Shen +5

The Chinese University of Hong Kong, Shenzhen · The University of Hong Kong · University of Electronic Science and Technology of China +2

AI Agents for Scientific Discovery Language Model Scaling Laws AI Agent Benchmarks

Ask PDF

Abstract

Language Model Performance Prediction

Explore similar work

Prescriptive Scaling Reveals the Evolution of Language Model Capabilities

Predictable Scaling: Efficient Model Exploration via Scaling-Law Prediction

SLDBench v2: Active Scaling Law Discovery