SLDBench v2: Active Scaling Law Discovery

Lingbei Meng; Rui Huang; Zhirong Shen; Jinxin Liu; Lingxiao Jing; Haorui Chen; Yaosong Du; Haowei Lin

cs.LGOct 12, 2026

SLDBench v2: Active Scaling Law Discovery

Authors: Lingbei Meng, Rui Huang, Zhirong Shen, Jinxin Liu, Lingxiao Jing, Haorui Chen, Yaosong Du, Haowei Lin

Organizations: The Chinese University of Hong Kong, Shenzhen · The University of Hong Kong · University of Electronic Science and Technology of China · Tsinghua University ·

Abstract

Scaling laws predict large-model performance from small experiments, but finding a reliable law itself requires costly experiments. SLDBench tests whether language-model agents can discover scaling laws when all experimental results are given upfront. We introduce SLDBench v2 with two extensions. First, we expand the corpus from 8 to 127 scenarios spanning language-model pretraining, downstream performance, and vision-language transfer, and select 30 challenging scenarios on which a reference power-law fit reaches $R^2 \le 0.8$ ; they represent at least $1.09 \times 10^{23}$ FLOPs of source-reported training compute. Second, we introduce active scaling law discovery: an agent chooses which experiments to run, pays each one's recorded cost (e.g., training FLOPs), and revises its law as outcomes arrive, and is scored on the same hidden extrapolation targets as in the passive setting. Across 8 frontier agent systems, median $R^2$ drops from 0.72-0.75 (passive) to 0.24-0.57 (active), and prediction error rises 1.5-4.7 times, depending on the system. Agents favor cheap experiments, spending 8-44% of the full-pool cost on average; low-cost acquisition can yield strong predictions, but cheap purchases also leave experimental conditions needed for extrapolation unobserved. Experimental planning thus remains a key open challenge for agents that assist model development in the prevailing AI-for-AI study. Project page: https://lingbei-meng.github.io/SLDBench-v2/.

Figures & tables

Figure 1: Passive and active scaling law discovery. (a) Passive SLD supplies the outcomes of all candidate experiments; the agent fits a law that must extrapolate to hidden larger-scale runs. (b) Active SLD supplies only a priced catalogue. The agent buys batches of experiments over rounds (filled markers, shaded by round; area grows with price), refits its law after each round, and stops; it is scored on the same hidden runs, and its spending $\rho$ is reported. A rival law (dashed) that fits the cheap points diverges at the test scale. (c) Median $R^{2}$ of the 8 agent systems on the 30 tasks: nearly identical in the Passive setting and widely spread in the Active one; shaded bands mark the range of median scores in each setting (Table 1 ).

Figure 2: Construction and anatomy of SLDBench v2. (a) The three-stage funnel from the collected corpus to the benchmark, colored by research domain; domains are assigned by keyword rules. (b) The 30 selected tasks. Each dot is a task and the star is the running example T22; the bar and the number on the right give the median, and the band the interquartile range; the last row divides the median test price by the median candidate price. (c) The Harbor task directory and the two containers of one task. The agent reads the instruction and the priced catalogue, buys outcomes through a root-only oracle that signs every charge into a ledger, and submits an executable law; a separate verifier checks the ledger and scores the law on hidden larger-scale runs. File names, commands, and limits are those of the released task.

		Passive		Active			Active cost $\rho$ (%)
Model	Harness	$R^{2}$	${>}.5$	$R^{2}$	${>}.5$	$\Delta R^{2}$	mean	median	full
Claude Opus 5.5	Claude Code	0.751	80.0	0.570	55.6	$-$ 0.181	7.8	4.01	1.1
Kimi K3	Claude Code	0.742	72.2	0.555	45.6	$-$ 0.187	32.0	17.75	10.0
DeepSeek V4 Pro	Claude Code	0.728	64.4	0.540	51.1	$-$ 0.188	34.8	21.51	7.8
GPT-5.6 Sol	Codex	0.728	70.0	0.514	51.1	$-$ 0.214	43.8	35.20	11.1
Qwen3.8 Flash	Claude Code	0.746	74.4	0.398	42.2	$-$ 0.348	15.8	5.35	2.2

Table 1: Prediction quality and acquisition cost, sorted by median Active

R^{2}

R^{2}

columns are medians over finite scores;

{>}.5

is the percentage of the 90 scheduled runs per system and setting with

R^{2}>0.5

;

\Delta R^{2}

is the Active minus the Passive median. Cost is the Active acquisition cost as a percentage of the full-pool cost, and full is the percentage of Active runs that bought every candidate. Bold marks the best agent value of each quality column. † The reference law of the difficulty screen (Appendix A.2 ), fitted once to all candidates, where the two settings coincide; the screen keeps only tasks on which it scores

R^{2}\leq 0.8

Figure 3: Where the Active loss comes from. (a) Percentage of scheduled runs with $R^{2}\leq 0$ or no score and (b) with $R^{2}>0.5$ , in the Passive (blue) and Active (orange) settings. (c) Active/Passive error multiplier of each system, aggregated within and then across the 14 sources; bars are 95% source-bootstrap intervals, and the dashed line marks equal error. (d) Median Active cost of each system against the Passive-to-Active rise in its failure share of panel (a); $r_{s}$ is the Spearman rank correlation over the 8 systems.

Figure 4: What agents buy. (a) Active cost of every run as a percentage of the full-pool cost (dots) and the per-system median (diamonds). (b) Median within-group price rank of the first (circle) and last (arrowhead) batch of multi-batch runs; 0 is the cheapest candidate of a group and 1 the most expensive. (c) Runs that observe every group with the catalogue’s full independent parameter variation, every group with reduced variation, or not every group. (d) Median share of candidates bought against median share of the full-pool cost, each computed separately; points below the diagonal pay less than the catalogue’s mean price per candidate.

Figure 5: What the purchased data are worth. (a) Median

R^{2}

over the 30 tasks when the fixed reference law is fitted to candidates chosen by four non-agent rules at budget

\rho

(lines), with the agents’ Passive medians at

\rho=100\%

and their Active medians at their median cost; each agent marker pairs two separate medians. (b) Share of the test squared error per group in the DeepSeek V4 Pro run on T22, sorted; dark bars are groups with two purchased points. (c) Share of the 30 tasks on which a rule gives the reference law a higher

R^{2}

than cheapest-first selection at the same budget; ties count as not higher, and the dashed line marks parity.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Code	Task	Pool	Test	$G$	$d$
T01	IBM BoolQ	47	15	7	2
T02	IBM OpenBookQA	47	15	7	2
T03	IBM OpenBookQA norm.	47	15	7	2
T04	IBM RACE-high	47	15	7	2
T05	IBM WinoGrande	46	15	7	2
T06	OLMo COPA	1,366	342	3	2

Appendix

Table 2: Data points of the 30 selected tasks, 339,092 in total. Pool and test count the candidate and hidden test records;

G

counts experimental groups and

d

input features, as defined in Section 2 .

Source	Upstream project	Tasks
ColPret	IBM/ColPret	12
LAION comparisons	LAION-AI/scaling-laws-for-comparison	5
Misfitting	hadasah/scaling_laws	2
Compute-optimal tokenization	facebookresearch/compute-optimal-tokenization	1
Datablations	huggingface/datablations	1
DCLM	mlfoundations/dclm	1

Appendix

Table 3: Upstream sources of the selected tasks. Tasks from one source are not independent campaigns.

Source	Tier	Accounting unit	Units	FLOPs
Compute-optimal tokenization	C3	Configuration	85	$2.70\times 10^{22}$
ProteinLM	C3	Source run	196	$8.24\times 10^{22}$
OpenWebText	C2	Source run	15	$1.55\times 10^{18}$

Appendix

Table 4: Training compute represented by the selected tasks. Units are training runs or configurations, and totals take the maximum recorded cost of each unit over pool and test. C3 and C2 totals are kept separate.

System	Passive > 0	Active > 0	Passive > .5	Active > .5	Passive > .9	Active > .9
Claude Opus 5.5	96.7%	77.8%	80.0%	55.6%	26.7%	15.6%
DeepSeek V4 Pro	86.7%	78.9%	64.4%	51.1%	15.6%	11.1%
DeepSeek V4 Flash	84.4%	64.4%	66.7%	41.1%	11.1%	5.6%
GPT-5.6 Sol	85.6%	71.1%	70.0%	51.1%	20.0%	12.2%
Gemini 3.8 Flash	85.6%	62.2%	70.0%	42.2%	18.9%	5.6%
Qwen3.8 Flash	92.2%	65.6%	74.4%	42.2%	20.0%	3.3%

Appendix

Table 5: Percentage of all scheduled runs whose

R^{2}

exceeds each threshold.

Model	Passive best@3 $R^{2}$	Active best@3 $R^{2}$
Claude Opus 5.5	0.8179	0.6877
Kimi K3	0.7684	0.6919
DeepSeek V4 Pro	0.7545	0.7583
GPT-5.6 Sol	0.7744	0.7438
Qwen3.8 Flash	0.7978	0.6673
Hy4-Preview	0.7970	0.5659

Appendix

Table 6: Best@3

R^{2}

: median over 30 tasks of the highest score across three repeats per task and setting. Bold marks the best value in each column.

	Median $R^{2}$				Beats cheapest-first (%)
$\rho$ (%)	cheapest	random	per group	spread	random	per group	spread
0.5	$-$ 1.104	$-$ 0.312	$-$ 2.541	$-$ 0.533	66.7	23.3	66.7
1	$-$ 0.646	$-$ 0.297	$-$ 2.029	$-$ 0.488	63.3	23.3	63.3
2	$-$ 0.811	$-$ 0.082	$-$ 1.456	$-$ 0.005	73.3	16.7	76.7
5	$-$ 0.182	$-$ 0.003	$-$ 0.664	0.063	73.3	46.7	70.0
10	0.065	0.085	$-$ 0.044	0.027	60.0	30.0	50.0

Appendix

Table 7: Non-agent acquisition baselines under the fixed reference law. Median

R^{2}

is over the 30 tasks, all of which have finite scores. “Beats cheapest-first” is the percentage of the 30 tasks on which a rule scores a strictly higher

R^{2}

than cheapest-first at the same budget, so ties count against the rule.

System	Full pool	Point share	Cost share	Price ratio	First-batch price position	Last-batch price position
Claude Opus 5.5	1.1%	21.5%	4.01%	0.237	0.197	0.608
DeepSeek V4 Pro	7.8%	53.0%	21.51%	0.586	0.347	0.512
DeepSeek V4 Flash	26.7%	28.6%	17.59%	0.904	0.287	0.500
GPT-5.6 Sol	11.1%	69.5%	35.20%	0.716	0.442	0.500
Gemini 3.8 Flash	15.6%	47.8%	7.78%	0.271	0.126	0.375
Qwen3.8 Flash	2.2%	34.0%	5.35%	0.248	0.309	0.414

Appendix

Table 8: Purchase behavior. Full pool is the percentage of runs that buy every candidate; the other columns are separate medians over runs, and those of batch price positions over multi-batch runs (lower is cheaper).

System	Cost median (%)	Cost ${<}1\%$	Cost ${>}90\%$	Full pool	Sign changes	${>}10\times$ error
Claude Opus 5.5	4.01	23.3%	1.1%	1.1%	20.0%	6.7%
DeepSeek V4 Pro	21.51	12.2%	8.9%	7.8%	26.7%	23.3%
DeepSeek V4 Flash	17.59	22.2%	27.8%	26.7%	33.3%	23.3%
GPT-5.6 Sol	35.20	7.8%	13.3%	11.1%	43.3%	33.3%
Gemini 3.8 Flash	7.78	21.1%	15.6%	15.6%	46.7%	20.0%
Qwen3.8 Flash	5.35	23.3%	2.2%	2.2%	46.7%	10.0%

Appendix

Table 9: Spending spread and repeat stability. Costs are percentages of the full-pool cost and the next three columns percentages of Active runs; sign changes of

R^{2}

and more than tenfold error ranges across the three repeats are percentages of the tasks with three finite Active repeats.

System	All groups	Missing groups	Independent variation lost	All groups; variation lost
Claude Opus 5.5	80.0%	20.0%	46.7%	26.7%
DeepSeek V4 Pro	88.9%	11.1%	24.4%	13.3%
DeepSeek V4 Flash	83.3%	16.7%	28.9%	12.2%
GPT-5.6 Sol	100.0%	0.0%	3.3%	3.3%
Gemini 3.8 Flash	96.7%	3.3%	14.4%	11.1%
Qwen3.8 Flash	86.7%	13.3%	34.4%	21.1%

Appendix

Table 10: Experimental coverage, as percentages of all Active runs of each system. Independent variation is measured under the fixed input representation defined above; a run that observes every group can still leave fewer parameter contributions distinguishable than the full catalogue.

System	Run	Points	Batch	Learning rate	$R^{2}$
Hy4-Preview	A	44	4	0.0001	$-$ 2.392
Hy4-Preview	A	44	64	0.0001	$-$ 2.392
Hy4-Preview	B	51	64	0.0001	$-$ 83.305
Hy4-Preview	B	51	128	0.0001	$-$ 83.305
Kimi K3	A	135	16	0.0001	$-$ 10.580

Appendix

Table 11: OpenWebText parameter combinations left unobserved by 12.5% of the runs of this task.

Group	Bought/pool	Bought D	Test D	RMSE	SSE share
ViT-B-16_mammut	2/8	1.28–3.072	640–3072	0.2867	28.16%
ViT-B-32_mammut	2/8	1.28–3.072	640–3072	0.2826	27.35%
ViT-B-16-text-plus_clip	2/8	1.28–3.072	640–3072	0.2273	17.70%
ViT-S-14_mammut	2/8	1.28–3.072	640–3072	0.1469	7.40%
ViT-M-32_mammut	2/8	1.28–3.072	640–3072	0.1319	5.96%
ViT-S-14_clip	2/7	1.28–3.072	1280–3072	0.1083	2.68%

Appendix

Table 12: All 22 groups of the T22 case, sorted by share of the squared test error (SSE);

D

in millions.

Benchmark	Evaluated activity	Principal outcome
MLAgentBench / MLE-bench	ML experimentation and engineering	Model or competition score
MLGym	Open-ended AI research tasks	Task-specific improvement
RE-Bench	Research engineering	Progress relative to experts
ResearchGym	Rediscovering a withheld method	Gain over supplied baselines
AIRS-Bench	Research without baseline code	Metrics versus published SOTA
PaperBench / CORE-Bench	Paper replication / reproduction	Rubric score / reproduction

Appendix

Table 13: Benchmarks discussed in Section 4 , summarized by their primary evaluated activity and outcome.

AI Agents for Scientific Discovery

AI agents for scientific discovery use language models, tools, and experiments to generate, test, and refine scientific knowledge.

See all 287 papers

ScAn-Bench: Evaluating Scaling Analysis Methodology
Sep 28, 2026Artin Sermaxhaj, Nastaran Alipour, Donat Sinani +4VLM Evaluation LLM Evaluation
University of Freiburg · Zuse School ELIZA · Juelich Supercomputing Center (JSC), Research Center Juelich (FZJ)
EdgeBench: Unveiling Scaling Laws of Learning from Real-World Environments
Jul 6, 2026Deyao Zhu, Xin Zhou, Shengling Qin +44Long-Horizon Agent Evaluation AI Agent Benchmarks
Amortizing Scaling Law Construction Costs
Sep 4, 2026Abhash Kumar Jha, Diana Alexandra Onuţu, Neeratyoy Mallik +6Language Model Scaling Laws Bayesian Optimization
ELLIS Institute Tübingen · OpenEuroLLM · Eindhoven University of Technology +4

Sep 28, 2026J/K move · Enter open · S save

ScAn-Bench: Evaluating Scaling Analysis Methodology

Artin Sermaxhaj, Nastaran Alipour, Donat Sinani, Johannes Hog+3

University of Freiburg · Zuse School ELIZA · Juelich Supercomputing Center (JSC), Research Center Juelich (FZJ)

VLM Evaluation LLM Evaluation Language Model Scaling Laws Scaling Laws

Open paper AskPDF arXiv

Recent progress in machine learning is driven by large-scale foundation models, where scaling laws and finding optimal scaling prescriptions for architecture, data, and hyperparameters are key in advancing the state-of-the-art. Therefore, it is surprising that no systematic study evaluates the methodology to obtain scaling laws and prescriptions across different model types. To shed light on this crucial blind spot and facilitate future research, we introduce the surrogate benchmarks ScAn-Bench-LLM and ScAn-Bench-VLM based on 4524 and 8024 checkpoints of language and vision-language model pipelines. On our benchmarks, we perform the first systematic evaluation of both data acquisition and extrapolation methodology for scaling analysis across different data modalities.

Abstract

AI Agents for Scientific Discovery

Explore similar work

ScAn-Bench: Evaluating Scaling Analysis Methodology

EdgeBench: Unveiling Scaling Laws of Learning from Real-World Environments

Amortizing Scaling Law Construction Costs