Authors: Lingbei Meng, Rui Huang, Zhirong Shen, Jinxin Liu, Lingxiao Jing, Haorui Chen, Yaosong Du, Haowei Lin
Organizations: The Chinese University of Hong Kong, Shenzhen · The University of Hong Kong · University of Electronic Science and Technology of China · Tsinghua University · Peking University
Scaling laws predict large-model performance from small experiments, but finding a reliable law itself requires costly experiments. SLDBench tests whether language-model agents can discover scaling laws when all experimental results are given upfront. We introduce SLDBench v2 with two extensions. First, we expand the corpus from 8 to 127 scenarios spanning language-model pretraining, downstream performance, and vision-language transfer, and select 30 challenging scenarios on which a reference power-law fit reaches R2≤0.8; they represent at least 1.09×1023 FLOPs of source-reported training compute. Second, we introduce active scaling law discovery: an agent chooses which experiments to run, pays each one's recorded cost (e.g., training FLOPs), and revises its law as outcomes arrive, and is scored on the same hidden extrapolation targets as in the passive setting. Across 8 frontier agent systems, median R2 drops from 0.72-0.75 (passive) to 0.24-0.57 (active), and prediction error rises 1.5-4.7 times, depending on the system. Agents favor cheap experiments, spending 8-44% of the full-pool cost on average; low-cost acquisition can yield strong predictions, but cheap purchases also leave experimental conditions needed for extrapolation unobserved. Experimental planning thus remains a key open challenge for agents that assist model development in the prevailing AI-for-AI study. Project page: https://lingbei-meng.github.io/SLDBench-v2/.
Figures & tables
Figure 1: Passive and active scaling law discovery. (a) Passive SLD supplies the outcomes of all candidate experiments; the agent fits a law that must extrapolate to hidden larger-scale runs. (b) Active SLD supplies only a priced catalogue. The agent buys batches of experiments over rounds (filled markers, shaded by round; area grows with price), refits its law after each round, and stops; it is scored on the same hidden runs, and its spending ρ is reported. A rival law (dashed) that fits the cheap points diverges at the test scale. (c) Median R2 of the 8 agent systems on the 30 tasks: nearly identical in the Passive setting and widely spread in the Active one; shaded bands mark the range of median scores in each setting (Table 1 ).
Figure 2: Construction and anatomy of SLDBench v2. (a) The three-stage funnel from the collected corpus to the benchmark, colored by research domain; domains are assigned by keyword rules. (b) The 30 selected tasks. Each dot is a task and the star is the running example T22; the bar and the number on the right give the median, and the band the interquartile range; the last row divides the median test price by the median candidate price. (c) The Harbor task directory and the two containers of one task. The agent reads the instruction and the priced catalogue, buys outcomes through a root-only oracle that signs every charge into a ledger, and submits an executable law; a separate verifier checks the ledger and scores the law on hidden larger-scale runs. File names, commands, and limits are those of the released task.
Passive
Active
Active cost ρ (%)
Model
Harness
R2
>.5
R2
>.5
ΔR2
mean
median
full
Claude Opus 5.5
Claude Code
0.751
80.0
0.570
55.6
− 0.181
7.8
4.01
1.1
Kimi K3
Claude Code
0.742
72.2
0.555
45.6
− 0.187
32.0
17.75
10.0
DeepSeek V4 Pro
Claude Code
0.728
64.4
0.540
51.1
− 0.188
34.8
21.51
7.8
GPT-5.6 Sol
Codex
0.728
70.0
0.514
51.1
− 0.214
43.8
35.20
11.1
Qwen3.8 Flash
Claude Code
0.746
74.4
0.398
42.2
− 0.348
15.8
5.35
2.2
Table 1: Prediction quality and acquisition cost, sorted by median Active R2 . R2 columns are medians over finite scores; >.5 is the percentage of the 90 scheduled runs per system and setting with R2>0.5 ; ΔR2 is the Active minus the Passive median. Cost is the Active acquisition cost as a percentage of the full-pool cost, and full is the percentage of Active runs that bought every candidate. Bold marks the best agent value of each quality column. † The reference law of the difficulty screen (Appendix A.2 ), fitted once to all candidates, where the two settings coincide; the screen keeps only tasks on which it scores R2≤0.8 .
Figure 3: Where the Active loss comes from. (a) Percentage of scheduled runs with R2≤0 or no score and (b) with R2>0.5 , in the Passive (blue) and Active (orange) settings. (c) Active/Passive error multiplier of each system, aggregated within and then across the 14 sources; bars are 95% source-bootstrap intervals, and the dashed line marks equal error. (d) Median Active cost of each system against the Passive-to-Active rise in its failure share of panel (a); rs is the Spearman rank correlation over the 8 systems.
Figure 4: What agents buy. (a) Active cost of every run as a percentage of the full-pool cost (dots) and the per-system median (diamonds). (b) Median within-group price rank of the first (circle) and last (arrowhead) batch of multi-batch runs; 0 is the cheapest candidate of a group and 1 the most expensive. (c) Runs that observe every group with the catalogue’s full independent parameter variation, every group with reduced variation, or not every group. (d) Median share of candidates bought against median share of the full-pool cost, each computed separately; points below the diagonal pay less than the catalogue’s mean price per candidate.
Figure 5: What the purchased data are worth. (a) Median R2 over the 30 tasks when the fixed reference law is fitted to candidates chosen by four non-agent rules at budget ρ (lines), with the agents’ Passive medians at ρ=100% and their Active medians at their median cost; each agent marker pairs two separate medians. (b) Share of the test squared error per group in the DeepSeek V4 Pro run on T22, sorted; dark bars are groups with two purchased points. (c) Share of the 30 tasks on which a rule gives the reference law a higher R2 than cheapest-first selection at the same budget; ties count as not higher, and the dashed line marks parity.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Code
Task
Pool
Test
G
d
T01
IBM BoolQ
47
15
7
2
T02
IBM OpenBookQA
47
15
7
2
T03
IBM OpenBookQA norm.
47
15
7
2
T04
IBM RACE-high
47
15
7
2
T05
IBM WinoGrande
46
15
7
2
T06
OLMo COPA
1,366
342
3
2
Appendix
Table 2: Data points of the 30 selected tasks, 339,092 in total. Pool and test count the candidate and hidden test records; G counts experimental groups and d input features, as defined in Section 2 .
Source
Upstream project
Tasks
ColPret
IBM/ColPret
12
LAION comparisons
LAION-AI/scaling-laws-for-comparison
5
Misfitting
hadasah/scaling_laws
2
Compute-optimal tokenization
facebookresearch/compute-optimal-tokenization
1
Datablations
huggingface/datablations
1
DCLM
mlfoundations/dclm
1
Appendix
Table 3: Upstream sources of the selected tasks. Tasks from one source are not independent campaigns.
Source
Tier
Accounting unit
Units
FLOPs
Compute-optimal tokenization
C3
Configuration
85
2.70×1022
ProteinLM
C3
Source run
196
8.24×1022
OpenWebText
C2
Source run
15
1.55×1018
Appendix
Table 4: Training compute represented by the selected tasks. Units are training runs or configurations, and totals take the maximum recorded cost of each unit over pool and test. C3 and C2 totals are kept separate.
System
Passive > 0
Active > 0
Passive > .5
Active > .5
Passive > .9
Active > .9
Claude Opus 5.5
96.7%
77.8%
80.0%
55.6%
26.7%
15.6%
DeepSeek V4 Pro
86.7%
78.9%
64.4%
51.1%
15.6%
11.1%
DeepSeek V4 Flash
84.4%
64.4%
66.7%
41.1%
11.1%
5.6%
GPT-5.6 Sol
85.6%
71.1%
70.0%
51.1%
20.0%
12.2%
Gemini 3.8 Flash
85.6%
62.2%
70.0%
42.2%
18.9%
5.6%
Qwen3.8 Flash
92.2%
65.6%
74.4%
42.2%
20.0%
3.3%
Appendix
Table 5: Percentage of all scheduled runs whose R2 exceeds each threshold.
Model
Passive best@3 R2
Active best@3 R2
Claude Opus 5.5
0.8179
0.6877
Kimi K3
0.7684
0.6919
DeepSeek V4 Pro
0.7545
0.7583
GPT-5.6 Sol
0.7744
0.7438
Qwen3.8 Flash
0.7978
0.6673
Hy4-Preview
0.7970
0.5659
Appendix
Table 6: Best@3 R2 : median over 30 tasks of the highest score across three repeats per task and setting. Bold marks the best value in each column.
Median R2
Beats cheapest-first (%)
ρ (%)
cheapest
random
per group
spread
random
per group
spread
0.5
− 1.104
− 0.312
− 2.541
− 0.533
66.7
23.3
66.7
1
− 0.646
− 0.297
− 2.029
− 0.488
63.3
23.3
63.3
2
− 0.811
− 0.082
− 1.456
− 0.005
73.3
16.7
76.7
5
− 0.182
− 0.003
− 0.664
0.063
73.3
46.7
70.0
10
0.065
0.085
− 0.044
0.027
60.0
30.0
50.0
Appendix
Table 7: Non-agent acquisition baselines under the fixed reference law. Median R2 is over the 30 tasks, all of which have finite scores. “Beats cheapest-first” is the percentage of the 30 tasks on which a rule scores a strictly higher R2 than cheapest-first at the same budget, so ties count against the rule.
System
Full pool
Point share
Cost share
Price ratio
First-batch price position
Last-batch price position
Claude Opus 5.5
1.1%
21.5%
4.01%
0.237
0.197
0.608
DeepSeek V4 Pro
7.8%
53.0%
21.51%
0.586
0.347
0.512
DeepSeek V4 Flash
26.7%
28.6%
17.59%
0.904
0.287
0.500
GPT-5.6 Sol
11.1%
69.5%
35.20%
0.716
0.442
0.500
Gemini 3.8 Flash
15.6%
47.8%
7.78%
0.271
0.126
0.375
Qwen3.8 Flash
2.2%
34.0%
5.35%
0.248
0.309
0.414
Appendix
Table 8: Purchase behavior. Full pool is the percentage of runs that buy every candidate; the other columns are separate medians over runs, and those of batch price positions over multi-batch runs (lower is cheaper).
System
Cost median (%)
Cost <1%
Cost >90%
Full pool
Sign changes
>10× error
Claude Opus 5.5
4.01
23.3%
1.1%
1.1%
20.0%
6.7%
DeepSeek V4 Pro
21.51
12.2%
8.9%
7.8%
26.7%
23.3%
DeepSeek V4 Flash
17.59
22.2%
27.8%
26.7%
33.3%
23.3%
GPT-5.6 Sol
35.20
7.8%
13.3%
11.1%
43.3%
33.3%
Gemini 3.8 Flash
7.78
21.1%
15.6%
15.6%
46.7%
20.0%
Qwen3.8 Flash
5.35
23.3%
2.2%
2.2%
46.7%
10.0%
Appendix
Table 9: Spending spread and repeat stability. Costs are percentages of the full-pool cost and the next three columns percentages of Active runs; sign changes of R2 and more than tenfold error ranges across the three repeats are percentages of the tasks with three finite Active repeats.
System
All groups
Missing groups
Independent variation lost
All groups; variation lost
Claude Opus 5.5
80.0%
20.0%
46.7%
26.7%
DeepSeek V4 Pro
88.9%
11.1%
24.4%
13.3%
DeepSeek V4 Flash
83.3%
16.7%
28.9%
12.2%
GPT-5.6 Sol
100.0%
0.0%
3.3%
3.3%
Gemini 3.8 Flash
96.7%
3.3%
14.4%
11.1%
Qwen3.8 Flash
86.7%
13.3%
34.4%
21.1%
Appendix
Table 10: Experimental coverage, as percentages of all Active runs of each system. Independent variation is measured under the fixed input representation defined above; a run that observes every group can still leave fewer parameter contributions distinguishable than the full catalogue.
System
Run
Points
Batch
Learning rate
R2
Hy4-Preview
A
44
4
0.0001
− 2.392
Hy4-Preview
A
44
64
0.0001
− 2.392
Hy4-Preview
B
51
64
0.0001
− 83.305
Hy4-Preview
B
51
128
0.0001
− 83.305
Kimi K3
A
135
16
0.0001
− 10.580
Appendix
Table 11: OpenWebText parameter combinations left unobserved by 12.5% of the runs of this task.
Group
Bought/pool
Bought D
Test D
RMSE
SSE share
ViT-B-16_mammut
2/8
1.28–3.072
640–3072
0.2867
28.16%
ViT-B-32_mammut
2/8
1.28–3.072
640–3072
0.2826
27.35%
ViT-B-16-text-plus_clip
2/8
1.28–3.072
640–3072
0.2273
17.70%
ViT-S-14_mammut
2/8
1.28–3.072
640–3072
0.1469
7.40%
ViT-M-32_mammut
2/8
1.28–3.072
640–3072
0.1319
5.96%
ViT-S-14_clip
2/7
1.28–3.072
1280–3072
0.1083
2.68%
Appendix
Table 12: All 22 groups of the T22 case, sorted by share of the squared test error (SSE); D in millions.
Benchmark
Evaluated activity
Principal outcome
MLAgentBench / MLE-bench
ML experimentation and engineering
Model or competition score
MLGym
Open-ended AI research tasks
Task-specific improvement
RE-Bench
Research engineering
Progress relative to experts
ResearchGym
Rediscovering a withheld method
Gain over supplied baselines
AIRS-Bench
Research without baseline code
Metrics versus published SOTA
PaperBench / CORE-Bench
Paper replication / reproduction
Rubric score / reproduction
Appendix
Table 13: Benchmarks discussed in Section 4 , summarized by their primary evaluated activity and outcome.
Recent progress in machine learning is driven by large-scale foundation models, where scaling laws and finding optimal scaling prescriptions for architecture, data, and hyperparameters are key in advancing the state-of-the-art. Therefore, it is surprising that no systematic study evaluates the methodology to obtain scaling laws and prescriptions across different model types. To shed light on this crucial blind spot and facilitate future research, we introduce the surrogate benchmarks ScAn-Bench-LLM and ScAn-Bench-VLM based on 4524 and 8024 checkpoints of language and vision-language model pipelines. On our benchmarks, we perform the first systematic evaluation of both data acquisition and extrapolation methodology for scaling analysis across different data modalities.
Pretraining scaling laws reveal that model capability improves predictably with data and compute. But learning from real world environments after deployment remains far less understood. Analyzing roughly 38,000 hours of agent interaction with the environment across 134 real world tasks, we find, to the best of our knowledge, the first evidence that overall performance during environment learning follows a log-sigmoid scaling law with remarkably high precision, reaching R^2 = 0.998. Across model generations, we also find that agent learning speed roughly doubles every three months. This discovery stems from EdgeBench, a suite of 134 real world tasks with ultra-long horizons, spanning scientific discovery, software engineering, combinatorial optimization, professional knowledge work, formal mathematics, and interactive games. Each task sustains at least 12 hours of continuous agent operation under rich, multilevel feedback, and is built through substantial expert effort. We publicly release 51 tasks and our full evaluation framework to accelerate the study of how agents learn from real world experience.
Scaling laws guide the design choices for training large foundation models, but deriving them involves training an exhaustive grid over hyperparameters, token budgets, and parameter counts, which is computationally expensive. Fitting a scaling law, however, only requires the best-loss frontier across compute scales, discarding most of the trained configurations. We propose a framework for efficient scaling law construction that formulates data collection as a Bayesian optimization problem, and introduce metrics for comparing scaling law fitting methods under constrained compute budgets. We find that progressively expanding the compute budget during acquisition, mirroring the compute-ordered evaluation of configurations in practice, substantially improves recovery efficiency. Augmenting the observed configurations with surrogate-fantasized evaluations then recovers the broader experimental grid, allowing accurate scaling law fitting without training every configuration. Together, these can closely match scaling law fits over a full dense grid at computational savings of up to 10–100×.
Abhash Kumar Jha, Diana Alexandra Onuţu, Neeratyoy Mallik +6
ELLIS Institute Tübingen · OpenEuroLLM · Eindhoven University of Technology +4