ScAn-Bench: Evaluating Scaling Analysis Methodology

Artin Sermaxhaj; Nastaran Alipour; Donat Sinani; Johannes Hog; Neeratyoy Mallik; Jenia Jitsev; Danny Stoll

cs.LGSep 28, 2026

ScAn-Bench: Evaluating Scaling Analysis Methodology

Authors: Artin Sermaxhaj, Nastaran Alipour, Donat Sinani, Johannes Hog, Neeratyoy Mallik, Jenia Jitsev, Danny Stoll

Organizations: University of Freiburg · Zuse School ELIZA · Juelich Supercomputing Center (JSC), Research Center Juelich (FZJ)

Abstract

Recent progress in machine learning is driven by large-scale foundation models, where scaling laws and finding optimal scaling prescriptions for architecture, data, and hyperparameters are key in advancing the state-of-the-art. Therefore, it is surprising that no systematic study evaluates the methodology to obtain scaling laws and prescriptions across different model types. To shed light on this crucial blind spot and facilitate future research, we introduce the surrogate benchmarks ScAn-Bench-LLM and ScAn-Bench-VLM based on 4524 and 8024 checkpoints of language and vision-language model pipelines. On our benchmarks, we perform the first systematic evaluation of both data acquisition and extrapolation methodology for scaling analysis across different data modalities.

Figures & tables

Model family	Surrogate query (CPU-s)	Training (GPU-h)
VLM	29	419.58
LLM	17	3519.42

Table 1 : Runtime comparison for 100 evaluations. Surrogate querying is measured in CPU hours on AMD EPYC 9655 CPU using 8 cores, while training cost is reported in GPU hours on A100 GPUs.

Benchmark	#HPs	#Scale	FLOP Range	Cont.	Disc.	#Down.
ScAn-Bench-VLM	6	3	$3.6\times 10^{13}$ – $7.8\times 10^{18}$	7	2	40
ScAn-Bench-LLM	5	4	$3.2\times 10^{16}$ – $2.9\times 10^{20}$	6	3	-

Table 2 : Summary of the two surrogate benchmarks. The table reports the number of hyperparameters (#HPs), scale parameters (#Scale), FLOP ranges, continuous and discrete dimensions, and the number of downstream tasks (#Down.) modeled by each surrogate benchmark.

	VLM		LLM
Surrogate	Spearman $\uparrow$	RMSE $\downarrow$	Spearman $\uparrow$	RMSE $\downarrow$
TabPFN	0.98	0.21	0.95	0.27
AutoGluon	0.98	0.26	0.91	0.30
Ensemble (XGB)	0.96	0.35	0.85	0.35
Ensemble (Mix)	0.95	0.37	0.86	0.33
Ensemble (LGB)	0.96	0.42	0.87	0.33

Table 3: Comparison of surrogate performance across model families in predicting upstream performance, measured by Spearman rank correlation (higher is better) and RMSE (lower is better).

	LLM			OpenCLIP
Strategy	0.1x	0.5x	Best	0.1x	0.5x	Best
CARBS	$2.02\pm 0.23$	$\mathbf{1.76\pm 0.17}$	$1.31$	$2.92\pm 0.41$	$2.49\pm 0.14$	$0.41$
RS	$\mathbf{1.71\pm 0.18}$	$1.89\pm 0.37$	$1.31$	$\mathbf{2.28\pm 0.29}$	$\mathbf{1.81\pm 0.30}$	$0.41$

Table 4 : Actual loss at predicted configurations. Evaluation of full-parameter extrapolations using CARBS and Random Search (RS) at various

C_{\text{target}}

budget ratios.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Parameters	Type	Range / Set	Sampling
Learning rate	Float	$[1\times 10^{-5},1\times 10^{-2}]$	Log
Weight decay	Float	$[1\times 10^{-3},0.2]$	Log
$\beta_{1}$	Float	$[0.8,0.99]$	Log
$\beta_{2}$	Float	$[0.9,0.999]$	Log
Cooldown fraction	Float	$[0.0,0.3]$	Linear
Embedding size ( $d_{model}$ )	Categorical	$\{256,320,\dots,1600\}$	Linear

Appendix

Table 5 : LLM ScAn Space Configuration

Token Budget	Global Batch Size
$[0,10^{8})$	32768
$[10^{8},10^{9})$	65536
$[10^{9},10^{10})$	131072
$[10^{10},2\times 10^{10})$	262144
$[2\times 10^{10},4\times 10^{10})$	524288

Appendix

Table 6 : Batch size heuristic ranges derived from Step Law.

Parameters	Type	Range / Set	Sampling
Learning rate	Float	$[1\times 10^{-5},5\times 10^{-2}]$	Log
Weight decay	Float	$[1\times 10^{-6},2\times 10^{-1}]$	Log
Warmup fraction	Float	$[0.0,0.75]$	Linear
$\beta_{1}$	Float	$[0.9,0.99]$	Log
$\beta_{2}$	Float	$[0.95,0.999]$	Log
$\epsilon$	Float	$[1\times 10^{-8},1\times 10^{-6}]$	Log

Appendix

Table 7 : VLM ScAn Space Configuration

Component	Choice
Optimizer	AdamW
Vision backbone	Vision Transformer (ViT)
Image patch size	32
Precision	Mixed precision
Layers (vision/text)	12 / 12
Shared embedding dimension(joint space)	384

Appendix

Table 8 : Fixed design choices across all VLM configurations.

Training Samples	Global Batch Size
$[0,3.4\text{M})$	512
$[3.4\text{M},14.6\text{M})$	2,048
$[14.6\text{M},70.4\text{M})$	4,096
$[70.4\text{M},145.92\text{M})$	8,192
$[145.92\text{M},340.48\text{M})$	16,384

Appendix

Table 9 : Batch size heuristic derived from Pareto-optimal OpenCLIP configurations.

Component	Choice
Optimizer	AdamW
Vocabulary size	50277
Sequence length	2048
Activation function	SwiGLU
Learning rate schedule	WSD
Normalization	RMSNorm

Appendix

Table 10 : Fixed training choices across all LLM configurations.

	Training		Downstream		Upstream		Total
GPU	VLM	LLM	VLM	LLM	VLM	LLM
RTX 2080 Ti	11.84	-	731.84	-	11.39	-	755.07
RTX 3080	142.42	-	373.53	-	13.47	-	529.42
L40	0.44	-	-	-	9.84	-	10.28
A100 40GB	3986.02	11042.45	828.75	935.35	198.66	3972.34	20962.55
H100 94GB	6653.11	26870.62	859.65	-	159.61	348.27	34891.26

Appendix

Table 11 : Compute cost (GPU hours) for data collection across different GPU types. The total column reports the sum of training and evaluation costs across both VLM and LLM benchmarks.

Family	Surrogate	RMSE $\downarrow$	MAE $\downarrow$	MDAE $\downarrow$	MARPD $\downarrow$	$R^{2}$ $\uparrow$	R $\uparrow$	Corr. $\uparrow$
VLM	TabPFN	0.21	0.06	0.02	2.69	0.96	0.98	0.98
	AutoGluon	0.26	0.12	0.07	5.79	0.94	0.97	0.98
	XGB	0.35	0.20	0.13	9.11	0.90	0.95	0.96
	Mix	0.37	0.21	0.14	10.11	0.89	0.95	0.95
	LGB	0.42	0.29	0.24	14.88	0.86	0.95	0.96
LLM	TabPFN	$\mathbf{0.27}$	$\mathbf{0.08}$	$\mathbf{0.01}$	$\mathbf{3.02}$	$\mathbf{0.77}$	$\mathbf{0.88}$	$\mathbf{0.95}$

Appendix

Table 12: Surrogate performance across model families on held-out final configurations.

Category	Field	Description
Input	Configuration	Sampled from ScAn space A.1
Input	Training progress	Fidelity in $[0,1]$ indicating training progress
Output	Upstream metrics	Validation or test loss
	Downstream metrics	Defined in A.4
	FLOPs	Cumulative training compute up to queried fidelity
	Model parameters	Number of model parameters

Appendix

Table 13 : Surrogate API query interface. Given a configuration and a fidelity value, the API returns performance predictions and additional metadata.

Explore similar work

Item Response Scaling Laws: A Measurement Theory Approach for Efficient and Generalizable Neural Scaling Estimation
May 29, 2026Sang Truong, Yuheng Tu, Rylan Schaeffer +1Item Response Theory Scaling Laws
1Stanford University · University of California, Los Angeles
On Test-Time Scaling for Vision-Language Models
Jun 27, 2026Fawaz Sammani, Tzoulio Chamiti, Nikos DeligiannisTest-Time Scaling Recent Vision-Language Models
ETRO Department, Vrije Universiteit Brussel, Belgium · imec, Kapeldreef 75, B-3001 Leuven, Belgium
When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
Feb 18, 2026Mubashara Akhtar, Anka Reuel, Prajna Soni +34Artificial Intelligence Benchmarks Large Language Model Benchmarks
ETH Zurich · ETH AI Center · Stanford University +23

May 29, 2026, cs.LGJ/K move · Enter open · S save

Item Response Scaling Laws: A Measurement Theory Approach for Efficient and Generalizable Neural Scaling Estimation

Sang Truong, Yuheng Tu, Rylan Schaeffer, Sanmi Koyejo

1Stanford University · University of California, Los Angeles

Item Response Theory Scaling Laws

Open paper AskPDF arXiv

Scaling laws provide a fundamental framework for understanding the performance of Language Models (LMs), yet deriving them requires prohibitively expensive evaluations across thousands of checkpoints or millions of inference samples. To address this, we introduce Item Response Scaling Laws (IRSL), a unified framework that integrates Item Response Theory (IRT) within the scaling law framework. Unlike traditional approaches that treat each model-benchmark pair in isolation, IRSL disentangles latent model ability from question characteristics, factorizing the scaling law estimation for

M

models and

N

questions to significantly reduce parameter complexity from

O(M \times N)

O(M + N)

. We instantiate IRSL with Beta-IRT, which leverages the empirical probability responses of LMs -- such as token probabilities in pre-training and pass rates in test-time sampling -- to capture richer signals than binary responses. We validate our approach across two prevalent scaling paradigms: (1) pre-training downstream scaling, using 6,612 LM checkpoints and 37,682 questions from 10 benchmarks; and (2) test-time scaling, using 12 LMs and 120 questions from 4 benchmarks with up to 2,500 samples per question. Given a one-time calibration on existing model responses, IRSL yields more reliable scaling estimates using only 50 questions per benchmark (a 99.9% reduction), achieving comparable or superior decision accuracy to traditional approaches. Furthermore, we show that the estimated latent model abilities are generalizable, enabling accurate performance forecasting across benchmarks that share the same measurement objective.

Abstract

Explore similar work

Item Response Scaling Laws: A Measurement Theory Approach for Efficient and Generalizable Neural Scaling Estimation

On Test-Time Scaling for Vision-Language Models

When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation