Cost-Efficient Estimation of General Abilities Across Benchmarks

Michael Krumdick; Adam Wiemerslage; Seth Ebner; Charles Lovering; Chris Tanner

cs.CLApr 1, 2026

Cost-Efficient Estimation of General Abilities Across Benchmarks

Authors: Michael Krumdick, Adam Wiemerslage, Seth Ebner, Charles Lovering, Chris Tanner

Organizations: Kensho Technologies · MIT

Abstract

Thousands of diverse benchmarks have been developed to measure the quality of large language models (LLMs). Yet prior work has demonstrated that LLM performance is often sufficiently explained by a small set of latent factors, or abilities. This suggests the potential for more efficient and principled benchmarking, but it remains difficult to compare the quality of different methods. Motivated by predictive validity, we argue that the quality of a benchmarking framework should be grounded in how efficiently it enables the prediction of model performance on unseen tasks. To analyze this objective, we collect the "Wide-scale Item Level Dataset" (WILD), a dataset of item-model response pairs, comprising evaluations of 65 models on 109,564 unique items spanning 163 tasks drawn from 27 datasets. This dataset enables the first analysis of how different techniques can predict a model's performance on a large, diverse collection of unseen tasks under different budget constraints. We demonstrate that combining a modified multidimensional item response theory (IRT) model with adaptive item selection driven by optimal experimental design can predict performance on 112 held-out benchmark tasks with a mean absolute error (MAE) of less than 7%, and can do so after observing only 16 items. We further demonstrate that incorporating cost-aware discount factors into our selection criteria can reduce the total tokens needed to reach 7% MAE from 141,000 tokens to only 22,000, an 85% reduction in evaluation cost.

Figures & tables

Panel of Figure 1 — Figure 1: Left : Token usage and accuracy averaged across all LLMs for the tasks in WILD. Tasks vary in token usage, leading to radically different item costs. Right : MAE for predicting the sample mean over 112 held-out tasks. The MIRT model with Optimal Experimental Design selection achieves low MAE for small sample size and token cost.

	Models	Tasks	Items
Perlitz et al. (2024)	44	16	65,000
Kipnis et al. (2025)	5055 *	63	28,632
Zhou et al. (2025)	15	63	16,108
Liao et al. (2025)	70	5	8,918
WILD	65	163	109,564

Table 1: Comparison of WILD to other item-level response datasets.

Figure 2: Left : Baseline selectors vs. Adaptive V-Optimal. Right : Prediction comparison. MIRT v.s. baseline predictors as we increase the number of extrapolation tasks. Model Task Mean (MTM) baselines are constant because extrapolation tasks are fully held out for the test models. See Figure 4 for interpolation results.

Panel of Figure 3 — Figure 3: Summary of main results comparing the performance of the Nested MIRT and 2PL IRT model. (Left) Comparison in the static selection setting. (Center) Comparison in the adaptive selection setting. (Right) Comparison of token efficiency in the adaptive setting. In all three plots, Random selection corresponds to the Prediction setting ( Section 6.2 ).

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Full comparison of the (Nested) MIRT model against the IRT and Model Task Mean predictors. Model Task Mean (EB) represents the empirical bayes version of the estimator. Assessors receive random samples from

T

calibration tasks (interpolation) and then must predict how the model will perform on those

T

calibration tasks and on

128-T

validation tasks (extrapolation). The MIRT model validation MAE drops both when the number of calibration tasks or the number of samples increases.

Method	2	4	8	16	32	64	128
				Items
IRT 1PL	11.76	10.03	8.65	6.91	5.33	4.19	3.52
IRT 2PL + Fisher	8.93	7.68	6.48	5.39	4.67	3.90	3.41
MIRT + VOptimal	8.53	7.43	6.42	5.46	4.58	3.94	3.38
Linear Regression	10.20	8.86	7.90	6.33	5.29	4.29	3.74
ModelTaskMean (EB)	15.17	14.05	10.55	8.52	6.35	5.01	4.11

Appendix

Table 2: Interpolation MAE (%) for

n_{\text{tasks}}=1

. Each method uses its best available selector. Lower is better; bold marks the best method per column. This setting simulates prior work that focused on compressing a single benchmark.

Figure 5: Scree plot and parallel analysis showing the most important factors in the training response matrix. A single factor explains almost 70% of variance, and two more latent factors can be selected via the elbow method.

Figure 6: Comparison of cost discounted models to the minimum cost baseline. All results presented for 16 calibration tasks.

Figure 7: Analysis of interpolation and extrapolation prediction performance for Sonnet 4.6 at different reasoning budgets. Our method is robust to the model behavior changes related to newer reasoning abilities with large thinking budgets.

Calibration Tasks	Held-out Tasks
BoolQ	BBH (Object Counting)
ChemBench (Organic Chemistry)	BigCodeBench
DROP	GSM8K
HellaSwag	MedQA
MATH (Precalculus)	MMLU-Pro (Health)
PAWS	MuSR

Appendix

Table 3: Tasks used for Sonnet 4.6 evaluation.

Figure 8: Ablation over

K

value for the MIRT model. Extrapolation MAE as a % difference from

K

=1. These results reflect the vanilla MIRT model, not Nested MIRT.

Figure 9: Comparison of MIRT with the Nested MIRT model for adaptive experiments. We plot the IRT 2PL model as a baseline comparison. Both MIRT models have

K

=3 parameters (including the general loading).

Figure 10: Comparison of adaptive and static selection on the same plot.

Figure 11: Comparison of the task selection rates between the V-Optimal and cost-aware V-Optimal (V-Optimal-

\tau

) when selecting 64 calibration items. Dotted line represents expected amount of occurrences if the selection was random.

Figure 12: Results breakdown per-task for adaptive models with 16 calibration tasks and 16 calibration items. Left : Relative MAE comparison of the IRT-2PL model with Fisher selection vs. MIRT with VOptimal selection. Further left in blue means that the 2PL model attained a lower MAE, whereas further right in red means that the MIRT model had a lower MAE. Right : Absolute MAE comparisons per task. Lower is better.

Figure 13: Results breakdown per-task for adaptive models with 16 calibration tasks and 128 calibration items. Left : Relative MAE comparison of the IRT-2PL model with Fisher selection vs. MIRT with VOptimal selection. Further left in blue means that the 2PL model attained a lower MAE, whereas further right in red means that the MIRT model had a lower MAE. Right : Absolute MAE comparisons per task. Lower is better.

Figure 14: Comparison of assessor performance for the 5% best-performing, i.e., lowest MAE tasks (left) and 5% worst-performing, i.e., highest MAE tasks (right).

Figure 15: Results breakdown per-model for adaptive models with 16 calibration tasks and 16 calibration items. Left : Relative MAE comparison of the IRT-2PL model with Fisher selection vs. MIRT with VOptimal selection. Further left in blue means that the 2PL model attained a lower MAE, whereas further right in red means that the MIRT model had a lower MAE. Right : Absolute MAE comparisons per task. Lower is better.

Figure 16: Results breakdown per-model for adaptive models with 16 calibration tasks and 128 calibration items. Left : Relative MAE comparison of the IRT-2PL model with Fisher selection vs. MIRT with VOptimal selection. Further left in blue means that the 2PL model attained a lower MAE, whereas further right in red means that the MIRT model had a lower MAE. Right : Absolute MAE comparisons per task. Lower is better.

Category	Models	Tasks	Access
Training	$\mathcal{M}_{train}$	$\mathcal{T}^{c}\cup\mathcal{T}^{v}$	Full obs.
Calibration	$\mathcal{M}_{test}$	$\mathcal{T}^{c}$ (item subset)	Test-time
Interpolation	$\mathcal{M}_{test}$	$\mathcal{T}^{c}$ (held-out items)	Predicted
Extrapolation	$\mathcal{M}_{test}$	$\mathcal{T}^{v}$	Predicted

Appendix

Table 4: Data partitioning schema for calibration and validation.

Dataset	Task	Items
WinoGrande Sakaguchi et al. (2021)		1,267
TruthfulQA Lin et al. (2022)		817
SQuAD Rajpurkar et al. (2016)		11,866
RACE-H Lai et al. (2017)		3,498
Pre-Flight Inspect Evals (2025)		300
PIQA Bisk et al. (2020)		1,803

Appendix

Table 5: Benchmarks, tasks, and number of unique evaluation items contained in WILD

Dataset	Task	Items
MMLU Hendrycks et al. (2021a)	Professional Law	1,534
	Moral Scenarios	895
	Miscellaneous	783
	Professional Psychology	609
	High School Psychology	534
	High School Macroeconomics	386

Appendix

Table 22

Dataset	Task	Items
GSM-Symbolic Mirzadeh et al. (2025)	P1	5,000
GSM-Symbolic Mirzadeh et al. (2025)	P2	2,500
Finance Fundamentals Krumdick et al. (2024)	Finknow	131
	TAT-QA (Extract)	129
	CodeTAT-QA	105
	ConvFinQA (Extract)	94

Appendix

Table 23

Model	Access?
Claude 3 Haiku	API
Claude 3 Sonnet	API
Claude 3.5 Haiku	API
Claude 3.5 Sonnet	API
GPT 4.1	API
GPT 4.1 mini	API

Appendix

Table 6: List of models contained in WILD.

Figure 17: Pearson correlation between LLM performance across all base datasets. Datasets are ordered based on hierarchical clustering of their correlation strength.

Figure 18: Pearson correlation between LLM performance across all sub-tasks. Tasks are ordered based on hierarchical clustering of their correlation strength.

Explore similar work

Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks
Oct 26, 2025Peiyu Li, Xiuxiu Tang, Si Chen +4LLM Evaluation Psychometric Validation
University of Notre Dame, Notre Dame, USA
Efficient Benchmarking Is Just Feature Selection and Multiple Regression
May 25, 2026Sam Bowyer, Acyr Locatelli, Kris CaoLLM Evaluation Feature Selection
Cohere, University of Bristol · Work done during an internship at Cohere. · Cohere
An Interpretable and Scalable Framework for Evaluating Large Language Models
May 7, 2026Xinhao Qu, Qiang Heng, Hao Zeng +1LLM Evaluation Item Response Theory
Department of Statistics, University of California, Riverside · School of Statistics and Data Science, Southeast University · Department of Statistics and Data Science, Southern University of Science and Technology

Oct 26, 2025J/K move · Enter open · S save

Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks

Peiyu Li, Xiuxiu Tang, Si Chen, Ying Cheng+3

University of Notre Dame, Notre Dame, USA

LLM Evaluation Psychometric Validation Item Response Theory

Open paper AskPDF arXiv

Evaluating large language models (LLMs) typically requires thousands of benchmark items, making the process expensive, slow, and increasingly impractical at scale. Existing evaluation protocols rely on average accuracy over fixed item sets, treating all items as equally informative despite substantial variation in difficulty and discrimination. We introduce ATLAS, an adaptive testing framework based on Item Response Theory (IRT) that estimates model ability using Fisher information-guided item selection. ATLAS reduces the number of required items by up to 90% while maintaining measurement precision. For instance, it matches whole-bank ability estimates using only 41 items (0.157 MAE) on HellaSwag (5,600 items). We further reconstruct accuracy from ATLAS's ability estimates and find that reconstructed accuracies closely match raw accuracies across all five benchmarks, indicating that ability

θ

preserves the global performance structure. At the same time,

θ

provides finer discrimination within accuracy-equivalent models: among more than 3,000 evaluated models, 23-31% shift by more than 10 rank positions, and models with identical accuracies receive meaningfully different ability estimates. Code and calibrated item banks are available at https://github.com/Peiyu-Georgia-Li/ATLAS.git.

Abstract

Explore similar work

Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks

Efficient Benchmarking Is Just Feature Selection and Multiple Regression

An Interpretable and Scalable Framework for Evaluating Large Language Models