Thousands of diverse benchmarks have been developed to measure the quality of large language models (LLMs). Yet prior work has demonstrated that LLM performance is often sufficiently explained by a small set of latent factors, or abilities. This suggests the potential for more efficient and principled benchmarking, but it remains difficult to compare the quality of different methods. Motivated by predictive validity, we argue that the quality of a benchmarking framework should be grounded in how efficiently it enables the prediction of model performance on unseen tasks. To analyze this objective, we collect the "Wide-scale Item Level Dataset" (WILD), a dataset of item-model response pairs, comprising evaluations of 65 models on 109,564 unique items spanning 163 tasks drawn from 27 datasets. This dataset enables the first analysis of how different techniques can predict a model's performance on a large, diverse collection of unseen tasks under different budget constraints. We demonstrate that combining a modified multidimensional item response theory (IRT) model with adaptive item selection driven by optimal experimental design can predict performance on 112 held-out benchmark tasks with a mean absolute error (MAE) of less than 7%, and can do so after observing only 16 items. We further demonstrate that incorporating cost-aware discount factors into our selection criteria can reduce the total tokens needed to reach 7% MAE from 141,000 tokens to only 22,000, an 85% reduction in evaluation cost.
Figures & tables
Figure 1: Left : Token usage and accuracy averaged across all LLMs for the tasks in WILD. Tasks vary in token usage, leading to radically different item costs. Right : MAE for predicting the sample mean over 112 held-out tasks. The MIRT model with Optimal Experimental Design selection achieves low MAE for small sample size and token cost.
Models
Tasks
Items
Perlitz et al. (2024)
44
16
65,000
Kipnis et al. (2025)
5055 *
63
28,632
Zhou et al. (2025)
15
63
16,108
Liao et al. (2025)
70
5
8,918
WILD
65
163
109,564
Table 1: Comparison of WILD to other item-level response datasets.
Figure 2: Left : Baseline selectors vs. Adaptive V-Optimal. Right : Prediction comparison. MIRT v.s. baseline predictors as we increase the number of extrapolation tasks. Model Task Mean (MTM) baselines are constant because extrapolation tasks are fully held out for the test models. See Figure 4 for interpolation results.
Figure 3: Summary of main results comparing the performance of the Nested MIRT and 2PL IRT model. (Left) Comparison in the static selection setting. (Center) Comparison in the adaptive selection setting. (Right) Comparison of token efficiency in the adaptive setting. In all three plots, Random selection corresponds to the Prediction setting ( Section 6.2 ).
Appendix figures & tables22 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Full comparison of the (Nested) MIRT model against the IRT and Model Task Mean predictors. Model Task Mean (EB) represents the empirical bayes version of the estimator. Assessors receive random samples from T calibration tasks (interpolation) and then must predict how the model will perform on those T calibration tasks and on 128−T validation tasks (extrapolation). The MIRT model validation MAE drops both when the number of calibration tasks or the number of samples increases.
Items
Method
2
4
8
16
32
64
128
IRT 1PL
11.76
10.03
8.65
6.91
5.33
4.19
3.52
IRT 2PL + Fisher
8.93
7.68
6.48
5.39
4.67
3.90
3.41
MIRT + VOptimal
8.53
7.43
6.42
5.46
4.58
3.94
3.38
Linear Regression
10.20
8.86
7.90
6.33
5.29
4.29
3.74
ModelTaskMean (EB)
15.17
14.05
10.55
8.52
6.35
5.01
4.11
Appendix
Table 2: Interpolation MAE (%) for ntasks=1 . Each method uses its best available selector. Lower is better; bold marks the best method per column. This setting simulates prior work that focused on compressing a single benchmark.
Figure 5: Scree plot and parallel analysis showing the most important factors in the training response matrix. A single factor explains almost 70% of variance, and two more latent factors can be selected via the elbow method.
Figure 6: Comparison of cost discounted models to the minimum cost baseline. All results presented for 16 calibration tasks.
Figure 7: Analysis of interpolation and extrapolation prediction performance for Sonnet 4.6 at different reasoning budgets. Our method is robust to the model behavior changes related to newer reasoning abilities with large thinking budgets.
Calibration Tasks
Held-out Tasks
BoolQ
BBH (Object Counting)
ChemBench (Organic Chemistry)
BigCodeBench
DROP
GSM8K
HellaSwag
MedQA
MATH (Precalculus)
MMLU-Pro (Health)
PAWS
MuSR
Appendix
Table 3: Tasks used for Sonnet 4.6 evaluation.
Figure 8: Ablation over K value for the MIRT model. Extrapolation MAE as a % difference from K =1. These results reflect the vanilla MIRT model, not Nested MIRT.
Figure 9: Comparison of MIRT with the Nested MIRT model for adaptive experiments. We plot the IRT 2PL model as a baseline comparison. Both MIRT models have K =3 parameters (including the general loading).
Figure 10: Comparison of adaptive and static selection on the same plot.
Figure 11: Comparison of the task selection rates between the V-Optimal and cost-aware V-Optimal (V-Optimal- τ ) when selecting 64 calibration items. Dotted line represents expected amount of occurrences if the selection was random.
Figure 12: Results breakdown per-task for adaptive models with 16 calibration tasks and 16 calibration items. Left : Relative MAE comparison of the IRT-2PL model with Fisher selection vs. MIRT with VOptimal selection. Further left in blue means that the 2PL model attained a lower MAE, whereas further right in red means that the MIRT model had a lower MAE. Right : Absolute MAE comparisons per task. Lower is better.
Figure 13: Results breakdown per-task for adaptive models with 16 calibration tasks and 128 calibration items. Left : Relative MAE comparison of the IRT-2PL model with Fisher selection vs. MIRT with VOptimal selection. Further left in blue means that the 2PL model attained a lower MAE, whereas further right in red means that the MIRT model had a lower MAE. Right : Absolute MAE comparisons per task. Lower is better.
Figure 14: Comparison of assessor performance for the 5% best-performing, i.e., lowest MAE tasks (left) and 5% worst-performing, i.e., highest MAE tasks (right).
Figure 15: Results breakdown per-model for adaptive models with 16 calibration tasks and 16 calibration items. Left : Relative MAE comparison of the IRT-2PL model with Fisher selection vs. MIRT with VOptimal selection. Further left in blue means that the 2PL model attained a lower MAE, whereas further right in red means that the MIRT model had a lower MAE. Right : Absolute MAE comparisons per task. Lower is better.
Figure 16: Results breakdown per-model for adaptive models with 16 calibration tasks and 128 calibration items. Left : Relative MAE comparison of the IRT-2PL model with Fisher selection vs. MIRT with VOptimal selection. Further left in blue means that the 2PL model attained a lower MAE, whereas further right in red means that the MIRT model had a lower MAE. Right : Absolute MAE comparisons per task. Lower is better.
Category
Models
Tasks
Access
Training
Mtrain
Tc∪Tv
Full obs.
Calibration
Mtest
Tc (item subset)
Test-time
Interpolation
Mtest
Tc (held-out items)
Predicted
Extrapolation
Mtest
Tv
Predicted
Appendix
Table 4: Data partitioning schema for calibration and validation.
Dataset
Task
Items
WinoGrande Sakaguchi et al. (2021)
1,267
TruthfulQA Lin et al. (2022)
817
SQuAD Rajpurkar et al. (2016)
11,866
RACE-H Lai et al. (2017)
3,498
Pre-Flight Inspect Evals (2025)
300
PIQA Bisk et al. (2020)
1,803
Appendix
Table 5: Benchmarks, tasks, and number of unique evaluation items contained in WILD
Dataset
Task
Items
MMLU Hendrycks et al. (2021a)
Professional Law
1,534
Moral Scenarios
895
Miscellaneous
783
Professional Psychology
609
High School Psychology
534
High School Macroeconomics
386
Appendix
Table 22
Dataset
Task
Items
GSM-Symbolic Mirzadeh et al. (2025)
P1
5,000
P2
2,500
Finance Fundamentals Krumdick et al. (2024)
Finknow
131
TAT-QA (Extract)
129
CodeTAT-QA
105
ConvFinQA (Extract)
94
Appendix
Table 23
Model
Access?
Claude 3 Haiku
API
Claude 3 Sonnet
API
Claude 3.5 Haiku
API
Claude 3.5 Sonnet
API
GPT 4.1
API
GPT 4.1 mini
API
Appendix
Table 6: List of models contained in WILD.
Figure 17: Pearson correlation between LLM performance across all base datasets. Datasets are ordered based on hierarchical clustering of their correlation strength.
Figure 18: Pearson correlation between LLM performance across all sub-tasks. Tasks are ordered based on hierarchical clustering of their correlation strength.
Evaluating large language models (LLMs) typically requires thousands of benchmark items, making the process expensive, slow, and increasingly impractical at scale. Existing evaluation protocols rely on average accuracy over fixed item sets, treating all items as equally informative despite substantial variation in difficulty and discrimination. We introduce ATLAS, an adaptive testing framework based on Item Response Theory (IRT) that estimates model ability using Fisher information-guided item selection. ATLAS reduces the number of required items by up to 90% while maintaining measurement precision. For instance, it matches whole-bank ability estimates using only 41 items (0.157 MAE) on HellaSwag (5,600 items). We further reconstruct accuracy from ATLAS's ability estimates and find that reconstructed accuracies closely match raw accuracies across all five benchmarks, indicating that ability θ preserves the global performance structure. At the same time, θ provides finer discrimination within accuracy-equivalent models: among more than 3,000 evaluated models, 23-31% shift by more than 10 rank positions, and models with identical accuracies receive meaningfully different ability estimates. Code and calibrated item banks are available at https://github.com/Peiyu-Georgia-Li/ATLAS.git.
Efficient benchmarking techniques aim to lower the computational cost of evaluating LLMs by predicting full benchmark scores using only a subset of a benchmark's questions. By reframing this problem as an instance of multiple regression with feature selection, we find that existing efficient benchmarking methods can be greatly improved by simply using kernel ridge regression at the prediction stage. Additionally, using an information-theoretic feature-selection algorithm called minimum redundancy maximum relevance (mRMR), we can further improve upon these methods by selecting question subsets that will be maximally useful for prediction. Except in very data-poor settings, these approaches consistently achieve smaller prediction errors (in both MAE and RMSE), and greater ranking correlation between predicted and true scores (in both Spearman ρ and Kendall τ) across a range of benchmarks using both binary and continuous metrics. Furthermore, mRMR subsampling is much faster than competitor methods (which often involve fitting probabilistic models or running clustering algorithms), and is more likely to select the same questions under different random seeds or training data splits. Tutorial code can be found at https://github.com/sambowyer/mrmr_eval .
Sam Bowyer, Acyr Locatelli, Kris Cao
Cohere, University of Bristol · Work done during an internship at Cohere. · Cohere
Evaluation of large language models (LLMs) is increasingly critical, yet standard benchmarking methods rely on average accuracy, overlooking both the inherent stochasticity of LLM outputs and the heterogeneity of benchmark items. Item Response Theory (IRT) offers a principled framework for modeling latent model abilities and item characteristics, but conventional methods are computationally expensive and numerically unstable, limiting large-scale implementations. To address these challenges, we propose an interpretable and scalable framework for LLM evaluation based on the majorization-minimization principle. Our approach reformulates the problem as a sequence of constrained matrix factorization subproblems, enabling stable and efficient parameter estimation with theoretical guarantees for identifiability and convergence. Experiments on synthetic and real-world datasets, including MATH-500 and six Open LLM Leaderboard benchmarks, demonstrate that our method achieves superior scalability and interpretability. It delivers orders-of-magnitude speedups over competing methods while maintaining comparable or even higher estimation accuracy. Our results align with established scaling laws and offer insights into item difficulty and discrimination, informing more principled benchmark design.
Xinhao Qu, Qiang Heng, Hao Zeng +1
Department of Statistics, University of California, Riverside · School of Statistics and Data Science, Southeast University · Department of Statistics and Data Science, Southern University of Science and Technology