Growing Pains: Extensible and Efficient LLM Benchmarking Via Fixed Parameter Calibration

Eliya Habba; Itay Itzhak; Asaf Yehudai; Yotam Perlitz; Elron Bandel; Michal Shmueli-Scheuer; Leshem Choshen; Gabriel Stanovsky

cs.CLApr 14, 2026

Growing Pains: Extensible and Efficient LLM Benchmarking Via Fixed Parameter Calibration

Authors: Eliya Habba, Itay Itzhak, Asaf Yehudai, Yotam Perlitz, Elron Bandel, Michal Shmueli-Scheuer, Leshem Choshen, Gabriel Stanovsky

Organizations: The Hebrew University of Jerusalem · Technion · IBM Research · MIT ·

Abstract

The rapid release of both language models and benchmarks makes it increasingly costly to evaluate every model on every dataset. In practice, models are often evaluated on different samples, making scores difficult to compare across studies. To address this, we propose a framework based on multidimensional Item Response Theory (IRT) that uses anchor items to calibrate new benchmarks to the evaluation suite while holding previously calibrated item parameters fixed. Our approach supports a realistic evaluation setting in which datasets are introduced over time and models are evaluated only on the datasets available at the time of evaluation, while a fixed anchor set for each dataset is used so that results from different evaluation periods can be compared directly. In large-scale experiments on more than 400 models, our framework predicts full-evaluation performance within 2-3 percentage points using only 100 anchor questions per dataset, with Spearman $ρ\geq 0.9$ for ranking preservation preservation. This shows that benchmark suites can grow over time while preserving score comparability, since adding a new dataset requires running existing models only on that dataset's anchors. Code and data are available at: https://eliyahabba.github.io/growing-pains/

Figures & tables

Figure 1: Fixed parameter calibration enables extensible evaluation as new benchmarks are added over time. (1) Base datasets (e.g., MMLU, GSM8K) are calibrated jointly on reference models to define initial anchor item parameters. (2) At each subsequent step t, a new dataset is integrated by estimating its item parameters while holding all previously calibrated anchor parameters fixed (locked icons). (3) Once calibrated, the accumulated anchors serve as a compact proxy for the full suite, enabling performance prediction from anchor responses alone.

Benchmark Suite	# Models	# Datasets	Included Datasets (# examples)
Open LLM Leaderboard	395	6	ARC Challenge (1,212), GSM8K (1,359), HellaSwag (10,082), MMLU (14,042), TruthfulQA (857), Winogrande (1,307)
MMLU	428	57	57 subdomains; (100–1,534 per subject)

Table 1: Summary of the benchmark suites used in the experiments.

Dataset	Total Items	N=50	N=100
ARC Challenge	1,212	4.1%	8.3%
GSM8K	1,359	3.7%	7.4%
HellaSwag	10,082	0.5%	1.0%
MMLU	14,042	0.4%	0.7%
TruthfulQA	857	5.8%	11.7%
Winogrande	1,307	3.8%	7.7%

Table 2: Anchor coverage as a percentage of total items for the Open LLM Leaderboard suite (left) and for representative subjects in the MMLU suite (right).

Figure 2: Fixed parameter calibration evaluates previously tested models only on the anchors of each newly added dataset, keeping the evaluation size constant as the suite grows. Concurrent calibration uses an increasingly large anchor set as the suite grows. Random sampling has the same fixed evaluation size but higher prediction error.

Figure 3: A small anchor budget suffices for accurate prediction across chain steps, with diminishing returns from larger sets. Across both benchmark suites, even compact sets (e.g.,

N=10

25

) allow fixed parameter calibration and concurrent calibration to maintain low and stable MAE. As the number of anchors increases, random sampling approaches the performance of IRT-based methods, narrowing the gap between all three approaches. Shaded regions denote 95% confidence intervals.

Figure 4: Prediction error remains low under time-ordered and family-held-out model splits on the Open LLM Leaderboard. Fixed parameter calibration tracks concurrent calibration and outperforms random sampling under both splits.

Figure 5: Prediction error stays low when an out-of-domain dataset is linked last, at increasing chain distance (the number of datasets added before it): GSM8K on the Open LLM Leaderboard (left), and the MMLU STEM (center) or mathematics (right) subjects. Fixed parameter calibration stays no worse than concurrent calibration and ahead of random sampling. Shaded regions denote 95% confidence intervals.

	Open LLM Leaderboard			MMLU
Method \ Anchors ( $N$ )	25	100	200	10	25	50	100
Random	0.82	0.91	0.96	0.68	0.85	0.91	0.98
Concurrent Calibration	0.89	0.94	0.97	0.73	0.83	0.90	0.98
Fixed Parameter Calibration	0.88	0.94	0.97	0.72	0.84	0.90	0.98

Table 3: Ranking preservation (Spearman

\rho

) between predicted and full-evaluation model orderings. IRT-based calibration methods are competitive with random anchor sampling, with the clearest gains at small anchor counts.

Figure 6: The effect of reference model count varies across suites. On the Open LLM Leaderboard (top), a reliable fixed parameter calibration is only achieved once the reference pool reaches 100 or more models. On MMLU (bottom), prediction quality is relatively robust even with only 25 reference models. Shaded regions denote 95% confidence intervals.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Item maps showing the difficulty ( $b$ ) and discrimination ( $a$ ) of all items, with selected anchor items highlighted. Clustering-based selection (circles) distributes anchors across the full parameter space, while top- $K$ selection (diamonds) concentrates them in a narrow high-discrimination region. Representative MMLU subjects.

Figure 8: Item maps showing the difficulty ( $b$ ) and discrimination ( $a$ ) of all items, with selected anchor items highlighted. Clustering-based selection (circles) distributes anchors across the full parameter space, while top- $K$ selection (diamonds) concentrates them in a narrow high-discrimination region. Open LLM Leaderboard datasets.

Figure 9: Replacing the clustering-based anchor selection with top-

K

selection by discrimination parameter leads to substantially higher MAE on the Open LLM Leaderboard, while the rest of the fixed parameter calibration pipeline remains identical. Shaded regions denote 95% confidence intervals.

Figure 10: Replacing the clustering-based anchor selection with top-

K

selection by discrimination parameter leads to substantially higher MAE on MMLU, while the rest of the fixed parameter calibration pipeline remains identical. Shaded regions denote 95% confidence intervals.

Figure 11: Prediction error remains low under time-ordered (top) and family-held-out (bottom) model splits, on the Open LLM Leaderboard (left) and MMLU (right). Fixed parameter calibration tracks concurrent calibration and outperforms random sampling, matching the random-split result in Figure 3 . Shaded regions denote 95% confidence intervals.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

DualEval: Joint Model-Item Calibration for Unified LLM Evaluation
Jun 24, 2026Aaron J. Li, Hao Huang, Youngmin Park +6LLM Evaluation Language Model Generation Evaluation
University of California, Berkeley · Arena · University of California, Los Angeles
Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks
Oct 26, 2025Peiyu Li, Xiuxiu Tang, Si Chen +4LLM Evaluation Psychometric Validation
University of Notre Dame, Notre Dame, USA
Efficient Safety Benchmarking via Item Response Theory
May 26, 2026Fabio Spagliardi, Mírian Silva, Ayan Datta +3LLM Safety Benchmarks Language Model Safety Evaluation
Supervised Program for Alignment Research (SPAR) · Yale University · Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)

Jun 24, 2026J/K move · Enter open · S save

DualEval: Joint Model-Item Calibration for Unified LLM Evaluation

Aaron J. Li, Hao Huang, Youngmin Park, Yitong Ma+5

University of California, Berkeley · Arena · University of California, Los Angeles

LLM Evaluation Language Model Generation Evaluation Item Response Theory

Open paper AskPDF arXiv

Current LLM evaluation relies on two complementary but often disconnected signals: static benchmarks with objective correctness labels and arena-style preference data that better reflect open-ended user interactions. We introduce DualEval, a latent model-item calibration framework that represents models and evaluation items in a shared space, jointly estimating model ability together with item difficulty and sharpness. We apply DualEval across four domains: coding, math, miscellaneous domain-knowledge tasks, and generic everyday user queries. Our evaluation uses 18 frontier LLMs, static benchmark labels, and reward-model scores validated against held-out human preferences for open-ended model responses. Empirically, our framework produces reliable and balanced model rankings, and its learned item-level profiles support downstream applications such as benchmark compression for sample-efficient evaluation and anomaly detection for contamination or outlier analysis. Overall, DualEval unifies static and arena-style evaluation through joint model-item calibration, producing model rankings and item-level diagnostics that support more sample-efficient, interpretable, and auditable evaluation pipelines.

Abstract

LLM Evaluation

Explore similar work

DualEval: Joint Model-Item Calibration for Unified LLM Evaluation

Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks

Efficient Safety Benchmarking via Item Response Theory