A benchmark is not merely a collection of test questions; it naturally induces an empirical measure over difficulty. In this work, we develop this measure-based view of benchmark evaluation. This perspective allows us to identify and formalize two central failure modes: measure mismatch, where the benchmark difficulty distribution differs from the intended difficulty distribution, and coverage gaps, where uncovered regions along the difficulty axis prevent benchmarks from detecting meaningful localized training progress. We propose two diagnostic metrics that capture these two failure modes and provide rigorous theoretical guarantees. Empirically, we analyze 25 popular benchmarks spanning 7 domains using carefully estimated item difficulties. Our results show that many benchmarks implicitly overweight certain difficulty regions, so their leaderboard conclusions should be interpreted with caution; others leave substantial portions of the difficulty axis uncovered, limiting their ability to monitor the model improvement process. Finally, we turn diagnosis into action by introducing BenchFill, a simple difficulty-targeted item rewriting algorithm that effectively reduces local coverage gaps.
Figures & tables
Figure 1: Two benchmark failure cases. In both panels, the curves show true model conditional performance as a function of question difficulty . (a) Measure Mismatch. The benchmark emphasizes easy questions, where the models perform similarly, while the difficulty distribution we care about places more weight on harder questions, where they differ substantially. Because the benchmark and the target distribution weight the difficulty axis differently, they lead to different conclusions for the same two models. (b) Coverage Gaps. A model actually improves substantially over a localized difficulty region. Because the benchmark has no questions in that region, this improvement is not reflected in the observed benchmark score, causing meaningful training progress to be missed.
Figure 2: Overview of our benchmark difficulty estimation and metric calculation pipeline. We collect empirical success rates from multiple LLMs on benchmark items, diagnose the response matrix, fit a 1PL IRT model to estimate item difficulties, perform post-fit IRT diagnostics, and compute two metrics over the normalized difficulty distribution.
Figure 3: Benchmark difficulty distributions. Each panel shows the ECDF of normalized item difficulties for one benchmark, together with the neutral uniform reference. Colored shaded intervals mark the five largest adjacent difficulty gaps, with labels denoting the corresponding gap sizes. Additional benchmark examples are provided in Fig. 6 .
Benchmark
DDM
MaxGap
Top-5Sum
#
MATH-500
0.164
0.058
0.175
500
Minerva-Math
0.122
0.031
0.149
272
OlympiadBench
0.064
0.048
0.142
675
Gaokao2023en
0.088
0.055
0.183
385
AMC23
0.089
0.217
0.539
40
AIME24
0.077
0.179
0.502
30
Table 1: Summary of proposed metrics across benchmarks. DDM denotes the 1-Wasserstein distance between the benchmark empirical difficulty distribution and a same-size empirical uniform distribution on [0,1] . MaxGap is the largest gap between consecutive normalized difficulties, and Top-5Sum is the sum of the five largest such gaps. # denotes the number of benchmark items. Lower is better for all three metrics. Highlighted entries mark comparatively large values for each metric: DDM≥0.10 , MaxGap≥0.10 , or Top-5Sum≥0.30 .
Figure 4: Overview of BenchFill . BenchFill iteratively rewrites parent items into child variants, annotates their difficulties, and updates the memory to guide the next round of gap filling.
Benchmark
Top-3 Gap Sum Reduction (%) ↑
Target Dev. ↓
Baseline
BenchFill
Oracle
Baseline
BenchFill
AIME24
6.90
35.57
35.57
0.057
0.005
AIME25
27.44
37.56
37.56
0.016
0.005
AIME26
17.94
33.04
33.04
0.045
0.002
AMC23
24.47
35.19
35.78
0.009
0.003
Gaokao2023en
11.45
38.88
45.47
0.152
0.004
Table 2: Gap-filling results. Top-3 Gap Sum Reduction reports the relative reduction (%) from the original benchmark, where larger is better. Target Dev. is the mean absolute deviation between the selected item difficulty and the target difficulty over three filled gaps, where lower is better. Baseline uses a budget-matched best-of- N uniform-parent rewriting strategy. Oracle is an idealized upper bound that sequentially inserts three items, each at the midpoint of the current largest gap, rather than an actual gap-filling method.
Figure 5: Difficulty distributions of four benchmarks before and after gap filling. Orange and blue curves show the empirical difficulty distributions before and after applying BenchFill , respectively. Shaded regions indicate the largest original coverage gaps. The annotated values show the maximum gaps before and after filling.
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Domain
# Examples
# Rollouts / Item
MATH-500 [ 30 ]
Math
500
8
Minerva-Math [ 27 ]
Math
272
8
OlympiadBench [ 15 ]
Math
675
8
Gaokao2023en [ 72 ]
Math
385
8
AMC23 [ 39 ]
Math
40
32
AIME24 [ 40 ]
Math
30
32
Appendix
Table 3: Summary of the benchmarks used in our analysis. For compactness, we abbreviate SWE-bench Verified (Mini) as SWE-bench-Mini, and BIG-Bench Hard-Logical Deduction-{3,5,7} objects as BBH-{3,5,7} objects throughout the paper.
Model
Size
Provider
Temperature
Top- p
Qwen3.5-397B-A17B
397B
Alibaba Qwen
0.6
0.95
GLM-4.6
357B
Zhipu AI
1.0
0.95
GLM-4.5-Air
110B
Zhipu AI
0.6
0.95
Qwen3-Next-80B-A3B-Instruct
80B
Alibaba Qwen
0.7
0.8
QwQ-32B
32B
Alibaba Qwen
0.6
0.95
Olmo-3.1-32B-Think
32B
Allen AI
0.6
0.95
Appendix
Table 4: Summary of selected models for non-agentic tasks.
Benchmark
Model Information ↑
Item Information ↑
Model Inversion Rate ↓
Item Inversion Rate ↓
AMC23
0.458
0.867
0.062
0.177
AIME24
0.514
0.824
0.040
0.101
AIME25
0.515
0.806
0.057
0.118
AIME26
0.465
0.831
0.046
0.089
Gaokao2023en
0.661
0.681
0.101
0.112
HumanEval
0.546
0.675
0.165
0.205
Appendix
Table 5: Response matrix diagnostic results. ↑ indicates that larger values are preferred and ↓ indicates that smaller values are preferred. We use model information ≥0.3 , item information ≥0.3 , model inversion rate ≤0.3 , and item inversion rate ≤0.3 as basic thresholds for response matrix diagnostics. All benchmarks pass these checks.
Benchmark
Max R^↓
Min Bulk ESS ↑
# Divergent Transitions ↓
AMC23
1.01
334
0
AIME24
1.02
272
0
AIME25
1.01
246
0
AIME26
1.00
385
0
Gaokao2023en
1.01
569
0
HumanEval
1.01
826
0
Appendix
Table 6: MCMC diagnostic results for IRT fitting. ↑ indicates that larger values are preferred and ↓ indicates that smaller values are preferred.
Benchmark
Pearson Mean
Pearson Min
Spearman Mean
Spearman Min
AMC23
0.9962
0.9820
0.9921
0.9484
AIME24
0.9985
0.9940
0.9977
0.9920
AIME25
0.9978
0.9904
0.9941
0.9809
AIME26
0.9979
0.9937
0.9972
0.9888
Gaokao2023en
0.9974
0.9953
0.9957
0.9905
HumanEval
0.9902
0.9771
0.9871
0.9735
Appendix
Table 7: Leave-one-out robustness of IRT difficulty estimation. For each benchmark, we fit IRT using the full set of 12 models, then refit IRT after removing one model at a time and compare the resulting item difficulties with those from the full fit. We report the mean and minimum Pearson and Spearman correlations across the 12 leave-one-out refits. The consistently high correlations indicate that our selected 12 models are sufficient for stable item difficulty estimation.
Benchmark
BIC1PL
BIC2PL
ΔBIC
Selected
ScienceQA
126817
151726
+24909
1PL
MMLU-Pro-Physics
44290
52569
+8279
1PL
MedQA
52414
60340
+7926
1PL
ZebraLogic
22000
29319
+7319
1PL
MMLU-Pro-Chemistry
37446
44645
+7199
1PL
MMLU-Pro-Biology
21712
25399
+3687
1PL
Appendix
Table 8: BIC comparison between 1PL and 2PL across all benchmarks. Lower BIC is preferred. We define ΔBIC=BIC2PL−BIC1PL , so positive values favor 1PL. BIC selects 1PL for 21 out of 25 benchmarks.
Benchmark
1PL
2PL
Max R^↓
Min Bulk ESS ↑
# Div. ↓
Max R^↓
Min Bulk ESS ↑
# Div. ↓
Gaokao2023en
1.01
569
0
1.06
63
983
AIME25
1.01
246
0
1.02
188
18
AIME24
1.02
272
0
1.04
118
2
AMC23
1.01
334
0
1.03
146
8
Appendix
Table 9: Posterior diagnostics for benchmarks where BIC favors 2PL. We report the maximum R^ , minimum bulk ESS, and number of divergent transitions across fitted parameters. Although BIC favors 2PL on these four benchmarks, the 2PL fits exhibit substantially worse sampling diagnostics, while the corresponding 1PL fits remain well-behaved.
Benchmark
MIRT-2
MIRT-3
Max R^↓
Min Bulk ESS ↑
# Div. ↓
Max R^↓
Min Bulk ESS ↑
# Div. ↓
AIME24
1.02
202
0
1.68
6
0
AIME25
1.75
6
0
1.02
172
0
AIME26
1.54
7
0
1.44
8
0
AMC23
1.75
6
0
1.35
10
0
Minerva-Math
1.73
6
0
1.37
10
0
Appendix
Table 10: Convergence diagnostics for multidimensional IRT fits on smaller benchmarks. We report the maximum R^ , minimum bulk ESS, and number of divergent transitions across fitted parameters. The multidimensional fits generally exhibit poor mixing, with high maximum R^ and very small minimum bulk ESS, indicating that they do not provide stable alternative difficulty estimates.
Figure 6: Additional benchmark difficulty distributions. Each panel shows the ECDF of normalized item difficulties for one benchmark. The black curve denotes the benchmark ECDF, and the gray dashed line denotes the uniform ECDF, which we use as the target difficulty distribution. Deviations from the uniform ECDF indicate mismatch relative to this neutral target distribution. Colored shaded intervals mark the five largest adjacent gaps between sorted item difficulties.
Figure 7: First-round alignment of specified target difficulty and annotated difficulty. Each panel corresponds to one benchmark. The horizontal axis shows the specified target difficulty, and the vertical axis shows the annotated difficulty of valid first-round children. Blue points denote individual generated children, and red markers denote the mean and standard deviation for each target difficulty. The dashed diagonal indicates perfect alignment. For readability, both axes are shown on a 0–10 scale, corresponding to normalized difficulties in [0,1] multiplied by 10. Within most benchmarks, higher specified targets produce higher annotated difficulties already in the first round, before any iterative feedback is incorporated.
Figure 8: Multi-round target alignment during BenchFill search. Each panel shows one representative gap-filling case from a different benchmark. The horizontal axis denotes the generation round, and the vertical axis denotes the absolute target deviation, ∣z^−z⋆∣ , for generated children. For readability, target difficulties and absolute deviations are multiplied by 10. Blue points denote individual generated children, and red markers denote the round-level mean and standard deviation. Across most cases, the mean target deviation decreases over rounds, and the generated children become more concentrated around the target, showing that the search-and-annotate loop further improves target alignment beyond the rewriter’s initial difficulty sensitivity.
Case
Benchmark variant
Gain
One-sided p
Detected
AIME24 Olmo-3.1-32B-Think
Original
+0.104
0.500100
×
Original + BenchFill
+2.419
0.005940
✓
Original + Baseline
+0.202
0.458000
×
AIME25 Phi-4
Original
+1.042
0.184100
×
Original + BenchFill
+3.730
0.000450
✓
Original + Baseline
+0.907
0.214200
×
Appendix
Table 12: Localized-improvement detection results. Each augmentation method adds one item to the original 30-item benchmark. Gain denotes the change in average accuracy. The one-sided stratified randomization test randomly reassigns rollout outcomes between the pre- and post-SFT conditions within each item using 100,000 permutations. Boldface indicates variants for which a positive gain is detected at p<0.05 .
Benchmark
Original benchmark ancestor
Selected gap-filling item
Item
Diff.
Item
Diff.
AMC23
For how many integers n does the expression logn−\pagecolorred!203log(n2)−(logn)2 represent a real number, where log denotes the base- 10 logarithm?
0.535
Find the number of integer values of n such that logn−\pagecolorred!201log(n2)−(logn)2 represents a real number, where log denotes the base- 10 logarithm.
0.597(0.598)
Gaokao2023en
On an 8×8 chessboard, 6 black rooks and k white rooks are placed on different cells so that each rook only attacks rooks of the opposite color. Compute the maximum possible value of k . (Two rooks attack each other if they are in the same row or column and no rooks are between them.)
0.864
On a 9×9 chessboard, 7 black rooks and k white rooks are placed on distinct squares so that each rook attacks only rooks of the opposite color. What is the maximum possible value of k ? (Two rooks attack each other if they are in the same row or column and there is no rook between them.)
0.886(0.885)
AIME25
Alex divides a disk into four quadrants with two perpendicular diameters intersecting at the center of the disk. He draws 25 more line segments through the disk, each connecting two points at random on the perimeter in different quadrants . Find the expected number of regions into which these 27 line segments divide the disk.
0.750
A circle has a diameter drawn, splitting it into two semicircular regions . Then 8 chords are drawn, each connecting a random point on the boundary in one semicircle to a random point on the boundary in the other semicircle . Assuming all points are chosen uniformly and independently, what is the expected number of regions into which the interior of the circle is divided?
0.674(0.671)
Appendix
Table 13: Representative pairs of original benchmark ancestors and final selected items produced by BenchFill . All difficulty values are normalized to [0,1] . The main change from the ancestor item to the selected item is highlighted. For selected items, values in parentheses denote the target difficulty.
Do language model benchmarks actually measure what practitioners intend them to ? High-level metadata is too coarse to convey the granular reality of benchmarks: a "poetry" benchmark may never test for haikus, while "instruction-following" benchmarks will often test for an arbitrary mix of skills. This opacity makes verifying alignment with practitioner goals a laborious process, risking an illusion of competence even when models fail on untested facets of user interests. We introduce BenchBrowser, a retriever that surfaces evaluation items relevant to natural language use cases over 20 benchmark suites. Validated by a human study confirming high retrieval precision, BenchBrowser generates evidence to help practitioners diagnose low content validity (narrow coverage of a capability's facets) and low convergent validity (lack of stable rankings when measuring the same capability). BenchBrowser, thus, helps quantify a critical gap between practitioner intent and what benchmarks actually test.
Language Technologies Institute, Carnegie Mellon University, Pittsburgh (PA), United States · Thomas Lord Department of Computer Science, University of Southern California, Los Angeles (CA), United States
The rapid release of both language models and benchmarks makes it increasingly costly to evaluate every model on every dataset. In practice, models are often evaluated on different samples, making scores difficult to compare across studies. To address this, we propose a framework based on multidimensional Item Response Theory (IRT) that uses anchor items to calibrate new benchmarks to the evaluation suite while holding previously calibrated item parameters fixed. Our approach supports a realistic evaluation setting in which datasets are introduced over time and models are evaluated only on the datasets available at the time of evaluation, while a fixed anchor set for each dataset is used so that results from different evaluation periods can be compared directly. In large-scale experiments on more than 400 models, our framework predicts full-evaluation performance within 2-3 percentage points using only 100 anchor questions per dataset, with Spearman ρ≥0.9 for ranking preservation preservation. This shows that benchmark suites can grow over time while preserving score comparability, since adding a new dataset requires running existing models only on that dataset's anchors. Code and data are available at: https://eliyahabba.github.io/growing-pains/
Eliya Habba, Itay Itzhak, Asaf Yehudai +5
The Hebrew University of Jerusalem · Technion · IBM Research +2
LLM benchmarks are increasingly dynamic: instead of containing a fixed set of questions, they define templates and parameters that can generate an effectively unlimited number of question variants. This flexibility is valuable, but it makes evaluation expensive -- especially when the goal is not just determining an average score, but reliably identifying a model's weak spots. This paper introduces a new methodology for identifying hard questions in dynamic benchmarks. It leverages COUP, a recent Bayesian optimization algorithm (Graham, Velez & Leyton-Brown, 2026), after introducing several substantive modifications to make the algorithm suitable for practical LLM pipelines. We also wrap it in a tool that supports flexible choices of datasets and utility functions, enabling users to target the kinds of questions they care about (e.g., low-accuracy questions; questions that are unusually hard relative to their measured complexity). In experiments across a range of benchmarks, we show that our method, dubbed QuickScope, discovers truly difficult questions more sample efficiently than standard baselines, while also reducing false positives from noisy outcomes.
Taylor Lundy, Narun K. Raman, Kevin Leyton-Brown
Department of Computer Science University of British Columbia Vancouver, BC