Your Benchmark Is an Empirical Measure Over Difficulty

Yifan Sun; Naicheng Yu; Jiawen Gong; Jingyan Shen; Huan Zhang

cs.LGOct 21, 2026

Your Benchmark Is an Empirical Measure Over Difficulty

Authors: Yifan Sun, Naicheng Yu, Jiawen Gong, Jingyan Shen, Huan Zhang

Organizations: University of Illinois Urbana-Champaign · University of California San Diego · New York University

Abstract

A benchmark is not merely a collection of test questions; it naturally induces an empirical measure over difficulty. In this work, we develop this measure-based view of benchmark evaluation. This perspective allows us to identify and formalize two central failure modes: measure mismatch, where the benchmark difficulty distribution differs from the intended difficulty distribution, and coverage gaps, where uncovered regions along the difficulty axis prevent benchmarks from detecting meaningful localized training progress. We propose two diagnostic metrics that capture these two failure modes and provide rigorous theoretical guarantees. Empirically, we analyze 25 popular benchmarks spanning 7 domains using carefully estimated item difficulties. Our results show that many benchmarks implicitly overweight certain difficulty regions, so their leaderboard conclusions should be interpreted with caution; others leave substantial portions of the difficulty axis uncovered, limiting their ability to monitor the model improvement process. Finally, we turn diagnosis into action by introducing BenchFill, a simple difficulty-targeted item rewriting algorithm that effectively reduces local coverage gaps.

Figures & tables

Figure 1: Two benchmark failure cases. In both panels, the curves show true model conditional performance as a function of question difficulty . (a) Measure Mismatch. The benchmark emphasizes easy questions, where the models perform similarly, while the difficulty distribution we care about places more weight on harder questions, where they differ substantially. Because the benchmark and the target distribution weight the difficulty axis differently, they lead to different conclusions for the same two models. (b) Coverage Gaps. A model actually improves substantially over a localized difficulty region. Because the benchmark has no questions in that region, this improvement is not reflected in the observed benchmark score, causing meaningful training progress to be missed.

Figure 2: Overview of our benchmark difficulty estimation and metric calculation pipeline. We collect empirical success rates from multiple LLMs on benchmark items, diagnose the response matrix, fit a 1PL IRT model to estimate item difficulties, perform post-fit IRT diagnostics, and compute two metrics over the normalized difficulty distribution.

Figure 3: Benchmark difficulty distributions. Each panel shows the ECDF of normalized item difficulties for one benchmark, together with the neutral uniform reference. Colored shaded intervals mark the five largest adjacent difficulty gaps, with labels denoting the corresponding gap sizes. Additional benchmark examples are provided in Fig. 6 .

Benchmark	$\mathrm{DDM}$	$\mathrm{Max\ Gap}$	$\mathrm{Top\text{-}5\ Sum}$	#
MATH-500	0.164	0.058	0.175	500
Minerva-Math	0.122	0.031	0.149	272
OlympiadBench	0.064	0.048	0.142	675
Gaokao2023en	0.088	0.055	0.183	385
AMC23	0.089	0.217	0.539	40
AIME24	0.077	0.179	0.502	30

Table 1: Summary of proposed metrics across benchmarks.

\mathrm{DDM}

denotes the 1-Wasserstein distance between the benchmark empirical difficulty distribution and a same-size empirical uniform distribution on

[0,1]

\mathrm{Max\ Gap}

is the largest gap between consecutive normalized difficulties, and

\mathrm{Top\text{-}5\ Sum}

is the sum of the five largest such gaps. # denotes the number of benchmark items. Lower is better for all three metrics. Highlighted entries mark comparatively large values for each metric:

\mathrm{DDM}\geq 0.10

\mathrm{Max\ Gap}

\geq 0.10

, or

\mathrm{Top\text{-}5\ Sum}

\geq 0.30

Figure 4: Overview of BenchFill . BenchFill iteratively rewrites parent items into child variants, annotates their difficulties, and updates the memory to guide the next round of gap filling.

Benchmark	Top-3 Gap Sum Reduction (%) $\uparrow$			Target Dev. $\downarrow$
	Baseline	BenchFill	Oracle	Baseline	BenchFill
AIME24	6.90	35.57	35.57	0.057	0.005
AIME25	27.44	37.56	37.56	0.016	0.005
AIME26	17.94	33.04	33.04	0.045	0.002
AMC23	24.47	35.19	35.78	0.009	0.003
Gaokao2023en	11.45	38.88	45.47	0.152	0.004

Table 2: Gap-filling results. Top-3 Gap Sum Reduction reports the relative reduction (%) from the original benchmark, where larger is better. Target Dev. is the mean absolute deviation between the selected item difficulty and the target difficulty over three filled gaps, where lower is better. Baseline uses a budget-matched best-of-

N

uniform-parent rewriting strategy. Oracle is an idealized upper bound that sequentially inserts three items, each at the midpoint of the current largest gap, rather than an actual gap-filling method.

Figure 5: Difficulty distributions of four benchmarks before and after gap filling. Orange and blue curves show the empirical difficulty distributions before and after applying BenchFill , respectively. Shaded regions indicate the largest original coverage gaps. The annotated values show the maximum gaps before and after filling.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Domain	# Examples	# Rollouts / Item
MATH-500 [ 30 ]	Math	500	8
Minerva-Math [ 27 ]	Math	272	8
OlympiadBench [ 15 ]	Math	675	8
Gaokao2023en [ 72 ]	Math	385	8
AMC23 [ 39 ]	Math	40	32
AIME24 [ 40 ]	Math	30	32

Appendix

Table 3: Summary of the benchmarks used in our analysis. For compactness, we abbreviate SWE-bench Verified (Mini) as SWE-bench-Mini, and BIG-Bench Hard-Logical Deduction-{3,5,7} objects as BBH-{3,5,7} objects throughout the paper.

Model	Size	Provider	Temperature	Top- $p$
Qwen3.5-397B-A17B	397B	Alibaba Qwen	0.6	0.95
GLM-4.6	357B	Zhipu AI	1.0	0.95
GLM-4.5-Air	110B	Zhipu AI	0.6	0.95
Qwen3-Next-80B-A3B-Instruct	80B	Alibaba Qwen	0.7	0.8
QwQ-32B	32B	Alibaba Qwen	0.6	0.95
Olmo-3.1-32B-Think	32B	Allen AI	0.6	0.95

Appendix

Table 4: Summary of selected models for non-agentic tasks.

Benchmark	Model Information $\uparrow$	Item Information $\uparrow$	Model Inversion Rate $\downarrow$	Item Inversion Rate $\downarrow$
AMC23	0.458	0.867	0.062	0.177
AIME24	0.514	0.824	0.040	0.101
AIME25	0.515	0.806	0.057	0.118
AIME26	0.465	0.831	0.046	0.089
Gaokao2023en	0.661	0.681	0.101	0.112
HumanEval	0.546	0.675	0.165	0.205

Appendix

Table 5: Response matrix diagnostic results.

\uparrow

indicates that larger values are preferred and

\downarrow

indicates that smaller values are preferred. We use model information

\geq 0.3

, item information

\geq 0.3

, model inversion rate

\leq 0.3

, and item inversion rate

\leq 0.3

as basic thresholds for response matrix diagnostics. All benchmarks pass these checks.

Benchmark	Max $\hat{R}$ $\downarrow$	Min Bulk ESS $\uparrow$
AMC23	1.01	334
AIME24	1.02	272
AIME25	1.01	246
AIME26	1.00	385
Gaokao2023en	1.01	569
HumanEval	1.01	826

Appendix

Table 6: MCMC diagnostic results for IRT fitting.

\uparrow

indicates that larger values are preferred and

\downarrow

indicates that smaller values are preferred.

Benchmark	Pearson Mean	Pearson Min	Spearman Mean	Spearman Min
AMC23	0.9962	0.9820	0.9921	0.9484
AIME24	0.9985	0.9940	0.9977	0.9920
AIME25	0.9978	0.9904	0.9941	0.9809
AIME26	0.9979	0.9937	0.9972	0.9888
Gaokao2023en	0.9974	0.9953	0.9957	0.9905
HumanEval	0.9902	0.9771	0.9871	0.9735

Appendix

Table 7: Leave-one-out robustness of IRT difficulty estimation. For each benchmark, we fit IRT using the full set of 12 models, then refit IRT after removing one model at a time and compare the resulting item difficulties with those from the full fit. We report the mean and minimum Pearson and Spearman correlations across the 12 leave-one-out refits. The consistently high correlations indicate that our selected 12 models are sufficient for stable item difficulty estimation.

Benchmark	$\mathrm{BIC}_{\mathrm{1PL}}$	$\mathrm{BIC}_{\mathrm{2PL}}$	$\Delta\mathrm{BIC}$	Selected
ScienceQA	126817	151726	+24909	1PL
MMLU-Pro-Physics	44290	52569	+8279	1PL
MedQA	52414	60340	+7926	1PL
ZebraLogic	22000	29319	+7319	1PL
MMLU-Pro-Chemistry	37446	44645	+7199	1PL
MMLU-Pro-Biology	21712	25399	+3687	1PL

Appendix

Table 8: BIC comparison between 1PL and 2PL across all benchmarks. Lower BIC is preferred. We define

\Delta\mathrm{BIC}=\mathrm{BIC}_{\mathrm{2PL}}-\mathrm{BIC}_{\mathrm{1PL}}

, so positive values favor 1PL. BIC selects 1PL for 21 out of 25 benchmarks.

Benchmark	1PL			2PL
Benchmark	Max $\hat{R}$ $\downarrow$	Min Bulk ESS $\uparrow$	# Div. $\downarrow$	Max $\hat{R}$ $\downarrow$	Min Bulk ESS $\uparrow$	# Div. $\downarrow$
Gaokao2023en	1.01	569	0	1.06	63	983
AIME25	1.01	246	0	1.02	188	18
AIME24	1.02	272	0	1.04	118	2
AMC23	1.01	334	0	1.03	146	8

Appendix

Table 9: Posterior diagnostics for benchmarks where BIC favors 2PL. We report the maximum

\hat{R}

, minimum bulk ESS, and number of divergent transitions across fitted parameters. Although BIC favors 2PL on these four benchmarks, the 2PL fits exhibit substantially worse sampling diagnostics, while the corresponding 1PL fits remain well-behaved.

Benchmark	MIRT-2			MIRT-3
Benchmark	Max $\hat{R}$ $\downarrow$	Min Bulk ESS $\uparrow$	# Div. $\downarrow$	Max $\hat{R}$ $\downarrow$	Min Bulk ESS $\uparrow$	# Div. $\downarrow$
AIME24	1.02	202	0	1.68	6	0
AIME25	1.75	6	0	1.02	172	0
AIME26	1.54	7	0	1.44	8	0
AMC23	1.75	6	0	1.35	10	0
Minerva-Math	1.73	6	0	1.37	10	0

Appendix

Table 10: Convergence diagnostics for multidimensional IRT fits on smaller benchmarks. We report the maximum

\hat{R}

, minimum bulk ESS, and number of divergent transitions across fitted parameters. The multidimensional fits generally exhibit poor mixing, with high maximum

\hat{R}

and very small minimum bulk ESS, indicating that they do not provide stable alternative difficulty estimates.

Figure 6: Additional benchmark difficulty distributions. Each panel shows the ECDF of normalized item difficulties for one benchmark. The black curve denotes the benchmark ECDF, and the gray dashed line denotes the uniform ECDF, which we use as the target difficulty distribution. Deviations from the uniform ECDF indicate mismatch relative to this neutral target distribution. Colored shaded intervals mark the five largest adjacent gaps between sorted item difficulties.

\tau_{t}\leftarrow\frac{t-\frac{1}{2}}{k}

Appendix

Algorithm 1 Quantile-Matching Benchmark Calibration

\{x_{p,1},\dots,x_{p,C}\}\leftarrow\mathcal{G}(x_{p},z^{\star},\mathcal{R}_{t}).

Appendix

Algorithm 2 BenchFill

Figure 7: First-round alignment of specified target difficulty and annotated difficulty. Each panel corresponds to one benchmark. The horizontal axis shows the specified target difficulty, and the vertical axis shows the annotated difficulty of valid first-round children. Blue points denote individual generated children, and red markers denote the mean and standard deviation for each target difficulty. The dashed diagonal indicates perfect alignment. For readability, both axes are shown on a 0–10 scale, corresponding to normalized difficulties in

[0,1]

multiplied by 10. Within most benchmarks, higher specified targets produce higher annotated difficulties already in the first round, before any iterative feedback is incorporated.

Figure 8: Multi-round target alignment during BenchFill search. Each panel shows one representative gap-filling case from a different benchmark. The horizontal axis denotes the generation round, and the vertical axis denotes the absolute target deviation,

|\hat{z}-z^{\star}|

, for generated children. For readability, target difficulties and absolute deviations are multiplied by 10. Blue points denote individual generated children, and red markers denote the round-level mean and standard deviation. Across most cases, the mean target deviation decreases over rounds, and the generated children become more concentrated around the target, showing that the search-and-annotate loop further improves target alignment beyond the rewriter’s initial difficulty sensitivity.

Case	Benchmark variant	Gain	One-sided $p$	Detected
AIME24 Olmo-3.1-32B-Think	Original	+0.104	0.500100	$\times$
	Original + BenchFill	+2.419	0.005940	$\checkmark$
	Original + Baseline	+0.202	0.458000	$\times$
AIME25 Phi-4	Original	+1.042	0.184100	$\times$
	Original + BenchFill	+3.730	0.000450	$\checkmark$
	Original + Baseline	+0.907	0.214200	$\times$

Appendix

Table 12: Localized-improvement detection results. Each augmentation method adds one item to the original 30-item benchmark. Gain denotes the change in average accuracy. The one-sided stratified randomization test randomly reassigns rollout outcomes between the pre- and post-SFT conditions within each item using 100,000 permutations. Boldface indicates variants for which a positive gain is detected at

p<0.05

Benchmark	Original benchmark ancestor		Selected gap-filling item
	Item	Diff.	Item	Diff.
AMC23	For how many integers $n$ does the expression $\sqrt{\frac{\log(n^{2})-(\log n)^{2}}{\log n-\hbox{\pagecolor{red!20}3}}}$ represent a real number, where $\log$ denotes the base- $10$ logarithm?	$0.535$	Find the number of integer values of $n$ such that $\sqrt{\frac{\log(n^{2})-(\log n)^{2}}{\log n-\hbox{\pagecolor{red!20}1}}}$ represents a real number, where $\log$ denotes the base- $10$ logarithm.	$0.597$ $(0.598)$
Gaokao2023en	On an $8\times 8$ chessboard, 6 black rooks and $k$ white rooks are placed on different cells so that each rook only attacks rooks of the opposite color. Compute the maximum possible value of $k$ . (Two rooks attack each other if they are in the same row or column and no rooks are between them.)	$0.864$	On a $9\times 9$ chessboard, 7 black rooks and $k$ white rooks are placed on distinct squares so that each rook attacks only rooks of the opposite color. What is the maximum possible value of $k$ ? (Two rooks attack each other if they are in the same row or column and there is no rook between them.)	$0.886$ $(0.885)$
AIME25	Alex divides a disk into four quadrants with two perpendicular diameters intersecting at the center of the disk. He draws $25$ more line segments through the disk, each connecting two points at random on the perimeter in different quadrants . Find the expected number of regions into which these $27$ line segments divide the disk.	$0.750$	A circle has a diameter drawn, splitting it into two semicircular regions . Then 8 chords are drawn, each connecting a random point on the boundary in one semicircle to a random point on the boundary in the other semicircle . Assuming all points are chosen uniformly and independently, what is the expected number of regions into which the interior of the circle is divided?	$0.674$ $(0.671)$

Appendix

Table 13: Representative pairs of original benchmark ancestors and final selected items produced by BenchFill . All difficulty values are normalized to

[0,1]

. The main change from the ancestor item to the selected item is highlighted. For selected items, values in parentheses denote the target difficulty.

Benchmark Design

Benchmark design creates tasks, datasets, metrics, and protocols that make AI capabilities measurable and comparisons credible.

See all 394 papers

BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validity
Feb 25, 2026Harshita Diddee, Gregory Yauney, Swabha Swayamdipta +1Benchmark Validity Benchmark Design
Language Technologies Institute, Carnegie Mellon University, Pittsburgh (PA), United States · Thomas Lord Department of Computer Science, University of Southern California, Los Angeles (CA), United States
Growing Pains: Extensible and Efficient LLM Benchmarking Via Fixed Parameter Calibration
Apr 14, 2026Eliya Habba, Itay Itzhak, Asaf Yehudai +5LLM Evaluation Benchmark Design
The Hebrew University of Jerusalem · Technion · IBM Research +2
QuickScope: Certifying Hard Questions in Dynamic LLM Benchmarks
Apr 20, 2026Taylor Lundy, Narun K. Raman, Kevin Leyton-BrownLLM Evaluation Bayesian Optimization
Department of Computer Science University of British Columbia Vancouver, BC

Feb 25, 2026J/K move · Enter open · S save

BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validity

Harshita Diddee, Gregory Yauney, Swabha Swayamdipta, Daphne Ippolito

Language Technologies Institute, Carnegie Mellon University, Pittsburgh (PA), United States · Thomas Lord Department of Computer Science, University of Southern California, Los Angeles (CA), United States

Benchmark Validity Benchmark Design

Open paper AskPDF arXiv

Do language model benchmarks actually measure what practitioners intend them to ? High-level metadata is too coarse to convey the granular reality of benchmarks: a "poetry" benchmark may never test for haikus, while "instruction-following" benchmarks will often test for an arbitrary mix of skills. This opacity makes verifying alignment with practitioner goals a laborious process, risking an illusion of competence even when models fail on untested facets of user interests. We introduce BenchBrowser, a retriever that surfaces evaluation items relevant to natural language use cases over 20 benchmark suites. Validated by a human study confirming high retrieval precision, BenchBrowser generates evidence to help practitioners diagnose low content validity (narrow coverage of a capability's facets) and low convergent validity (lack of stable rankings when measuring the same capability). BenchBrowser, thus, helps quantify a critical gap between practitioner intent and what benchmarks actually test.

Abstract

Benchmark Design

Explore similar work

BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validity

Growing Pains: Extensible and Efficient LLM Benchmarking Via Fixed Parameter Calibration

QuickScope: Certifying Hard Questions in Dynamic LLM Benchmarks