Are Benchmarks Reliable? Toward Structural Diagnosis via Item-Level Capability Boundaries

Haiquan Hu; Yuzhu Liang; Weicheng Tang; Yanzeng Li; Yao Shi; Li Xu; Tian Wang

cs.AISep 27, 2026

Are Benchmarks Reliable? Toward Structural Diagnosis via Item-Level Capability Boundaries

Authors: Haiquan Hu, Yuzhu Liang, Weicheng Tang, Yanzeng Li, Yao Shi, Li Xu, Tian Wang

Organizations: Beijing Normal University

Abstract

Evaluating large language models (LLMs) relies heavily on benchmark scores, yet aggregate metrics can obscure whether individual items serve their intended measurement purpose. BSDProbe provides an item-level framework for benchmark structural diagnosis that jointly characterizes correctness transitions and their supporting response evidence. Repeated responses along an empirically ordered model axis yield complementary correctness and normalized entropy trajectories, from which four-dimensional profiles of boundary position, boundary width, boundary-signal validity, and order consistency are constructed for benchmark-level aggregation. Experiments on six benchmarks identify varying proportions of potentially anomalous items. With Qwen3 as both the selection and reference axis, BSDProbe achieves a 17.9% mean relative improvement in normalized discrimination ratio over the best baseline per benchmark and matches or exceeds all baselines in ranking agreement with the full benchmark.

Figures & tables

Figure 1: Schematic comparison of two hypothetical benchmarks with the same target-model score but different item compositions.

Figure 2: Overview of BSDProbe. Repeated responses along an empirically ordered model axis yield correctness and normalized entropy trajectories, which jointly support four-dimensional item profiles. Aggregating these profiles enables benchmark-level structural diagnosis, while order-consistency screening identifies items for further review.

Figure 3: Benchmark structural profiles along the Qwen3 axis. Each row corresponds to one benchmark, showing (a) boundary-position distributions

x_{q}

, (b) boundary-width distributions

w_{q}

, and (c) boundary-signal validity

v_{q}

Figure 4: Benchmark structural profiles along the Qwen2.5 axis.

Figure 5: Benchmark structural profiles along the empirically ordered cross-model axis.

Benchmark	Qwen3	Qwen2.5	Cross-model
MMLU	31.80	26.78	31.04
GPQA	56.25	48.21	57.59
GSM8K	8.87	3.79	28.13
MATH	8.18	4.25	7.27
PopQA	20.04	20.34	27.48
TriviaQA	25.28	25.82	36.96

Table 1: Order-anomaly rates (%) across model axes. Each rate uses all analyzed items in the corresponding benchmark–axis condition.

Benchmark	Random	Tiny Benchmarks	Anchor Points	Bayesian	PSN-IRT	BSDProbe (Qwen3)	BSDProbe (Qwen2.5)	BSDProbe (Cross-model)
MMLU	$1.00_{\pm 0.09}$	$1.48_{\pm 0.06}$	$0.69_{\pm 0.06}$	$1.03_{\pm 0.15}$	$1.65_{\pm 0.12}$	$\mathbf{2.04_{\pm 0.06}}$	$1.27_{\pm 0.10}$	$1.35_{\pm 0.09}$
GPQA	$0.99_{\pm 0.19}$	$2.23_{\pm 0.14}$	$0.80_{\pm 0.08}$	$1.26_{\pm 0.21}$	$1.08_{\pm 0.00}$	$\mathbf{2.90_{\pm 0.14}}$	$1.75_{\pm 0.12}$	$1.53_{\pm 0.15}$
GSM8K	$1.09_{\pm 0.24}$	$0.57_{\pm 0.12}$	$1.44_{\pm 0.12}$	$1.77_{\pm 0.00}$	$2.45_{\pm 0.18}$	$\mathbf{2.55_{\pm 0.18}}$	$1.93_{\pm 0.18}$	$1.53_{\pm 0.15}$
MATH	$1.12_{\pm 0.14}$	$1.08_{\pm 0.07}$	$1.51_{\pm 0.11}$	$1.83_{\pm 0.13}$	$0.44_{\pm 0.00}$	$\mathbf{2.07_{\pm 0.11}}$	$1.39_{\pm 0.10}$	$1.23_{\pm 0.14}$
PopQA	$1.00_{\pm 0.36}$	$0.36_{\pm 0.08}$	$1.23_{\pm 0.09}$	$3.10_{\pm 0.46}$	$4.42_{\pm 0.00}$	$\mathbf{6.06_{\pm 0.19}}$	$4.20_{\pm 0.26}$	$3.28_{\pm 0.28}$
TriviaQA	$0.96_{\pm 0.09}$	$\mathbf{1.68_{\pm 0.03}}$	$0.76_{\pm 0.04}$	$1.28_{\pm 0.09}$	$1.40_{\pm 0.06}$	$\mathbf{1.68_{\pm 0.03}}$	$1.30_{\pm 0.06}$	$1.04_{\pm 0.05}$

Table 2: Subset discriminability measured by NDR for 100-item subsets, with all NDR values evaluated on the Qwen3 reference axis (

\uparrow

indicates higher discrimination). BSDProbe labels indicate the axis used for subset selection. Values are reported as mean

\pm

standard deviation over repeated subset sampling. Bold marks the best result in each row.

Benchmark	Random	Tiny Benchmarks	Anchor Points	Bayesian	PSN-IRT	BSDProbe (Qwen3)	BSDProbe (Qwen2.5)	BSDProbe (Cross-model)
MMLU	$0.943_{\pm 0.092}$	$0.874_{\pm 0.068}$	$0.997_{\pm 0.011}$	$0.906_{\pm 0.088}$	$0.983_{\pm 0.042}$	$\mathbf{1.000_{\pm 0.000}}$	$0.990_{\pm 0.016}$	$\mathbf{1.000_{\pm 0.000}}$
GPQA	$0.895_{\pm 0.148}$	$\mathbf{1.000_{\pm 0.000}}$	$0.820_{\pm 0.127}$	$0.599_{\pm 0.154}$	$\mathbf{1.000_{\pm 0.000}}$	$\mathbf{1.000_{\pm 0.000}}$	$0.973_{\pm 0.056}$	$0.976_{\pm 0.042}$
GSM8K	$0.845_{\pm 0.166}$	$0.741_{\pm 0.099}$	$0.952_{\pm 0.047}$	$\mathbf{0.966_{\pm 0.000}}$	$0.941_{\pm 0.043}$	$\mathbf{0.966_{\pm 0.000}}$	$0.843_{\pm 0.087}$	$0.948_{\pm 0.047}$
MATH	$0.783_{\pm 0.106}$	$0.860_{\pm 0.074}$	$0.910_{\pm 0.105}$	$0.873_{\pm 0.090}$	$0.775_{\pm 0.000}$	$\mathbf{1.000_{\pm 0.000}}$	$0.753_{\pm 0.167}$	$0.720_{\pm 0.237}$
PopQA	$0.893_{\pm 0.076}$	$0.649_{\pm 0.177}$	$0.872_{\pm 0.157}$	$0.896_{\pm 0.106}$	$\mathbf{1.000_{\pm 0.000}}$	$\mathbf{1.000_{\pm 0.000}}$	$0.967_{\pm 0.054}$	$0.983_{\pm 0.054}$
TriviaQA	$0.876_{\pm 0.086}$	$0.909_{\pm 0.055}$	$0.987_{\pm 0.042}$	$0.930_{\pm 0.067}$	$0.997_{\pm 0.011}$	$\mathbf{1.000_{\pm 0.000}}$	$\mathbf{1.000_{\pm 0.000}}$	$\mathbf{1.000_{\pm 0.000}}$

Table 3: Ranking agreement measured by Kendall

\tau_{b}

for 100-item subsets evaluated on the Qwen3 reference axis (

\uparrow

indicates higher agreement). BSDProbe labels indicate the axis used for subset selection. Values are reported as mean

\pm

standard deviation over repeated subset sampling. Bold marks the best result in each row.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Qwen3 axis	Qwen2.5 axis	Cross-model axis
Qwen3-0.6B	Qwen2.5-0.5B-Instruct	Llama-3.2-1B-Instruct
Qwen3-1.7B	Qwen2.5-1.5B-Instruct	Llama-3.2-3B-Instruct
Qwen3-4B	Qwen2.5-3B-Instruct	Llama-3.1-8B-Instruct
Qwen3-8B	Qwen2.5-7B-Instruct	GLM-4-9B-0414
Qwen3-14B	Qwen2.5-14B-Instruct	DeepSeek-R1-Distill-Qwen-14B
Qwen3-32B	Qwen2.5-32B-Instruct	GLM-4-32B-0414

Appendix

Table 4: Model sets used in the evaluated axes.

Axis	Model	MMLU	GPQA	GSM8K	MATH	PopQA	TriviaQA
Qwen3 Axis	Qwen3-0.6B	41.20%	29.02%	68.16%	56.22%	12.94%	18.98%
	Qwen3-1.7B	61.15%	29.69%	85.29%	76.87%	15.08%	35.38%
	Qwen3-4B	74.69%	37.28%	93.56%	86.00%	18.17%	52.20%
	Qwen3-8B	78.97%	45.54%	94.39%	85.86%	23.38%	66.43%
	Qwen3-14B	81.95%	50.89%	95.98%	88.12%	26.29%	71.95%
	Qwen3-32B	85.22%	57.81%	95.98%	88.10%	28.31%	76.54%

Appendix

Table 5: Benchmark-level accuracy for the evaluated model sets across six benchmarks. The Qwen3 and Qwen2.5 axes follow benchmark-specific empirical orderings, while the heterogeneous cross-model setting uses benchmark-specific empirical orderings reported in Table 6 .

Task type	Benchmarks	Ordered cross-model axis
MCQ	MMLU, GPQA	Llama-3.2-1B-Instruct $\rightarrow$ Llama-3.2-3B-Instruct $\rightarrow$ Llama-3.1-8B-Instruct $\rightarrow$ GLM-4-9B-0414 $\rightarrow$ DeepSeek-R1-Distill-Qwen-14B $\rightarrow$ GLM-4-32B-0414
Math	GSM8K, MATH	Llama-3.2-1B-Instruct $\rightarrow$ Llama-3.2-3B-Instruct $\rightarrow$ Llama-3.1-8B-Instruct $\rightarrow$ GLM-4-9B-0414 $\rightarrow$ DeepSeek-R1-Distill-Qwen-14B $\rightarrow$ GLM-4-32B-0414
QA	PopQA, TriviaQA	Llama-3.2-1B-Instruct $\rightarrow$ GLM-4-9B-0414 $\rightarrow$ Llama-3.2-3B-Instruct $\rightarrow$ DeepSeek-R1-Distill-Qwen-14B $\rightarrow$ Llama-3.1-8B-Instruct $\rightarrow$ GLM-4-32B-0414

Appendix

Table 6: Task-type-specific ordering of the heterogeneous cross-model axis. The same candidate model set is used, but the ordering is selected according to capability trends within each task type.

Benchmark	$t=5$	$t=10$	$t=15$	$t=20$	$t=25$
GPQA	0.0921	0.0504	0.0345	0.0261	0.0216
GSM8K	0.0182	0.0119	0.0101	0.0084	0.0073
MATH	0.0680	0.0513	0.0440	0.0394	0.0363
MMLU	0.0245	0.0145	0.0102	0.0080	0.0066
PopQA	0.1998	0.1442	0.1169	0.0998	0.0880
TriviaQA	0.0593	0.0389	0.0299	0.0246	0.0214

Appendix

Table 7: Mean within-item answer-state JSD between disjoint blocks of

t

responses, using the first random partition and natural logarithms.

Benchmark	Items	Self-JSD	Cross-JSD	$G_{20}$	95% CI for $G_{20}$
GPQA	448	0.0113	0.1878	0.1766	[0.1631, 0.1893]
GSM8K	1317	0.0025	0.0673	0.0648	[0.0534, 0.0768]
MATH	5000	0.0060	0.1521	0.1461	[0.1394, 0.1529]
MMLU	14042	0.0046	0.1909	0.1863	[0.1822, 0.1904]
PopQA	14267	0.0019	0.2184	0.2165	[0.2115, 0.2212]
TriviaQA	7993	0.0024	0.2877	0.2853	[0.2812, 0.2893]

Appendix

Table 8: Outcome-role JSD at

t=20

and paired item-level bootstrap confidence intervals for

G_{20}=\text{Cross-JSD}-\text{Self-JSD}

. Item counts refer to the Qwen3-8B reliability cohort.

	$T=5$			$T=10$			$T=15$
Benchmark	Agree. (%)	$x_{q}$ MAE	$w_{q}$ MAE	Agree. (%)	$x_{q}$ MAE	$w_{q}$ MAE	Agree. (%)	$x_{q}$ MAE	$w_{q}$ MAE
GPQA	64.8	0.080	0.096	80.7	0.049	0.066	89.8	0.029	0.041
GSM8K	94.0	0.028	0.033	96.7	0.016	0.021	98.3	0.009	0.012
MATH	93.6	0.051	0.064	95.8	0.031	0.043	97.6	0.018	0.026
MMLU	82.2	0.032	0.041	91.3	0.019	0.026	95.9	0.011	0.016
PopQA	91.0	0.027	0.032	95.9	0.016	0.021	98.2	0.009	0.013

Appendix

Table 9: Response-budget sensitivity on Qwen3 relative to

T=20

. Values are means over 100 subsampling repetitions. Agreement refers to item-level order-review decisions; position and width MAEs use items with positive fitted gain in both conditions.

	$n=3$			$n=4$			$n=5$
Benchmark	Agree. (%)	$x_{q}$ MAE	$w_{q}$ MAE	Agree. (%)	$x_{q}$ MAE	$w_{q}$ MAE	Agree. (%)	$x_{q}$ MAE	$w_{q}$ MAE
GPQA	65.6–75.7	0.166	0.187	75.0–86.4	0.114	0.133	83.5–93.3	0.066	0.080
GSM8K	93.9–94.8	0.142	0.161	95.5–96.8	0.073	0.084	97.3–98.2	0.033	0.038
MATH	93.4–93.8	0.147	0.164	95.1–95.5	0.080	0.095	97.0–97.6	0.038	0.049
MMLU	79.2–84.2	0.158	0.179	85.1–90.5	0.095	0.109	90.8–95.5	0.049	0.058
PopQA	88.4–90.3	0.161	0.185	92.1–94.5	0.102	0.112	95.5–97.4	0.053	0.058

Appendix

Table 10: Endpoint-preserving model reduction on Qwen3 relative to the six-model setting, with

T=20

. Agreement ranges cover all subaxes at each model count. Position and width MAEs are averaged across subaxes after mapping crossings to the original model coordinates.

Benchmark	Random	Tiny Benchmarks	Anchor Points	Bayesian	PSN-IRT	BSDProbe (Qwen3)	BSDProbe (Qwen2.5)	BSDProbe (Cross-model)
MMLU	$1.02_{\pm 0.12}$	$\mathbf{1.57_{\pm 0.06}}$	$0.89_{\pm 0.06}$	$1.06_{\pm 0.15}$	$1.12_{\pm 0.00}$	$1.15_{\pm 0.06}$	$1.57_{\pm 0.07}$	$1.09_{\pm 0.09}$
GPQA	$1.13_{\pm 0.22}$	$2.02_{\pm 0.22}$	$1.59_{\pm 0.31}$	$2.02_{\pm 0.17}$	$2.53_{\pm 0.00}$	$1.88_{\pm 0.12}$	$\mathbf{3.32_{\pm 0.16}}$	$2.14_{\pm 0.20}$
GSM8K	$1.07_{\pm 0.07}$	$1.31_{\pm 0.06}$	$0.96_{\pm 0.08}$	$\mathbf{1.55_{\pm 0.04}}$	$1.42_{\pm 0.00}$	$1.34_{\pm 0.05}$	$1.38_{\pm 0.07}$	$1.26_{\pm 0.07}$
MATH	$0.99_{\pm 0.06}$	$1.47_{\pm 0.03}$	$1.22_{\pm 0.03}$	$\mathbf{1.56_{\pm 0.06}}$	$0.84_{\pm 0.00}$	$1.02_{\pm 0.07}$	$1.30_{\pm 0.06}$	$1.21_{\pm 0.09}$
PopQA	$0.98_{\pm 0.27}$	$0.10_{\pm 0.05}$	$1.93_{\pm 0.22}$	$3.18_{\pm 0.40}$	$5.61_{\pm 0.00}$	$4.42_{\pm 0.31}$	$\mathbf{6.53_{\pm 0.25}}$	$3.34_{\pm 0.39}$
TriviaQA	$1.03_{\pm 0.09}$	$\mathbf{1.78_{\pm 0.06}}$	$0.85_{\pm 0.04}$	$1.36_{\pm 0.10}$	$1.57_{\pm 0.00}$	$1.40_{\pm 0.07}$	$1.78_{\pm 0.05}$	$1.17_{\pm 0.07}$

Appendix

Table 11: Subset discriminability measured by NDR for 100-item subsets evaluated on the Qwen2.5 reference axis (

\uparrow

indicates higher discrimination).

Benchmark	Random	Tiny Benchmarks	Anchor Points	Bayesian	PSN-IRT	BSDProbe (Qwen3)	BSDProbe (Qwen2.5)	BSDProbe (Cross-model)
MMLU	$0.980_{\pm 0.042}$	$0.949_{\pm 0.044}$	$0.997_{\pm 0.011}$	$0.913_{\pm 0.061}$	$\mathbf{1.000_{\pm 0.000}}$	$0.997_{\pm 0.011}$	$\mathbf{1.000_{\pm 0.000}}$	$0.997_{\pm 0.011}$
GPQA	$0.794_{\pm 0.152}$	$0.848_{\pm 0.124}$	$0.765_{\pm 0.063}$	$0.900_{\pm 0.054}$	$0.867_{\pm 0.000}$	$0.842_{\pm 0.095}$	$\mathbf{0.970_{\pm 0.055}}$	$0.877_{\pm 0.031}$
GSM8K	$0.824_{\pm 0.157}$	$0.915_{\pm 0.060}$	$0.896_{\pm 0.079}$	$0.987_{\pm 0.042}$	$0.894_{\pm 0.000}$	$0.997_{\pm 0.011}$	$\mathbf{1.000_{\pm 0.000}}$	$0.896_{\pm 0.048}$
MATH	$0.963_{\pm 0.063}$	$0.859_{\pm 0.016}$	$0.873_{\pm 0.035}$	$0.976_{\pm 0.054}$	$0.966_{\pm 0.000}$	$0.960_{\pm 0.064}$	$\mathbf{1.000_{\pm 0.000}}$	$0.959_{\pm 0.059}$
PopQA	$0.802_{\pm 0.161}$	$-0.438_{\pm 0.299}$	$0.960_{\pm 0.052}$	$0.903_{\pm 0.093}$	$\mathbf{1.000_{\pm 0.000}}$	$0.930_{\pm 0.067}$	$\mathbf{1.000_{\pm 0.000}}$	$0.970_{\pm 0.055}$
TriviaQA	$0.926_{\pm 0.096}$	$0.963_{\pm 0.053}$	$0.909_{\pm 0.065}$	$0.927_{\pm 0.090}$	$\mathbf{1.000_{\pm 0.000}}$	$0.983_{\pm 0.042}$	$\mathbf{1.000_{\pm 0.000}}$	$0.973_{\pm 0.056}$

Appendix

Table 12: Ranking agreement measured by Kendall

\tau_{b}

for 100-item subsets evaluated on the Qwen2.5 reference axis (

\uparrow

indicates higher agreement).

Benchmark	Random	Tiny Benchmarks	Anchor Points	Bayesian	PSN-IRT	BSDProbe (Qwen3)	BSDProbe (Qwen2.5)	BSDProbe (Cross-model)
MMLU	$1.10_{\pm 0.10}$	$1.08_{\pm 0.09}$	$0.88_{\pm 0.06}$	$1.31_{\pm 0.19}$	$1.72_{\pm 0.25}$	$1.60_{\pm 0.15}$	$1.41_{\pm 0.14}$	$\mathbf{2.12_{\pm 0.13}}$
GPQA	$1.07_{\pm 0.19}$	$1.80_{\pm 0.17}$	$1.06_{\pm 0.11}$	$1.62_{\pm 0.23}$	$1.92_{\pm 0.00}$	$1.88_{\pm 0.11}$	$1.61_{\pm 0.18}$	$\mathbf{2.28_{\pm 0.10}}$
GSM8K	$1.01_{\pm 0.12}$	$1.48_{\pm 0.09}$	$1.11_{\pm 0.07}$	$\mathbf{1.53_{\pm 0.15}}$	$1.46_{\pm 0.00}$	$1.43_{\pm 0.11}$	$1.31_{\pm 0.07}$	$1.30_{\pm 0.06}$
MATH	$0.95_{\pm 0.07}$	$1.31_{\pm 0.03}$	$0.97_{\pm 0.04}$	$\mathbf{1.54_{\pm 0.04}}$	$0.81_{\pm 0.00}$	$1.29_{\pm 0.06}$	$1.20_{\pm 0.09}$	$1.30_{\pm 0.05}$
PopQA	$1.04_{\pm 0.21}$	$0.65_{\pm 0.12}$	$2.17_{\pm 0.25}$	$3.13_{\pm 0.43}$	$\mathbf{4.59_{\pm 0.00}}$	$2.66_{\pm 0.25}$	$2.46_{\pm 0.13}$	$4.41_{\pm 0.26}$
TriviaQA	$1.04_{\pm 0.11}$	$0.99_{\pm 0.11}$	$1.11_{\pm 0.08}$	$1.69_{\pm 0.14}$	$1.80_{\pm 0.16}$	$1.48_{\pm 0.07}$	$1.59_{\pm 0.15}$	$\mathbf{2.39_{\pm 0.11}}$

Appendix

Table 13: Subset discriminability measured by NDR for 100-item subsets evaluated on the cross-model reference axis (

\uparrow

indicates higher discrimination).

Benchmark	Random	Tiny Benchmarks	Anchor Points	Bayesian	PSN-IRT	BSDProbe (Qwen3)	BSDProbe (Qwen2.5)	BSDProbe (Cross-model)
MMLU	$0.926_{\pm 0.096}$	$0.927_{\pm 0.036}$	$0.997_{\pm 0.011}$	$0.899_{\pm 0.115}$	$0.920_{\pm 0.069}$	$0.987_{\pm 0.042}$	$0.960_{\pm 0.090}$	$\mathbf{1.000_{\pm 0.000}}$
GPQA	$0.862_{\pm 0.113}$	$0.957_{\pm 0.063}$	$0.742_{\pm 0.095}$	$0.837_{\pm 0.118}$	$0.733_{\pm 0.000}$	$0.859_{\pm 0.056}$	$0.963_{\pm 0.053}$	$\mathbf{0.973_{\pm 0.041}}$
GSM8K	$0.720_{\pm 0.134}$	$0.450_{\pm 0.245}$	$0.695_{\pm 0.091}$	$0.743_{\pm 0.030}$	$0.733_{\pm 0.000}$	$0.733_{\pm 0.000}$	$0.765_{\pm 0.063}$	$\mathbf{1.000_{\pm 0.000}}$
MATH	$0.983_{\pm 0.042}$	$\mathbf{1.000_{\pm 0.000}}$	$0.970_{\pm 0.055}$	$\mathbf{1.000_{\pm 0.000}}$	$0.966_{\pm 0.000}$	$\mathbf{1.000_{\pm 0.000}}$	$0.987_{\pm 0.042}$	$\mathbf{1.000_{\pm 0.000}}$
PopQA	$0.912_{\pm 0.066}$	$0.703_{\pm 0.132}$	$0.747_{\pm 0.076}$	$0.815_{\pm 0.125}$	$0.733_{\pm 0.000}$	$0.886_{\pm 0.104}$	$0.922_{\pm 0.099}$	$\mathbf{1.000_{\pm 0.000}}$
TriviaQA	$0.884_{\pm 0.069}$	$0.454_{\pm 0.267}$	$0.903_{\pm 0.068}$	$0.997_{\pm 0.011}$	$0.969_{\pm 0.065}$	$0.963_{\pm 0.063}$	$0.896_{\pm 0.067}$	$\mathbf{1.000_{\pm 0.000}}$

Appendix

Table 14: Ranking agreement measured by Kendall

\tau_{b}

for 100-item subsets evaluated on the cross-model reference axis (

\uparrow

indicates higher agreement).

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,355 papers

Feb 17, 2026

Quantifying construct validity in large language model evaluations

The LLM community often reports benchmark results as if they are synonymous with general model capabilities. However, benchmarks can have problems that distort performance, like test set contamination and annotator error. How can we know that a benchmark is a reliable indicator of some capability that we want to measure? This question concerns the construct validity of LLM benchmarks, and it requires separating benchmark results from capabilities when we model and predict LLM performance. Both social scientists and computer scientists propose formal models - latent factor models and scaling laws - for identifying the capabilities underlying benchmark scores. However, neither technique is satisfactory for construct validity. Latent factor models ignore scaling laws, and as a result, the capabilities they extract often proxy model size. Scaling laws ignore measurement error, and as a result, the capabilities they extract are both uninterpretable and overfit to the observed benchmarks. This thesis presents the structured capabilities model, the first model to extract interpretable and generalisable capabilities from a large collection of LLM benchmark results. I fit this model and its two alternatives on a large sample of results from the OpenLLM Leaderboard. Structured capabilities outperform latent factor models on parsimonious fit indices, and exhibit better out-of-distribution benchmark prediction than scaling laws. These improvements are possible because neither existing approach separates model scale from capabilities in the appropriate way. Model scale should inform capabilities, as in scaling laws, and these capabilities should inform observed results up to measurement error, as in latent factor models. In combining these two insights, structured capabilities demonstrate better explanatory and predictive power for quantifying construct validity in LLM evaluations.

Ryan Othniel Kearns

St Catherine’s College

LLM Evaluation Latent Variable Models

Ask PDF

Jul 30, 2026

Benchmarks Are Not Monolithic: Sample-Level Auditing and Orchestration for LLM Evaluation

Benchmark datasets are central to evaluating Large Language Models (LLMs), yet they are typically conceived as monolithic tasks, obscuring substantial variation in the demands of individual samples. We introduce a dataset-centric meta-evaluation framework that audits benchmark datasets at the sample level along five latent dimensions: 1. Cognitive and Knowledge Demands, 2. Language and Content Quality, 3. Task Properties, 4. Context, and 5. Ethics, Safety, and Fairness. Applying this framework, we annotate five influential benchmarks -- MMLU, ARC, WinoGrande, HellaSwag, and TruthfulQA -- revealing pronounced internal heterogeneity that is not captured by aggregate accuracy scores. We show how these annotations enable criterion-driven orchestration of composite benchmark subsets across datasets, supporting targeted evaluation of model capabilities such as Reasoning Depth or Ethical Sensitivity. This approach reframes benchmark evaluation as dataset introspection, providing a principled methodology for analyzing and re-composing existing benchmarks to better reflect diverse evaluation needs.

Philipp D. Siedler, Jordan Sassoon

Aleph Alpha Research, Heidelberg, Germany.

LLM Evaluation Benchmark Auditing Benchmark Construction LLM Auditing

Ask PDF

Feb 25, 2026

BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validity

Do language model benchmarks actually measure what practitioners intend them to ? High-level metadata is too coarse to convey the granular reality of benchmarks: a "poetry" benchmark may never test for haikus, while "instruction-following" benchmarks will often test for an arbitrary mix of skills. This opacity makes verifying alignment with practitioner goals a laborious process, risking an illusion of competence even when models fail on untested facets of user interests. We introduce BenchBrowser, a retriever that surfaces evaluation items relevant to natural language use cases over 20 benchmark suites. Validated by a human study confirming high retrieval precision, BenchBrowser generates evidence to help practitioners diagnose low content validity (narrow coverage of a capability's facets) and low convergent validity (lack of stable rankings when measuring the same capability). BenchBrowser, thus, helps quantify a critical gap between practitioner intent and what benchmarks actually test.

Harshita Diddee, Gregory Yauney, Swabha Swayamdipta +1

Language Technologies Institute, Carnegie Mellon University, Pittsburgh (PA), United States · Thomas Lord Department of Computer Science, University of Southern California, Los Angeles (CA), United States

Benchmark Validity Benchmark Design

Ask PDF

Abstract

LLM Evaluation

Explore similar work

Quantifying construct validity in large language model evaluations

Benchmarks Are Not Monolithic: Sample-Level Auditing and Orchestration for LLM Evaluation

BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validity