The Evaluation Blind Spot: A Stereological Theory of Benchmark Coverage for Large Language Models

Jason Z Wang

cs.LGApr 15, 2026

The Evaluation Blind Spot: A Stereological Theory of Benchmark Coverage for Large Language Models

Authors: Jason Z Wang

Organizations: Independent

Abstract

We give a stereological theory of LLM benchmark coverage. For any suite with effective dimensionality d_eff, the visible Hausdorff distance between two convex capability profiles consistent with the same scores is bounded by epsilon + C R m^(-1/(d_eff-1)), with matching Lipschitz lower bound. Empirically, three independent leaderboards (Open LLM v2, an extended 12-benchmark suite, LiveBench) all have d_eff in [2.86, 4.80] on their competitive frontier; the structural blind spot exceeds the observed runner-up score gap by two orders of magnitude and dominates statistical noise by 52-127x. Under a chi-squared projection model, the isotropic prior is the optimistic case; across six hidden-capability priors and four ambient dimensions the simulated half-split swap rate of the top two models stays in [0.38, 0.49], and a 500-trial random visible/held-out split shows that 92% of trials swap the top-1 ranking with on average 2.83 of 5 top-5 models changing. A submodular greedy algorithm with the Nemhauser (1 - 1/e) guarantee finds a stable core of 4 benchmarks; 7 of 12 suffice for 90% coverage, and the trained subset transfers across temporal quarters with 93-97% retention. A counterfactual validation across 12 internal benchmarks and 27 Chatbot Arena categories confirms that the eigenstructure predicts which evaluations are irreplaceable (rho = -0.69, p = 0.013 for removal disruption) and which external evaluations bring new information (rho = +0.38). As a second, independent theoretical contribution, we resolve Gardner's Problem 1.5 (1995) for C^2 support functions, establishing the minimax rate Theta(R/(kappa m^(2/(D-1)))) in general dimension via optimal recovery theory on S^(D-1).

Figures & tables

Leaderboard	Type	Slice	$k$	$n$	$d_{\mathrm{eff}}$ [CI]	Spearman
Open LLM v2	accuracy	full	6	458	1.88 [1.77, 2.02]	1.70
Open LLM v2	accuracy	frontier	6	229	2.86 [2.60, 3.11]	2.77
Extended (v1+v2)	accuracy	full	12	295	2.11 [1.98, 2.27]	1.89
Extended (v1+v2)	accuracy	frontier	12	148	4.80 [4.15, 5.20]	4.62
LiveBench	mixed	full	7	37	2.63 [2.08, 3.34]	2.55
LiveBench	mixed	frontier	7	19	4.74	4.62

Table 1: Effective dimensionality across leaderboard families. Frontier slices the top 50% by mean score. CIs are 95% percentile bootstrap (

300

resamples). Spearman column shows the same statistic computed on rank correlations.

Figure 1: Eigenvalue spectrum (full and frontier populations, both suites), with the Marchenko–Pastur upper edge

\lambda_{+}=(1+\sqrt{k/n})^{2}

and the Kaiser line

\lambda=1

. A single dominant eigenvalue captures most variance on the full population; on the frontier (bottom row) the residual mass becomes visible and

d_{\mathrm{eff}}

rises into

[3,5]

Figure 2: PCA biplot of the 12-benchmark extended frontier. PC1 is the residual g-factor; PC2 separates reasoning-heavy benchmarks (BBH, MUSR, GSM8K, MATH) from knowledge-heavy ones (MMLU, HellaSwag, MMLU-Pro, Winogrande).

Suite	$k$	$n$	$d_{\mathrm{eff}}$	$R$	$\omega_{\mathrm{emp}}$	$\delta_{H}^{\mathrm{vis}}$	$\delta_{H}^{\mathrm{vis}}/\Delta_{2}$
OLLM v2	6	229	2.86	5.05	2.10	21.2	$123\times$
Extended	12	148	4.80	7.30	1.89	27.6	$385\times$
LiveBench	7	19	4.74	4.30	1.88	16.1	$26\times$

Table 2: Visible indistinguishability radius per suite (frontier slices,

z

-score standardisation).

Figure 3: Left: cumulative coverage of the greedy subset on the extended frontier suite. Right: uncovered eigen-mass before versus after selecting

7

benchmarks; the blind spot shrinks by

\approx 90\%

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

#	Experiment	Key result	Section
1	$d_{\mathrm{eff}}$ across $3$ leaderboards	$d_{\mathrm{eff}}^{\mathrm{frontier}}\in[2.86,4.80]$ universally	§ 1 , H.8
2	Permutation null for eigenvalues	Agrees with MP edge	H.1
3	Split-half reliability of $d_{\mathrm{eff}}$	MAD $\leq 0.12$	H.2
4	Saturation curve	Stable from $n^{\prime}\geq 100$	H.3
5	Greedy out-of-sample $\tau$	$+0.18$ vs random at $r=2$	§ 4 , H.4
6	Five-way subset comparison	Spectral matches/exceeds at every $r$	§ 4 , H.5

Appendix

Table 3: Summary of empirical validation experiments.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,355 papers

Benchmarks Are Not Atomic: Composition-Aware LLM Evaluation using BenchHub
May 31, 2025Eunsu Kim, Haneul Yoo, Guijin Son +3LLM Evaluation
KAIST · Yonsei University · OnelineAI +1
The Capability Frontier: Benchmarks Miss 82% of Model Performance
Jun 25, 2026Bradley Fowler, Ryan Smith, Daniel Thi Graviet +8LLM Evaluation Language Model Generation Evaluation
1Martian · 3ThoughtWorks · University of Oxford
Do Not Trust the Benchmark: Limitations of General LLM Rankings and a Case for Task-Specific Evaluation
Sep 19, 2026Danial AminLLM Evaluation Benchmark Validity

May 31, 2025J/K move · Enter open · S save

Benchmarks Are Not Atomic: Composition-Aware LLM Evaluation using BenchHub

Eunsu Kim, Haneul Yoo, Guijin Son, Hitesh Patel+2

KAIST · Yonsei University · OnelineAI+1

LLM Evaluation

Open paper AskPDF arXiv

LLM benchmarks are often treated as coherent measurement units, yet they are heterogeneous collections of instances spanning diverse domains, skills, formats, and contexts. As a result, aggregate benchmark scores can conflate model capability with benchmark composition, obscuring what existing benchmarks actually cover. We introduce BenchHub, a composition-aware evaluation framework that represents benchmarks as distributions over instance-level attributes. BenchHub integrates 54 benchmarks comprising 839K samples across 10 languages. It enables researchers to inspect benchmark contents, uncover reusable coverage hidden behind benchmark names, transparently compare benchmarks, and construct controllable evaluation sets for application-aligned model selection. Using BenchHub, we show that similarly motivated benchmarks can differ substantially in internal composition, existing benchmarks often contain reusable coverage beyond their stated purposes, models exhibit fine-grained category-level performance variation hidden by aggregate scores, and model rankings can shift under different reweighting and resampling configurations. Our results motivate evaluation practices that make benchmark composition explicit, inspectable, and controllable.

Abstract

LLM Evaluation

Explore similar work

Benchmarks Are Not Atomic: Composition-Aware LLM Evaluation using BenchHub

The Capability Frontier: Benchmarks Miss 82% of Model Performance

Do Not Trust the Benchmark: Limitations of General LLM Rankings and a Case for Task-Specific Evaluation