cs.LGApr 15, 2026

The Evaluation Blind Spot: A Stereological Theory of Benchmark Coverage for Large Language Models

Authors: Jason Z Wang

Organizations: Independent

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We give a stereological theory of LLM benchmark coverage. For any suite with effective dimensionality d_eff, the visible Hausdorff distance between two convex capability profiles consistent with the same scores is bounded by epsilon + C R m^(-1/(d_eff-1)), with matching Lipschitz lower bound. Empirically, three independent leaderboards (Open LLM v2, an extended 12-benchmark suite, LiveBench) all have d_eff in [2.86, 4.80] on their competitive frontier; the structural blind spot exceeds the observed runner-up score gap by two orders of magnitude and dominates statistical noise by 52-127x. Under a chi-squared projection model, the isotropic prior is the optimistic case; across six hidden-capability priors and four ambient dimensions the simulated half-split swap rate of the top two models stays in [0.38, 0.49], and a 500-trial random visible/held-out split shows that 92% of trials swap the top-1 ranking with on average 2.83 of 5 top-5 models changing. A submodular greedy algorithm with the Nemhauser (1 - 1/e) guarantee finds a stable core of 4 benchmarks; 7 of 12 suffice for 90% coverage, and the trained subset transfers across temporal quarters with 93-97% retention. A counterfactual validation across 12 internal benchmarks and 27 Chatbot Arena categories confirms that the eigenstructure predicts which evaluations are irreplaceable (rho = -0.69, p = 0.013 for removal disruption) and which external evaluations bring new information (rho = +0.38). As a second, independent theoretical contribution, we resolve Gardner's Problem 1.5 (1995) for C^2 support functions, establishing the minimax rate Theta(R/(kappa m^(2/(D-1)))) in general dimension via optimal recovery theory on S^(D-1).

Figures & tables

LeaderboardTypeSlicekknndeffd_{\mathrm{eff}} [CI]Spearman
Open LLM v2accuracyfull64581.88 [1.77, 2.02]1.70
Open LLM v2accuracyfrontier62292.86 [2.60, 3.11]2.77
Extended (v1+v2)accuracyfull122952.11 [1.98, 2.27]1.89
Extended (v1+v2)accuracyfrontier121484.80 [4.15, 5.20]4.62
LiveBenchmixedfull7372.63 [2.08, 3.34]2.55
LiveBenchmixedfrontier7194.744.62
Table 1: Effective dimensionality across leaderboard families. Frontier slices the top 50% by mean score. CIs are 95% percentile bootstrap ( 300300 resamples). Spearman column shows the same statistic computed on rank correlations.
Figure 1: Eigenvalue spectrum (full and frontier populations, both suites), with the Marchenko–Pastur upper edge λ+=(1+k/n)2\lambda_{+}=(1+\sqrt{k/n})^{2} and the Kaiser line λ=1\lambda=1 . A single dominant eigenvalue captures most variance on the full population; on the frontier (bottom row) the residual mass becomes visible and deffd_{\mathrm{eff}} rises into [3,5][3,5] .
Figure 2: PCA biplot of the 12-benchmark extended frontier. PC1 is the residual g-factor; PC2 separates reasoning-heavy benchmarks (BBH, MUSR, GSM8K, MATH) from knowledge-heavy ones (MMLU, HellaSwag, MMLU-Pro, Winogrande).
Suitekknndeffd_{\mathrm{eff}}RRωemp\omega_{\mathrm{emp}}δHvis\delta_{H}^{\mathrm{vis}}δHvis/Δ2\delta_{H}^{\mathrm{vis}}/\Delta_{2}
OLLM v262292.865.052.1021.2123×123\times
Extended121484.807.301.8927.6385×385\times
LiveBench7194.744.301.8816.126×26\times
Table 2: Visible indistinguishability radius per suite (frontier slices, zz -score standardisation).
Figure 3: Left: cumulative coverage of the greedy subset on the extended frontier suite. Right: uncovered eigen-mass before versus after selecting 77 benchmarks; the blind spot shrinks by ≈90%\approx 90\% .
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

#ExperimentKey resultSection
1deffd_{\mathrm{eff}} across 33 leaderboardsdefffrontier∈[2.86,4.80]d_{\mathrm{eff}}^{\mathrm{frontier}}\in[2.86,4.80] universally§ 1 , H.8
2Permutation null for eigenvaluesAgrees with MP edgeH.1
3Split-half reliability of deffd_{\mathrm{eff}}MAD ≤0.12\leq 0.12H.2
4Saturation curveStable from n′≥100n^{\prime}\geq 100H.3
5Greedy out-of-sample τ\tau+0.18+0.18 vs random at r=2r=2§ 4 , H.4
6Five-way subset comparisonSpectral matches/exceeds at every rr§ 4 , H.5
Appendix
Table 3: Summary of empirical validation experiments.
More in this topic

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,355 papers

Explore similar work

CardsList
May 31, 2025

Benchmarks Are Not Atomic: Composition-Aware LLM Evaluation using BenchHub

LLM benchmarks are often treated as coherent measurement units, yet they are heterogeneous collections of instances spanning diverse domains, skills, formats, and contexts. As a result, aggregate benchmark scores can conflate model capability with benchmark composition, obscuring what existing benchmarks actually cover. We introduce BenchHub, a composition-aware evaluation framework that represents benchmarks as distributions over instance-level attributes. BenchHub integrates 54 benchmarks comprising 839K samples across 10 languages. It enables researchers to inspect benchmark contents, uncover reusable coverage hidden behind benchmark names, transparently compare benchmarks, and construct controllable evaluation sets for application-aligned model selection. Using BenchHub, we show that similarly motivated benchmarks can differ substantially in internal composition, existing benchmarks often contain reusable coverage beyond their stated purposes, models exhibit fine-grained category-level performance variation hidden by aggregate scores, and model rankings can shift under different reweighting and resampling configurations. Our results motivate evaluation practices that make benchmark composition explicit, inspectable, and controllable.

Eunsu Kim, Haneul Yoo, Guijin Son +3

KAIST · Yonsei University · OnelineAI +1

LLM Evaluation
AskPDF
Jun 25, 2026

The Capability Frontier: Benchmarks Miss 82% of Model Performance

Existing benchmarks typically report accuracy for a single model on a single run. This systematically understates real-world LLM capabilities, particularly under heterogeneous data distributions: (i) different models get different questions correct according to their specializations, and (ii) given a budget, multiple generations can be sampled and selectively retained. To quantify this gap, we introduce the Capability Frontier: a Pareto frontier over a set of models that characterizes the best achievable performance at each cost level under optimal selection across models and generations (i.e., via an oracle). Our construction corrects for two opposing biases: underestimation from single-model evaluation and overestimation from taking maxima over noisy samples. We study 21 LLMs across 16 widely used benchmarks spanning coding, reasoning, medicine, factuality, instruction following, and agentic tasks, comparing Capability Frontier performance at matched cost to each benchmark's top-performing model. Correcting for single-model evaluation yields a 54% error rate reduction; additionally correcting for single runs yields an 82% improvement, with SOTA accuracy matched at 85% cost reduction. Complementing these empirical results, we use controlled probabilistic simulations to show that higher query topic entropy produces a near-monotonic increase in the performance gap between oracle routing and the best single model. Our findings suggest collective LLM capabilities are substantially underestimated, with implications for evaluation and deployment in data-heterogeneous, multi-domain settings.

Bradley Fowler, Ryan Smith, Daniel Thi Graviet +8

1Martian · 3ThoughtWorks · University of Oxford

LLM EvaluationLanguage Model Generation EvaluationTest-Time ScalingLLM RoutingLanguage Model Ensembles
AskPDF
Sep 19, 2026

Do Not Trust the Benchmark: Limitations of General LLM Rankings and a Case for Task-Specific Evaluation

Benchmark scores increasingly influence the development, marketing, and selection of large language models (LLMs). Yet an overall score is interpretable only in relation to the system tested, the questions included, and the conditions of evaluation. This perspective examines five connected limitations of general LLM rankings: differences between evaluated and publicly available systems; commercial incentives and dependencies in external evaluation; benchmark saturation, defective tests, and data contamination; models exploiting scoring procedures; and the limited relevance of general scores to users' tasks. Documented cases illustrate why these problems require different responses. I argue for evaluation procedures that disclose the tested configuration, validate questions and successful task completion, report performance alongside cost and execution time, and make the scope of generalization explicit. I then discuss Isotanta, a crowdsourced benchmarking platform, as a practical example of contributed questions and repeated evaluation. A larger question pool may improve task coverage, while repeated sampling can improve the stability of estimates on that pool; neither guarantees validity or personalization. The paper distinguishes the platform's current shared ranking from proposed task-specific and user-provided evaluations. Its central argument is that model selection requires evidence about performance on the intended work, not simply a high position on a general leaderboard.

Danial Amin

LLM EvaluationBenchmark Validity
AskPDF