BenchScope: How Many Independent Signals Does Your Benchmark Provide?

Tommy Sha; Stella Zhao

cs.AIMar 31, 2026

BenchScope: How Many Independent Signals Does Your Benchmark Provide?

Authors: Tommy Sha, Stella Zhao

Organizations: Stony Brook University · University of Minnesota Twin Cities

Abstract

AI evaluation suites often report many scores without checking whether those scores carry independent information. We introduce Effective Dimensionality (ED), the participation ratio of a centered benchmark-score spectrum, as a fast, population-conditional upper-bound diagnostic of measurement breadth. Applied at per-instance granularity to 22 benchmarks across 8 domains and more than 8,400 model evaluations, ED reveals substantial redundancy: the six-score Open LLM Leaderboard behaves like roughly two effective measurement axes (ED = 1.7), BBH and MMLU-Pro are near-interchangeable (rho = 0.96, stable across seven subpopulations), and measurement breadth varies more than 20x across current benchmarks. We show that relative ED rankings are stable under matched-dimension controls and that ED can flag redundant suite components, monitor performance-conditional compression, and guide benchmark maintenance. Because binary spectra overestimate absolute latent dimensionality, we interpret ED as a screening statistic rather than a literal factor count and complement it with null, reliability, and saturation analyses. We provide a 22-benchmark reference atlas and a four-step diagnostic workflow that benchmark maintainers can run with a score matrix and a few lines of code.

Figures & tables

Figure 1: Redundancy structure in the Open LLM Leaderboard v2. (a) The pairwise correlation matrix (full 4,576-model population) shows that the six-score suite behaves like roughly two effective measurement axes ( $\text{ED}=1.7$ ): BBH and MMLU-Pro are near-duplicates ( $\rho=0.96$ ) while other pairs are only weakly correlated (e.g., IFEval–MuSR: $\rho=0.32$ ). (b) A subset-specific pattern ( $n=188$ only): within a curated subset, MATH and MuSR are negatively correlated ( $\rho=-0.64$ ); in the full population, the correlation is positive ( $\rho=0.50$ ; see Figure 5 ).

Figure 2: From matrix to metric: ED computation pipeline. A binary pass/fail matrix is task-centered, spectrally decomposed via SVD, and summarized by the participation ratio, a single closed-form scalar requiring no thresholds, no permutations, and no iterations.

Benchmark	$T$	$N$	$\textbf{ED}_{\textbf{null}}$	$\textbf{ED}_{\textbf{obs}}$	Ratio
SWE-bench Verified	500	134	105.7	7.9	0.075
Open LLM (43-sub)	43	188	35.0	4.5	0.129
BigCodeBench-complete	1140	153	134.9	28.9	0.214

Table 1: Observed ED vs. Marchenko–Pastur null (Eq. 3 ). The ratio

\text{ED}_{\text{obs}}/\text{ED}_{\text{null}}

quantifies how much more structured than random each benchmark is, on a 0–1 scale.

Unconstrained ( $\min(T,N)>6$ ):
Benchmark	Domain	Size ( $T\!\times\!N$ )	ED	ED ${}_{\text{tet}}$	PC1%	ED/ED ${}_{\text{n}}$
BigCodeBench-instruct	Coding (prac.)	$1140\!\times\!126$	33.0	${\sim}$ 17	14.7	.291
BigCodeBench-complete	Coding (prac.)	$1140\!\times\!153$	28.9	${\sim}$ 16	16.5	.214
BigCodeBench-hard-comp.	Coding (prac.)	$148\!\times\!199$	18.5	${\sim}$ 10	20.6	.218
BigCodeBench-hard-inst.	Coding (prac.)	$148\!\times\!173$	18.3	${\sim}$ 10	20.3	.229
SWE-bench Verified	Coding (repo)	$500\!\times\!134$	7.9	${\sim}$ 4	34.4	.075

Table 2: Effective dimensionality of 22 benchmarks. ED is the Pearson-based upper bound;

\text{ED}_{\text{tet}}

is the tetrachoric-corrected estimate (

{\approx}0.5\times

ED). Top : unconstrained. Bottom : shape-constrained (

\dagger

\min(T,N)\leq 6

). Relative ordering survives matched-dimension controls (

\rho=1.0

300\times 50

Figure 3: Taxonomy $\neq$ measurement breadth. All three examples use per-instance ED as the common metric. BFCL’s 20 categories yield per-instance ED = 1.5; MMLU-Pro’s 14 disciplines yield category-level ED = 1.1; BigCodeBench’s single label yields ED = 29 (raw), of which ${\sim}8$ PCs exceed the permutation null. Category counts can overstate or understate empirical breadth by an order of magnitude.

Figure 4: ED across 22 benchmarks, sorted by descending ED. BigCodeBench (

\text{ED}=29

) spans more than

20\times

the raw dimensionality of GAIA and EvalPlus (

\text{ED}=1.2

). The variation is not explained by benchmark size alone; task-set design and matrix shape both contribute (Section 5.2 ).

Benchmark Pair	$\bm{\rho}$	95% CI	Partial $\bm{\rho}$	Type
MATH $\times$ MuSR	$-0.635$	$[-0.714,\;-0.540]$	$-0.625$	Negative
GPQA $\times$ MATH	$-0.340$	$[-0.470,\;-0.200]$	$-0.584$	Negative
IFEval $\times$ MuSR	$-0.310$	$[-0.440,\;-0.170]$	$-0.611$	Negative
IFEval $\times$ MMLU-Pro	$-0.150$	$[-0.290,\;-0.010]$	$-0.479$	Negative
GPQA $\times$ IFEval	$-0.270$	$[-0.410,\;-0.130]$	$-0.568$	Negative
BBH $\times$ MATH	$-0.210$	$[-0.350,\;-0.060]$	$-0.022$	Negative

Table 3: Pairwise correlations from the Open LLM Leaderboard v2 (

n=188

curated models). Negative : 95% bootstrap CI entirely below 0. Redundant : CI entirely above 0.5.

\rho

and CI are unconditional Spearman correlations; partial

\rho

controls for log parameter count. These correlations are specific to this 188-model subset; all 15 pairs are positive in the full 4,576-model population.

Pair	Spearman $\bm{\rho}$	Ceiling $\sqrt{(1{+}\rho)/2}$
MATH $\times$ MuSR	$-0.64$	$0.42$
GPQA $\times$ MATH	$-0.34$	$0.57$
IFEval $\times$ MuSR	$-0.31$	$0.59$
GPQA $\times$ IFEval	$-0.27$	$0.60$
BBH $\times$ MATH	$-0.21$	$0.63$
IFEval $\times$ MMLU-Pro	$-0.15$	$0.65$

Table 4: Composite correlation ceiling (Theorem 1 ) for selected benchmark pairs within the 188-model subset.

Figure 5: Pairwise correlations in the Open LLM Leaderboard v2. (a) MATH vs. MuSR scatter for the 188-model subset, colored by model size: the negative correlation (

\rho=-0.64

) is visible within size bands, but in the full 4,576-model population the correlation is positive (

\rho=+0.50

). (b) All 15 benchmark pairs sorted by Spearman

\rho

with bootstrap 95% CIs. Six pairs are significantly negative (blue); two are redundant (red,

\rho>0.5

). These negative correlations are specific to the 188-model subset and do not generalize to the full population (Section 4.3 ).

Figure 6: Eigenvalue spectra reveal how many PCs are real. BigCodeBench (blue) has 8 PCs exceeding the 95% permutation null (shaded signal region); SWE-bench (purple) has ${\sim}4$ – $6$ ; BFCL (red) collapses to one dominant axis. Raw ED overstates true dimensionality by $3$ – $4\times$ .

Figure 7: BFCL’s 20 claimed categories collapse into 7 effective groups (redundancy $=2.8\times$ ). Group 7: simple_python $\approx$ simple_java $\approx$ simple_javascript ( $r=0.84$ ), three languages testing the same skill.

Figure 8: (a) Controlled experiment: structural heterogeneity, not topic diversity, drives ED. (b) Saturation curves: SWE-bench saturates at

\text{ED}_{\infty}\approx 8

; BigCodeBench saturates slowly (

\text{ED}_{\infty}\approx 35

Figure 9: Performance-conditional compression on the Open LLM Leaderboard (4,576 models, sorted by composite score, sliding window of 500). ED declines from 2.65 (weakest quintile) to 1.40 (upper-middle plateau) then partially recovers to 2.52 among frontier models. Note: the x-axis is a performance-ranked window, not a time axis. The frontier recovery ( $p>0.999$ , Cohen’s $d=7.8$ ) likely reflects heterogeneous training philosophies among top-performing models.

Figure 10: Benchmark aging. (a) SWE-bench ED declines 17% from early to late agents. (b) Solve rate improves 60% (40% to 64%). (c) Pairwise agent correlations shift rightward (mean

\rho=0.43\to 0.55

): late agents are 29% more similar to each other. A fixed-variance control shows the SWE-bench decline is partly a ceiling effect; the leaderboard-level decline survives variance standardization.

Figure 11: Semantic validation. (a) SWE-bench: PC2 separates 2024-era agents (red) from 2025-era (blue),

\text{AUC}=0.97

. (b) BigCodeBench: PC2 separates code-specialized (red) from general-purpose models (blue),

\text{AUC}=0.94

Figure 12: Four-step diagnostic workflow for benchmark maintainers. (1) Flag redundant pairs ( $\rho>0.9$ ); (2) check suite-level ED; (3) monitor temporal decline; (4) vet new additions ( $\max\rho<0.7$ ). All computations are closed-form and require only a score matrix.

Benchmark	Task-center	Model-center	Double-center
BigCodeBench	28.9	4.4	62.3
SWE-bench Verified	7.9	3.8	30.8
BFCL	1.5	3.1	3.6

Table 5: ED under three centering methods. The relative ordering (BigCodeBench

\gg

SWE-bench

>

BFCL) is stable across all methods.

Benchmark	ED	PA	Kaiser	ED/PA
BFCL	1.5	3	8	0.5
LiveCodeBench	6.2	3	6	2.1
SWE-bench	7.9	4	27	2.0
BigCodeBench	28.9	9	43	3.2

Table 6: ED vs. parallel analysis (PA) and Kaiser criterion on four per-instance benchmarks. ED overestimates absolute dimensionality but preserves the rank ordering.

Benchmark Auditing

Benchmark auditing checks whether evaluation datasets, metrics, and infrastructure measure the capabilities and risks they claim to assess.

See all 97 papers

AI Cartography: Mapping the Latent Landscape of AI Benchmark Ecosystems
May 24, 2026Michael Hardy, Anka Reuel, Lijin Zhang +6LLM Evaluation Benchmark Design
Stanford University, Stanford, CA, USA · BroadMetrics, Edison, NJ, USA
BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validity
Feb 25, 2026Harshita Diddee, Gregory Yauney, Swabha Swayamdipta +1Benchmark Validity Benchmark Design
Language Technologies Institute, Carnegie Mellon University, Pittsburgh (PA), United States · Thomas Lord Department of Computer Science, University of Southern California, Los Angeles (CA), United States
The Evaluation Blind Spot: A Stereological Theory of Benchmark Coverage for Large Language Models
Apr 15, 2026Jason Z WangLLM Evaluation
Independent

May 24, 2026J/K move · Enter open · S save

AI Cartography: Mapping the Latent Landscape of AI Benchmark Ecosystems

Michael Hardy, Anka Reuel, Lijin Zhang, Jodi M. Casabianca+5

Stanford University, Stanford, CA, USA · BroadMetrics, Edison, NJ, USA

LLM Evaluation Benchmark Design Psychometric Validation Benchmark Validity

Open paper AskPDF arXiv

While aggregate leaderboard scores drive AI development, they contain substantial measurement noise whose sources and magnitudes remain unquantified, making it unclear when rankings reflect genuine capability differences versus evaluation artifacts. We introduce a framework for measuring the latent landscape in AI benchmark ecosystems. Applying Confirmatory Factor Analysis (CFA) and Generalizability Theory to 4,000+ models from the Open LLM Leaderboard, we decompose sources of ranking variance and establish: (1) structures assumed in current reporting practice underestimate the strength of relationships between benchmarks; (2) evidence of local dependence among leaderboard items, undermining uses of benchmarks as measurement instruments under current scoring systems; (3) contributor metadata explains more rank-relevant variance (

\approx9\%

) than architecture or deployment categories in this context; (4) a manifest-score "scaling law" slope has low reliability (

R_β=0.53

); by contrast, the latent general-factor size slope is highly stable across ecosystem controls (

R_g=0.97

). We are able to provide unique insights into benchmark dynamics, such as which benchmarks are a function of LLM size and which can be oppositely impacted by post-training practices. We provide actionable diagnostics to determine how benchmark rankings can be trusted and how benchmark design can be improved.

Abstract

Benchmark Auditing

Explore similar work

AI Cartography: Mapping the Latent Landscape of AI Benchmark Ecosystems

BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validity

The Evaluation Blind Spot: A Stereological Theory of Benchmark Coverage for Large Language Models