Large Language Models (LLMs) are advancing rapidly, yet the benchmarks used to measure this progress are becoming increasingly unreliable. Score inflation and selective reporting have eroded the authority of standard benchmarks, leaving the community uncertain about which evaluation results remain trustworthy. We introduce the Benchmark Health Index (BHI), a pure data-driven framework for auditing evaluation sets along three orthogonal and complementary axes: (1) Capability Discrimination, measuring how sharply a benchmark separates model performance beyond noise; (2) Anti-Saturation, estimating remaining headroom before ceiling effects erode resolution and thus the benchmark's expected longevity; and (3) Impact, quantifying influence across academic and industrial ecosystems via adoption breadth and practice-shaping power. By distilling 106 validated benchmarks from the technical reports of 91 representative models in 2025, we systematically characterize the evaluation landscape. BHI is the first framework to quantify benchmark health at a macro level, providing a principled basis for benchmark selection and enabling dynamic lifecycle management for next-generation evaluation protocols.
Figures & tables
Figure 1: Top 10 Benchmarks Ranked by BHI
Figure 2: Overview of BHI Framework. Top Row: Three challenges characterize current benchmarks: weak discrimination, performance saturation, and popularity–adoption decoupling. Middle Row: We introduce three data-driven metrics: SDisc quantifies benchmark sensitivity, SAS assesses challenge headroom, and SImp measures capability-weighted adoption across the model ecosystem. Bottom Row: The final BHI score is synthesized through objective weighting.
Figure 3: The BHI Data Architecture. (a) The systematic process of data construction. (b) The taxonomic distribution of the validated benchmark set across 13 functional domains. (c) The distribution of evaluated models across diverse global AI vendors.
Figure 4: Performance evolution on Humanity’s Last Exam. The dashed line represents the fitted linear trend, whose slope ( k ) indicates the rate of score improvement over time.
Rank
Benchmark
SDisc ( RCV , EDR )
SAS ( SSta , SDyn )
SImp ( NUsage )
BHI
1
Humanity’s Last Exam [ 101 ]
0.6608 (0.2954, 0.9131)
0.6299 (0.6211, 0.6653)
0.7710 (56)
0.6846
3
SimpleQA [ 130 ]
0.8130 (0.4599, 0.9347)
0.5043 (0.5011, 0.5169)
0.4435 (27)
0.5857
4
BrowseComp [ 131 ]
0.8042 (0.4490, 0.9365)
0.3281 (0.3200, 0.3607)
0.6475 (36)
0.5845
7
ZeroBench [ 110 ]
0.5205 (0.1925, 0.8000)
0.8687 (0.8594, 0.9059)
0.2498 (6)
0.5593
16
AIME 2025 [ 152 ]
0.6923 (0.3445, 0.8889)
0.1154 (0.1122, 0.1284)
0.7194 (61)
0.4947
20
Tau2-bench-Telecom [ 9 ]
0.9005 (0.6419, 0.7800)
0.1108 (0.1058, 0.1309)
0.5016 (25)
0.4921
Table 1: BHI Metrics and Sub-metric Decomposition for Representative Benchmarks
Removed Component
Avg. Shift
Max Shift
SDisc
15.53
49
SAS
16.18
82
SImp
11.34
66
Table 2: Ablation results showing ranking shifts after removing each component.
Figure 5: Indicator correlation analysis using Pearson (left) and Spearman (right) coefficients.
Figure 6: Temporal adoption patterns of known-saturated and active frontier benchmarks.
Category
Benchmark
SDisc
SAS
SImp
BHI
Rank
Known Saturated
HumanEval [ 76 ]
0.5520
0.1669
0.1232
0.2787
113
MMLU [ 52 ]
0.3215
0.1193
0.3248
0.2502
126
MBPP [ 57 ]
0.4225
0.1744
0.1168
0.2371
129
GSM8K [ 23 ]
0.3920
0.0865
0.1931
0.2199
140
HellaSwag [ 149 ]
0.4395
0.0835
0.1263
0.2132
142
C-Eval [ 56 ]
0.2674
0.0935
0.2816
0.2098
143
Table 3: BHI Distribution Across Known-Saturated and Active Frontier Benchmark Cohorts
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
δ (%)
Spearman
Top-10 Overlap
wDisc
wAS
wImp
0.5
0.9582
10
0.3643
0.3386
0.2971
1.0
0.9870
9
0.3255
0.3598
0.3147
1.5
0.9963
9
0.3168
0.3644
0.3188
2.0
1.0000
10
0.3263
0.3574
0.3163
2.5
0.9955
10
0.3383
0.3465
0.3152
3.0
0.9865
10
0.3603
0.3354
0.3043
Appendix
Table 4: Sensitivity of BHI rankings to the EDR threshold δ . Spearman correlation is computed against the default setting δ=2.0% .
Figure 7: Sensitivity of BHI ranking stability and the CRITIC-assigned discrimination weight under different EDR thresholds.
RCV interval
Spearman
Kendall
Top-10
Top-20
Rank shift
Max weight
ρ
τb
overlap
overlap
Median
Max
drift (pp)
P0–P100
0.9935
0.9463
9/10
17/20
2
29
1.806
P5–P95
0.9978
0.9716
9/10
19/20
1
21
0.772
P10–P90
1.0000
1.0000
10/10
20/20
0
0
0
P15–P85
0.9986
0.9740
10/10
18/20
1
9
0.293
P20–P80
0.9962
0.9547
10/10
18/20
2
14
0.376
Appendix
Table 5: Sensitivity of BHI rankings to the RCV percentile interval. All comparisons use P10–P90 as the reference. Rank shifts are absolute; weight drift is the maximum absolute change across the three CRITIC weights, measured in percentage points (pp).
Figure 8: Chronological distribution of the 91 mainstream large language models released throughout 2025.
σ
Spearman
Kendall
Score Stability
Top-10 Overlap
Mean Rank Shift
0.01
0.9911
0.9353
0.9898
9.67
3.70
0.02
0.9827
0.9027
0.9852
9.45
5.53
0.05
0.9544
0.8329
0.9755
8.90
9.44
0.10
0.9184
0.7674
0.9650
8.67
13.00
0.15
0.8854
0.7189
0.9543
8.44
15.69
0.20
0.8575
0.6804
0.9458
8.36
17.75
Appendix
Table 6: Noise robustness under CRITIC weighting. Score stability is measured as 1−MAE between perturbed and baseline BHI scores.
σ
Spearman
Kendall
Score Stability
Top-10 Overlap
Mean Rank Shift
0.01
0.9909
0.9306
0.9892
9.19
3.99
0.02
0.9819
0.8967
0.9845
9.14
5.91
0.05
0.9523
0.8258
0.9744
8.78
9.89
0.10
0.9106
0.7527
0.9619
8.36
13.81
0.15
0.8748
0.7013
0.9504
7.93
16.67
0.20
0.8425
0.6595
0.9405
7.72
18.90
Appendix
Table 7: Noise robustness under equal weighting.
Figure 9: Noise robustness of BHI under CRITIC weighting.
Figure 10: Noise robustness of BHI under equal weighting.
Dropout Ratio
Spearman
Kendall
Top-10 Overlap
Mean Rank Shift
Models Used
0.05
0.9961
0.9667
9.31
2.99
112
0.10
0.9911
0.9389
9.14
5.12
106
0.15
0.9851
0.9134
9.04
7.56
100
0.20
0.9814
0.8986
8.91
9.31
94
0.25
0.9769
0.8835
8.75
11.34
88
0.30
0.9716
0.8684
8.48
13.53
82
Appendix
Table 8: Model dropout robustness under CRITIC weighting.
Dropout Ratio
Spearman
Kendall
Top-10 Overlap
Mean Rank Shift
Models Used
0.05
0.9956
0.9586
8.83
3.30
112
0.10
0.9909
0.9353
8.90
5.20
106
0.15
0.9854
0.9131
8.81
7.57
100
0.20
0.9815
0.8984
8.75
9.32
94
0.25
0.9768
0.8833
8.69
11.37
88
0.30
0.9717
0.8692
8.55
13.55
82
Appendix
Table 9: Model dropout robustness under equal weighting.
Figure 11: Ranking stability under model dropout using CRITIC weighting.
Figure 12: Ranking stability under model dropout using equal weighting.
Window T
Spearman
Kendall
Top-10 Overlap
Top-20 Overlap
Avg. Rank Shift
20
0.9999
0.9966
10
20
0.2405
25
1.0000
0.9982
10
20
0.1392
30
1.0000
1.0000
10
20
0.0000
35
0.9999
0.9973
10
20
0.2152
40
0.9999
0.9955
10
20
0.3291
Appendix
Table 10: Sensitivity of BHI rankings to the temporal projection window in SDyn .
Figure 13: Sensitivity of BHI rankings to the temporal window used in the dynamic anti-saturation component.
wSta
wDyn
Spearman
Kendall
Top-10 Overlap
Top-20 Overlap
Avg. Rank Shift
0.6
0.4
0.9995
0.9889
10
20
0.7722
0.7
0.3
0.9998
0.9931
10
20
0.5063
0.8
0.2
1.0000
1.0000
10
20
0.0000
0.9
0.1
0.9998
0.9939
9
20
0.4177
Appendix
Table 11: Sensitivity of BHI rankings to the static–dynamic weighting in SAS .
Figure 14: Sensitivity of BHI rankings to the relative weighting between static resistance and dynamic saturation projection.
Metric
Pearson Correlation
Spearman Correlation
SDisc
0.2807
0.2424
SAS
-0.0275
-0.0333
SImp
0.9130
0.8862
BHI
0.4737
0.4000
Appendix
Table 12: Correlation between benchmark coverage and BHI components.
Figure 15: Relationship between benchmark coverage and final BHI score. Coverage is measured as the number of models with valid scores on each benchmark.
Indicator
Original submission
June 2026
Change
Models
117
133
+16
Eligible benchmarks
158
185
+27
wDisc
32.63%
32.72%
+0.09 pp
wAS
35.74%
37.45%
+1.71 pp
wImp
31.63%
29.83%
−1.80 pp
Appendix
Table 13: Data coverage and CRITIC weights in the original submission and the June 2026 update. Weight changes are measured in percentage points (pp).
Rank
Benchmark
SDisc ( RCV , EDR )
SAS ( SSta , SDyn )
SImp ( NUsage )
BHI
1
Humanity’s Last Exam [ 101 ]
0.6608 (0.2954, 0.9131)
0.6299 (0.6211, 0.6653)
0.7710 (56)
0.6846
2
SWE-Lancer(no limited) [ 88 ]
0.9979 (0.6396, 1.0000)
0.6102 (0.6129, 0.5992)
0.1757 (3)
0.5992
3
SimpleQA [ 130 ]
0.8130 (0.4599, 0.9347)
0.5043 (0.5011, 0.5169)
0.4435 (27)
0.5857
4
BrowseComp [ 131 ]
0.8042 (0.4490, 0.9365)
0.3281 (0.3200, 0.3607)
0.6475 (36)
0.5845
5
ARC-AGI-2 [ 21 ]
0.8531 (0.4871, 0.9714)
0.4411 (0.4487, 0.4104)
0.4513 (15)
0.5787
6
LiveCodeBench-V6 [ 59 ]
0.8266 (0.4733, 0.9393)
0.2786 (0.2680, 0.3213)
0.6593 (56)
0.5779
Appendix
Table 14: Detailed BHI Metric Results and Sub-metric Results
LLM benchmarks are often treated as coherent measurement units, yet they are heterogeneous collections of instances spanning diverse domains, skills, formats, and contexts. As a result, aggregate benchmark scores can conflate model capability with benchmark composition, obscuring what existing benchmarks actually cover. We introduce BenchHub, a composition-aware evaluation framework that represents benchmarks as distributions over instance-level attributes. BenchHub integrates 54 benchmarks comprising 839K samples across 10 languages. It enables researchers to inspect benchmark contents, uncover reusable coverage hidden behind benchmark names, transparently compare benchmarks, and construct controllable evaluation sets for application-aligned model selection. Using BenchHub, we show that similarly motivated benchmarks can differ substantially in internal composition, existing benchmarks often contain reusable coverage beyond their stated purposes, models exhibit fine-grained category-level performance variation hidden by aggregate scores, and model rankings can shift under different reweighting and resampling configurations. Our results motivate evaluation practices that make benchmark composition explicit, inspectable, and controllable.
Benchmark scores increasingly influence the development, marketing, and selection of large language models (LLMs). Yet an overall score is interpretable only in relation to the system tested, the questions included, and the conditions of evaluation. This perspective examines five connected limitations of general LLM rankings: differences between evaluated and publicly available systems; commercial incentives and dependencies in external evaluation; benchmark saturation, defective tests, and data contamination; models exploiting scoring procedures; and the limited relevance of general scores to users' tasks. Documented cases illustrate why these problems require different responses. I argue for evaluation procedures that disclose the tested configuration, validate questions and successful task completion, report performance alongside cost and execution time, and make the scope of generalization explicit. I then discuss Isotanta, a crowdsourced benchmarking platform, as a practical example of contributed questions and repeated evaluation. A larger question pool may improve task coverage, while repeated sampling can improve the stability of estimates on that pool; neither guarantees validity or personalization. The paper distinguishes the platform's current shared ranking from proposed task-specific and user-provided evaluations. Its central argument is that model selection requires evidence about performance on the intended work, not simply a high position on a general leaderboard.
Benchmarks are essential for evaluating and improving large language models (LLMs). Yet many widely used benchmarks are redundant, making evaluation unnecessarily expensive. Benchmark compression lowers this cost by estimating full-benchmark scores and rankings from a partial evaluation. Research in this area has grown rapidly since 2024, but to our knowledge no survey has systematically organized its methods and findings. We review 119 papers to explain how methods choose what to evaluate, recover full-benchmark results, and verify their reliability. From this analysis, we develop a taxonomy covering compression within individual benchmarks, across benchmark suites, and during adaptive evaluation, together with work on redundancy and reliability. We compare their information needs, total costs, reliability checks, and limitations. Methods reduce evaluation costs by selecting representative subsets, using cheaper or reusable information, and testing adaptively, but risk poor coverage, inaccurate estimates, and sequential delays. The resulting taxonomy and practical guide help researchers choose, validate, and deploy methods for more efficient and reliable LLM evaluation.