Benchmark compression reduces the often prohibitive time and cost of model evaluation by evaluating models on a small benchmark subset that closely matches full-benchmark results. To quantify estimation error, we derive an accurate analytical approximation to the error distribution when estimating full-benchmark scores from a subset, showing that most subsets yield small errors, while a few produce much larger errors, resulting in a long-tailed distribution. This means random sampling alone can achieve surprisingly strong performance with high probability, but remains unstable due to rare large errors. We further show that this instability is inherent to modern foundation models and persists even as more models are evaluated. Based on these findings, we introduce BestRand, a stable and low-cost benchmark compression method that preserves the efficiency of random sampling while substantially reducing its high-error tail. We validate BestRand on 70+ text, multimodal, and agent benchmarks, and further extend it to VLA evaluation. Compared with competitive methods, BestRand reduces high-error risk by 50.2% and estimation error by 16.4% on average, while requiring only 51.3% of the construction cost. It further reduces OOD estimation error by 21.8% on average across score-range, model-family, and temporal shifts. Together, our theory provides practical guidance for understanding and predicting the accuracy and reliability of benchmark compression, while BestRand enables stable and accurate compression at low cost. The code and data are publicly available onhttps://github.com/George-Ling3/BestRand.
Figures & tables
Figure 1: Empirical results are consistent with our theory. Top: empirical MAE distributions and modes closely match theoretical results on representative LLM, VLM, and VLA benchmarks. Bottom: ρ is generally positive; higher ρ theoretically yields longer tails, while ρ and s strongly predict empirical MAE P95, which captures long-tail risk. Additional validation is provided in Appendix F .
Figure 2: Error distributions of Random and BestRand on six representative benchmarks at 20% retention. Random exhibits a pronounced right tail, whereas BestRand concentrates errors near zero and suppresses high-error outcomes. Results on additional benchmarks are shown in Figure 8 .
Average MAE (%) ↓ mean ± std.
Method
2%
5%
10%
15%
20%
50%
Mean
Text 44 benchmarks
SparseEval (Linear)
19.88 ± 9.48
21.08 ± 6.99
22.25 ± 7.72
21.18 ± 10.10
21.44 ± 9.60
20.04 ± 7.95
20.98 ± 8.67
SparseEval (MLP)
12.53 ± 5.85
11.94 ± 5.16
11.70 ± 5.38
11.49 ± 5.32
11.74 ± 5.51
11.80 ± 5.80
11.87 ± 5.46
Anchor Points
7.68 ± 3.24
7.23 ± 2.66
6.78 ± 2.73
6.32 ± 2.66
6.41 ± 2.73
4.72 ± 2.58
6.52 ± 2.90
TinyBenchmarks
7.45 ± 4.49
4.94 ± 3.34
3.26 ± 2.22
2.56 ± 1.82
2.26 ± 1.70
1.45 ± 1.55
3.65 ± 3.37
Table 1: Compression accuracy across 70 benchmarks at six retention ratios. Results are grouped into 44 text, 16 multimodal, and 10 agent benchmarks. Each entry reports average MAE (%) and the standard deviation across benchmarks. The final column averages all benchmarks and retention ratios. Lower is better. Best mean values are bold and second-best mean values are underlined .
Figure 3: High-error risk across text, multimodal, and agent benchmarks. Worst-30% entry and worse-than-average entry use Random’s 70th-percentile and mean errors as thresholds, respectively. Whiskers show 95% bootstrap confidence intervals. BestRand achieves the lowest risk in all settings. Rand, Zip, Tiny, AP, SE-M, and SE-L denote Random, ZipBench, TinyBenchmarks, Anchor Points, SparseEval MLP, and SparseEval Linear, respectively. Detailed results are shown in Figure 6 .
Figure 4: BestRand ablations at 20% retention. (a) Effects of the number of probe models, the number of candidate subsets, and the candidate-pool size. (b) Performance comparison with more construction models. (c) Effect of probe-model selection. (d) Effect of the matching norm.
Figure 5: Low-cost compression at 20% retention. (a) At the same construction cost, BestRand achieves substantially lower average MAE across 12 text benchmarks. Response-free methods appear at Cost=0 but generally have much larger errors. BestRand remains accurate at Cost<1 , while learnable baselines require Cost≥1 . (b) Over the displayed accuracy range, BestRand requires substantially less construction cost to reach the same average MAE on six text benchmarks.
Score Range OOD
Family OOD
Temporal OOD
Method
Mid → Tails
Low → High
High → Low
Held-out Families
Past → Future
Mean
SparseEval (Linear)
18.50 ± 3.24
21.50 ± 5.74
18.31 ± 4.58
22.45 ± 3.63
24.32 ± 6.65
22.07 ± 2.98
SparseEval (MLP)
16.13 ± 7.04
16.43 ± 7.57
21.59 ± 9.64
11.87 ± 5.89
11.71 ± 6.05
13.88 ± 6.12
Anchor Points
4.94 ± 1.62
5.63 ± 3.03
6.96 ± 4.52
7.26 ± 1.93
8.23 ± 2.64
7.11 ± 1.57
TinyBenchmarks
2.34 ± 1.17
2.46 ± 1.35
3.03 ± 1.93
2.20 ± 0.93
2.27 ± 1.01
2.36 ± 0.89
ZipBench
2.03 ± 0.94
2.43 ± 1.62
2.30 ± 1.06
2.16 ± 1.28
2.15 ± 1.28
2.19 ± 1.21
Table 2: OOD performance under score-range, model-family, and temporal shifts. Arrows in the column headers point from the construction group to the test group. BestRand achieves the lowest average MAE in every setting. The final column averages the three OOD types equally.
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
Type
Benchmark
Public Link
# Items
# Test Models
Metric (Single?)
Multimodal Benchmarks
CountBenchQA ( Beyer et al., 2024 )
HF/CountBenchQA
491
11
Accuracy ( ✓ )
DynaMath ( Zou et al., 2025 )
HF/DynaMath_Sample
5,010
20
Accuracy ( ✗ )
LogicVista ( Xiao et al., 2024 )
GH/LogicVista
448
20
Accuracy ( ✓ )
MathVerse ( Zhang et al., 2024 )
HF/MathVerse
788
20
Accuracy ( ✓ )
MathVista ( Lu et al., 2024 )
HF/MathVista
1,000
10
Accuracy ( ✓ )
MME ( Fu et al., 2025a )
GH/MME
2,374
16
MME Score ( ✓ )
Appendix
Table 3: Details of the 16 multimodal and 10 agent benchmarks used in the main experiments and the two VLA benchmarks used in the extension. Metric (Single?) reports the metric used in our experiments and whether it is the benchmark’s only official metric.
Type
Benchmark
Public Link
# Items
# Test Models
Metric (Single?)
Text Benchmarks
ACPBench Hard ( Kokel et al., 2026 )
HF/acp_bench
1,040
20
Accuracy ( ✓ )
AGC-Bench ( Beaty et al., 2026 )
HF/agcbench-2026
100
20
Accuracy ( ✗ )
AIME ( MAA, 2026 )
↗ MAA/AIME
60
20
Accuracy ( ✓ )
ARC-Challenge ( Clark et al., 2018 )
HF/ai2_arc
1,172
20
Accuracy ( ✓ )
ARC-Easy ( Clark et al., 2018 )
HF/ai2_arc
2,376
20
Accuracy ( ✓ )
BBEH ( Kazemi et al., 2025 )
GH/bbeh
4,520
20
Accuracy ( ✓ )
Appendix
Table 4: Details of the 44 text benchmarks used in the main experiments. Each benchmark uses five construction models and the remaining models for testing. Metric (Single?) reports the metric used in our experiments and whether it is the benchmark’s only official metric.
Construction Information
Type
Method
Text
Embed.
Scores
All items
Full score
Response-free
Random
—
—
—
—
—
Facility Location ( Yao et al., 2026 )
—
✓
—
—
—
MIS ( Kjorvezir et al., 2026 )
—
✓
—
—
—
Agreement ( Purificato et al., 2026 )
—
✓
—
—
—
Coverage ( Purificato et al., 2026 )
—
✓
—
—
—
Appendix
Table 5: Comparison of the construction information required by BestRand and 15 diverse baselines. Text and Embed. denote direct prompt features and pretrained text embeddings. Scores denotes historical scores for individual models and items, All items indicates whether these scores must cover the entire benchmark, and Full score denotes the corresponding full benchmark aggregate. Among response-based methods, BestRand has lower information requirements because it uses scores from only a subset of items, reducing construction cost while retaining strong empirical performance.
Figure 6: High-error risk across six representative text, multimodal, and agent benchmarks. Labels T, M, and A denote the text, multimodal, and agent benchmark categories, respectively. Columns use Random’s empirical mean, 70th percentile, and 95th percentile MAE as thresholds for the worse-than-average, worst 30% tail, and worst 5% tail region. Results are averaged equally across six retention ratios. Random and BestRand use 10,000 trials per ratio, while the other baselines use three trials. Rand, Zip, Tiny, AP, SE-M, and SE-L denote Random, ZipBench, TinyBenchmarks, Anchor Points, SparseEval MLP, and SparseEval Linear, respectively.
Figure 7: We conduct an additional experiment on the same six representative benchmarks as in Figure 6 . Random uses the same 10,000 trials, while BestRand and all other methods are evaluated with ten seeds per retention ratio. The high error risk patterns remain similar across the two experiments. Averaged over the worse-than-average and worst-30% metrics, BestRand reduces high error risk relative to Random, ZipBench, and TinyBenchmarks by 68.06% on the six benchmarks in Figure 6 and by 71.06% in this new experiment. This close agreement confirms that the main conclusion is robust to the number of seeds and that BestRand consistently reduces high error risk.
Figure 8: MAE distributions of Random and BestRand across 18 text, multimodal, and agent benchmarks. BestRand consistently concentrates errors near zero and suppresses high-error outcomes.
Figure 9: Empirical MAE distributions closely match theoretical predictions across representative text, multimodal, and agent benchmarks, including the predicted shapes and modes.
Prediction error ↓ mean ± std.
Method
SR
DS
Eff. MRE
Smooth.
Avg. Error
Bench2Drive, removal ratio =50%
SparseEval (Linear)
16.14 ± 0.24
1.22 ± 0.16
2.73 ± 1.73
1.38 ± 0.31
5.37 ± 0.49
SparseEval (MLP)
3.50 ± 0.15
1.78 ± 0.19
2.31 ± 2.13
1.85 ± 0.73
2.36 ± 0.37
Anchor Points
5.21 ± 1.48
4.55 ± 1.62
2.40 ± 1.71
3.44 ± 2.15
3.90 ± 1.38
TinyBenchmarks
2.15 ± 0.12
2.38 ± 0.50
3.80 ± 3.18
2.02 ± 0.62
2.58 ± 0.83
Appendix
Table 6: Bench2Drive compression results using five training models. Within each run, all train-dependent methods use the same SR-stratified model sample. We report percentage-point MAE for SR and Smoothness, score MAE for DS, and mean relative error in percent for Efficiency. Avg. Error averages these four errors. Lower is better. Standard deviations appear in smaller type. The best and second-best means in each removal-ratio block are bolded and underlined , respectively.
Prediction error ↓ mean ± std.
Method
ID SR
ID DS
ID HM
Gen SR
Gen DS
Gen HM
Avg. Error
Fail2Drive, removal ratio =50%
SparseEval (Linear)
17.40 ± 8.19
4.63 ± 0.62
10.44 ± 7.40
18.48 ± 2.77
5.56 ± 0.78
13.17 ± 4.26
11.62 ± 3.38
SparseEval (MLP)
4.56 ± 0.67
3.53 ± 0.95
3.07 ± 0.66
6.71 ± 2.20
3.86 ± 0.55
3.17 ± 0.81
4.15 ± 0.36
Anchor Points
2.79 ± 0.09
2.51 ± 0.63
2.09 ± 0.43
5.33 ± 1.30
3.99 ± 0.42
3.88 ± 1.18
3.43 ± 0.39
TinyBenchmarks
2.52 ± 0.23
1.81 ± 0.49
1.89 ± 0.28
3.49 ± 0.93
1.63 ± 0.58
2.57 ± 0.56
2.32 ± 0.23
Appendix
Table 7: Fail2Drive paired-route compression results using five uniformly sampled training records. The same selected route pairs are used for ID and generalization evaluation. Random and BestRand use 100 seeds, while fitted baselines use three. We report test-record MAE as mean ± standard deviation. Lower is better. Standard deviations appear in smaller type. The best and second-best means in each removal-ratio block are bolded and underlined , respectively.
Figure 10: Performance of data-only compression methods at 20% retention on six representative benchmarks. Facility Location, MIS, Agreement, and Coverage select items from prompt-derived representations without historical model responses. Random also requires no historical responses, while BestRand evaluates a few probe models only on its candidate pool. Lower MAE is better.
Average Spearman correlation ↑ mean ± std.
Method
5%
10%
15%
20%
50%
Mean
Text 44 benchmarks
SparseEval (Linear)
0.381 ± 0.238
0.364 ± 0.252
0.384 ± 0.242
0.347 ± 0.271
0.407 ± 0.265
0.377 ± 0.252
SparseEval (MLP)
0.466 ± 0.288
0.514 ± 0.263
0.586 ± 0.263
0.588 ± 0.303
0.638 ± 0.308
0.558 ± 0.290
Anchor Points
0.756 ± 0.190
0.791 ± 0.165
0.801 ± 0.165
0.800 ± 0.172
0.862 ± 0.146
0.802 ± 0.170
TinyBenchmarks
0.841 ± 0.181
0.883 ± 0.152
0.900 ± 0.164
0.932 ± 0.100
0.966 ± 0.053
0.904 ± 0.143
Appendix
Table 8: Spearman rank correlation across 70 benchmarks at 5–50% retention. Values are mean ± sample standard deviation across benchmarks, and the Mean column pools all displayed settings. Higher is better. Best means are bold and second-best means are underlined .
Average Spearman correlation ↑ mean ± std.
Method
5%
10%
15%
20%
50%
Mean
TruthfulQA
SparseEval (Linear)
-0.002 ± 0.420
0.627 ± 0.227
0.672 ± 0.174
0.697 ± 0.109
0.729 ± 0.071
0.545 ± 0.347
SparseEval (MLP)
-0.145 ± 0.776
0.255 ± 0.951
0.415 ± 0.554
0.522 ± 0.364
0.684 ± 0.303
0.346 ± 0.614
Anchor Points
0.884 ± 0.010
0.974 ± 0.010
0.905 ± 0.054
0.974 ± 0.006
0.974 ± 0.006
0.942 ± 0.046
TinyBenchmarks
0.896 ± 0.075
0.925 ± 0.042
0.959 ± 0.014
0.970 ± 0.009
0.981 ± 0.013
0.946 ± 0.047
Appendix
Table 9: Text benchmark Spearman correlations at 5–50% retention. Values are mean ± sample standard deviation; higher is better. Best means are bold and second-best means are underlined .
Average Spearman correlation ↑ mean ± std.
Method
5%
10%
15%
20%
50%
Mean
MathVerse
SparseEval (Linear)
0.199 ± 0.372
-0.318 ± 0.355
-0.306 ± 0.128
0.043 ± 0.435
0.107 ± 0.411
-0.055 ± 0.375
SparseEval (MLP)
-0.177 ± 0.440
0.450 ± 0.423
0.618 ± 0.160
0.747 ± 0.048
0.445 ± 0.065
0.417 ± 0.407
Anchor Points
0.771 ± 0.142
0.783 ± 0.071
0.802 ± 0.114
0.846 ± 0.053
0.958 ± 0.010
0.832 ± 0.104
TinyBenchmarks
0.853 ± 0.054
0.936 ± 0.026
0.946 ± 0.016
0.962 ± 0.020
0.989 ± 0.004
0.937 ± 0.053
Appendix
Table 10: Multimodal benchmark Spearman correlations at 5–50% retention. Values are mean ± sample standard deviation; higher is better. Best means are bold and second-best are underlined .
Average Spearman correlation ↑ mean ± std.
Method
5%
10%
15%
20%
50%
Mean
Drafter-Bench
SparseEval (Linear)
0.287 ± 0.353
-0.065 ± 0.383
-0.055 ± 0.471
0.011 ± 0.322
0.240 ± 0.153
0.084 ± 0.336
SparseEval (MLP)
0.667 ± 0.170
0.685 ± 0.101
0.227 ± 0.627
0.556 ± 0.080
0.683 ± 0.097
0.564 ± 0.311
Anchor Points
0.797 ± 0.067
0.868 ± 0.056
0.853 ± 0.111
0.951 ± 0.038
0.964 ± 0.024
0.887 ± 0.086
TinyBenchmarks
0.842 ± 0.029
0.897 ± 0.009
0.919 ± 0.013
0.927 ± 0.029
0.938 ± 0.006
0.905 ± 0.039
Appendix
Table 11: Agent benchmark Spearman correlations at 5–50% retention. Values are mean ± sample standard deviation; higher is better. Best means are bold and second-best means are underlined .
Average MAE (%) ↓ mean ± std.
Method
2%
5%
10%
15%
20%
50%
Mean
TruthfulQA
SparseEval (Linear)
15.04 ± 5.08
28.42 ± 3.90
18.03 ± 5.94
16.60 ± 3.22
15.08 ± 5.39
19.15 ± 5.62
18.72 ± 6.29
SparseEval (MLP)
9.21 ± 3.15
10.38 ± 1.97
9.66 ± 3.54
9.83 ± 3.86
9.72 ± 3.59
9.52 ± 2.94
9.72 ± 2.74
Anchor Points
7.15 ± 1.00
7.81 ± 0.87
4.81 ± 0.64
5.24 ± 0.08
2.70 ± 0.33
2.35 ± 0.17
5.01 ± 2.16
TinyBenchmarks
6.21 ± 1.15
4.50 ± 1.04
3.03 ± 0.24
2.25 ± 0.28
2.27 ± 0.37
1.26 ± 0.26
3.25 ± 1.79
Appendix
Table 12: Text benchmark MAE at 2–50% retention. Values are mean ± sample standard deviation in percent; lower is better. Best means are bold and second-best means are underlined .
Average MAE (%) ↓ mean ± std.
Method
2%
5%
10%
15%
20%
50%
Mean
MathVerse
SparseEval (Linear)
42.87 ± 30.75
33.96 ± 26.14
42.56 ± 27.64
44.15 ± 17.58
30.57 ± 23.97
16.27 ± 4.08
35.06 ± 22.11
SparseEval (MLP)
18.20 ± 4.98
15.64 ± 3.33
17.82 ± 1.94
13.49 ± 2.57
10.64 ± 2.59
13.95 ± 1.79
14.96 ± 3.72
Anchor Points
9.14 ± 1.59
7.81 ± 1.58
7.46 ± 1.23
7.81 ± 1.84
7.63 ± 1.61
4.41 ± 0.87
7.38 ± 1.93
TinyBenchmarks
9.54 ± 0.11
5.57 ± 0.42
3.79 ± 0.98
3.40 ± 1.05
2.54 ± 0.16
1.26 ± 0.47
4.35 ± 2.79
Appendix
Table 13: Multimodal benchmark MAE at 2–50% retention. Values are mean ± sample standard deviation in percent; lower is better. Best means are bold and second-best means are underlined .
Average MAE (%) ↓ mean ± std.
Method
2%
5%
10%
15%
20%
50%
Mean
Drafter-Bench
SparseEval (Linear)
23.58 ± 11.47
27.37 ± 13.35
41.79 ± 42.16
39.31 ± 40.39
17.49 ± 5.47
14.58 ± 4.61
27.35 ± 23.54
SparseEval (MLP)
11.92 ± 13.27
11.08 ± 12.11
12.94 ± 15.06
13.69 ± 14.96
13.94 ± 16.42
13.01 ± 15.91
12.77 ± 12.39
Anchor Points
3.06 ± 1.67
2.44 ± 0.40
2.12 ± 0.66
1.88 ± 0.30
1.48 ± 0.12
1.13 ± 0.07
2.02 ± 0.91
TinyBenchmarks
6.84 ± 2.08
3.66 ± 0.62
3.69 ± 0.55
3.79 ± 0.70
3.88 ± 0.67
3.26 ± 0.32
4.19 ± 1.50
Appendix
Table 14: Agent benchmark MAE at 2–50% retention. Values are mean ± sample standard deviation in percent; lower is better. Best means are bold and second-best means are underlined .
Average MAE (%) ↓ mean ± std.
Method
2%
5%
10%
15%
20%
50%
Mean
SparseEval (Linear)
19.73 ± 6.78
21.72 ± 5.86
24.98 ± 8.09
18.65 ± 5.32
19.43 ± 5.88
21.39 ± 7.08
20.98 ± 6.67
SparseEval (MLP)
11.94 ± 6.77
11.72 ± 5.06
11.30 ± 4.28
10.32 ± 4.44
10.81 ± 4.47
10.90 ± 5.64
11.17 ± 5.03
DatBench
14.70 ± 5.29
12.12 ± 5.22
11.11 ± 5.12
9.57 ± 3.75
8.44 ± 2.78
6.54 ± 2.02
10.41 ± 4.86
Anchor Points
7.85 ± 2.60
8.04 ± 1.51
6.99 ± 1.27
7.32 ± 2.30
7.71 ± 2.73
5.25 ± 2.15
7.19 ± 2.29
EssenceBench
10.00 ± 6.47
8.14 ± 6.33
8.06 ± 5.31
6.75 ± 4.12
6.22 ± 4.67
2.97 ± 2.30
7.02 ± 5.35
Appendix
Table 15: Compared with 15 baselines, BestRand delivers the strongest overall performance, achieving the lowest average MAE. Each entry reports the average MAE (%) and sample standard deviation across benchmarks; the final column aggregates all benchmark–budget combinations. Lower is better; the best and second-best results are shown in bold and underlined , respectively.
Figure 11: MAE of BestRand and 15 baselines across six retention ratios on 12 benchmarks. Each panel shows one benchmark, with MAE plotted on a logarithmic scale. BestRand consistently achieves the lowest or near-lowest error across benchmarks and retention ratios.
Mean Absolute Error (%) ↓ mean ± std.
Benchmark
Method
2%
5%
10%
15%
20%
50%
ARC-Challenge
Random
7.70 ± 2.89
4.73 ± 1.78
3.27 ± 1.23
2.59 ± 0.98
2.18 ± 0.82
1.09 ± 0.41
BestRand
6.25 ± 1.66
3.82 ± 1.00
2.61 ± 0.64
2.03 ± 0.49
1.67 ± 0.38
0.81 ± 0.18
Reduction
18.9%/42.5%
19.1%/43.6%
20.2%/47.8%
21.5%/49.9%
23.6%/53.5%
25.2%/56.9%
BBH
Random
3.56 ± 0.92
2.22 ± 0.57
1.53 ± 0.39
1.21 ± 0.31
1.02 ± 0.26
0.51 ± 0.13
BestRand
3.19 ± 0.68
1.99 ± 0.42
1.36 ± 0.28
1.07 ± 0.22
0.89 ± 0.18
0.44 ± 0.09
Appendix
Table 16: Stability of Random and BestRand on the 12 benchmarks. Entries report mean MAE and sample standard deviation across seeds. Reduction rows show BestRand’s relative reductions in mean MAE and standard deviation compared with Random, formatted as mean/std. Random and BestRand use 10,000 seeds. Across all settings, BestRand achieves both a lower mean MAE and a lower standard deviation than Random, demonstrating consistently better accuracy and stability.
Benchmark
Adiag↑
P+↑
Aoff↑
Benchmark
Adiag↑
P+↑
Aoff↑
ACEBench
98.8
100.0
99.2
MathVista
94.9
100.0
95.3
AGC-Bench
98.7
100.0
94.9
MedQA
99.3
100.0
92.8
ARC-Challenge
99.5
100.0
94.1
MMMU
97.4
100.0
95.3
BBEH
92.5
100.0
87.7
MMMU-Pro
98.9
100.0
92.0
BBH
99.8
96.4
85.1
MMStar
98.2
100.0
95.9
Bench2Drive
99.2
100.0
95.6
MMVP
96.9
100.0
95.1
Appendix
Table 17: Empirical results strongly support Assumption 2 . Across diverse benchmarks, model-score variances are highly similar, while pairwise model-score covariances are overwhelmingly positive and close to a common value. Adiag measures variance similarity, P+ is the proportion of positive pairwise covariances, and Aoff measures covariance similarity. All are percentages; higher values indicate closer agreement with the assumption, and 100% indicates perfect agreement.
Figure 12: Item-level numerical validation on 30 benchmarks using ten probe models. The panels show a subset of the evaluated benchmarks, while aggregate results across all benchmarks are reported in Figure 13 . Across the displayed benchmarks, R2 is consistently well above zero, all p -values are small, and every regression is statistically significant. The dashed line denotes equality.
Figure 13: Numerical experiments consistently validate Assumption 2 . Each box summarizes the distribution of explained variance R2 across benchmarks for a given number of probe models, with whiskers spanning the full observed range. The small plot in the lower-right corner reports the largest benchmark-level p -value at each probe count. All R2 values are positive and all benchmark-level results are statistically significant, providing strong empirical support for the assumption.
Figure 14: Finite sample stability of the empirical covariance and its derived summaries. Each benchmark uses the same model set across all item subsets. The quantities Cn , σn , and Varn(g) are recomputed from nested random item subsets. Each point is the median over 30 random item orderings, and the vertical bars cover the middle half of the results. The text boxes report the error reduction from 1% to 10% and 25%. Across all eight benchmarks, the errors decrease sharply after only 10% of the items and become small by 25%, showing that partial item sets provide accurate estimates of the full benchmark covariance structure and its derived summaries.
Figure 15: Empirical MAE distributions closely match the theoretical predictions of Theorem 2.1 across representative benchmarks. Their shapes and empirical modes agree with theory even when only a small number of models is available. Blue histograms show empirical distributions, orange curves show theoretical distributions, and the solid blue and dashed orange lines indicate the empirical and theoretical modes. The three panels in each row use the same horizontal scale.
Figure 16: RMS probe imbalance as the candidate count increases. Full-reference selection reports E[D∗2] for the rule in Theorem 2.2 . For finite-pool BestRand, Dpool∗ is the optimized imbalance and Dfull(S∗) is the imbalance relative to the full benchmark probe mean.
Figure 17: Common-shift scale ratio sS∗/s as the candidate count increases. Curves compare the joint Gaussian full-reference model, empirical full-reference selection, and finite-pool BestRand. The horizontal line marks the Gaussian full-reference limit 1−R2 as E[D∗2]→0 .
Benchmarks are essential for evaluating and improving large language models (LLMs). Yet many widely used benchmarks are redundant, making evaluation unnecessarily expensive. Benchmark compression lowers this cost by estimating full-benchmark scores and rankings from a partial evaluation. Research in this area has grown rapidly since 2024, but to our knowledge no survey has systematically organized its methods and findings. We review 119 papers to explain how methods choose what to evaluate, recover full-benchmark results, and verify their reliability. From this analysis, we develop a taxonomy covering compression within individual benchmarks, across benchmark suites, and during adaptive evaluation, together with work on redundancy and reliability. We compare their information needs, total costs, reliability checks, and limitations. Methods reduce evaluation costs by selecting representative subsets, using cheaper or reusable information, and testing adaptively, but risk poor coverage, inaccurate estimates, and sequential delays. The resulting taxonomy and practical guide help researchers choose, validate, and deploy methods for more efficient and reliable LLM evaluation.
Comprehensive benchmark suites are essential for improving large language models (LLMs), but many widely used benchmarks are redundant, making evaluation unnecessarily expensive. Although recent benchmark compression methods (BCMs) can mitigate this cost, many strong BCMs rely on large collections of per-sample evaluation results from numerous LLMs to identify representative samples. Building such collections is also expensive unless they are already public, making these methods difficult to extend to newly released benchmarks. To address this challenge, we present ZipBench, a simple and low-cost BCM with theoretical error and rank-consistency guarantees. ZipBench evaluates only a small set of anchor LLMs, synthesizes pseudo evaluation results to broaden coverage, learns compact sample representations, and selects a small yet representative subset. Building on it, we create ZipBench Zoo, a collection of compact versions of 100+ benchmark proxies spanning text, multimodal, and agent tasks. These benchmark achieve mean absolute errors of 0.002--0.02 and average Spearman correlations of ~0.98 with the full benchmarks. Overall, ZipBench reduces the cost of both LLM evaluation and compact benchmark construction, lowering the barrier to broad LLM research for compute-constrained researchers. The code has been released in https://github.com/MilkThink-Lab/ZipBench.
Benchmark compression reduces evaluation cost by selecting a small set of samples. Existing methods typically guide selection using a single metric, although benchmarks often assess models along multiple dimensions. We find that model rankings are often weakly correlated across metrics, indicating that different metrics capture distinct aspects of model performance. Samples selected for one metric do not reliably preserve scores and rankings on others when used for compressed evaluation. At 80% compression, each of the five single-metric baselines yields higher reconstruction error than equal-size random sampling in over 70% of cross-metric comparisons. To address this challenge, we present HoloCosm, which jointly minimizes normalized score-reconstruction error across source models and metrics while constraining ranking distortion. We evaluate HoloCosm across multiple compression rates on 15 language, multimodal, VLA, and agent benchmark configurations with 85 component metrics. With 80% compression, HoloCosm outperforms Random, the strongest baseline overall: average NMAE drops by 35.3% to 0.236, worst-component NMAE drops by 39.5% to 0.330, and average Spearman correlation rises from 0.884 to 0.915. In addition, HoloCosm remains competitive with existing baselines in single-metric evaluation. We will release our code and experimental results.
Yupei Lin, Zhongzhan Huang, Junxin Li +5
Sun Yat-sen University · Guangdong University of Technology