Random Is Almost Enough? Stable and Low-Cost Benchmark Compression via BestRand
Organizations: Bosch Research · Carnegie Mellon University
Abstract
Benchmark compression reduces the often prohibitive time and cost of model evaluation by evaluating models on a small benchmark subset that closely matches full-benchmark results. To quantify estimation error, we derive an accurate analytical approximation to the error distribution when estimating full-benchmark scores from a subset, showing that most subsets yield small errors, while a few produce much larger errors, resulting in a long-tailed distribution. This means random sampling alone can achieve surprisingly strong performance with high probability, but remains unstable due to rare large errors. We further show that this instability is inherent to modern foundation models and persists even as more models are evaluated. Based on these findings, we introduce BestRand, a stable and low-cost benchmark compression method that preserves the efficiency of random sampling while substantially reducing its high-error tail. We validate BestRand on 70+ text, multimodal, and agent benchmarks, and further extend it to VLA evaluation. Compared with competitive methods, BestRand reduces high-error risk by 50.2% and estimation error by 16.4% on average, while requiring only 51.3% of the construction cost. It further reduces OOD estimation error by 21.8% on average across score-range, model-family, and temporal shifts. Together, our theory provides practical guidance for understanding and predicting the accuracy and reliability of benchmark compression, while BestRand enables stable and accurate compression at low cost. The code and data are publicly available onhttps://github.com/George-Ling3/BestRand.
Figures & tables
| Average MAE (%) mean std. | |||||||
| Method | 2% | 5% | 10% | 15% | 20% | 50% | Mean |
| Text 44 benchmarks | |||||||
| SparseEval (Linear) | 19.88 9.48 | 21.08 6.99 | 22.25 7.72 | 21.18 10.10 | 21.44 9.60 | 20.04 7.95 | 20.98 8.67 |
| SparseEval (MLP) | 12.53 5.85 | 11.94 5.16 | 11.70 5.38 | 11.49 5.32 | 11.74 5.51 | 11.80 5.80 | 11.87 5.46 |
| Anchor Points | 7.68 3.24 | 7.23 2.66 | 6.78 2.73 | 6.32 2.66 | 6.41 2.73 | 4.72 2.58 | 6.52 2.90 |
| TinyBenchmarks | 7.45 4.49 | 4.94 3.34 | 3.26 2.22 | 2.56 1.82 | 2.26 1.70 | 1.45 1.55 | 3.65 3.37 |
| Score Range OOD | Family OOD | Temporal OOD | ||||
|---|---|---|---|---|---|---|
| Method | Mid Tails | Low High | High Low | Held-out Families | Past Future | Mean |
| SparseEval (Linear) | 18.50 3.24 | 21.50 5.74 | 18.31 4.58 | 22.45 3.63 | 24.32 6.65 | 22.07 2.98 |
| SparseEval (MLP) | 16.13 7.04 | 16.43 7.57 | 21.59 9.64 | 11.87 5.89 | 11.71 6.05 | 13.88 6.12 |
| Anchor Points | 4.94 1.62 | 5.63 3.03 | 6.96 4.52 | 7.26 1.93 | 8.23 2.64 | 7.11 1.57 |
| TinyBenchmarks | 2.34 1.17 | 2.46 1.35 | 3.03 1.93 | 2.20 0.93 | 2.27 1.01 | 2.36 0.89 |
| ZipBench | 2.03 0.94 | 2.43 1.62 | 2.30 1.06 | 2.16 1.28 | 2.15 1.28 | 2.19 1.21 |
Appendix figures & tables27 assets
Supplementary material from the paper’s appendix.
Appendix
| Type | Benchmark | Public Link | # Items | # Test Models | Metric (Single?) |
|---|---|---|---|---|---|
| Multimodal Benchmarks | CountBenchQA ( Beyer et al., 2024 ) | HF/CountBenchQA | 491 | 11 | Accuracy ( ✓ ) |
| DynaMath ( Zou et al., 2025 ) | HF/DynaMath_Sample | 5,010 | 20 | Accuracy ( ✗ ) | |
| LogicVista ( Xiao et al., 2024 ) | GH/LogicVista | 448 | 20 | Accuracy ( ✓ ) | |
| MathVerse ( Zhang et al., 2024 ) | HF/MathVerse | 788 | 20 | Accuracy ( ✓ ) | |
| MathVista ( Lu et al., 2024 ) | HF/MathVista | 1,000 | 10 | Accuracy ( ✓ ) | |
| MME ( Fu et al., 2025a ) | GH/MME | 2,374 | 16 | MME Score ( ✓ ) |
| Type | Benchmark | Public Link | # Items | # Test Models | Metric (Single?) |
|---|---|---|---|---|---|
| Text Benchmarks | ACPBench Hard ( Kokel et al., 2026 ) | HF/acp_bench | 1,040 | 20 | Accuracy ( ✓ ) |
| AGC-Bench ( Beaty et al., 2026 ) | HF/agcbench-2026 | 100 | 20 | Accuracy ( ✗ ) | |
| AIME ( MAA, 2026 ) | MAA/AIME | 60 | 20 | Accuracy ( ✓ ) | |
| ARC-Challenge ( Clark et al., 2018 ) | HF/ai2_arc | 1,172 | 20 | Accuracy ( ✓ ) | |
| ARC-Easy ( Clark et al., 2018 ) | HF/ai2_arc | 2,376 | 20 | Accuracy ( ✓ ) | |
| BBEH ( Kazemi et al., 2025 ) | GH/bbeh | 4,520 | 20 | Accuracy ( ✓ ) |
| Construction Information | ||||||
| Type | Method | Text | Embed. | Scores | All items | Full score |
| Response-free | Random | — | — | — | — | — |
| Facility Location ( Yao et al., 2026 ) | — | ✓ | — | — | — | |
| MIS ( Kjorvezir et al., 2026 ) | — | ✓ | — | — | — | |
| Agreement ( Purificato et al., 2026 ) | — | ✓ | — | — | — | |
| Coverage ( Purificato et al., 2026 ) | — | ✓ | — | — | — | |
| Prediction error mean std. | |||||
| Method | SR | DS | Eff. MRE | Smooth. | Avg. Error |
| Bench2Drive, removal ratio | |||||
| SparseEval (Linear) | 16.14 0.24 | 1.22 0.16 | 2.73 1.73 | 1.38 0.31 | 5.37 0.49 |
| SparseEval (MLP) | 3.50 0.15 | 1.78 0.19 | 2.31 2.13 | 1.85 0.73 | 2.36 0.37 |
| Anchor Points | 5.21 1.48 | 4.55 1.62 | 2.40 1.71 | 3.44 2.15 | 3.90 1.38 |
| TinyBenchmarks | 2.15 0.12 | 2.38 0.50 | 3.80 3.18 | 2.02 0.62 | 2.58 0.83 |
| Prediction error mean std. | |||||||
| Method | ID SR | ID DS | ID HM | Gen SR | Gen DS | Gen HM | Avg. Error |
| Fail2Drive, removal ratio | |||||||
| SparseEval (Linear) | 17.40 8.19 | 4.63 0.62 | 10.44 7.40 | 18.48 2.77 | 5.56 0.78 | 13.17 4.26 | 11.62 3.38 |
| SparseEval (MLP) | 4.56 0.67 | 3.53 0.95 | 3.07 0.66 | 6.71 2.20 | 3.86 0.55 | 3.17 0.81 | 4.15 0.36 |
| Anchor Points | 2.79 0.09 | 2.51 0.63 | 2.09 0.43 | 5.33 1.30 | 3.99 0.42 | 3.88 1.18 | 3.43 0.39 |
| TinyBenchmarks | 2.52 0.23 | 1.81 0.49 | 1.89 0.28 | 3.49 0.93 | 1.63 0.58 | 2.57 0.56 | 2.32 0.23 |
| Average Spearman correlation mean std. | ||||||
| Method | 5% | 10% | 15% | 20% | 50% | Mean |
| Text 44 benchmarks | ||||||
| SparseEval (Linear) | 0.381 0.238 | 0.364 0.252 | 0.384 0.242 | 0.347 0.271 | 0.407 0.265 | 0.377 0.252 |
| SparseEval (MLP) | 0.466 0.288 | 0.514 0.263 | 0.586 0.263 | 0.588 0.303 | 0.638 0.308 | 0.558 0.290 |
| Anchor Points | 0.756 0.190 | 0.791 0.165 | 0.801 0.165 | 0.800 0.172 | 0.862 0.146 | 0.802 0.170 |
| TinyBenchmarks | 0.841 0.181 | 0.883 0.152 | 0.900 0.164 | 0.932 0.100 | 0.966 0.053 | 0.904 0.143 |
| Average Spearman correlation mean std. | ||||||
| Method | 5% | 10% | 15% | 20% | 50% | Mean |
| TruthfulQA | ||||||
| SparseEval (Linear) | -0.002 0.420 | 0.627 0.227 | 0.672 0.174 | 0.697 0.109 | 0.729 0.071 | 0.545 0.347 |
| SparseEval (MLP) | -0.145 0.776 | 0.255 0.951 | 0.415 0.554 | 0.522 0.364 | 0.684 0.303 | 0.346 0.614 |
| Anchor Points | 0.884 0.010 | 0.974 0.010 | 0.905 0.054 | 0.974 0.006 | 0.974 0.006 | 0.942 0.046 |
| TinyBenchmarks | 0.896 0.075 | 0.925 0.042 | 0.959 0.014 | 0.970 0.009 | 0.981 0.013 | 0.946 0.047 |
| Average Spearman correlation mean std. | ||||||
| Method | 5% | 10% | 15% | 20% | 50% | Mean |
| MathVerse | ||||||
| SparseEval (Linear) | 0.199 0.372 | -0.318 0.355 | -0.306 0.128 | 0.043 0.435 | 0.107 0.411 | -0.055 0.375 |
| SparseEval (MLP) | -0.177 0.440 | 0.450 0.423 | 0.618 0.160 | 0.747 0.048 | 0.445 0.065 | 0.417 0.407 |
| Anchor Points | 0.771 0.142 | 0.783 0.071 | 0.802 0.114 | 0.846 0.053 | 0.958 0.010 | 0.832 0.104 |
| TinyBenchmarks | 0.853 0.054 | 0.936 0.026 | 0.946 0.016 | 0.962 0.020 | 0.989 0.004 | 0.937 0.053 |
| Average Spearman correlation mean std. | ||||||
| Method | 5% | 10% | 15% | 20% | 50% | Mean |
| Drafter-Bench | ||||||
| SparseEval (Linear) | 0.287 0.353 | -0.065 0.383 | -0.055 0.471 | 0.011 0.322 | 0.240 0.153 | 0.084 0.336 |
| SparseEval (MLP) | 0.667 0.170 | 0.685 0.101 | 0.227 0.627 | 0.556 0.080 | 0.683 0.097 | 0.564 0.311 |
| Anchor Points | 0.797 0.067 | 0.868 0.056 | 0.853 0.111 | 0.951 0.038 | 0.964 0.024 | 0.887 0.086 |
| TinyBenchmarks | 0.842 0.029 | 0.897 0.009 | 0.919 0.013 | 0.927 0.029 | 0.938 0.006 | 0.905 0.039 |
| Average MAE (%) mean std. | |||||||
| Method | 2% | 5% | 10% | 15% | 20% | 50% | Mean |
| TruthfulQA | |||||||
| SparseEval (Linear) | 15.04 5.08 | 28.42 3.90 | 18.03 5.94 | 16.60 3.22 | 15.08 5.39 | 19.15 5.62 | 18.72 6.29 |
| SparseEval (MLP) | 9.21 3.15 | 10.38 1.97 | 9.66 3.54 | 9.83 3.86 | 9.72 3.59 | 9.52 2.94 | 9.72 2.74 |
| Anchor Points | 7.15 1.00 | 7.81 0.87 | 4.81 0.64 | 5.24 0.08 | 2.70 0.33 | 2.35 0.17 | 5.01 2.16 |
| TinyBenchmarks | 6.21 1.15 | 4.50 1.04 | 3.03 0.24 | 2.25 0.28 | 2.27 0.37 | 1.26 0.26 | 3.25 1.79 |
| Average MAE (%) mean std. | |||||||
| Method | 2% | 5% | 10% | 15% | 20% | 50% | Mean |
| MathVerse | |||||||
| SparseEval (Linear) | 42.87 30.75 | 33.96 26.14 | 42.56 27.64 | 44.15 17.58 | 30.57 23.97 | 16.27 4.08 | 35.06 22.11 |
| SparseEval (MLP) | 18.20 4.98 | 15.64 3.33 | 17.82 1.94 | 13.49 2.57 | 10.64 2.59 | 13.95 1.79 | 14.96 3.72 |
| Anchor Points | 9.14 1.59 | 7.81 1.58 | 7.46 1.23 | 7.81 1.84 | 7.63 1.61 | 4.41 0.87 | 7.38 1.93 |
| TinyBenchmarks | 9.54 0.11 | 5.57 0.42 | 3.79 0.98 | 3.40 1.05 | 2.54 0.16 | 1.26 0.47 | 4.35 2.79 |
| Average MAE (%) mean std. | |||||||
| Method | 2% | 5% | 10% | 15% | 20% | 50% | Mean |
| Drafter-Bench | |||||||
| SparseEval (Linear) | 23.58 11.47 | 27.37 13.35 | 41.79 42.16 | 39.31 40.39 | 17.49 5.47 | 14.58 4.61 | 27.35 23.54 |
| SparseEval (MLP) | 11.92 13.27 | 11.08 12.11 | 12.94 15.06 | 13.69 14.96 | 13.94 16.42 | 13.01 15.91 | 12.77 12.39 |
| Anchor Points | 3.06 1.67 | 2.44 0.40 | 2.12 0.66 | 1.88 0.30 | 1.48 0.12 | 1.13 0.07 | 2.02 0.91 |
| TinyBenchmarks | 6.84 2.08 | 3.66 0.62 | 3.69 0.55 | 3.79 0.70 | 3.88 0.67 | 3.26 0.32 | 4.19 1.50 |
| Average MAE (%) mean std. | |||||||
|---|---|---|---|---|---|---|---|
| Method | 2% | 5% | 10% | 15% | 20% | 50% | Mean |
| SparseEval (Linear) | 19.73 6.78 | 21.72 5.86 | 24.98 8.09 | 18.65 5.32 | 19.43 5.88 | 21.39 7.08 | 20.98 6.67 |
| SparseEval (MLP) | 11.94 6.77 | 11.72 5.06 | 11.30 4.28 | 10.32 4.44 | 10.81 4.47 | 10.90 5.64 | 11.17 5.03 |
| DatBench | 14.70 5.29 | 12.12 5.22 | 11.11 5.12 | 9.57 3.75 | 8.44 2.78 | 6.54 2.02 | 10.41 4.86 |
| Anchor Points | 7.85 2.60 | 8.04 1.51 | 6.99 1.27 | 7.32 2.30 | 7.71 2.73 | 5.25 2.15 | 7.19 2.29 |
| EssenceBench | 10.00 6.47 | 8.14 6.33 | 8.06 5.31 | 6.75 4.12 | 6.22 4.67 | 2.97 2.30 | 7.02 5.35 |
| Mean Absolute Error (%) mean std. | |||||||
|---|---|---|---|---|---|---|---|
| Benchmark | Method | 2% | 5% | 10% | 15% | 20% | 50% |
| ARC-Challenge | Random | 7.70 2.89 | 4.73 1.78 | 3.27 1.23 | 2.59 0.98 | 2.18 0.82 | 1.09 0.41 |
| BestRand | 6.25 1.66 | 3.82 1.00 | 2.61 0.64 | 2.03 0.49 | 1.67 0.38 | 0.81 0.18 | |
| Reduction | 18.9%/42.5% | 19.1%/43.6% | 20.2%/47.8% | 21.5%/49.9% | 23.6%/53.5% | 25.2%/56.9% | |
| BBH | Random | 3.56 0.92 | 2.22 0.57 | 1.53 0.39 | 1.21 0.31 | 1.02 0.26 | 0.51 0.13 |
| BestRand | 3.19 0.68 | 1.99 0.42 | 1.36 0.28 | 1.07 0.22 | 0.89 0.18 | 0.44 0.09 | |
| Benchmark | Benchmark | ||||||
| ACEBench | 98.8 | 100.0 | 99.2 | MathVista | 94.9 | 100.0 | 95.3 |
| AGC-Bench | 98.7 | 100.0 | 94.9 | MedQA | 99.3 | 100.0 | 92.8 |
| ARC-Challenge | 99.5 | 100.0 | 94.1 | MMMU | 97.4 | 100.0 | 95.3 |
| BBEH | 92.5 | 100.0 | 87.7 | MMMU-Pro | 98.9 | 100.0 | 92.0 |
| BBH | 99.8 | 96.4 | 85.1 | MMStar | 98.2 | 100.0 | 95.9 |
| Bench2Drive | 99.2 | 100.0 | 95.6 | MMVP | 96.9 | 100.0 | 95.1 |
LLM Evaluation
Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.
See all 2,539 papers