Few Trials, Many Metrics: Beta Mixtures for Low-Budget AI Evaluation
Organizations: Getulio Vargas Foundation · Institute of Foundation Models, MBZUAI
Abstract
Evaluating a stochastic AI system often requires many repeated samples per instance, especially for metrics defined over long horizons. Recent work has shown that modeling instance-level success probabilities with a Beta distribution recovers Pass@ from far fewer samples than direct estimation. We show that this framework supports a broader range of evaluation targets. We fit a mixture of Beta distributions by marginal maximum likelihood, with the number of components chosen by BIC. From that one fit, we compute Pass and AUC@ at any horizon in closed form. We also derive delta-method confidence intervals for these metrics and for Pass@, which require no additional model calls. Treating configurations rather than tasks as instances extends the same model to sensitivity analysis. For repeated success metrics, we find that mixture priors yield more accurate estimators than naive estimation across competition mathematics and graduate-level science benchmarks when extrapolating beyond the observed number of trials, and the resulting intervals cover the full-data reference. When prompts are treated as instances, it estimates the distribution of prompt sensitivity more accurately than a comparable item-response baseline without external covariates. Direct estimators remain preferable once trials reach the target horizon.
Figures & tables
| Dataset | Tasks | Naive | Mixture | Win rate (%) |
|---|---|---|---|---|
| Trace | 30 | 12.90 | 4.84 | 96 |
| Math | 30–53 | 14.77 | 6.55 | 95 |
| GPQA | 3,600 | 13.65 | 1.42 | 100 |
| Dataset | Metric | Naive | Mixture | Naive | Mixture |
|---|---|---|---|---|---|
| Trace | Pass k | 1.58 | 1.85 | 7.07 | 3.51 |
| Trace | AUC@ | 0.83 | 1.46 | 3.24 | 2.21 |
| Math | Pass k | 1.93 | 2.34 | 9.46 | 4.97 |
| Math | AUC@ | 0.66 | 1.38 | 2.38 | 2.06 |
| GPQA | Pass k | 1.19 | 0.16 | 5.95 | 0.83 |
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Short name | Collections |
|---|---|---|
| AceReason-Nemotron-1.1-7B | AceReason-7B | Trace |
| Bespoke-Stratos-7B | Bespoke-7B | Trace |
| DeepSeek-R1-Distill-Qwen-1.5B | R1-Distill-1.5B | Trace |
| EXAONE-4.0-1.2B | EXAONE-1.2B | Trace |
| FuseO1-DeepSeekR1-QwQ-SkyT1-Flash-32B-Preview | FuseO1-32B | Trace |
| gpt-oss-20b_high | gpt-oss-high | Trace, Math, GPQA |
| Trace | Math | GPQA | All | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Tol. (pp) | Mix. | Naive | Fewer (%) | Mix. | Naive | Fewer (%) | Mix. | Naive | Fewer (%) | Mix. | Naive | Fewer (%) |
| 2 | 30 | 50 | 62 | 45 | 60 | 44 | 5 | 55 | 94 | 30 | 50 | 62 |
| 3 | 15 | 35 | 81 | 22.5 | 40 | 72 | 5 | 37.5 | 94 | 15 | 35 | 81 |
| 4 | 10 | 25 | 88 | 17.5 | 32.5 | 78 | 5 | 27.5 | 94 | 10 | 25 | 88 |
| 5 | 5 | 20 | 94 | 10 | 20 | 88 | 5 | 22.5 | 94 | 5 | 20 | 94 |
| 6 | 5 | 15 | 94 | 7.5 | 20 | 91 | 5 | 20 | 94 | 5 | 15 | 94 |
| Dataset | Metric | Naive | Comb. | Mixture | Naive | Mixture |
|---|---|---|---|---|---|---|
| Trace | Pass@ | 1.19 | 0.88 | 1.72 | 4.89 | 2.91 |
| Trace | Pass k | 1.58 | 1.03 | 1.85 | 7.07 | 3.51 |
| Trace | AUC@ | 0.83 | 0.65 | 1.46 | 3.24 | 2.21 |
| Math | Pass@ | 0.95 | 0.68 | 1.68 | 3.70 | 2.79 |
| Math | Pass k | 1.93 | 1.10 | 2.34 | 9.46 | 4.97 |
| Metric | Naive | Comb. | Mixture | |
|---|---|---|---|---|
| Pass@60 | 5 | 12.29 | – | 4.57 |
| Pass@60 | 10 | 8.03 | – | 3.75 |
| Pass@60 | 20 | 4.46 | – | 2.96 |
| Pass@60 | 40 | 2.29 | – | 2.12 |
| Pass@60 | 60 | 1.61 | 1.16 | 1.69 |
| Pass@60 | 80 | 1.27 | 0.00 | 1.37 |
| Metric | Naive | Comb. | Mixture | |
|---|---|---|---|---|
| Pass@60 | 5 | 8.82 | – | 3.77 |
| Pass@60 | 10 | 5.85 | – | 3.38 |
| Pass@60 | 20 | 3.25 | – | 2.96 |
| Pass@60 | 40 | 1.98 | – | 2.08 |
| Pass@60 | 60 | 1.35 | 0.92 | 1.82 |
| Pass@60 | 80 | 1.07 | 0.00 | 1.45 |
| Metric | Naive | Comb. | Mixture | |
|---|---|---|---|---|
| Pass@60 | 5 | 17.93 | – | 1.59 |
| Pass@60 | 10 | 11.67 | – | 1.24 |
| Pass@60 | 20 | 6.64 | – | 0.61 |
| Pass@60 | 40 | 3.45 | – | 0.38 |
| Pass@60 | 60 | 2.32 | 0.09 | 0.20 |
| Pass@60 | 80 | 1.74 | 0.00 | 0.13 |
| Dataset | Metric | Estimator | Seed 0 | Seed 1 | Seed 2 | Mean | SD |
|---|---|---|---|---|---|---|---|
| Trace | Pass@ | Naive | 4.96 | 5.09 | 4.63 | 4.89 | 0.24 |
| Trace | Pass@ | Mixture | 2.93 | 2.77 | 3.01 | 2.91 | 0.12 |
| Trace | Pass k | Naive | 6.79 | 6.99 | 7.44 | 7.07 | 0.34 |
| Trace | Pass k | Mixture | 3.34 | 3.55 | 3.64 | 3.51 | 0.15 |
| Trace | AUC@ | Naive | 3.31 | 3.40 | 3.03 | 3.24 | 0.19 |
| Trace | AUC@ | Mixture | 2.24 | 2.11 | 2.27 | 2.21 | 0.08 |
Generative AI Evaluation
Generative AI evaluation measures the quality, reliability, diversity, and real-world usefulness of generated outputs and AI-assisted workflows.
See all 335 papers