cs.AIOct 31, 2026

Few Trials, Many Metrics: Beta Mixtures for Low-Budget AI Evaluation

Authors: Gustavo Tironi, Felipe Maia Polo, Diego Mesquita

Organizations: Getulio Vargas Foundation · Institute of Foundation Models, MBZUAI

Abstract

Evaluating a stochastic AI system often requires many repeated samples per instance, especially for metrics defined over long horizons. Recent work has shown that modeling instance-level success probabilities with a Beta distribution recovers Pass@kk from far fewer samples than direct estimation. We show that this framework supports a broader range of evaluation targets. We fit a mixture of Beta distributions by marginal maximum likelihood, with the number of components chosen by BIC. From that one fit, we compute Passk^k and AUC@kk at any horizon in closed form. We also derive delta-method confidence intervals for these metrics and for Pass@kk, which require no additional model calls. Treating configurations rather than tasks as instances extends the same model to sensitivity analysis. For repeated success metrics, we find that mixture priors yield more accurate estimators than naive estimation across competition mathematics and graduate-level science benchmarks when extrapolating beyond the observed number of trials, and the resulting intervals cover the full-data reference. When prompts are treated as instances, it estimates the distribution of prompt sensitivity more accurately than a comparable item-response baseline without external covariates. Direct estimators remain preferable once trials reach the target horizon.

Figures & tables

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Fast Bayesian Methods for Uncertainty Quantification in Sparse AI Evaluation

    Nov 18, 2026Dario Deniz Ergun, Haohua Wang, Brendon Tao +2Language Model Performance PredictionBayesian Inference

  2. Can We Trust Item Response Theory for AI Evaluation?

    Jul 16, 2026Han Jiang, Sunbeom Kwon, Jinwen Luo +2LLM EvaluationItem Response Theory

  3. How to Run Statistics over LLM Judges and Trust the Results: Calibrated Inference for Small-Sample AI Evaluation with evalstats

    Sep 20, 2026Ian ArawjoStatistical Hypothesis TestingLLM Evaluation