cs.AINov 18, 2026

Fast Bayesian Methods for Uncertainty Quantification in Sparse AI Evaluation

Authors: Dario Deniz Ergun, Haohua Wang, Brendon Tao, Jonah Kollenberg, Chen Ye Gan

Organizations: Institute of Data and Information, Tsinghua University · Delft Institute of Applied Mathematics, Delft University of Technology

Abstract

Estimating how an AI system will perform on a new benchmark need not require another large model. We use public evaluation counts to choose a weak Beta prior, then update it exactly from at most 31 revealed outcomes. The predictor uses no item semantics, pretrained weights, or inference API. Its single online evaluation in the AIMS competition returned a team-reported Brier ALC of 0.188477, a 24.6% reduction from the constant-0.5 reference. The complete submission is 7,754 bytes; a server microbenchmark measures 12.71 microseconds per prediction with 31 matching labels. We describe how source evaluations are compressed into 301 prior means and show why a concentration of two lets new evidence rapidly determine the prediction. The resulting posterior gives exact uncertainty about the modelled success rate without numerical integration or neural inference. A public evaluation covers 79 system-benchmark pairs and eight seeds, reaching Brier ALC 0.200250 for the submitted method. Accompanying code and sufficient statistics reconstruct the fitted prior byte for byte and replay all 632 evaluation trajectories. The result demonstrates how much predictive value a compact Bayesian model can extract from sparse target feedback at very low online cost.

Figures & tables

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Few Trials, Many Metrics: Beta Mixtures for Low-Budget AI Evaluation

    Oct 31, 2026Gustavo Tironi, Felipe Maia Polo, Diego MesquitaGenerative AI EvaluationUncertainty Quantification

  2. Unstable Rankings in Bayesian Deep Learning Evaluation

    Apr 25, 2026Qishi Zhan, Minxuan Hu, Guansu Wang +2Bayesian Neural NetworksStatistical Power Analysis

  3. You Don't Need to Run Every Eval

    Jun 22, 2026Yuchen Zeng, Dimitris PapailiopoulosLLM EvaluationRobust Matrix Completion