Efficient Evaluation of LLM Performance with Statistical Guarantees

Skyler Wu; Yash Nair; Emmanuel J. Candès

stat.MLJan 28, 2026

Efficient Evaluation of LLM Performance with Statistical Guarantees

Authors: Skyler Wu, Yash Nair, Emmanuel J. Candès

Organizations: Department of Statistics, Stanford University

Abstract

Exhaustively evaluating many large language models (LLMs) on a large suite of benchmarks is expensive. We cast benchmarking as finite-population inference and, under a fixed query budget, seek tight confidence intervals (CIs) for model accuracy with asymptotic frequentist coverage. We propose Factorized Active Querying (FAQ), which (a) leverages historical information through a Bayesian factor model; (b) adaptively selects questions using a hybrid variance-reduction/active-learning sampling policy; and (c) maintains validity through Pro-Active Inference---a finite-population extension of active inference (Zrnic & Candès, 2024) that enables direct question selection while preserving asymptotic frequentist coverage. In our experiments, FAQ delivers up to $5\times$ effective sample size gains over strong baselines on two benchmark suites, across varying historical-data missingness levels, with negligible computational overhead: this means that it matches the CI width of uniform sampling while using up to $5\times$ fewer queries. We release our source code and our curated datasets to support reproducibility and future research.

Figures & tables

Figure 1: High-level overview of Factorized Active Querying (FAQ). (a) Using partially-observed historical outcomes

H

, we fit historical model and question factors (Eq. 6 ). For a new model, we initialize its latent factor from the historical fit. Then, at each sampling round

t

, we (b) compute the hybrid sampling policy

q_{t}(\cdot)

from the current factor posterior (Eq. 4 ), (c) sample a question

I_{t}\sim q_{t}(\cdot)

, and (d) update the model factor using the observed outcome

z_{I_{t}}

(Eqs. 7 - 8 ). Finally, we (e) output the PAI estimate and confidence interval (Eqs. 2 , 9 ).

q_{t}(j)=\frac{\tau}{N_{q}}+(1-\tau)\mathrm{Norm}(h^{(t)}_{\mathrm{cat}})(j),

Algorithm 1 Factorized Active Querying

Figure 2: Evaluating

2.2

K LLMs on MMLU-Pro and BBH+GPQA+IFEval+MATH+MuSR with fully-observed historical data. (Top Row) Effective sample size (ESS) vs. budget for FAQ, the strongest baseline per budget (selected post-hoc, favoring baselines), and uniform sampling. Numbers annotate the ESS multiplier relative to uniform sampling. For example, on MMLU-Pro at budget

1500

, an ESS multiplier of

5.01

means FAQ matches uniform sampling’s CI width at budget

7515

5.01\times

gain). (Bottom Row) Empirical coverage of the corresponding

95\%

normal-approximation CIs, averaged over

2.2

K test models and

100

seeds. Standard errors are shown, but negligible on the plot scale. Higher ESS and ESS-multiplier values indicate better performance.

Figure 3: ESS under missing historical data on BBH+GPQA+IFEval+MATH+MuSR. ESS vs. budget across missingness settings (Section 4 ). Standard errors are shown, but negligible on the plot scale. MMLU-Pro and coverage results are in Appendix E .

Figure 4: Per-model coverages on BBH+GPQA+IFEval+MATH+MuSR with fully-observed historical data at budget

7.5\%

. Per-model coverages (averaged over

100

seeds) vs. (Left) model release date and (Right) true accuracy. Gray dots: individual models; solid-blue curve: locally-weighted mean with a

K=501

nearest-neighbor Gaussian kernel; light-blue band: local

\pm 1

SD.

Figure 5: Cold-start FAQ on GPQA (All). ESS multipliers relative to uniform sampling across budgets for cold-start FAQ with zero target-suite historical data for factor-model fitting vs. strongest post-hoc baseline per budget with access to

0.1\%

target-suite historical data. Standard errors are shown, but negligible on the plot scale. Higher ESS-multiplier values indicate better performance.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Coverage and RMSE of FAQ with/without replacement on MMLU-Pro with fully-observed historical data. (Left) coverage and (Right) RMSE of

\hat{\theta}_{n_{b}}

as a function of budget

n_{b}

, for FAQ sampling with/without replacement. Standard errors are shown, but negligible on the plot scale. Targeted nominal coverage for all experiments shown was

95\%

Figure 7: CI-width ratio (post-hoc best traditional active inference ablation / FAQ) across budgets/missingness Missingness uses

n_{\text{full-obs}}=0

and MCAR

p_{\text{obs}}

(legend values) on the remaining entries. Standard errors are shown, but negligible at the plot scale. Width ratios

>1

favor FAQ (narrower CIs).

Figure 8: ESS under missing historical data on MMLU-Pro and BBH+GPQA+IFEval +MATH+MuSR. ESS vs. budget for FAQ, the strongest baseline per budget (selected post-hoc), and uniform sampling, per missingness setting. Missingness is parameterized by

n_{\text{full-obs}}

fully-observed historical rows (out of

2.2

K) and MCAR entrywise observation probability

p_{\text{obs}}

on the remaining rows (e.g.,

n_{\text{full-obs}}=50

and

p_{\text{obs}}=0.1

\sim

12\%

observed). Standard errors are shown, but negligible on the plot scale. This figure shows the full results corresponding to Figure 3 in the main text.

Figure 9: Coverage under missing historical data on MMLU-Pro and BBH+GPQA+IFEval +MATH+MuSR. Coverage vs. budget for FAQ, the strongest baseline per budget (selected post-hoc), and uniform sampling, per missingness setting. Missingness is parameterized by

n_{\text{full-obs}}

fully-observed historical rows (out of

2.2

K) and MCAR entrywise observation probability

p_{\text{obs}}

on the remaining rows (e.g.,

n_{\text{full-obs}}=50

and

p_{\text{obs}}=0.1

\sim

12\%

observed). Standard errors are shown, but negligible on the plot scale. This figure confirms that the results in Figure 3 in the main text were achieved with valid coverage.

Figure 10: Confidence interval (CI) widths under missing historical data on MMLU-Pro and BBH+GPQA+IFEval+MATH+MuSR. CI width vs. budget for FAQ, the strongest baseline per budget (selected post-hoc), and uniform sampling, per missingness setting. Missingness is parameterized by

n_{\text{full-obs}}

fully-observed historical rows (out of

2.2

K) and MCAR entrywise observation probability

p_{\text{obs}}

on the remaining rows (e.g.,

n_{\text{full-obs}}=50

and

p_{\text{obs}}=0.1

\sim

12\%

observed). Standard errors are shown, but negligible on the plot scale. Smaller CI widths indicate stronger performance.

Figure 11: Per-model coverages on MMLU-Pro with fully-observed historical data at budget

7.5\%

. Per-model coverages (averaged over

100

seeds) vs. (Left) model release date and (Right) true model accuracy. Gray dots: individual models; solid-blue curve: locally-weighted mean with a

K=501

nearest-neighbor Gaussian kernel; light-blue band: local

\pm 1

SD band. This figure shows the MMLU-Pro counterpart to Figure 4 in the main text.

Figure 12: Coverage for cold-start FAQ on GPQA (All). Coverage vs. budget for cold-start FAQ with zero target-suite historical data for factor-model fitting , the strongest post-hoc baseline per budget with access to

0.1\%

target-suite historical data , and uniform sampling. Standard errors are shown, but negligible on the plot scale. This figure confirms that main text Figure 5 ’s results were achieved with valid coverage.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,432 papers

Audit Me If You Can: Query-Efficient Active Fairness Auditing of Black-Box LLMs
Jan 6, 2026David Hartmann, Lena Pohlmann, Lelia Hanslik +3LLM Auditing Algorithmic Fairness
Weizenbaum Institut Berlin · Technische Universität Berlin · FIZ Karlsruhe +2
Efficient Sequential Evaluation of Large Language Models
Jul 19, 2026Chia-Yu Hsu, Shubhanshu ShekharLLM Evaluation Confidence Estimation in Language Models
EECS Department, University of Michigan
Efficient Benchmarking Is Just Feature Selection and Multiple Regression
May 25, 2026Sam Bowyer, Acyr Locatelli, Kris CaoLLM Evaluation Feature Selection
Cohere, University of Bristol · Cohere

Jan 6, 2026J/K move · Enter open · S save

Audit Me If You Can: Query-Efficient Active Fairness Auditing of Black-Box LLMs

David Hartmann, Lena Pohlmann, Lelia Hanslik, Noah Gießing+2

Weizenbaum Institut Berlin · Technische Universität Berlin · FIZ Karlsruhe+2

LLM Auditing Algorithmic Fairness Adaptive Sampling Demographic Bias in Language Models Black-Box Model Auditing

Open paper AskPDF arXiv

Large Language Models (LLMs) exhibit systematic biases across demographic groups. Auditing is proposed as an accountability tool for black-box LLM applications, but suffers from resource-intensive query access. We conceptualise auditing as uncertainty estimation over a target fairness metric and introduce BAFA, the Bounded Active Fairness Auditor for query-efficient auditing of black-box LLMs. BAFA maintains a version space of surrogate models consistent with queried scores and computes uncertainty intervals for fairness metrics (e.g.,

Δ

AUC) via constrained empirical risk minimisation. Active query selection narrows these intervals to reduce estimation error. We evaluate BAFA on two standard fairness dataset case studies: CivilComments and Bias-in-Bios, comparing against stratified sampling, power sampling, and ablations. BAFA achieves target error thresholds with up to 40

\times

fewer queries than stratified sampling (e.g., 144 vs 5,956 queries at

\varepsilon=0.02

for CivilComments) for tight thresholds, demonstrates substantially better performance over time, and shows lower variance across runs. These results suggest that active sampling can reduce resources needed for independent fairness auditing with LLMs, supporting continuous model evaluations.

Abstract

LLM Evaluation

Explore similar work

Audit Me If You Can: Query-Efficient Active Fairness Auditing of Black-Box LLMs

Efficient Sequential Evaluation of Large Language Models

Efficient Benchmarking Is Just Feature Selection and Multiple Regression