Measuring all the noises of LLM Evals

Sida I. Wang

cs.LGDec 24, 2025

Measuring all the noises of LLM Evals

Authors: Sida I. Wang

Organizations: FAIR at Meta

Abstract

Applying well-established statistical methods effectively to LLM evals requires consideration of model correlations and prediction noises. Leading evals often ignore data noise, have low statistical power, or make unnecessary trade-offs. We define and measure three types of noise: 1) prediction noise from generating different answers on a given question, 2) data noise from sampling questions, and 3) their combined total noise following the law of total variance. To emphasize relative comparisons and gain statistical power, we propose all-pairs comparison, which applies paired comparison to all pairs of LLMs and measures all the noise components based on millions of question-level predictions across many evals and settings, revealing clear trends. First, most evals exhibit a characteristic and highly predictable total noise level across all model pairs. Second, paired prediction noise typically exceeds paired data noise, which means reducing prediction noise by averaging can significantly increase statistical power. By measuring all the noises together, we can assess eval results in context and make better empirical decisions.

Figures & tables

Panel of Figure 1 — Figure 1: paired total standard errors SE(A-B) vs. the accuracy E(A) . Each dot is a paired comparison of model A vs. B, agreeing with the beta theory (trend line). Left: on SWEBench estimated from 1 sample per question, Right: on MATH500 from multiple samples per question and thus also showing the irreducible data noise SE_x(A-B) .

Test Type	$\operatorname{SE}(\bar{A}-\bar{B})$	Difference for $p<0.05$
Unpaired	6%	11%
Unpaired + averaging	4%	8%
Paired	4%	8%
Paired + averaging	1–2%	2-4%

Table 2

Figure 2: Increasingly sensitive training curves on SWEBench-Verified with $N=500$ , A and B both start from the same checkpoint at step 0. Column 1) Original training curve. One prediction accuracy, showing B is better than A over all steps. 2) Unpaired bootstrap. A is better than B on many resampled set of 500 questions, showing a meaningless difference using this analysis method. 3) Paired bootstrap. Slightly significant with a $z$ -score of 1.7 at 1 prediction per question. 4) Paired, averaged bootstrap. The result is very significant with a $z$ -score of 3.5 when the question level predictions are averaged over 5 consecutive training checkpoints to reduce the prediction noise.

Panel of Figure 3 — Figure 3: Results heatmap. Each row is a different question, sorted from the easiest to the hardest. Each column is a model whose $x$ -coordinate is the overall accuracy. Left: SWEbench-verified, 1 prediction per question; Right: MATH500, 1000 predictions per question. See § 5.1 for more details.

Panel of Figure 4 — Figure 4: All noise components on CRUXEval at temperature 0.2 (left) and 0.8 (right). The prediction noise dominates at 0.8 temperature, whereas the data noise dominates at 0.2 temperature, while both still have similar total noise.

Panel of Figure 5 — Figure 5: SE(A-B) vs. the difference E(A-B) . constant p-values are straight-lines on this figure and every model pair to the right of the line are more significant.

Panel of Figure 6 — Figure 6: SE(A-B) vs. the difference E(A-B) . Each point is a pair of random training seeds. Under the null hypothesis, the proportion of points on the right side of each $p$ -value line is expected to be $p$ . The eval nq produced more significant results than the null hypothesis whereas hellaswag produced even less than the null. constant p-values are straight-lines on this figure and every model pair to the right of the line are more significant.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Relative errors of the paired variances. $K=5$ samples are drawn from the 500 questions from MATH500, where 1000 real predictions are drawn from models $A$ and $B$ on each question, which is treated as the ground truth. The root mean squared relative errors (rms) of variance components are plotted vs. bootstrap sample size $N$ , extending beyond the population size 500. The left figure without the correction on $K$ has an unacceptably high 70% relative error even with 2000 data points. The middle figure is with the correction of § A.2 and the right figure uses $1/K$ instead of $1/(K-1)$ .

Name	Formula	Code	Bernoulli
total variance	$\mathrm{Var}_{x,\epsilon}[A]$	var(A)	$\bar{p}(1-\bar{p})$
data variance	$\mathrm{Var}_{x}[\mathrm{E}_{\epsilon}[A]]$	var(mean(A, axis=1))-b	$\frac{1}{N}\sum_{i}(p_{i}-\bar{p})^{2}$
prediction variance	$\mathrm{E}_{x}[\mathrm{Var}_{\epsilon}[A]]$	mean(var(A, axis=1))+b	$\frac{1}{N}\sum_{i}p_{i}(1-p_{i})$

Appendix

Table 1: Unpaired estimators where

A\in\mathbb{R}^{N\times K}

with

K

samples for each of

N

questions.

Panel of Figure 8 — Figure 8: Empirical cumulative distribution curves (CDF, solid) and their respective Beta models (dots). Each color represents a different bin by model accuracy. The average accuracy of each question in each bin of models is computed and sorted to produce the CDF curves. The fit is good.

Formula	Win rates
$\mathrm{Var}_{x,\epsilon}[A-B]$	$q_{A}+q_{B}-(q_{A}-q_{B})^{2}$
$\mathrm{Var}_{x}[\mathrm{E}_{\epsilon}[A-B]]$	$\frac{1}{N}\sum_{i}(q_{A,i}-q_{B,i}-(q_{A}-q_{B}))^{2}$
$\mathrm{E}_{x}[\mathrm{Var}_{\epsilon}[A-B]]$	$\frac{1}{N}\sum_{i}q_{A,i}+q_{B,i}-(q_{A,i}-q_{B,i})^{2}$

Appendix

Table 3: where

A,B\in\{0,1\}

with

\Pr[A(x_{i})>B(x_{i})]=q_{A,i}

\Pr[A(x_{i})<B(x_{i})]=q_{B,i}

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

CELEUS: Certifiable and Efficient LLM Evaluation via E-Processes
Jun 18, 2026Zhijian Zhou, Zesheng Ye, Zhaorun Chen +2LLM Evaluation Anytime-Valid Inference
The University of Melbourne · Department of Computer Science, University of Chicago · Department of Computer Science, University of Illinois Urbana-Champaign
How to Run Statistics over LLM Judges and Trust the Results: Calibrated Inference for Small-Sample AI Evaluation with evalstats
Sep 20, 2026Ian ArawjoStatistical Hypothesis Testing LLM Evaluation
Department of Computer Science and Operations Research (DIRO) Université de Montréal Montréal, Quebec, Canada
Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking
Apr 13, 2026Solomon MessingLLM Evaluation
ML Commons

Jun 18, 2026J/K move · Enter open · S save

CELEUS: Certifiable and Efficient LLM Evaluation via E-Processes

Zhijian Zhou, Zesheng Ye, Zhaorun Chen, Bo Li+1

The University of Melbourne · Department of Computer Science, University of Chicago · Department of Computer Science, University of Illinois Urbana-Champaign

LLM Evaluation Anytime-Valid Inference Adaptive Sampling Uncertainty Sampling

Open paper AskPDF arXiv

Can we trust evaluation scores to capture an LLM's true real-world performance? Certifiable evaluation answers this question by providing guarantee for LLM evaluation. In particular, existing methods sequentially curate evaluation samples and keep updating confidence intervals (CIs) that cover the true performance with high probability (e.g., 95%) until some conditions are satisfied, e.g., the CI width reaches a target precision. However, existing methods are not generally anytime-valid: the claimed coverage (e.g., 95%) may fail when CIs are repeatedly updated and used to decide when to stop, leaving a gap between theoretical rigor and practice. This paper bridges this gap by proposing Celeus, a Certifiable framework for Efficient LLM evaluation, which leverages E-processes to build anytime-valid CIs. Concretely, we propose signals that combine two ingredients: (i) Uncertainty-guided sampling to select informative samples for evaluation, and (ii) Surrogate-assisted approximations for unevaluated samples. We prove that such signals remain unbiased for the evaluation score conditional on the past, enabling statistically-grounded and anytime-valid

e

-process CIs. More importantly, the two ingredients reduce estimation variance and help reach the target precision with fewer evaluated samples. We also prove that CIs obtained by Celeus can shrink at a near-parametric rate up to logarithmic factors and analyze the oracle variance-optimal sampling rule that motivates the empirical uncertainty-guided one. Experiments show that Celeus reaches the target precision using 54-62% fewer evaluated samples than baselines, while preserving anytime-valid coverage.

Abstract

LLM Evaluation

Explore similar work

CELEUS: Certifiable and Efficient LLM Evaluation via E-Processes

How to Run Statistics over LLM Judges and Trust the Results: Calibrated Inference for Small-Sample AI Evaluation with evalstats

Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking