Knowing When to Stop: Bayesian Optimal Stopping for LLM Evaluations

Toby D. Pilditch

cs.AIAug 14, 2026

Knowing When to Stop: Bayesian Optimal Stopping for LLM Evaluations

Authors: Toby D. Pilditch

Organizations: UK AI Security Institute, London, UK

Abstract

LLM evaluations often use fixed sampling budgets, testing every item the same number of times even after estimates are precise. We introduce optstop, a precision-based adaptive stopping framework that treats evaluation as a sequential measurement problem: keep sampling where uncertainty remains high, and stop where estimates are precise or stable enough. The framework builds on hierarchical Bayesian inference, supports binary, ordinal, and continuous outcomes, and keeps every benchmark item eligible for sampling, without requiring a calibrated item bank. It runs live or retrospectively, and includes a safeguard that samples more cautiously as measured performance approaches zero, where rare successes matter most. In an illustrative 200-item, 10-epoch evaluation, it removes 57%-97% of planned trials across nine validation settings, with overall conclusions equivalent to the full run. These results show that LLM evaluation compute can be allocated by uncertainty rather than by fixed repetition counts, with the magnitude of savings depending on evaluation design.

Figures & tables

Figure 1: Example hierarchically nested structure of evaluation data. Taken from [ 24 ] .

	Binary	Ordinal	Continuous bounded
Scores	0/1 (correct/incorrect)	Ordered categories $0,\ldots,K{-}1$ (e.g., 0–10 rubric, $K=11$ )	Bounded continuous, normalised to $[0,1]$
Routing	Default pathway	ordinal_tasks name matching ( inspect_ai and standalone)	score_agg aggregation ( inspect_ai ) or continuous_tasks name matching (standalone)
Item-level model	Adaptive Beta (data-dependent prior)	Bayesian bootstrap over the modal category	Beta via method-of-moments
Grouping-level model	Logit-normal hierarchical	Hierarchical ordered logistic (cumulative link); Dirichlet-Multinomial fallback	Logit-normal hierarchical on item-level means
Group-level priors	$\mu_{\text{group}}\sim\text{Normal}(\mu_{0},1.5)$ ; $\sigma_{\text{group}}\sim\text{Exponential}(1)$	$\mu_{\text{group}}\sim\text{Normal}(\mu_{0},2)$ ; $\sigma_{\text{group}}\sim\text{Exponential}(1)$	$\mu_{\text{group}}\sim\text{Normal}(\mu_{0},1.5)$ ; $\sigma_{\text{group}}\sim\text{Exponential}(1)$ ; $\phi_{\text{group}}\sim\text{Gamma}(2,1)$
Estimand	Population mean success probability	Modal category of the item-averaged distribution	Population mean of item-level means

Table 1: Summary of the three inference pathways.

\mu_{0}=0

by default; the prior scales shown are the pathway defaults (both configurable via prior_mu and prior_sigma );

\mu_{\text{group}}

\sigma_{\text{group}}

, and

\phi_{\text{group}}

denote the group-level mean, between-item scale, and precision parameters respectively. Stopping thresholds apply at both item and grouping levels, and the asymmetric conservatism adjustment (Section 2.2 ) applies to all three pathways when estimated performance falls below the low-performance threshold. Full specifications - including item-level priors, cutpoint parameterisation, and the priors of the Dirichlet-Multinomial fallback model - are given in Appendix A.4 .

Figure 2: Stopping efficiency and score fidelity across the $3\times 3$ validation matrix. Bars show the number of trials: full run (blue, 2,000 planned; 1,980 for mid-binary) versus truncated (orange, with stop trial annotated). Overlaid markers show normalised task performance: full run (blue circles) versus truncated (orange squares). The close alignment of score markers demonstrates high fidelity despite substantial trial reductions, particularly in the continuous pathway (54–142 trials of 2,000). Error bars are item-level 95% confidence intervals (mean $\pm$ 1.96 SEM).

Figure 3: Bayesian hierarchical meta-analysis of the truncation effect across all nine cells. (a) Forest plot showing independent estimates (grey squares) and hierarchical estimates after partial pooling (coloured circles). Dashed arrows connect paired estimates, illustrating shrinkage toward the group mean. The bottom row shows the overall estimate $\mu$ . Cell labels report $N$ , the number of items matched in the paired comparison (fewer in cells that stopped before a full pass over all items). Grey error bars: Wald 95% confidence intervals from observed standard errors; coloured error bars: 97% HDIs from the hierarchical posterior. (b) Posterior density of $\mu$ (overall mean difference, normalised $[0,1]$ scale). Shaded region: ROPE ( $\pm 0.02$ ); vertical dashed lines: 97% HDI boundaries. The entire HDI falls within the ROPE, yielding an accept-null verdict.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Grouping	Model	Task	Assumed Pattern
G1	Model A	Task 1 (Binary)	High accuracy, consistent
G2	Model A	Task 2 (Ordinal)	Scores clustered in upper range
G3	Model B	Task 1 (Binary)	Low accuracy, conservatism active
G4	Model B	Task 2 (Ordinal)	Low, diffuse score distribution

Appendix

Table 2: Model groupings, tasks, and assumed performance characteristics

Pathway	Benchmarks	Scale	Modes Tested
Binary	TruthfulQA, RACE-H	{0, 1}	Live, shadow, multi-model
Ordinal	WritingBench, SciKnowEval	11-cat, 6-cat	Live, shadow
Continuous	WritingBench	[0, 10]	Live, shadow, multi-model

Appendix

Table 3: Inference pathway validation coverage

$\dagger$ Continuous pathway CI widths reported on a normalised [0,1] scale; not directly comparable to binary posterior widths.
	$\delta=0.2$			$\delta=0.05$
Task (Pathway)	Trials	Efficiency	Final CI	Trials	Efficiency	Final CI
TruthfulQA (Binary)	63	87.4%	0.163	238	52.4%	0.047
RACE-H (Binary)	24	95.2%	0.192	270	46.0%	0.049
WritingBench (Continuous)	30	94.0%	$\dagger$	100	80.0%	$\dagger$
Overall	117	92.2%		608	59.5%

Appendix

Table 4: Precision-efficiency trade-off across threshold settings

	Abs. Bias		CI Width		Coverage		Diverg.
Region	L-N	B-B	L-N	B-B	L-N	B-B	Ratio
Mid-range (0.1–0.9)	0.022	0.022	0.070	0.070	0.80	0.81	120 $\times$
Near-boundary (0.05, 0.95)	0.012	0.012	0.039	0.038	0.78	0.78	192 $\times$
Boundary (0.0, 1.0)	0.007	0.005	0.011	0.011	0.00	0.00	9,206 $\times$

Appendix

Table 5: Logit-normal (L-N) vs. Beta-Binomial (B-B) hierarchical model comparison under a Beta-Binomial data-generating process. Metrics averaged across sample sizes (20, 50, 100 items) and 25 replications per condition. Divergence ratio reports total B-B divergences divided by total L-N divergences within each region.

Cell	Benchmark	Model	Pathway	max_tokens
low_binary	MATH Level 5	GPT-3.5 Turbo	Binary	–
mid_binary	GPQA Diamond	GPT-4o	Binary	–
high_binary	MMLU (0-shot)	GPT-4o	Binary	–
low_ordinal	WritingBench	Claude Sonnet 4.5	Ordinal	50
mid_ordinal	WritingBench	Claude Sonnet 4.5	Ordinal	500
high_ordinal	WritingBench	Claude Sonnet 4.5	Ordinal	5,000

Appendix

Table 6: Cell configurations for the

3\times 3

matrix validation experiment. All cells used 200 items, 10 epochs, seed 42,

\delta=0.05

, and 97% credible intervals. Planned trials = items

\times

epochs = 2,000 per cell (reduced by 20 for mid-binary due to 2 items failing to load).

Cell	$n$	$\bar{d}$	$\mu_{\text{diff}}$	94% HDI	ROPE
low_binary	200	$+$ 0.006	$+$ 0.007	[ $-$ 0.010, $+$ 0.025]	undecided
mid_binary	198	$-$ 0.011	$-$ 0.011	[ $-$ 0.029, $+$ 0.007]	undecided
high_binary	200	$+$ 0.005	$+$ 0.005	[ $-$ 0.003, $+$ 0.012]	accept
low_ordinal	200	$+$ 0.000	$+$ 0.000	[ $-$ 0.016, $+$ 0.017]	accept
mid_ordinal	200	$+$ 0.005	$+$ 0.006	[ $-$ 0.018, $+$ 0.029]	accept
high_ordinal	152	$-$ 0.060	$-$ 0.059	[ $-$ 0.114, $-$ 0.001]	undecided

Appendix

Table 7: Item-matched paired comparison results (Analysis A, within-shadow).

n

= number of matched items;

\bar{d}

= observed mean difference (full

-

truncated);

\mu_{\text{diff}}

= posterior mean; HDI = 94% highest density interval; ROPE = equivalence decision. All models converged with

\hat{R}=1.0

, 0 divergences, and bulk ESS

>4{,}600

for

\mu_{\text{diff}}

and

>2{,}500

for

\sigma_{\text{diff}}

Figure 4: Posterior distribution of $\mu_{\text{diff}}$ for the mid-binary cell (Analysis A). The grey shaded band marks the ROPE ( $\pm 0.02$ ); the dashed vertical line marks zero. The posterior is centred near $-0.011$ , with the 94% HDI $[-0.029,+0.007]$ extending beyond the ROPE due to the high per-item Bernoulli variance of this near-50% binary benchmark, yielding an undecided verdict.

Figure 5: ROPE sensitivity analysis across six ROPE widths (Analysis A, within-shadow). Each row is a cell; columns are ROPE widths. Green: accept; orange: undecided. No cell rejects at any width. At the pathway-specific primary widths ( $\pm 0.02$ for binary and continuous, $\pm 0.10$ for ordinal), 6 of 9 cells accept. At $\pm 0.05$ (matching the framework’s precision threshold $\delta$ ), 8 of 9 cells accept; high-ordinal accepts only at $\pm 0.20$ .

Figure 6: Forest plot of model effects from hierarchical ordered logistic regression (Analysis B). Blue: full run; orange: truncated. Error bars show 94% HDIs. The ranking $\texttt{max\_tokens}=5{,}000>500>50$ is preserved in both conditions, with non-overlapping HDIs between all adjacent levels.

Figure 7: Forest plot of model effects from hierarchical continuous model (Analysis C). Blue: full run; orange: truncated. The ranking $\texttt{max\_tokens}=5{,}000>500>50$ is preserved with clean convergence (0 divergences) in both conditions.

Figure 8: Convergence of posterior variance over trials. Curves show the running posterior variance of each cell’s group-level estimate, normalised by the cell’s full-run between-item variance; vertical dotted lines indicate would-have-stopped points. Diminishing returns are evident across all pathways, with posterior variance collapsing within the first 200–400 trials.

Explore similar work

Stop Guessing When to Stop Testing: Efficient Model Evaluation with Just Enough Data
Jul 9, 2026Ofir Arviv, Kristjan Greenewald, Yotam Perlitz +3Early Stopping Optimal Stopping
IBM Research
Speculative Evaluation of Stochastic LLMs
Sep 23, 2026Qianli Shen, Xiang Li, Ruomeng Ding +3LLM Evaluation Language Model Generation Evaluation
Alibaba Group · National University of Singapore · University of North Carolina at Chapel Hill
CELEUS: Certifiable and Efficient LLM Evaluation via E-Processes
Jun 18, 2026Zhijian Zhou, Zesheng Ye, Zhaorun Chen +2LLM Evaluation Anytime-Valid Inference
The University of Melbourne · Department of Computer Science, University of Chicago · Department of Computer Science, University of Illinois Urbana-Champaign

Jul 9, 2026J/K move · Enter open · S save

Stop Guessing When to Stop Testing: Efficient Model Evaluation with Just Enough Data

Ofir Arviv, Kristjan Greenewald, Yotam Perlitz, Hadar Mulian+2

IBM Research

Early Stopping Optimal Stopping Automated Evaluation Sequential Hypothesis Testing Statistical Hypothesis Testing

Open paper AskPDF arXiv

The inherent rigidity of fixed-size benchmarks makes them an inefficient tool for model evaluation. Diverse evaluation objectives, including model ranking, model selection and testing throughout development, demand varying levels of statistical power. The mismatch between fixed sample sizes and these diverse needs results in either excessive computational cost or compromised reliability - a critical concern for model evaluation. To overcome these limitations, we call for adoption of sequential testing in our field. We provide an adaptive evaluation framework, that provides a principled way to navigate the trade-off between efficiency and reliability in model evaluation. Our framework combines the established statistical paradigm of sequential testing with stopping criteria tailored to common evaluation needs such as diminishing returns detection, and minimum detectable effect size. We demonstrate its ability to adaptively manage the efficiency-reliability trade-off on the Open VLM Leaderboard, including, for example, a 80% reduction in computational cost compared to fixed-size evaluation (with a 2.5-point CI width allowance) while maintaining statistical significance.

Abstract

Explore similar work

Stop Guessing When to Stop Testing: Efficient Model Evaluation with Just Enough Data

Speculative Evaluation of Stochastic LLMs

CELEUS: Certifiable and Efficient LLM Evaluation via E-Processes