Human Evaluation with Automatic Scores: Separating Sampling and Estimation Gains

Yang Liu; Ran Zou; Zhengtao Gui; Chao Xue

cs.CLOct 28, 2026

Human Evaluation with Automatic Scores: Separating Sampling and Estimation Gains

Authors: Yang Liu, Ran Zou, Zhengtao Gui, Chao Xue

Organizations: Department of Statistics and Operations Research, University of North Carolina at Chapel Hill, USA · Department of Statistics, University of California, Irvine, USA · School of Computer Science and Engineering, University of New South Wales, Australia

Abstract

Automatic judges offer inexpensive information about model outputs, but an evaluation team must decide how to turn that information into fewer human annotations. Scores can guide sampling, correct an estimate, or support adaptive acquisition, and these uses may capture overlapping gains. We compare their incremental value on fixed evaluation pools from three preference datasets, and 19 attributed-question-answering systems. Matched comparisons use the same scores, account for pilot labels and budget rules, and separate acquisition from estimation. Score stratification is a strong reference: after ten-bin stratification, the best fixed local linear correction offers only 1.00--5.27% further MSE reduction in the primary configurations. Unequal allocation offers larger gains in some pools, but can increase error when confident scores conceal variable human outcomes. Comparison on AttributedQA, shrinking learned quotas toward score-only quotas lowers pooled MSE for 17 of 19 systems relative to the score-only procedure, but ordinary pooled 95% intervals cover only 88.09--93.54%. The same shrinkage improves on the score-only reference in only one of eight preference configurations. Ranking experiments further show that lower individual MSE need not improve system selection. These findings support a simple starting procedure, proportional sampling within score strata followed by a stratified human mean, and identify what additional evidence is needed to justify correction or adaptation for the intended evaluation output.

Figures & tables

Pool	Items	Human target	Automatic score
AlpacaFarm	648	Preference	Mistral and Qwen judges
Chatbot Arena	2,870	Preference; contrast	Same judges; contrast uses 1,247 items
HelpSteer3	2,017	Preference	Same judges; four task groups
AttributedQA	1,000/system	Attribution; ranking	Released AutoAIS; 19 systems

Table 1: Primary evaluation pools. Preference scores average both response orders. One acquired label reveals one released outcome, which can aggregate several human judgments. Appendix A specifies annotation units; Appendix H reports scoring resources and cost definitions.

A. Preference targets
		No score bins	Ten bins: proportional allocation			Ten bins: working allocation
Target / system	Score	Original mean	Linear	Ridge	Oracle	Mean ( $b=0$ )	Ridge
Alpaca	M	1.225	1.009 (0.0014)	1.005 (0.0010)	0.987	0.996	1.000 (0.0011)
Alpaca	Q	1.404	1.005 (0.0022)	1.002 (0.0014)	0.965	1.028	1.032 (0.0012)
Arena mean	M	1.072	1.008 (0.0018)	1.003 (0.0014)	0.990	0.998	1.000 (0.0013)
Arena mean	Q	1.163	1.002 (0.0016)	1.001 (0.0012)	0.986	1.039	1.040 (0.0011)

Table 2: Separating gains from stratification, correction, and allocation. Primary-budget MSE divided by the matching ten-bin proportional-mean risk (reference

1

; lower is better). Mean sets

b_{h}=0

: scores guide sampling, while estimation uses weighted human averages without correction. Parentheses give one MCSE from 10,000 paired draws; means and oracles are exact. Among ten-bin non-oracle columns, bold marks displayed ratios

\leq 0.95

\geq 1.05

, highlighting improvement and deterioration. Italics identify the full-label oracle. M/Q denote Mistral/Qwen; AQA uses AutoAIS.

1 bin/group: no score stratification
Calibration	MSE ratio	MCSE
IsoCal	0.648–0.965	0.0045–0.0078
LinearCal	0.616–0.945	0.0018–0.0080
MonoSpline	0.622–0.944	0.0032–0.0082
Local correction	0.618–0.946	0.0019–0.0080
5 bins/group

Table 3: Calibration gains depend on how finely scores are stratified. Results for all eight preference configurations at the primary budget. Each block uses its own matching proportional stratified mean as the MSE reference

1

; lower is better. Entries are ranges across configurations; MSE ranges are bold throughout. MCSE gives the range of Monte Carlo standard errors for the MSE ratios.

Figure 1: Score-based uncertainty helps when it tracks human variation. Two AQA systems at

L=250

labels each. The upper panels compare human-outcome SD

S_{y,h}

with working SD

\sqrt{v_{h}^{\rm work}}

; the lower panels show proportional and working quotas. Strata are ordered by score. Complete human outcomes explain these results retrospectively.

Figure 2: Learning quotas and reusing labels are separate decisions. MSE relative to the one-stage proportional mean, using the same pilot samples and total budgets within each configuration. M and Q denote Mistral and Qwen. Blue indicates lower MSE. Within each group of three columns, the acquired labels are identical and only the estimator changes.

Final-quota rule	Lower MSE	MSE ratio median (range)	95% CI coverage range (%)
Score-only (reference)	—	1.000	93.75–95.08
Learned	0/19	1.096 (1.023–1.716)	75.24–91.22
Half learned + proportional	8/19	1.016 (0.836–1.259)	92.17–94.02
Half learned + score-only	17/19	0.926 (0.866–1.061)	88.09–93.54

Table 4: Shrinkage improves MSE, but ordinary pooled intervals can still undercover. All methods use Pooling on the 19 AQA systems, with matched pilot samples and total label budgets. The reference is score-only quotas with Pooling. "Lower MSE" counts systems with lower MSE than this reference. MSE ratios are summarized by their median and range across systems. Coverage is the percentage of repeated samples whose ordinary pooled 95% normal interval contains the full-pool target; the last column gives its range across systems. The paired-comparison protocol and Monte Carlo checks are given in Appendix E.2 .

Figure 3: The reference changes the apparent benefit of active acquisition. AQA at nominal budget

L=250

. (a) RobustASI relative to its same-pilot uniform control; (b) capped ASI and RobustASI relative to the proportional ten-stratum human mean. Ratios below one favor the active method. Counts use point estimates; Monte Carlo standard errors are reported in Table 11 .

Figure 4: The label budget changes how clearly selection errors differ. Probability of selecting the wrong best system among 19 AQA systems. Proportional and working quotas use the same stratified-mean estimator and the same score strata for each system. Each system receives exactly

L

labels, for

19L

per leaderboard. Samples use the shared question ordering described in the text. Error bars are

\pm 2

Monte Carlo standard errors.

Procedure	Inverted pairs	Best-system selection	Top-three recall
Proportional mean	6.68 (0.01)	99.07 (0.04)	96.16 (0.05)
Working mean	6.85 (0.01)	97.21 (0.08)	94.49 (0.06)
Score-only + Pooling	6.41 (0.02)	98.67 (0.07)	96.08 (0.09)
Learned + Pooling	6.43 (0.02)	99.30 (0.05)	94.21 (0.12)
Proportional-shrink + Pooling	6.27 (0.02)	99.29 (0.05)	96.25 (0.08)
Score-shrink + Pooling	5.99 (0.01)	99.40 (0.04)	95.86 (0.09)

Table 5: AQA ranking at 250 labels per system. Entries are percentages, with MCSE in parentheses. Lower inverted-pair rates and higher selection and recall rates are better. All rows use the same joint replay with shared question orderings. The last two rows shrink learned final quotas halfway toward proportional or score-only quotas, respectively, and use Pooling.

Figure 5: An accurate average variance can hide unreliable interval widths. Learned quotas with Composite on the 19 AQA systems. Left: coverage of the same point estimates using either sample-based variance or a fixed reference variance; bars are

\pm 1.96

MCSE. Right: the mean and median sample variance relative to the reference. The reference uses complete pool outcomes to diagnose the source of undercoverage.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

A. Preference targets
		Unrestricted		Clipped to $[0,1]$
Target / system	Score	Local	Shared	Local	Shared
Alpaca	M	1.076 (0.0052)	1.009 (0.0014)	1.002 (0.0009)	1.002 (0.0009)
Alpaca	Q	1.032 (0.0053)	1.005 (0.0022)	0.994 (0.0016)	1.000 (0.0017)
Arena mean	M	1.062 (0.0045)	1.008 (0.0018)	1.002 (0.0010)	1.002 (0.0010)
Arena mean	Q	1.041 (0.0045)	1.002 (0.0016)	0.998 (0.0014)	0.998 (0.0013)

Appendix

Table 6: Coefficient sharing and clipping at the primary budgets, using ten score bins per natural group and proportional quotas. Entries are MSE divided by

R_{0}

, the risk of the matching stratified human mean; lower is better. Parentheses give one Monte Carlo standard error from 10,000 paired draws. Local fits estimate one coefficient per stratum; Shared fits estimate one coefficient per natural group. M/Q denote Mistral/Qwen; AQA uses AutoAIS. The Shared column reproduces the proportional-allocation Linear results in the main table.

Bins/group	Calibration	Coverage (%)
1	IsoCal	92.42–94.26
	LinearCal	94.52–95.28
	MonoSpline	93.92–94.80
	Local correction	94.50–95.42
5	IsoCal	92.98–94.60
	LinearCal	94.46–95.52

Appendix

Table 7: Coverage of nominal 95% residual normal intervals for the calibration comparison. Each entry gives the range of empirical coverage across the eight preference configurations at the primary budget. The corresponding point-estimation results appear in Table 3 .

Temperature	Calibration error	Quota distance	MSE
$T=0.5$	3/27	0/27	1/27
$T=2$	20/27	25/27	20/27

Appendix

Table 8: Temperature sensitivity across the 27 primary configurations. Entries count decreases relative to

T=1

. Configurations share data and are descriptive comparisons, not independent replications.

	Proportional	Working
Target / judge	Fixed	Fixed	Linear
Alpaca / M	1.001	0.997	1.004 (0.0014)
Alpaca / Q	0.997	1.030	1.036 (0.0023)
Arena mean / M	0.999	0.997	1.005 (0.0018)
Arena mean / Q	1.001	1.039	1.042 (0.0016)
Arena contrast / M	0.997	1.000	1.010 (0.0023)

Appendix

Table 9: Additional correction–allocation comparisons. Every entry is MSE divided by the matching proportional stratified-mean risk

R_{0}

. Fixed correction sets

b_{h}=1

; Linear fits a group-shared coefficient. Parentheses give one MCSE. Mean and Ridge results are reported in the main table. Bold marks ratios outside

[0.95,1.05]

. M/Q denote Mistral/Qwen.

	Toward proportional			Toward score-only
Configuration	Completion	Composite	Pooling	Completion	Composite	Pooling
Alpaca / M	1.228	1.028	1.048	1.227	1.027	1.047
Alpaca / Q	1.202	1.007	1.011	1.211	1.014	1.014
Arena contrast / M	1.275	1.062	1.089	1.275	1.063	1.088
Arena contrast / Q	1.192	1.000	1.005	1.193	1.003	1.001
Arena mean / M	1.229	1.031	1.070	1.230	1.031	1.070

Appendix

Table 10: Shrinkage results for all 27 primary configurations. Every ratio uses the MSE of score-only quotas with Pooling as its denominator. Columns compare the three label-reuse estimators under each shrinkage rule. All procedures use the same pilot samples and total label budget within a configuration.

System	ASI nominal	Actual $K$	Robust nominal	Actual $K$	Robust/uniform
LLM-as-retriever	1.066 $\pm$ 0.005	1.036	1.025 $\pm$ 0.004	1.019	0.850 $\pm$ 0.003
Post+rerank	1.651 $\pm$ 0.007	1.604	1.353 $\pm$ 0.006	1.347	1.028 $\pm$ 0.004
Post-1	1.131 $\pm$ 0.005	1.099	1.049 $\pm$ 0.004	1.043	0.866 $\pm$ 0.003
Post-2	1.157 $\pm$ 0.005	1.125	1.056 $\pm$ 0.003	1.051	0.882 $\pm$ 0.003
Post-3	1.173 $\pm$ 0.006	1.140	1.076 $\pm$ 0.004	1.070	0.872 $\pm$ 0.003
Post-4	1.253 $\pm$ 0.006	1.218	1.109 $\pm$ 0.003	1.104	0.898 $\pm$ 0.003

Appendix

Table 11: AQA active comparisons. Nominal ratios use

R_{\rm prop}(L)

; actual-

K

ratios use

\mathbb{E}_{K}[R_{\rm prop}(K)]

. The last column uses the same-pilot uniform control. Reported errors are one MCSE.

Target / score	ASI nominal	Actual $K$	Robust nominal	Actual $K$	Robust/uniform
Alpaca / M	1.101	1.060 $\pm$ 0.005	1.246	1.242 $\pm$ 0.001	1.000 $\pm$ 0.000
Alpaca / Q	1.254	1.208 $\pm$ 0.005	1.308	1.304 $\pm$ 0.002	1.005 $\pm$ 0.001
Arena mean / M	1.046	1.017 $\pm$ 0.005	1.181	1.181 $\pm$ 0.000	1.000 $\pm$ 0.000
Arena mean / Q	1.165	1.133 $\pm$ 0.005	1.175	1.175 $\pm$ 0.001	1.004 $\pm$ 0.001
Arena contrast / M	1.049	1.009 $\pm$ 0.005	1.312	1.309 $\pm$ 0.001	1.000 $\pm$ 0.000
Arena contrast / Q	1.211	1.167 $\pm$ 0.005	1.320	1.318 $\pm$ 0.002	1.008 $\pm$ 0.001

Appendix

Table 12: Preference comparisons with original groups respected. Denominators match Table 11 ; M/Q denote Mistral/Qwen. Errors are one MCSE. RobustASI MSE includes zero-support residual bias.

Scope	Quota	Estimator	MSE ratio	MCSE
AQA	Proportional	LURE	0.952–1.007	0.0047–0.0088
AQA	Proportional	Residual LURE	0.953–1.159	0.0075–0.0142
AQA	Working	LURE	0.926–1.757	0.0026–0.1532
AQA	Working	Residual LURE	0.908–1.746	0.0079–0.1538
Preference	Proportional	LURE	0.988–1.022	0.0030–0.0056
Preference	Proportional	Residual LURE	0.996–1.036	0.0022–0.0077

Appendix

Table 13: Fixed-budget active-testing controls. MSE and its MCSE both use the same-quota uniform mean as reference. Working stratum quotas and within-stratum second-moment proposals follow the annotation-specific model.

	Shared question priorities			Independent systems
$L$	Prop.	Working	$\Delta$ (MCSE)	Prop.	Working	$\Delta$ (MCSE)
50	15.256	15.109	-0.146 (0.029)	16.756	16.388	-0.369 (0.026)
100	11.108	11.270	+0.162 (0.024)	12.531	12.413	-0.118 (0.023)
150	9.051	9.211	+0.160 (0.010)	10.356	10.179	-0.177 (0.042)
200	7.672	7.810	+0.138 (0.019)	8.905	8.718	-0.187 (0.019)
250	6.683	6.832	+0.149 (0.008)	7.820	7.660	-0.161 (0.036)

Appendix

Table 14: AQA pairwise inversion rates under both joint sampling designs. Rates are percentages;

\Delta

is working minus proportional in percentage points, with paired MCSE in parentheses. Lower inversion is better. A ranking costs

19L

system–question labels.

		Proportional mean			Working, ten bins
Target / score	$L$	2 bins	5 bins	10 bins	Mean	Shared PPI
Alpaca / M	32	0.871	0.823	0.828	0.817	0.882 (0.0053)
Alpaca / Q	32	0.841	0.736	0.725	0.735	0.782 (0.0040)
Arena mean / M	143	0.948	0.934	0.936	0.940	1.007 (0.0053)
Arena mean / Q	143	0.907	0.871	0.863	0.881	0.905 (0.0030)
Arena contrast / M	62	0.947	0.940	0.938	0.939	1.025 (0.0068)

Appendix

Table 15: Five-percent budgets: every entry is MSE divided by the exact original-group proportional-mean MSE at the same

L=\lfloor 0.05N\rfloor

. This single denominator permits comparisons across partitions. The first three numeric risk columns use proportional means; the last two use ten-bin working quotas with a mean or group-shared PPI fit. Mean risks are exact. Parentheses give PPI MCSEs; preference fits use 10,000 draws and AQA fits use 20,000. Working quotas use the annotation-specific model. M/Q denote Mistral/Qwen, both using order-averaged scores. Rerank denotes AutoAIS reranking.

Pool	Protocol	Valid	Failed	Input tokens	Output tokens	Recorded time
Alpaca	Qwen	1,296	2	545,495	48,784	5.432 h, request sum
Arena	Qwen	5,740	0	5,921,032	217,142	48.546 h, request sum
HelpSteer3	Qwen	4,034	11	6,449,542	148,258	5.335 h, request sum
Alpaca	Mistral	1,296	0	497,730	0	68.865 s, batch share
Arena	Mistral	5,740	0	5,655,830	0	800.265 s, batch share
HelpSteer3	Mistral	4,034	0	6,586,190	0	947.061 s, batch share

Appendix

Table 16: Recorded resources for the both-order preference scores. Input counts exclude unobserved failed-call tokens and, for Mistral, batch padding. Zero Mistral output tokens reflects forward-only likelihood scoring. Times have the measurement units shown and should not be summed as a common hardware-cost quantity.

Human-in-the-Loop Evaluation

Human-in-the-Loop Evaluation studies how human judgment and feedback can assess, guide, and monitor AI systems in realistic settings.

See all 160 papers

Instance-Optimal Estimation with Multiple LLM Judges on a Budget
May 22, 2026Junghyun Lee, Sanghwa Kim, Yassir Jedra +2Parameter Estimation LLM Evaluation
KAIST AI · ICL EEE · KTH EECS
Scoring Rules! Statistical and Strategic Alignment for Text Evaluation Metrics
Aug 2, 2026Shengwei Xu, Yuxuan Lu, Yifan Wu +2Language Model Generation Evaluation Automated Evaluation
University of Michigan · Peking University · Microsoft Research +1
Select Smarter, Not More: Prompt-Aware Evaluation Scheduling with Submodular Guarantees
Apr 13, 2026Xiaoyu Ma, Yiwen Li, Haoyue Liu +4Automatic Prompt Optimization Automated Evaluation
The Chinese University of Hong Kong, Shenzhen · Xi’an Jiaotong University · Taobao and Tmall Group

May 22, 2026J/K move · Enter open · S save

Instance-Optimal Estimation with Multiple LLM Judges on a Budget

Junghyun Lee, Sanghwa Kim, Yassir Jedra, Alexandre Proutière+1

KAIST AI · ICL EEE · KTH EECS

Parameter Estimation LLM Evaluation LLM-as-a-Judge Minimax Estimation Adaptive Sampling

Open paper AskPDF arXiv

Evaluating large language models increasingly relies on LLM-as-a-judge protocols, but such evaluations remain costly: different judges have different prices and reliabilities, and the difficulty of each prompt-response pair can vary substantially. This raises a basic allocation question: under a fixed budget, how should one distribute evaluation queries across heterogeneous judges and instances to obtain the most accurate score estimates? We formalize this question as budgeted heteroskedastic multi-judge estimation. Given

K

prompt-response pairs,

J

judges with known costs, and unknown query-judge variances, the goal is to estimate a bounded score vector while minimizing an

\ell_p

-error. Our first contribution is to analyze the inverse-variance weighted estimator (IVWE) and to derive the oracle allocation that minimizes its error rate. Since this allocation depends on the unknown variances, we then address the practical unknown-variance setting by proposing EST-IVWE, an adaptive algorithm that constructs and leverages optimistically biased variance estimates to stabilize the empirical allocation. We prove that EST-IVWE matches the oracle IVWE rate up to lower-order terms in the budget. Our second and central theoretical contribution is a matching local minimax lower bound, which establishes the instance-optimality of the proposed algorithms. A key technical insight is that Fano-type high-probability arguments are too coarse for this problem: their packing construction loses the local variance structure that governs the optimal allocation. We instead use an Assouad-type in-expectation argument, based on local perturbations, which preserves this structure and yields the sharp allocation-dependent lower bound. Finally, we numerically validate the superiority of our approach over naïve uniform allocation on synthetic and HelpSteer2 datasets.

Abstract

Human-in-the-Loop Evaluation

Explore similar work

Instance-Optimal Estimation with Multiple LLM Judges on a Budget

Scoring Rules! Statistical and Strategic Alignment for Text Evaluation Metrics

Select Smarter, Not More: Prompt-Aware Evaluation Scheduling with Submodular Guarantees