Conditional Evaluation of Language Models with Cheap Auxiliary Signals

Zhi Zhang; Lingfeng Lyu; Yue Kang; Doudou Zhou

cs.LGAug 17, 2026

Conditional Evaluation of Language Models with Cheap Auxiliary Signals

Authors: Zhi Zhang, Lingfeng Lyu, Yue Kang, Doudou Zhou

Organizations: Department of Statistics and Data Science, University of California, Los Angeles · Department of Statistics and Finance, School of Management, University of Science and Technology of China · Microsoft · Department of Statistics and Data Science, National University of Singapore

Abstract

Aggregate accuracy hides where models succeed and fail. Estimating conditional performance profiles from gold labels alone is expensive, while cheap auxiliary signals such as LLM-judge scores, pairwise comparisons, confidence scores, and judge-disagreement features can be collected for every benchmark item but are often biased or miscalibrated. We propose LACE (Local Augmented Control-Variate Evaluation), a semi-supervised estimator for conditional LLM evaluation. The key step is local centering: after subtracting the conditional mean of a cheap signal within the target profile region, any linear augmentation has zero conditional mean and therefore cannot change the estimand. The augmentation coefficient is used only for efficiency, and a local ridge control variate combines a gold-label residual mean from the labeled subset with a cheap-signal mean from the full item pool. We prove calibration-free identification, unbiasedness for grouped profiles, local oracle optimality within centered linear augmentations, and first-order adaptivity to the estimated coefficient. The resulting gain formula is governed by a population local $R^2$ , which characterizes how the efficiency attainable from the cheap signals varies across profile values. We also derive corresponding estimators for direct paired model gaps and deployment-weighted scores. We empirically evaluate the primary performance-profile estimator on MATH-500, ScienceQA, MMLU, WinoGrande, HellaSwag, TruthfulQA, GSM8K, and ARC.

Figures & tables

Benchmark	Profiling axis $Z$	$M$	Domain
MATH-500	difficulty level ( $1,\ldots,5$ )	500	math reasoning
ScienceQA	grade level ( $2,\ldots,12$ )	500	science (text-only)
MMLU	question token length	500	general knowledge
WinoGrande	sentence length (tokens)	500	commonsense
HellaSwag	context sentence length	500	commonsense
TruthfulQA	question token length	500	factuality

Table 1: Benchmarks and profiling axes used in the main experiments.

Estimator	$n_{\mathrm{lab}}=50$	$n_{\mathrm{lab}}=100$	$n_{\mathrm{lab}}=200$	Overall
Naive	$1.000\,[1.000,1.000]$	$1.000\,[1.000,1.000]$	$1.000\,[1.000,1.000]$	$1.000\,[1.000,1.000]$
Plug-in (judge)	$3.128\,[2.342,4.177]$	$2.780\,[2.398,3.223]$	$2.093\,[1.863,2.351]$	$2.630\,[2.270,3.047]$
Plug-in (multi)	$3.214\,[2.438,4.236]$	$2.876\,[2.366,3.497]$	$2.006\,[1.770,2.272]$	$2.647\,[2.277,3.078]$
Aug. plug-in	$1.060\,[0.798,1.406]$	$1.227\,[1.090,1.382]$	$1.502\,[1.302,1.734]$	$1.250\,[1.098,1.423]$
Global CV	$1.438\,[1.383,1.494]$	$1.568\,[1.507,1.633]$	$1.703\,[1.612,1.799]$	$1.566\,[1.516,1.618]$
Per-signal CV	$0.891\,[0.873,0.909]$	$1.118\,[1.103,1.134]$	$1.175\,[1.160,1.190]$	$1.054\,[1.042,1.066]$

Table 2: Overall results. Entries are geometric-mean RE

[95\%\ \mathrm{MC\ CI}]

across the 24 benchmark–model cells at each label budget; Overall aggregates all 72 cells. Each cell is a ratio of mean MSEs over the same

B=100

label splits. Intervals apply the paired delta method within each cell; budget columns combine cellwise log-RE variances across independent cells, and the Overall column treats the three nested budgets within each cell jointly, using the label-split permutation as the joint resampling unit, so its variance includes the cross-budget covariances induced by the nested prefixes. Target: realized full-pool gold kernel profile; randomness: matched

L

-draws conditional on

T

Appendix figures & tables36 assets

Supplementary material from the paper’s appendix.

Appendix

Estimator	$n_{\mathrm{lab}}=50$	$n_{\mathrm{lab}}=100$	$n_{\mathrm{lab}}=200$
Naive	$1.00\,[1.00,1.00]\ (9.81)$	$1.00\,[1.00,1.00]\ (5.00)$	$1.00\,[1.00,1.00]\ (1.77)$
Plug-in (judge)	$2.59\,[0.96,6.98]\ (3.79)$	$2.43\,[1.56,3.79]\ (2.06)$	$2.53\,[1.72,3.72]\ (0.70)$
Plug-in (multi)	$2.77\,[1.34,5.73]\ (3.54)$	$2.67\,[1.86,3.83]\ (1.87)$	$2.37\,[1.77,3.18]\ (0.75)$
Aug. plug-in	$1.45\,[1.05,2.00]\ (6.76)$	$1.44\,[1.20,1.73]\ (3.48)$	$1.70\,[1.22,2.36]\ (1.04)$
Global CV	$1.86\,[1.43,2.41]\ (5.27)$	$1.81\,[1.63,2.01]\ (2.76)$	$1.90\,[1.55,2.33]\ (0.93)$
Per-signal CV	$1.24\,[1.16,1.33]\ (7.93)$	$1.39\,[1.31,1.47]\ (3.60)$	$1.41\,[1.33,1.50]\ (1.25)$

Appendix

Table 3: RE

[95\%\ \mathrm{MC\ CI}]

on MATH-500 (Claude Haiku 3, continuous

Z

B=100

); mean MSE

\times 10^{3}

is in parentheses. Intervals use paired split-level delta-method influence values. The target is the realized full-pool gold kernel profile; uncertainty is over matched labeled-subset draws conditional on the observed benchmark pool.

Estimator	$n_{\mathrm{lab}}=50$	$n_{\mathrm{lab}}=100$	$n_{\mathrm{lab}}=200$
Naive	$1.00\,[1.00,1.00]\ (1.71)$	$1.00\,[1.00,1.00]\ (0.74)$	$1.00\,[1.00,1.00]\ (0.31)$
Plug-in (judge)	$2.59\,[1.37,4.89]\ (0.66)$	$2.10\,[1.52,2.91]\ (0.35)$	$1.62\,[1.17,2.25]\ (0.19)$
Plug-in (multi)	$1.96\,[0.95,4.05]\ (0.88)$	$1.42\,[0.94,2.15]\ (0.52)$	$1.24\,[0.90,1.71]\ (0.25)$
Aug. plug-in	$0.74\,[0.34,1.61]\ (2.30)$	$0.73\,[0.53,1.00]\ (1.01)$	$1.07\,[0.78,1.46]\ (0.29)$
Global CV	$1.43\,[1.23,1.66]\ (1.20)$	$1.35\,[1.16,1.57]\ (0.55)$	$1.53\,[1.22,1.92]\ (0.20)$
Per-signal CV	$1.06\,[0.99,1.14]\ (1.62)$	$1.09\,[1.02,1.17]\ (0.68)$	$1.17\,[1.10,1.24]\ (0.27)$

Appendix

Table 4: RE

[95\%\ \mathrm{MC\ CI}]

on ScienceQA (Claude Haiku 3, continuous

Z

B=100

); mean MSE

\times 10^{3}

Estimator	$n_{\mathrm{lab}}=50$	$n_{\mathrm{lab}}=100$	$n_{\mathrm{lab}}=200$
Naive	$1.00\,[1.00,1.00]\ (28.09)$	$1.00\,[1.00,1.00]\ (9.59)$	$1.00\,[1.00,1.00]\ (4.17)$
Plug-in (judge)	$3.23\,[1.67,6.25]\ (8.69)$	$2.65\,[1.46,4.80]\ (3.61)$	$1.91\,[1.31,2.78]\ (2.18)$
Plug-in (multi)	$3.09\,[0.66,14.51]\ (9.10)$	$3.42\,[1.01,11.59]\ (2.80)$	$2.51\,[1.36,4.62]\ (1.66)$
Aug. plug-in	$0.69\,[0.34,1.39]\ (40.94)$	$1.06\,[0.69,1.62]\ (9.08)$	$1.00\,[0.48,2.09]\ (4.17)$
Global CV	$1.12\,[1.06,1.18]\ (25.03)$	$1.29\,[1.05,1.58]\ (7.41)$	$1.18\,[1.05,1.33]\ (3.53)$
Per-signal CV	$0.96\,[0.90,1.02]\ (29.18)$	$1.00\,[0.94,1.06]\ (9.60)$	$1.01\,[0.96,1.06]\ (4.12)$

Appendix

Table 5: RE

[95\%\ \mathrm{MC\ CI}]

on MMLU (Claude Haiku 3, continuous

Z

B=100

); mean MSE

\times 10^{3}

Estimator	$n_{\mathrm{lab}}=50$	$n_{\mathrm{lab}}=100$	$n_{\mathrm{lab}}=200$
Naive	$1.00\,[1.00,1.00]\ (14.40)$	$1.00\,[1.00,1.00]\ (5.51)$	$1.00\,[1.00,1.00]\ (2.21)$
Plug-in (judge)	$4.52\,[0.15,139.57]\ (3.18)$	$3.49\,[1.42,8.57]\ (1.58)$	$3.47\,[1.26,9.53]\ (0.64)$
Plug-in (multi)	$5.00\,[0.25,98.06]\ (2.88)$	$3.56\,[0.71,17.92]\ (1.55)$	$3.08\,[1.26,7.55]\ (0.72)$
Aug. plug-in	$1.55\,[1.25,1.92]\ (9.31)$	$1.56\,[0.45,5.43]\ (3.54)$	$2.16\,[1.60,2.92]\ (1.03)$
Global CV	$1.72\,[1.61,1.84]\ (8.35)$	$1.92\,[1.72,2.15]\ (2.87)$	$2.21\,[1.60,3.05]\ (1.00)$
Per-signal CV	$1.18\,[1.14,1.23]\ (12.23)$	$1.19\,[1.14,1.24]\ (4.63)$	$1.23\,[1.19,1.27]\ (1.81)$

Appendix

Table 6: RE

[95\%\ \mathrm{MC\ CI}]

on WinoGrande (Claude Haiku 3, continuous

Z

B=100

); mean MSE

\times 10^{3}

Estimator	$n_{\mathrm{lab}}=50$	$n_{\mathrm{lab}}=100$	$n_{\mathrm{lab}}=200$
Naive	$1.00\,[1.00,1.00]\ (6.70)$	$1.00\,[1.00,1.00]\ (3.23)$	$1.00\,[1.00,1.00]\ (1.05)$
Plug-in (judge)	$2.41\,[0.20,28.63]\ (2.79)$	$2.29\,[0.57,9.15]\ (1.41)$	$2.40\,[0.71,8.16]\ (0.44)$
Plug-in (multi)	$2.65\,[0.61,11.50]\ (2.53)$	$2.39\,[0.19,30.40]\ (1.35)$	$2.25\,[1.01,5.02]\ (0.47)$
Aug. plug-in	$1.12\,[0.53,2.36]\ (5.98)$	$1.31\,[0.99,1.74]\ (2.46)$	$1.65\,[1.13,2.41]\ (0.63)$
Global CV	$1.34\,[1.27,1.41]\ (4.99)$	$1.57\,[1.47,1.67]\ (2.05)$	$1.73\,[1.49,2.00]\ (0.60)$
Per-signal CV	$1.00\,[0.95,1.06]\ (6.69)$	$1.15\,[1.10,1.20]\ (2.82)$	$1.16\,[1.11,1.21]\ (0.90)$

Appendix

Table 7: RE

[95\%\ \mathrm{MC\ CI}]

on HellaSwag (Claude Haiku 3, continuous

Z

B=100

); mean MSE

\times 10^{3}

Estimator	$n_{\mathrm{lab}}=50$	$n_{\mathrm{lab}}=100$	$n_{\mathrm{lab}}=200$
Naive	$1.00\,[1.00,1.00]\ (20.94)$	$1.00\,[1.00,1.00]\ (9.29)$	$1.00\,[1.00,1.00]\ (3.82)$
Plug-in (judge)	$2.85\,[0.00,7723.82]\ (7.35)$	$3.20\,[0.17,61.80]\ (2.90)$	$3.65\,[0.30,44.55]\ (1.05)$
Plug-in (multi)	$3.10\,[0.00,1999.15]\ (6.75)$	$3.55\,[0.04,342.26]\ (2.62)$	$3.40\,[0.01,897.68]\ (1.12)$
Aug. plug-in	$1.65\,[0.29,9.27]\ (12.69)$	$2.10\,[0.54,8.19]\ (4.42)$	$2.85\,[0.55,14.76]\ (1.34)$
Global CV	$1.48\,[1.30,1.68]\ (14.15)$	$1.72\,[1.39,2.13]\ (5.40)$	$2.35\,[1.34,4.12]\ (1.63)$
Per-signal CV	$1.12\,[1.05,1.19]\ (18.69)$	$1.25\,[1.19,1.32]\ (7.43)$	$1.48\,[1.42,1.55]\ (2.58)$

Appendix

Table 8: RE

[95\%\ \mathrm{MC\ CI}]

on TruthfulQA (Claude Haiku 3, continuous

Z

B=100

); mean MSE

\times 10^{3}

Estimator	$n_{\mathrm{lab}}=50$	$n_{\mathrm{lab}}=100$	$n_{\mathrm{lab}}=200$
Naive	$1.00\,[1.00,1.00]\ (7.65)$	$1.00\,[1.00,1.00]\ (2.50)$	$1.00\,[1.00,1.00]\ (0.85)$
Plug-in (judge)	$3.71\,[1.63,8.43]\ (2.06)$	$2.30\,[1.46,3.61]\ (1.09)$	$1.28\,[0.79,2.07]\ (0.67)$
Plug-in (multi)	$5.84\,[1.66,20.53]\ (1.31)$	$4.10\,[2.39,7.03]\ (0.61)$	$1.68\,[0.88,3.19]\ (0.51)$
Aug. plug-in	$1.25\,[0.18,8.85]\ (6.12)$	$1.34\,[0.79,2.28]\ (1.87)$	$1.62\,[1.11,2.37]\ (0.53)$
Global CV	$1.45\,[1.32,1.59]\ (5.27)$	$1.63\,[1.48,1.79]\ (1.53)$	$1.73\,[1.38,2.17]\ (0.49)$
Per-signal CV	$0.96\,[0.85,1.08]\ (7.97)$	$1.18\,[1.07,1.30]\ (2.12)$	$1.26\,[1.13,1.40]\ (0.68)$

Appendix

Table 9: RE

[95\%\ \mathrm{MC\ CI}]

on GSM8K (Claude Haiku 3, continuous

Z

B=100

); mean MSE

\times 10^{3}

Estimator	$n_{\mathrm{lab}}=50$	$n_{\mathrm{lab}}=100$	$n_{\mathrm{lab}}=200$
Naive	$1.00\,[1.00,1.00]\ (7.04)$	$1.00\,[1.00,1.00]\ (3.17)$	$1.00\,[1.00,1.00]\ (1.26)$
Plug-in (judge)	$3.29\,[1.10,9.82]\ (2.14)$	$2.82\,[1.15,6.93]\ (1.12)$	$1.68\,[0.89,3.16]\ (0.75)$
Plug-in (multi)	$3.31\,[0.30,36.46]\ (2.12)$	$2.79\,[0.40,19.31]\ (1.14)$	$1.58\,[0.74,3.39]\ (0.80)$
Aug. plug-in	$0.98\,[0.05,17.94]\ (7.16)$	$1.15\,[0.60,2.19]\ (2.75)$	$1.27\,[0.87,1.85]\ (0.99)$
Global CV	$1.39\,[1.24,1.56]\ (5.07)$	$1.42\,[1.29,1.56]\ (2.23)$	$1.67\,[1.42,1.96]\ (0.76)$
Per-signal CV	$0.89\,[0.81,0.97]\ (7.88)$	$1.16\,[1.07,1.25]\ (2.74)$	$1.21\,[1.12,1.31]\ (1.04)$

Appendix

Table 10: RE

[95\%\ \mathrm{MC\ CI}]

on ARC (Claude Haiku 3, continuous

Z

B=100

); mean MSE

\times 10^{3}

Estimator	$n_{\mathrm{lab}}=50$	$n_{\mathrm{lab}}=100$	$n_{\mathrm{lab}}=200$
Naive	$1.00\,[1.00,1.00]\ (7.46)$	$1.00\,[1.00,1.00]\ (3.50)$	$1.00\,[1.00,1.00]\ (1.63)$
Plug-in (judge)	$1.91\,[1.15,3.18]\ (3.91)$	$2.08\,[1.40,3.10]\ (1.68)$	$2.31\,[1.54,3.46]\ (0.71)$
Plug-in (multi)	$1.85\,[0.63,5.42]\ (4.04)$	$2.18\,[1.47,3.23]\ (1.60)$	$2.20\,[1.17,4.15]\ (0.74)$
Aug. plug-in	$0.96\,[0.74,1.24]\ (7.79)$	$1.16\,[0.82,1.64]\ (3.02)$	$1.46\,[0.87,2.45]\ (1.12)$
Global CV	$1.28\,[1.13,1.45]\ (5.82)$	$1.44\,[1.18,1.76]\ (2.43)$	$1.53\,[1.28,1.84]\ (1.07)$
Per-signal CV	$0.81\,[0.74,0.89]\ (9.17)$	$1.05\,[0.97,1.14]\ (3.34)$	$1.21\,[1.13,1.29]\ (1.35)$

Appendix

Table 11: RE

[95\%\ \mathrm{MC\ CI}]

on MATH-500 (Ministral 3B, continuous

Z

B=100

); mean MSE

\times 10^{3}

Estimator	$n_{\mathrm{lab}}=50$	$n_{\mathrm{lab}}=100$	$n_{\mathrm{lab}}=200$
Naive	$1.00\,[1.00,1.00]\ (2.30)$	$1.00\,[1.00,1.00]\ (1.13)$	$1.00\,[1.00,1.00]\ (0.38)$
Plug-in (judge)	$3.34\,[0.09,123.13]\ (0.69)$	$2.57\,[0.46,14.27]\ (0.44)$	$2.25\,[0.97,5.22]\ (0.17)$
Plug-in (multi)	$2.82\,[0.78,10.25]\ (0.81)$	$2.43\,[0.92,6.41]\ (0.47)$	$1.77\,[0.91,3.45]\ (0.21)$
Aug. plug-in	$0.86\,[0.56,1.32]\ (2.68)$	$1.04\,[0.74,1.46]\ (1.09)$	$1.09\,[0.73,1.63]\ (0.35)$
Global CV	$1.38\,[1.27,1.50]\ (1.67)$	$1.45\,[1.28,1.65]\ (0.78)$	$1.53\,[1.29,1.82]\ (0.25)$
Per-signal CV	$0.71\,[0.64,0.79]\ (3.22)$	$1.18\,[1.09,1.28]\ (0.96)$	$1.15\,[1.07,1.23]\ (0.33)$

Appendix

Table 12: RE

[95\%\ \mathrm{MC\ CI}]

on ScienceQA (Ministral 3B, continuous

Z

B=100

); mean MSE

\times 10^{3}

Estimator	$n_{\mathrm{lab}}=50$	$n_{\mathrm{lab}}=100$	$n_{\mathrm{lab}}=200$
Naive	$1.00\,[1.00,1.00]\ (18.17)$	$1.00\,[1.00,1.00]\ (6.65)$	$1.00\,[1.00,1.00]\ (3.20)$
Plug-in (judge)	$3.36\,[1.83,6.17]\ (5.40)$	$2.33\,[1.67,3.25]\ (2.86)$	$1.81\,[1.50,2.18]\ (1.77)$
Plug-in (multi)	$4.19\,[1.62,10.81]\ (4.34)$	$2.77\,[1.89,4.06]\ (2.40)$	$2.38\,[1.90,2.98]\ (1.34)$
Aug. plug-in	$0.69\,[0.31,1.54]\ (26.32)$	$0.97\,[0.61,1.54]\ (6.83)$	$1.34\,[1.01,1.78]\ (2.38)$
Global CV	$1.20\,[1.10,1.30]\ (15.15)$	$1.28\,[0.68,2.40]\ (5.18)$	$1.41\,[1.21,1.65]\ (2.27)$
Per-signal CV	$0.84\,[0.78,0.91]\ (21.52)$	$0.99\,[0.91,1.07]\ (6.72)$	$1.09\,[1.03,1.16]\ (2.94)$

Appendix

Table 13: RE

[95\%\ \mathrm{MC\ CI}]

on MMLU (Ministral 3B, continuous

Z

B=100

); mean MSE

\times 10^{3}

Estimator	$n_{\mathrm{lab}}=50$	$n_{\mathrm{lab}}=100$	$n_{\mathrm{lab}}=200$
Naive	$1.00\,[1.00,1.00]\ (14.72)$	$1.00\,[1.00,1.00]\ (6.86)$	$1.00\,[1.00,1.00]\ (2.40)$
Plug-in (judge)	$3.72\,[2.60,5.33]\ (3.95)$	$3.83\,[2.18,6.73]\ (1.79)$	$2.91\,[2.30,3.68]\ (0.83)$
Plug-in (multi)	$3.81\,[2.76,5.26]\ (3.86)$	$4.15\,[2.54,6.77]\ (1.65)$	$2.80\,[2.14,3.66]\ (0.86)$
Aug. plug-in	$1.33\,[1.00,1.76]\ (11.07)$	$1.90\,[1.38,2.62]\ (3.61)$	$2.22\,[1.82,2.71]\ (1.08)$
Global CV	$1.84\,[1.47,2.31]\ (7.98)$	$2.20\,[1.81,2.68]\ (3.12)$	$2.20\,[1.81,2.67]\ (1.09)$
Per-signal CV	$1.11\,[1.05,1.17]\ (13.27)$	$1.21\,[1.14,1.28]\ (5.67)$	$1.27\,[1.21,1.33]\ (1.89)$

Appendix

Table 14: RE

[95\%\ \mathrm{MC\ CI}]

on WinoGrande (Ministral 3B, continuous

Z

B=100

); mean MSE

\times 10^{3}

Estimator	$n_{\mathrm{lab}}=50$	$n_{\mathrm{lab}}=100$	$n_{\mathrm{lab}}=200$
Naive	$1.00\,[1.00,1.00]\ (9.73)$	$1.00\,[1.00,1.00]\ (5.25)$	$1.00\,[1.00,1.00]\ (1.62)$
Plug-in (judge)	$5.02\,[2.53,9.97]\ (1.94)$	$5.99\,[2.54,14.12]\ (0.88)$	$4.74\,[2.91,7.72]\ (0.34)$
Plug-in (multi)	$3.98\,[1.64,9.65]\ (2.45)$	$4.24\,[2.05,8.75]\ (1.24)$	$3.05\,[1.87,4.99]\ (0.53)$
Aug. plug-in	$1.93\,[0.84,4.45]\ (5.05)$	$3.33\,[1.40,7.93]\ (1.58)$	$3.29\,[0.79,13.64]\ (0.49)$
Global CV	$3.02\,[2.19,4.16]\ (3.23)$	$4.15\,[3.28,5.25]\ (1.26)$	$3.76\,[2.17,6.51]\ (0.43)$
Per-signal CV	$1.36\,[1.25,1.48]\ (7.15)$	$1.53\,[1.44,1.63]\ (3.43)$	$1.49\,[1.39,1.60]\ (1.09)$

Appendix

Table 15: RE

[95\%\ \mathrm{MC\ CI}]

on HellaSwag (Ministral 3B, continuous

Z

B=100

); mean MSE

\times 10^{3}

Estimator	$n_{\mathrm{lab}}=50$	$n_{\mathrm{lab}}=100$	$n_{\mathrm{lab}}=200$
Naive	$1.00\,[1.00,1.00]\ (7.21)$	$1.00\,[1.00,1.00]\ (4.24)$	$1.00\,[1.00,1.00]\ (1.85)$
Plug-in (judge)	$3.17\,[1.90,5.30]\ (2.28)$	$4.49\,[3.27,6.17]\ (0.95)$	$3.64\,[2.86,4.63]\ (0.51)$
Plug-in (multi)	$2.68\,[1.49,4.83]\ (2.70)$	$4.48\,[2.53,7.92]\ (0.95)$	$3.35\,[2.14,5.24]\ (0.55)$
Aug. plug-in	$1.08\,[0.50,2.35]\ (6.67)$	$1.03\,[0.57,1.86]\ (4.13)$	$1.76\,[1.40,2.21]\ (1.05)$
Global CV	$1.34\,[0.75,2.39]\ (5.37)$	$1.51\,[1.11,2.05]\ (2.81)$	$1.72\,[1.21,2.44]\ (1.07)$
Per-signal CV	$0.90\,[0.85,0.96]\ (8.01)$	$1.08\,[1.02,1.14]\ (3.92)$	$1.09\,[1.04,1.14]\ (1.70)$

Appendix

Table 16: RE

[95\%\ \mathrm{MC\ CI}]

on TruthfulQA (Ministral 3B, continuous

Z

B=100

); mean MSE

\times 10^{3}

Estimator	$n_{\mathrm{lab}}=50$	$n_{\mathrm{lab}}=100$	$n_{\mathrm{lab}}=200$
Naive	$1.00\,[1.00,1.00]\ (4.68)$	$1.00\,[1.00,1.00]\ (2.03)$	$1.00\,[1.00,1.00]\ (0.70)$
Plug-in (judge)	$2.77\,[0.30,25.82]\ (1.69)$	$3.03\,[0.87,10.55]\ (0.67)$	$1.95\,[0.59,6.45]\ (0.36)$
Plug-in (multi)	$2.82\,[1.62,4.92]\ (1.66)$	$3.09\,[0.99,9.64]\ (0.66)$	$2.36\,[1.05,5.29]\ (0.30)$
Aug. plug-in	$0.77\,[0.36,1.64]\ (6.06)$	$0.86\,[0.55,1.36]\ (2.37)$	$1.20\,[0.76,1.91]\ (0.58)$
Global CV	$1.19\,[1.10,1.29]\ (3.94)$	$1.31\,[1.18,1.45]\ (1.54)$	$1.47\,[1.19,1.81]\ (0.47)$
Per-signal CV	$0.83\,[0.77,0.90]\ (5.61)$	$1.12\,[1.00,1.25]\ (1.81)$	$1.16\,[1.05,1.28]\ (0.61)$

Appendix

Table 17: RE

[95\%\ \mathrm{MC\ CI}]

on GSM8K (Ministral 3B, continuous

Z

B=100

); mean MSE

\times 10^{3}

Estimator	$n_{\mathrm{lab}}=50$	$n_{\mathrm{lab}}=100$	$n_{\mathrm{lab}}=200$
Naive	$1.00\,[1.00,1.00]\ (4.44)$	$1.00\,[1.00,1.00]\ (1.82)$	$1.00\,[1.00,1.00]\ (0.65)$
Plug-in (judge)	$5.14\,[2.38,11.08]\ (0.86)$	$4.48\,[2.81,7.15]\ (0.41)$	$3.89\,[2.66,5.70]\ (0.17)$
Plug-in (multi)	$5.79\,[0.38,87.23]\ (0.77)$	$5.09\,[2.28,11.39]\ (0.36)$	$4.73\,[1.46,15.31]\ (0.14)$
Aug. plug-in	$1.28\,[0.41,4.03]\ (3.48)$	$1.49\,[0.43,5.19]\ (1.23)$	$1.67\,[0.91,3.06]\ (0.39)$
Global CV	$1.32\,[1.21,1.44]\ (3.37)$	$1.47\,[1.26,1.72]\ (1.24)$	$1.76\,[1.21,2.57]\ (0.37)$
Per-signal CV	$0.90\,[0.83,0.98]\ (4.95)$	$1.06\,[0.98,1.15]\ (1.72)$	$1.18\,[1.08,1.29]\ (0.55)$

Appendix

Table 18: RE

[95\%\ \mathrm{MC\ CI}]

on ARC (Ministral 3B, continuous

Z

B=100

); mean MSE

\times 10^{3}

Estimator	$n_{\mathrm{lab}}=50$	$n_{\mathrm{lab}}=100$	$n_{\mathrm{lab}}=200$
Naive	$1.00\,[1.00,1.00]\ (6.14)$	$1.00\,[1.00,1.00]\ (2.66)$	$1.00\,[1.00,1.00]\ (0.98)$
Plug-in (judge)	$2.42\,[1.32,4.44]\ (2.54)$	$2.57\,[1.81,3.66]\ (1.03)$	$2.01\,[1.61,2.50]\ (0.49)$
Plug-in (multi)	$2.49\,[1.30,4.76]\ (2.46)$	$2.71\,[1.84,3.98]\ (0.98)$	$1.94\,[1.51,2.49]\ (0.50)$
Aug. plug-in	$1.32\,[0.94,1.86]\ (4.64)$	$1.37\,[1.04,1.81]\ (1.94)$	$1.47\,[1.17,1.84]\ (0.67)$
Global CV	$1.49\,[1.26,1.76]\ (4.13)$	$1.50\,[1.32,1.71]\ (1.78)$	$1.54\,[1.34,1.77]\ (0.63)$
Per-signal CV	$0.89\,[0.82,0.97]\ (6.90)$	$1.10\,[1.04,1.16]\ (2.42)$	$1.11\,[1.05,1.17]\ (0.88)$

Appendix

Table 19: RE

[95\%\ \mathrm{MC\ CI}]

on MATH-500 (Qwen3 32B, continuous

Z

B=100

); mean MSE

\times 10^{3}

Estimator	$n_{\mathrm{lab}}=50$	$n_{\mathrm{lab}}=100$	$n_{\mathrm{lab}}=200$
Naive	$1.00\,[1.00,1.00]\ (0.92)$	$1.00\,[1.00,1.00]\ (0.42)$	$1.00\,[1.00,1.00]\ (0.16)$
Plug-in (judge)	$3.07\,[1.25,7.56]\ (0.30)$	$2.34\,[1.54,3.56]\ (0.18)$	$1.30\,[0.91,1.85]\ (0.13)$
Plug-in (multi)	$2.78\,[1.53,5.07]\ (0.33)$	$1.43\,[0.79,2.59]\ (0.29)$	$0.76\,[0.52,1.11]\ (0.22)$
Aug. plug-in	$0.83\,[0.62,1.10]\ (1.12)$	$0.77\,[0.56,1.05]\ (0.55)$	$0.88\,[0.59,1.32]\ (0.18)$
Global CV	$1.19\,[1.16,1.22]\ (0.78)$	$1.34\,[1.21,1.48]\ (0.31)$	$1.44\,[1.15,1.80]\ (0.11)$
Per-signal CV	$0.24\,[0.17,0.33]\ (3.90)$	$0.85\,[0.78,0.93]\ (0.49)$	$1.11\,[1.05,1.18]\ (0.15)$

Appendix

Table 20: RE

[95\%\ \mathrm{MC\ CI}]

on ScienceQA (Qwen3 32B, continuous

Z

B=100

); mean MSE

\times 10^{3}

Estimator	$n_{\mathrm{lab}}=50$	$n_{\mathrm{lab}}=100$	$n_{\mathrm{lab}}=200$
Naive	$1.00\,[1.00,1.00]\ (17.93)$	$1.00\,[1.00,1.00]\ (5.00)$	$1.00\,[1.00,1.00]\ (2.02)$
Plug-in (judge)	$2.09\,[0.59,7.46]\ (8.57)$	$1.20\,[0.76,1.90]\ (4.17)$	$0.61\,[0.45,0.84]\ (3.31)$
Plug-in (multi)	$2.48\,[0.96,6.39]\ (7.22)$	$1.52\,[1.07,2.16]\ (3.30)$	$0.73\,[0.56,0.95]\ (2.77)$
Aug. plug-in	$0.89\,[0.01,137.87]\ (20.25)$	$0.78\,[0.45,1.34]\ (6.38)$	$1.11\,[0.66,1.86]\ (1.81)$
Global CV	$1.17\,[1.11,1.24]\ (15.28)$	$1.19\,[1.08,1.31]\ (4.22)$	$1.18\,[1.10,1.27]\ (1.71)$
Per-signal CV	$0.97\,[0.92,1.03]\ (18.43)$	$1.02\,[0.97,1.07]\ (4.90)$	$1.03\,[0.97,1.09]\ (1.96)$

Appendix

Table 21: RE

[95\%\ \mathrm{MC\ CI}]

on MMLU (Qwen3 32B, continuous

Z

B=100

); mean MSE

\times 10^{3}

Estimator	$n_{\mathrm{lab}}=50$	$n_{\mathrm{lab}}=100$	$n_{\mathrm{lab}}=200$
Naive	$1.00\,[1.00,1.00]\ (10.35)$	$1.00\,[1.00,1.00]\ (4.59)$	$1.00\,[1.00,1.00]\ (1.80)$
Plug-in (judge)	$5.03\,[1.94,13.05]\ (2.06)$	$3.54\,[1.84,6.79]\ (1.30)$	$3.69\,[1.95,6.99]\ (0.49)$
Plug-in (multi)	$4.92\,[0.81,29.84]\ (2.10)$	$3.68\,[2.34,5.78]\ (1.25)$	$3.14\,[1.78,5.54]\ (0.57)$
Aug. plug-in	$1.42\,[0.87,2.31]\ (7.27)$	$1.69\,[1.23,2.31]\ (2.72)$	$1.98\,[1.46,2.68]\ (0.91)$
Global CV	$1.66\,[1.36,2.02]\ (6.24)$	$2.04\,[1.65,2.53]\ (2.25)$	$2.08\,[1.43,3.03]\ (0.86)$
Per-signal CV	$1.02\,[0.97,1.08]\ (10.14)$	$1.11\,[1.07,1.15]\ (4.13)$	$1.16\,[1.11,1.21]\ (1.55)$

Appendix

Table 22: RE

[95\%\ \mathrm{MC\ CI}]

on WinoGrande (Qwen3 32B, continuous

Z

B=100

); mean MSE

\times 10^{3}

Estimator	$n_{\mathrm{lab}}=50$	$n_{\mathrm{lab}}=100$	$n_{\mathrm{lab}}=200$
Naive	$1.00\,[1.00,1.00]\ (8.01)$	$1.00\,[1.00,1.00]\ (4.57)$	$1.00\,[1.00,1.00]\ (1.47)$
Plug-in (judge)	$1.57\,[1.07,2.31]\ (5.11)$	$1.44\,[1.13,1.84]\ (3.17)$	$1.30\,[1.09,1.55]\ (1.13)$
Plug-in (multi)	$1.30\,[0.85,1.98]\ (6.16)$	$1.23\,[0.85,1.78]\ (3.73)$	$1.02\,[0.80,1.31]\ (1.44)$
Aug. plug-in	$0.61\,[0.48,0.78]\ (13.11)$	$0.78\,[0.61,1.00]\ (5.90)$	$0.95\,[0.82,1.10]\ (1.54)$
Global CV	$1.03\,[1.02,1.04]\ (7.79)$	$1.04\,[1.02,1.06]\ (4.39)$	$1.07\,[1.03,1.12]\ (1.37)$
Per-signal CV	$0.93\,[0.89,0.97]\ (8.66)$	$1.00\,[0.97,1.03]\ (4.59)$	$1.01\,[0.99,1.03]\ (1.45)$

Appendix

Table 23: RE

[95\%\ \mathrm{MC\ CI}]

on HellaSwag (Qwen3 32B, continuous

Z

B=100

); mean MSE

\times 10^{3}

Estimator	$n_{\mathrm{lab}}=50$	$n_{\mathrm{lab}}=100$	$n_{\mathrm{lab}}=200$
Naive	$1.00\,[1.00,1.00]\ (22.10)$	$1.00\,[1.00,1.00]\ (9.79)$	$1.00\,[1.00,1.00]\ (3.81)$
Plug-in (judge)	$3.85\,[2.40,6.18]\ (5.74)$	$3.46\,[2.12,5.66]\ (2.83)$	$1.39\,[0.77,2.52]\ (2.75)$
Plug-in (multi)	$4.87\,[1.19,19.93]\ (4.54)$	$4.38\,[2.69,7.14]\ (2.24)$	$1.74\,[1.10,2.76]\ (2.19)$
Aug. plug-in	$1.89\,[0.89,4.02]\ (11.68)$	$2.47\,[1.29,4.71]\ (3.97)$	$4.79\,[2.52,9.09]\ (0.80)$
Global CV	$2.16\,[1.84,2.54]\ (10.22)$	$2.00\,[1.70,2.35]\ (4.89)$	$3.81\,[2.31,6.30]\ (1.00)$
Per-signal CV	$1.02\,[0.95,1.10]\ (21.74)$	$1.21\,[1.13,1.29]\ (8.09)$	$1.32\,[1.24,1.41]\ (2.89)$

Appendix

Table 24: RE

[95\%\ \mathrm{MC\ CI}]

on TruthfulQA (Qwen3 32B, continuous

Z

B=100

); mean MSE

\times 10^{3}

Estimator	$n_{\mathrm{lab}}=50$	$n_{\mathrm{lab}}=100$	$n_{\mathrm{lab}}=200$
Naive	$1.00\,[1.00,1.00]\ (2.19)$	$1.00\,[1.00,1.00]\ (1.17)$	$1.00\,[1.00,1.00]\ (0.34)$
Plug-in (judge)	$2.49\,[1.55,4.01]\ (0.88)$	$2.29\,[1.71,3.08]\ (0.51)$	$1.11\,[0.84,1.47]\ (0.31)$
Plug-in (multi)	$3.90\,[1.18,12.90]\ (0.56)$	$3.29\,[1.77,6.11]\ (0.35)$	$1.37\,[0.79,2.38]\ (0.25)$
Aug. plug-in	$0.83\,[0.44,1.57]\ (2.65)$	$0.84\,[0.41,1.73]\ (1.38)$	$0.79\,[0.07,8.45]\ (0.43)$
Global CV	$1.24\,[1.15,1.34]\ (1.76)$	$1.27\,[1.17,1.38]\ (0.92)$	$1.22\,[0.89,1.67]\ (0.28)$
Per-signal CV	$0.56\,[0.50,0.62]\ (3.89)$	$1.07\,[1.01,1.13]\ (1.09)$	$1.02\,[0.96,1.09]\ (0.33)$

Appendix

Table 25: RE

[95\%\ \mathrm{MC\ CI}]

on GSM8K (Qwen3 32B, continuous

Z

B=100

); mean MSE

\times 10^{3}

Estimator	$n_{\mathrm{lab}}=50$	$n_{\mathrm{lab}}=100$	$n_{\mathrm{lab}}=200$
Naive	$1.00\,[1.00,1.00]\ (1.85)$	$1.00\,[1.00,1.00]\ (0.81)$	$1.00\,[1.00,1.00]\ (0.26)$
Plug-in (judge)	$4.41\,[2.91,6.69]\ (0.42)$	$4.00\,[2.43,6.59]\ (0.20)$	$2.30\,[1.81,2.93]\ (0.11)$
Plug-in (multi)	$3.65\,[2.64,5.04]\ (0.51)$	$3.28\,[2.10,5.12]\ (0.25)$	$1.60\,[1.05,2.44]\ (0.16)$
Aug. plug-in	$0.75\,[0.45,1.26]\ (2.47)$	$0.84\,[0.56,1.25]\ (0.97)$	$0.82\,[0.54,1.24]\ (0.32)$
Global CV	$1.26\,[0.93,1.70]\ (1.47)$	$1.42\,[1.27,1.58]\ (0.57)$	$1.21\,[1.06,1.39]\ (0.21)$
Per-signal CV	$0.79\,[0.74,0.85]\ (2.34)$	$1.03\,[0.96,1.10]\ (0.79)$	$1.04\,[1.00,1.08]\ (0.25)$

Appendix

Table 26: RE

[95\%\ \mathrm{MC\ CI}]

on ARC (Qwen3 32B, continuous

Z

B=100

); mean MSE

\times 10^{3}

Panel of Figure 1 — Figure 1: Pointwise MSE ( $\times 10^{3}$ ) for Claude Haiku 3. Each row: one dataset (label on left); columns are $n=50,100,200$ .

Panel of Figure 2 — Figure 2: Pointwise MSE ( $\times 10^{3}$ ) for Ministral 3B. Each row: one dataset (label on left), columns: $n=50,100,200$ .

Panel of Figure 3 — Figure 3: Pointwise MSE ( $\times 10^{3}$ ) for Qwen3 32B. Each row: one dataset (label on left), columns: $n=50,100,200$ .

Method	$n{=}50$	$n{=}100$	$n{=}200$	Overall
Naive	$1.000\,[1.000,1.000]$	$1.000\,[1.000,1.000]$	$1.000\,[1.000,1.000]$	$1.000\,[1.000,1.000]$
Plug-in (judge)	$2.960\,[2.780,3.150]$	$2.650\,[2.510,2.800]$	$2.030\,[1.930,2.140]$	$2.516\,[2.392,2.646]$
Plug-in (multi)	$3.050\,[2.860,3.250]$	$2.720\,[2.580,2.870]$	$1.950\,[1.840,2.070]$	$2.529\,[2.419,2.644]$
Aug. plug-in	$1.030\,[0.981,1.082]$	$1.190\,[1.136,1.247]$	$1.460\,[1.381,1.543]$	$1.214\,[1.174,1.256]$
Global CV	$1.370\,[1.305,1.438]$	$1.490\,[1.421,1.562]$	$1.617\,[1.555,1.681]$	$1.489\,[1.438,1.542]$
Per-signal CV	$0.930\,[0.887,0.975]$	$1.130\,[1.104,1.157]$	$1.205\,[1.183,1.228]$	$1.083\,[1.059,1.107]$

Appendix

Table 28: Within-

L

three-fold cross-validation. Entries are geometric-mean RE

[95\%\ \mathrm{CI}]

across the 24 benchmark–model cells; intervals use the

B=100

matched label splits, and the Overall column treats the three nested budgets within each cell jointly, with the label-split permutation as the joint resampling unit, as in Table 2 . The fixed-hyperparameter counterparts are the corresponding rows of Table 2 . Target: realized full-pool gold kernel profile; randomness: matched

L

-draws conditional on

T

Method	Opus 4.6 judge + anchor	Llama-3.1-8B judge + anchor	GPT-OSS-20B judge + anchor
Naive	$1.000\,[1.000,1.000]$	$1.000\,[1.000,1.000]$	$1.000\,[1.000,1.000]$
Plug-in (judge)	$2.630\,[2.270,3.047]$	$1.574\,[1.530,1.619]$	$1.896\,[1.843,1.950]$
Plug-in (multi)	$2.647\,[2.277,3.078]$	$1.899\,[1.835,1.966]$	$2.281\,[2.202,2.363]$
Aug. plug-in	$1.250\,[1.098,1.423]$	$1.061\,[1.030,1.093]$	$1.167\,[1.132,1.203]$
Global CV	$1.566\,[1.516,1.618]$	$1.254\,[1.223,1.286]$	$1.423\,[1.386,1.461]$
Per-signal CV	$1.054\,[1.042,1.066]$	$0.942\,[0.926,0.958]$	$1.003\,[0.985,1.021]$

Appendix

Table 29: Performance with jointly weaker judge and anchor models. Entries are overall geometric-mean RE

[95\%\ \mathrm{CI}]

across 72 cells; as in the Overall column of Table 2 , the three nested budgets within each cell are treated jointly, with the label-split permutation as the joint resampling unit. Target: realized full-pool gold kernel profile; randomness: matched

L

-draws conditional on

T

Benchmark / candidate	Jittered Gaussian	Triangular ordinal	Within group
MATH-500 / Claude Haiku 3	6.14/5.78/6.07	4.82/4.38/4.41	1.75/3.73/7.27
MATH-500 / Ministral 3B	4.19/4.73/5.20	2.84/3.72/4.16	1.36/1.76/3.23
MATH-500 / Qwen3 32B	5.47/5.38/4.32	3.27/3.99/3.17	2.10/3.35/3.63
ScienceQA / Claude Haiku 3	5.96/5.70/5.13	1.47/2.00/2.99	0.65/1.05/1.72
ScienceQA / Ministral 3B	5.72/6.56/5.67	1.78/2.72/3.14	0.61/0.70/1.59
ScienceQA / Qwen3 32B	5.71/5.90/5.12	1.22/2.49/2.70	0.34/0.52/1.45

Appendix

Table 30: Ordered-profile kernel ablation. Entries are LACE RE at

n_{\mathrm{lab}}=50/100/200

under jittered Gaussian smoothing, triangular ordinal smoothing, and exact within-group estimation. The jittered-Gaussian column is the main experimental protocol and reports the same runs as Tables 3–26; the remaining two columns rerun the estimator under the alternative locality definitions. Each estimator is compared with the naive estimator for its own matched full-pool target; the table therefore tests persistence of the efficiency gain rather than ranking targets or kernels. If a compact-support method has no labeled mass at a target level in a replication, that level is omitted for that replication.

Estimator	$n_{\mathrm{lab}}{=}500$	$n_{\mathrm{lab}}{=}1{,}000$	$n_{\mathrm{lab}}{=}1{,}500$
Naive	$1.00\times$ ( $2.09$ )	$1.00\times$ ( $1.20$ )	$1.00\times$ ( $0.79$ )
Plug-in (judge-only)	$0.17\times$ ( $12.17$ )	$0.10\times$ ( $11.74$ )	$0.07\times$ ( $11.74$ )
Plug-in (multivariate)	$0.14\times$ ( $14.92$ )	$0.08\times$ ( $14.62$ )	$0.05\times$ ( $14.62$ )
Augmented plug-in	$1.33\times$ ( $1.57$ )	$1.43\times$ ( $0.84$ )	$1.42\times$ ( $0.56$ )
Global CV	$1.32\times$ ( $1.58$ )	$1.34\times$ ( $0.89$ )	$1.31\times$ ( $0.61$ )
Per-signal CV ( $S_{1}$ )	$1.29\times$ ( $1.62$ )	$1.36\times$ ( $0.88$ )	$1.27\times$ ( $0.62$ )

Appendix

Table 31: Setting B (ordered discrete): conditional-on-design relative efficiency over

B=300

replications, with

\overline{\mathrm{MSE}}\times 10^{3}

in parentheses. Ridge-regularized estimators use the prespecified

\lambda=0.3

; bold marks the strongest feasible estimator. MSE is evaluated against the fixed conditional design profile

\theta(g)

Estimator	$n_{\mathrm{lab}}{=}500$	$n_{\mathrm{lab}}{=}1{,}000$	$n_{\mathrm{lab}}{=}1{,}500$
Naive	$1.00\times$ ( $4.77$ )	$1.00\times$ ( $2.17$ )	$1.00\times$ ( $1.51$ )
Plug-in (judge-only)	$0.16\times$ ( $28.93$ )	$0.08\times$ ( $28.16$ )	$0.05\times$ ( $28.24$ )
Plug-in (multivariate)	$0.24\times$ ( $19.55$ )	$0.12\times$ ( $18.51$ )	$0.08\times$ ( $18.48$ )
Augmented plug-in	$1.28\times$ ( $3.72$ )	$1.25\times$ ( $1.73$ )	$1.28\times$ ( $1.18$ )
Global CV	$1.31\times$ ( $3.65$ )	$1.25\times$ ( $1.74$ )	$1.26\times$ ( $1.19$ )
Per-signal CV ( $S_{1}$ )	$1.43\times$ ( $3.33$ )	$1.40\times$ ( $1.55$ )	$1.36\times$ ( $1.11$ )

Appendix

Table 32: Setting C (unordered categorical): conditional-on-design relative efficiency over

B=300

replications, with

\overline{\mathrm{MSE}}\times 10^{3}

in parentheses. Ridge-regularized estimators use the prespecified

\lambda=0.3

; bold marks the strongest feasible estimator. MSE is evaluated against the fixed conditional design profile

\theta(g)

Label fraction $\pi$	Theorem 4 prediction	Empirical RE (oracle) $h=M^{-1/3}$ ( $95\%$ CI)	Empirical RE (feasible) $h=M^{-1/3}$ ( $95\%$ CI)	Empirical RE (oracle) fixed $h$ ( $95\%$ CI)
0.05	3.245	3.216 $(3.101,3.332)$	3.159 $(3.045,3.272)$	2.332 $(2.194,2.470)$
0.10	2.902	2.916 $(2.818,3.014)$	2.904 $(2.807,3.002)$	2.197 $(2.070,2.324)$
0.15	2.625	2.610 $(2.522,2.698)$	2.601 $(2.512,2.689)$	1.994 $(1.878,2.109)$

Appendix

Table 33: Theorem 4 in its asymptotic regime, at

M=320{,}000

(

h=0.0146

). Entries are profile-level RE with paired Monte Carlo

95\%

CIs. MSE is evaluated against the population profile

\theta_{0}(z)=1/2

. The feasible column uses

b=h

and

\lambda=1/(nh)

Figure 4: Pointwise gain at

M=320{,}000

for

\pi\in\{0.05,0.10,0.15\}

. Each panel plots, across the

17

evaluation points, the Theorem 4 prediction

1/\{1-(1-\pi)R^{2}(z)\}

(dashed, computed from the DGP’s analytically known population

R^{2}(z)

), the oracle-coefficient empirical RE under the shrinking bandwidth

h=M^{-1/3}

(circles), and the feasible estimator with vanishing ridge

\lambda=1/(nh)

(squares); shaded regions are paired Monte Carlo

95\%

intervals. The fixed-bandwidth control

h=0.12

(dotted) stays below the prediction throughout. The oracle intervals contain the prediction at

15/17

17/17

, and

16/17

points and the feasible intervals at

15/17

16/17

, and

16/17

points.

Method	RE $n{=}50$	RE $n{=}100$	RE $n{=}200$	Overall	Best
Naive	$1.000\,[1.000,1.000]$	$1.000\,[1.000,1.000]$	$1.000\,[1.000,1.000]$	$1.000\,[1.000,1.000]$	0/72
LACE	$\mathbf{2.235}\,[2.212,2.260]$	$\mathbf{2.328}\,[2.306,2.353]$	$\mathbf{1.849}\,[1.833,1.867]$	$\mathbf{2.127}\,[2.112,2.144]$	72/72
Global CV	$0.953\,[0.948,0.958]$	$0.974\,[0.971,0.978]$	$0.990\,[0.988,0.992]$	$0.972\,[0.970,0.975]$	0/72
Per-signal CV	$1.168\,[1.164,1.173]$	$1.178\,[1.174,1.182]$	$1.138\,[1.134,1.141]$	$1.161\,[1.158,1.164]$	0/72
Residual-only	$0.824\,[0.812,0.837]$	$0.579\,[0.569,0.587]$	$0.354\,[0.349,0.360]$	$0.553\,[0.546,0.559]$	0/72
Plug-in (judge)	$1.063\,[1.053,1.073]$	$0.758\,[0.750,0.765]$	$0.493\,[0.488,0.498]$	$0.735\,[0.729,0.741]$	0/72

Appendix

Table 34: Repeated-pool superpopulation risk over

24

DGP settings formed by eight benchmark-derived profile marginals and three candidate-indexed outcome regimes (

B=500

M_{n}=10n

, and labeled fraction

\pi=0.1

). Budget columns are geometric-mean RE across the

24

settings and Overall is the geometric mean across all

72

setting–budget combinations. Brackets are paired Monte Carlo

95\%

percentile intervals from

5{,}000

bootstrap resamples of the

B=500

replications; “Best” counts setting–budget combinations where the method is top. MSE is evaluated against the population target

\theta_{0,c}(z)=p_{m}(z)

Signal	Description	API calls	Range
$S^{\mathrm{judge}}$	Mean of 3 pointwise judge scores	3	$[0,1]$
$S^{\mathrm{disagree}}$	Variance of 3 pointwise judge scores	0 (derived)	$[0,2/9]$
$S^{\mathrm{pair},1}$	Pairwise vs. weak anchor (Ministral 3B)	2	$\{0,0.25,0.5,0.75,1\}$
$S^{\mathrm{pair},2}$	Pairwise vs. medium anchor (Haiku 3)	2	$\{0,0.25,0.5,0.75,1\}$
$S^{\mathrm{pair},3}$	Pairwise vs. strong anchor (Opus 4.6)	2	$\{0,0.25,0.5,0.75,1\}$
$S^{\mathrm{conf}}$	Self-reported confidence	0 additional	$[0,1]$

Appendix

Table 35: Summary of the

K=6

cheap evaluator signals collected per item.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

Oct 6, 2026

Conditional Accuracy Profiles: Diagnosing LLM Judges across Deployment Conditions

LLM-as-judge is now a standard tool for scalable evaluation, but judge performance is still often summarized by a single accuracy number. This aggregate view hides the deployment conditions under which a judge succeeds or fails. We introduce Conditional Accuracy Profiling (CAP), a post-hoc diagnostic framework that decomposes pairwise LLM-judge accuracy into eight conditions organized into content sensitivity, robustness, and rationale quality. CAP is benchmark-agnostic: it can be applied directly when a benchmark provides the required annotations, approximately through task-subset proxies, or through controlled augmentation when perturbation pairs can be generated. We instantiate CAP on seven LLM judges across six pairwise judging benchmarks, including judgerEva-Standard, a controlled testbed we created to support all eight conditions. CAP exposes profile differences hidden by aggregate accuracy: on judgerEva's judge-independent Hard-Constructed subset, the two judges most sensitive to omitted qualifications rank in the bottom three of seven by overall accuracy, so omission sensitivity is not predicted by aggregate accuracy. Across benchmarks, Position Robustness shows the strongest rank stability (mean Spearman

\barρ{=}0.87

) but is itself fragile under JudgeBench-Pro adversarial stress, showing the largest mean accuracy drop among the shared conditions, though the dominant degradation channel varies by judge. Condition-level profiles provide a more actionable basis than aggregate accuracy for selecting LLM judges.

Wenqi Li, Bin Liu, Mindi Ruan +3

University at Albany, State University of New York · West Virginia University

LLM Evaluation LLM-as-a-Judge Language Model Robustness

Ask PDF

Feb 9, 2026

CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation

LLM-as-a-judge ensembles are the standard paradigm for scalable evaluation, but their aggregation mechanisms suffer from a fundamental flaw: they implicitly assume that judges provide independent estimates of true quality. However, in practice, LLM judges exhibit correlated errors caused by shared latent confounders -- such as verbosity, stylistic preferences, or training artifacts -- causing standard aggregation rules like majority vote or averaging to provide little gain or even amplify systematic mistakes. To address this, we introduce CARE, a confounder-aware aggregation framework that explicitly models LLM judge scores as arising from both a latent true-quality signal and shared confounding factors. Rather than heuristically re-weighting judges, CARE separates quality from confounders without access to ground-truth labels. We provide theoretical guarantees for identifiability and finite-sample recovery under shared confounders, and we quantify the systematic bias incurred when aggregation models omit confounding latent factors. Across 12 public benchmarks spanning continuous scoring, binary classification, and pairwise preference settings, CARE improves aggregation accuracy, reducing error by up to 26.8%. Code is released in https://github.com/SprocketLab/CARE.

Jitian Zhao, Changho Shin, Tzu-Heng Huang +2

Department of Statistics, University of Wisconsin–Madison, Madison, WI, USA · Department of Computer Sciences, University of Wisconsin–Madison, Madison, WI, USA.

LLM Evaluation LLM-as-a-Judge Language Model Generation Evaluation Language Model Ensembles

Ask PDF

Jun 18, 2026

CELEUS: Certifiable and Efficient LLM Evaluation via E-Processes

Can we trust evaluation scores to capture an LLM's true real-world performance? Certifiable evaluation answers this question by providing guarantee for LLM evaluation. In particular, existing methods sequentially curate evaluation samples and keep updating confidence intervals (CIs) that cover the true performance with high probability (e.g., 95%) until some conditions are satisfied, e.g., the CI width reaches a target precision. However, existing methods are not generally anytime-valid: the claimed coverage (e.g., 95%) may fail when CIs are repeatedly updated and used to decide when to stop, leaving a gap between theoretical rigor and practice. This paper bridges this gap by proposing Celeus, a Certifiable framework for Efficient LLM evaluation, which leverages E-processes to build anytime-valid CIs. Concretely, we propose signals that combine two ingredients: (i) Uncertainty-guided sampling to select informative samples for evaluation, and (ii) Surrogate-assisted approximations for unevaluated samples. We prove that such signals remain unbiased for the evaluation score conditional on the past, enabling statistically-grounded and anytime-valid

e

-process CIs. More importantly, the two ingredients reduce estimation variance and help reach the target precision with fewer evaluated samples. We also prove that CIs obtained by Celeus can shrink at a near-parametric rate up to logarithmic factors and analyze the oracle variance-optimal sampling rule that motivates the empirical uncertainty-guided one. Experiments show that Celeus reaches the target precision using 54-62% fewer evaluated samples than baselines, while preserving anytime-valid coverage.

Zhijian Zhou, Zesheng Ye, Zhaorun Chen +2

The University of Melbourne · Department of Computer Science, University of Chicago · Department of Computer Science, University of Illinois Urbana-Champaign

LLM Evaluation Anytime-Valid Inference Adaptive Sampling Uncertainty Sampling

Ask PDF

Abstract

LLM Evaluation

Explore similar work

Conditional Accuracy Profiles: Diagnosing LLM Judges across Deployment Conditions

CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation

CELEUS: Certifiable and Efficient LLM Evaluation via E-Processes