Localize-Then-Decide Guarantees for LLM Judgments

Xinyu Li; Yi Zhou; Guanqun Cao; Zeyu Fu; Tianjin Huang; Gaojie Jin

cs.CLAug 26, 2026

Localize-Then-Decide Guarantees for LLM Judgments

Authors: Xinyu Li, Yi Zhou, Guanqun Cao, Zeyu Fu, Tianjin Huang, Gaojie Jin

Organizations: University of Exeter · Cardiff University · University of the West of England · University of Macau

Abstract

Large language models (LLMs) are increasingly used as evaluators to assess output quality and preference alignment, yet providing reliable guarantees of agreement with human judgments remains challenging. Recent work introduces confidence-thresholding methods that provide such guarantees for pairwise comparisons, relying on the assumption that higher estimated confidence implies lower disagreement risk with humans. However, this assumption can break down when the number of candidate responses increases, since distributing probability mass across many alternatives can distort confidence estimates. To address this issue, we propose a Localize-Then-Decide framework. First, conformal prediction localizes a small shortlist that contains the human-preferred response with high probability. Then, a calibrated confidence-based rule selectively chooses a single response from this shortlist or abstains. This design restores the monotonic relationship between confidence and disagreement risk and enables high-probability agreement guarantees. Experiments with multiple candidate sizes across several datasets and judge LLMs demonstrate that our framework consistently achieves higher guarantee success rates and substantially higher coverage than single-stage baselines.

Figures & tables

Figure 1: An overview of the Localize-Then-Decide (two-stage) guarantees framework.

Figure 2: Simulations on confidence–agreement monotonicity. (a) As the number of candidates $m$ increases, the monotone relationship between confidence scores and (above confidence) human-agreement rate progressively degrades. (b) Fixing $m{=}20$ : although top- $1$ agreement loses monotonicity, top- $3$ agreement remains monotone with the same estimated confidence. (c) Within the localized (top- $3$ ) shortlist, the four estimated confidences (especially the margin ones) restore monotonicity. Details of the simulations are given in Appendix B .

Dataset	Judge	Direct $m$ -to- $1$			Stage I ( $m$ -to- $k$ )			Stage II ( $k$ -to- $1$ )
Dataset	Judge	$m{=}5$	$m{=}10$	$m{=}20$	$m{=}5$	$m{=}10$	$m{=}20$	$m{=}5$	$m{=}10$	$m{=}20$
TL;DR	Llama-3-8B	0.21	0.32	0.43	0.03	0.05	0.07	0.02	0.04	0.05
	Qwen2.5-72B	0.14	0.24	0.34	0.01	0.02	0.04	0.00	0.01	0.02
	GPT-OSS-120B	0.11	0.20	0.30	0.01	0.02	0.03	0.00	0.01	0.02
Chatbot Arena	Llama-3-8B	0.24	0.35	0.46	0.04	0.06	0.08	0.03	0.05	0.07
	Qwen2.5-72B	0.15	0.26	0.37	0.02	0.03	0.05	0.01	0.02	0.03

Table 1: Ranking loss (

\mathcal{L}_{\mathrm{rank}}

\downarrow

) measuring confidence–agreement monotonicity on real data (

T{=}20

thresholds,

\alpha{=}0.10

). Direct

m

-to-

1

: single-stage selection using top-1 EMP confidence. Stage I (

m

-to-

k

): top-

k

localization agreement using top-1 EMP confidence. Stage II (

k

-to-

1

): selection within the shortlist using KL margin.

Figure 3: Confidence–agreement curves on TL;DR and Qwen2.5-72B. (a) Single-stage $m$ -to- $1$ . (b) Stage I top- $k$ localization. (c) Stage II scoring within the top-3 shortlist ( $m{=}20$ ).

Judge	Method	TL;DR			Chatbot Arena			HH-RLHF			AlpacaEval
Judge	Method	Cov.	GSR	Agr.	Cov.	GSR	Agr.	Cov.	GSR	Agr.	Cov.	GSR	Agr.
Llama-3 8B	No Selection	100%	0%	36% $\pm$ 1.2	100%	0%	31% $\pm$ 1.4	100%	0%	40% $\pm$ 1.1	100%	0%	33% $\pm$ 1.3
	Single-Stage (top-1 EMP)	26% $\pm$ 3.2	53.8%	77% $\pm$ 2.3	21% $\pm$ 3.6	47.6%	74% $\pm$ 2.5	30% $\pm$ 2.9	58.4%	79% $\pm$ 2.0	25% $\pm$ 3.4	52.1%	76% $\pm$ 2.2
	Single-Stage (EMP margin)	31% $\pm$ 3.0	58.6%	78% $\pm$ 2.1	26% $\pm$ 3.4	53.2%	76% $\pm$ 2.3	35% $\pm$ 2.7	63.5%	80% $\pm$ 1.9	28% $\pm$ 3.2	55.8%	78% $\pm$ 2.2
	Single-Stage (KL margin)	35% $\pm$ 2.8	63.1%	80% $\pm$ 2.0	30% $\pm$ 3.1	57.8%	78% $\pm$ 2.1	39% $\pm$ 2.5	68.3%	82% $\pm$ 1.8	33% $\pm$ 3.0	61.7%	79% $\pm$ 2.1
	Single-Stage (Vote)	33% $\pm$ 2.9	60.7%	79% $\pm$ 2.0	28% $\pm$ 3.2	55.4%	77% $\pm$ 2.2	37% $\pm$ 2.7	65.2%	81% $\pm$ 1.8	31% $\pm$ 3.0	59.3%	79% $\pm$ 2.0

Table 2: Two-stage vs. single-stage across judge models and datasets (

m{=}10

, target agreement

=0.81

\delta{=}0.10

). Two-stage:

\varepsilon{=}\alpha{=}0.10

; single-stage:

\varepsilon^{\prime}{=}0.19

. For Two-Stage framework, Stage I always uses EMP scores. Agr. denotes average empirical human-agreement on accepted instances. Cov. and Agr. are reported as mean

\pm

std over 1,000 random splits. We provide more results in Appendix I .

Figure 4: Guarantee and cascade results (TL;DR dataset). (a) Single-model guarantee curve (Qwen2.5-72B, $m{=}10$ ): Two-Stage (blue) consistently meets the target (above diagonal); Single-Stage (red) falls below. Shaded bands show min/max over 1,000 splits. (b) Effect of $m$ on GSR (Qwen2.5-72B): Single-Stage drops from 82% to 50% as $m$ grows; Two-Stage remains $\geq 91\%$ . (c) Cascaded guarantee curve (Llama-8B $\to$ 70B $\to$ GPT-OSS-120B, $m{=}10$ ): empirical agreement tracks the target throughout. (d) Evaluator composition (same cascade as (c)) as fraction of all test instances: at lower targets the weakest tier handles most instances; as the target tightens, more instances escalate to stronger tiers.

Cascade Configuration	Single-Stage		Two-Stage (Ours)
	Cov.	GSR	Cov.	GSR	Comp. (%)
	Cov.	GSR	Cov.	GSR	$T_{1}$	$T_{2}$	$T_{3}$
Qwen2.5-72B only	50% $\pm$ 2.4	72.6%	64% $\pm$ 1.9	95.8%	–	–	100.0
GPT-oss-120B only	56% $\pm$ 2.1	76.4%	70% $\pm$ 1.7	96.8%	–	–	100.0
Qwen-7B $\to$ 32B $\to$ 72B	60% $\pm$ 2.8	65.7%	78% $\pm$ 1.9	93.1%	37.8	35.6	26.5
Llama-8B $\to$ 70B $\to$ GPT-120B	63% $\pm$ 2.5	68.3%	81% $\pm$ 1.8	94.2%	31.9	39.1	29.0

Table 3: Cascaded evaluation on TL;DR (

m{=}10

, target agreement

=0.81

\delta{=}0.10

). Instances abstained by one tier escalate to the next.

T_{i}

denotes the

i

-th model in the cascade (weakest first); “Comp.” shows each tier’s share of the covered instances (summing to 100%). Coverage and GSR are reported as mean

\pm

std over 1,000 random splits.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value	Description
$m$	$\{2,5,10,20\}$	Candidates per instance
$N$	25	Simulated annotators
Instances	$20{,}000$	Per $m$ setting
$q_{\mathrm{gap}}$	$\mathrm{Gamma}(2,0.8)$	Best-candidate advantage
Comp. prob. $\rho$	0.20	Competitive distractor
Non-best $q_{i}$	$-\mathrm{Gamma}(1.5,0.5)$	Weak-candidate quality

Appendix

Table 4: Simulation configuration for Figure 2 .

Figure 5: Confidence distributions for correct vs. wrong predictions (Qwen2.5-72B, TL;DR, $m{=}10$ ). (a) Single-stage direct $m$ -to-1 KL margin: the correct and wrong distributions overlap heavily (median gap $=0.06$ ), making threshold-based selection unreliable. (b) Two-stage KL margin within the shortlist: after Stage I localization, the distributions are well separated (median gap $=0.92$ ), confirming that localization improves the discriminative power of Stage II scoring.

Dataset	Judge	Direct $m$ -to- $1$			Stage I ( $m$ -to- $k$ )			Stage II ( $k$ -to- $1$ )
Dataset	Judge	$m{=}5$	$m{=}10$	$m{=}20$	$m{=}5$	$m{=}10$	$m{=}20$	$m{=}5$	$m{=}10$	$m{=}20$
TL;DR	Mistral-7B	0.22	0.33	0.44	0.03	0.05	0.06	0.02	0.03	0.05
	Qwen2.5-32B	0.17	0.28	0.38	0.02	0.03	0.05	0.01	0.02	0.03
	Llama-3-70B	0.15	0.25	0.35	0.02	0.03	0.04	0.01	0.02	0.03
Chatbot Arena	Mistral-7B	0.25	0.36	0.47	0.04	0.06	0.08	0.03	0.05	0.06
	Qwen2.5-32B	0.19	0.30	0.42	0.03	0.04	0.06	0.02	0.03	0.05

Appendix

Table 5: Ranking loss (additional judges) (

T{=}20

thresholds,

\alpha{=}0.10

). Same format as Table 1 .

Figure 11

Figure 6: Confidence–agreement curves on Chatbot Arena and Qwen2.5-72B. Same layout as Figure 3 . (a) Single-stage $m$ -to- $1$ . (b) Stage I top- $k$ localization. (c) Stage II scoring within the top-3 shortlist ( $m{=}20$ ).

Figure 7: Confidence–agreement curves on HH-RLHF and GPT-OSS-120B. Same layout as Figure 3 . (a) Single-stage $m$ -to- $1$ . (b) Stage I top- $k$ localization. (c) Stage II scoring within the top-3 shortlist ( $m{=}20$ ).

Judge	Method	TL;DR			Chatbot Arena			HH-RLHF			AlpacaEval
Judge	Method	Cov.	GSR	Agr.	Cov.	GSR	Agr.	Cov.	GSR	Agr.	Cov.	GSR	Agr.
Mistral 7B	No Selection	100%	0%	35% $\pm$ 1.3	100%	0%	30% $\pm$ 1.5	100%	0%	39% $\pm$ 1.1	100%	0%	33% $\pm$ 1.4
	Single-Stage (EMP)	25% $\pm$ 3.3	52.4%	77% $\pm$ 2.3	20% $\pm$ 3.7	46.8%	74% $\pm$ 2.5	29% $\pm$ 3.0	57.6%	79% $\pm$ 2.0	23% $\pm$ 3.5	50.3%	76% $\pm$ 2.3
	Single-Stage (EMP margin)	30% $\pm$ 3.1	57.3%	78% $\pm$ 2.1	25% $\pm$ 3.4	51.8%	76% $\pm$ 2.3	34% $\pm$ 2.8	62.7%	80% $\pm$ 1.9	28% $\pm$ 3.2	55.1%	77% $\pm$ 2.2
	Single-Stage (KL margin)	34% $\pm$ 2.9	62.4%	80% $\pm$ 2.0	29% $\pm$ 3.2	56.9%	78% $\pm$ 2.1	38% $\pm$ 2.6	67.8%	82% $\pm$ 1.8	32% $\pm$ 3.0	60.6%	79% $\pm$ 2.1
	Single-Stage (Vote)	32% $\pm$ 3.0	59.8%	79% $\pm$ 2.0	27% $\pm$ 3.3	54.1%	77% $\pm$ 2.2	36% $\pm$ 2.7	64.5%	81% $\pm$ 1.8	30% $\pm$ 3.1	57.9%	78% $\pm$ 2.1

Appendix

Table 6: Two-stage vs. single-stage (additional judges) (

m{=}10

, target agreement

=0.81

\delta{=}0.10

). Same format as Table 2 . Coverage and Agr. are reported as mean

\pm

std over 1,000 random splits.

Dataset	Cascade Configuration	Single-Stage		Two-Stage (Ours)
		Cov.	GSR	Cov.	GSR	Comp. (%)
		Cov.	GSR	Cov.	GSR	$T_{1}$	$T_{2}$	$T_{3}$
Chatbot Arena	Qwen2.5-72B only	44% $\pm$ 2.6	67.4%	58% $\pm$ 2.2	94.6%	–	–	100.0
	Qwen-7B $\to$ 32B $\to$ 72B	55% $\pm$ 3.0	61.8%	74% $\pm$ 2.1	92.3%	38.5	35.2	26.3
	Mistral-7B $\to$ DS-67B $\to$ GPT-120B	53% $\pm$ 3.2	60.4%	73% $\pm$ 2.3	91.7%	35.8	37.9	26.3
	DeepSeek-16B $\to$ Qwen-72B	52% $\pm$ 3.4	59.6%	70% $\pm$ 2.5	90.8%	46.1	53.9	–

Appendix

Table 7: Cascaded evaluation on additional datasets (

m{=}10

, target agreement

=0.81

\delta{=}0.10

). Same format as Table 3 .

(Judge, Dataset, $m$ )	$\lambda{=}0.0$	$\lambda{=}0.2$	$\lambda{=}0.4$	$\lambda{=}0.6$	$\lambda{=}0.8$
Llama-3-8B, TL;DR, $m{=}5$	9.6%	5.8%	3.1%	1.5%	0.6%
Llama-3-8B, Chatbot Arena, $m{=}10$	9.9%	6.5%	3.7%	1.7%	0.7%
Qwen2.5-72B, TL;DR, $m{=}10$	8.4%	5.2%	2.9%	1.3%	0.5%
GPT-OSS-120B, TL;DR, $m{=}10$	7.1%	4.3%	2.2%	0.9%	0.3%
GPT-OSS-120B, Chatbot Arena, $m{=}10$	8.2%	5.0%	2.7%	1.1%	0.4%
GPT-OSS-120B, HH-RLHF, $m{=}10$	6.5%	3.9%	1.9%	0.8%	0.3%

Appendix

Table 8: Direct validation of the cross-stage monotonicity condition. Conditional shortlist miss probability (%) as a function of the Stage II margin threshold.

Method	GSR	Cov.
Jung et al. ( $m{>}2$ adapted) + variance normalization	74.2%	53%
Single-Stage (KL margin)	76.4%	56%
Two-Stage (KL margin)	96.8%	70%

Appendix

Table 9: Strengthened single-stage comparison on TL;DR with GPT-OSS-120B (

m{=}10

, target agreement

=0.81

), averaged over 1,000 splits.

Variant	Renorm.	Exp.	Cond. Avg.	Var. Norm.	Scale	GSR	Cov.
Full KL margin	✓	✓	✓	✓	✓	96.8%	70%
$-$ Renormalization	–	✓	✓	✓	✓	83.5%	57%
$-$ Variance normalization	✓	✓	✓	–	✓	89.2%	61%
$-$ Scale factor	✓	✓	✓	✓	–	91.4%	64%
$-$ Conditional averaging	✓	✓	–	✓	✓	92.6%	65%
$-$ Exponential transform	✓	–	✓	✓	✓	95.1%	68%

Appendix

Table 10: Component-wise KL-margin ablation under the two-stage framework on TL;DR with GPT-OSS-120B (

m{=}10

, target agreement

=0.81

), averaged over 1,000 splits.

Figure 8: Guarantee curve and $m$ -effect on Chatbot Arena (Qwen2.5-72B). Same layout as Figure 4 (a)–(b).

Figure 9: Guarantee curve and $m$ -effect on HH-RLHF (Qwen2.5-72B). Same layout as Figure 4 (a)–(b).

Figure 10: Guarantee curve and $m$ -effect on TL;DR (GPT-OSS-120B). Same layout as Figure 4 (a)–(b).

Figure 11: Cascaded evaluation on Chatbot Arena (Qwen-7B $\to$ 32B $\to$ 72B, $m{=}10$ ). Same layout as Figure 4 (c)–(d).

LLM-as-a-Judge

Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.

See all 559 papers

Margin-Adaptive Confidence Ranking for Reliable LLM Judgement
May 14, 2026Gaojie Jin, Yong Tao, Lijia Yu +1LLM Evaluation LLM-as-a-Judge
Department of Computer Science, University of Exeter · Institute of AI for Industries, Chinese Academy of Sciences · Department of Mathematics and Computer Science, Eindhoven University of Technology
Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guarantees
Aug 18, 2026Sher Badshah, Ali Emami, Hassan SajjadLLM-as-a-Judge Selective Prediction
Dalhousie University · New York University Abu Dhabi · Emory University
When Consistency Does Not Mean Reliability: Evaluating Local LLM Judges Against Human Ratings
Sep 12, 2026Aakash Kumar TiwariLLM-as-a-Judge Bias in LLM-as-a-Judge
Department of Mathematics Indian Institute of Technology Kharagpur

May 14, 2026J/K move · Enter open · S save

Margin-Adaptive Confidence Ranking for Reliable LLM Judgement

Gaojie Jin, Yong Tao, Lijia Yu, Tianjin Huang

Department of Computer Science, University of Exeter · Institute of AI for Industries, Chinese Academy of Sciences · Department of Mathematics and Computer Science, Eindhoven University of Technology

LLM Evaluation LLM-as-a-Judge Learning to Rank Confidence Estimation in Language Models Human-AI Agreement LLM Uncertainty Estimation

Open paper AskPDF arXiv

Jung et al. (2025) introduce a hypothesis testing framework for guaranteeing agreement between large language models (LLMs) and human judgments, relying on the assumption that the model's estimated confidence is monotonic with respect to human-disagreement risk. In practice, however, this assumption may be violated, and the generalization behavior of the confidence estimator is not explicitly analyzed. We mitigate these issues by learning a dedicated confidence estimator instead of relying on heuristic confidence signals. Our approach leverages simulated annotator diversity and a margin-based ranking formulation to explicitly model how confidently an LLM distinguishes between human-agreement and human-disagreement cases. We further derive generalization guarantees for this estimator, revealing a margin-dependent trade-off that informs the design of an adaptive estimator training procedure. When integrated into fixed-sequence testing, the learned confidence estimator yields improved ranking accuracy and empirically strengthens the monotonic relationship between confidence and disagreement risk, leading to higher success rates in satisfying target agreement levels across multiple datasets and judge models.

Abstract

LLM-as-a-Judge

Explore similar work

Margin-Adaptive Confidence Ranking for Reliable LLM Judgement

Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guarantees

When Consistency Does Not Mean Reliability: Evaluating Local LLM Judges Against Human Ratings