DIAL: Position-Debiased LLM Judges with Adaptive Human Preference Calibration

Zesheng Cai; Yingqi Fan; Sichang Chen; Jin-Hong Du

cs.AISep 25, 2026

DIAL: Position-Debiased LLM Judges with Adaptive Human Preference Calibration

Authors: Zesheng Cai, Yingqi Fan, Sichang Chen, Jin-Hong Du

Organizations: The University of Hong Kong · Sun Yat-sen University

Abstract

Large language models (LLMs) as a judge enable scalable evaluation, but their judgments can be sensitive to response order and, even after removing such position effects, can still diverge systematically from human preferences.We introduce DIAL, a unified framework that combines abundant LLM comparisons with limited human comparisons to separate judge-specific position effects, learn shared structure in position-debiased LLM preferences, and adaptively calibrate that structure toward the human preference target. Theoretically, we study three aspects of DIAL: (i) identification of latent LLM preferences, position effects, and human calibration; (ii) adaptive estimation that balances LLM anchoring against limited human evidence; and (iii) fixed-weight uncertainty quantification for the calibrated human preference. Empirically, we evaluate position debiasing and human alignment separately in controlled simulations and on three human-preference benchmarks, showing that DIAL remains robust to unbalanced response order, achieves strong human-aligned rankings with limited labels, and adapts toward human evidence when LLM information is imperfect. Our real-data study collects over 410K judgments from 21 LLM judges in both display orders, providing a resource for future studies of LLM-judge bias, heterogeneity, and human alignment.

Figures & tables

Figure 1: Overview of DIAL framework. (a) Pairwise evaluation combines abundant, scalable LLM judgments with limited, higher-fidelity human judgments. (b) Empirical examples illustrate two distinct challenges: position bias and human-preference misalignment. (c) DIAL jointly models LLM and human comparisons to separate judge-specific position effects from latent LLM preferences, learn shared preference structure across judges, and adaptively calibrate toward human preference.

Figure 2: Synthetic evaluation with

N=10

K=4

r=1

s_{\text{h}}=\mu

, and canonical-first display probability

0.75

; curves show means

\pm 1.96

Monte Carlo standard errors over

50

replications. Columns report excess human risk, Kendall’s

\tau

95\%

interval coverage, and

\operatorname{RMSE}(\widehat{b},b)

. Top:

n_{\text{h}}

varies at

n_{\text{l}}=20{,}000

under correct specification; dotted lines are the first-order risk benchmarks for Human, calibration within

\operatorname{span}(\mu)

, and calibration within

\operatorname{col}(W)

. Bottom:

n_{\text{l}}

varies at

n_{\text{h}}=800

under judge-pair overdispersion that misspecifies the LLM likelihood while leaving the human calibration model correctly specified.

Figure 3: Real-data position robustness evaluation: excess held-out human log loss (zero is the test-pool BTL fit), averaged over up to

50

splits with

\pm 1.96

Monte Carlo standard errors. Panels show Chatbot Arena, MT-Bench, and PandaLM using the same five order-sensitive judges: Mistral-Small 3.2 , Claude Haiku 4.5 , Qwen3-30B without reasoning, GLM4-9B , and DeepSeek-R1-1.5B , under balanced, one-sided (

5\%

swapped), and one-sided plus

8

injected pure-position judges.

Figure 4: Human-label efficiency and adaptive calibration on Chatbot Arena: Kendall’s

\tau

with the held-out human ranking as

n_{\text{h}}

varies, using (a) all available judgments of the

21

LLM judges and (b)

n_{\text{l}}=2{,}000

. Curves show means over up to

50

splits with

\pm 1.96

Monte Carlo standard errors; Human and AtC are shown only where the pooled-human BTL fit is finite in all splits, and the dashed vertical line marks the smallest such budget (

n_{\text{h}}=800

Appendix		Content
Appendix A		Detailed comparisons to related work.
Appendix B	B.1	Identification of position-debiased LLM scores and order effects ( Proposition 1 ).
	B.2	Debiased versus order-averaged probabilities.
	B.3	Estimation of the position-debiased representation and judge diagnostics.
Appendix C	C.1	Identification and fixed-space risk preliminaries ( Proposition 2 ).
	C.2	Proof of Theorem 3 .

Table 5

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

M_{\theta}\leftarrow\mu,\qquad\text{or }M_{\theta}\leftarrow W_{\theta}=[\mu,V]\text{ for full-space calibration.}

Appendix

Algorithm E.1 DIAL-Anc estimation

(\widehat{c}_{\lambda},\widehat{\theta}_{\lambda})\in\mathop{\mathrm{argmin}}_{c,\theta\in\Theta_{r}}\left\{\ell_{\text{h}}(M_{\theta}c)+\lambda\ell_{\text{l}}(S_{\theta},b_{\theta})\right\},

Appendix

Algorithm E.2 DIAL-Ada estimation

Figure F1 : Larger-scale replicate of Figure 2 with

N=20

K=6

r=2

, and

50

replications; rows, columns, and estimators follow Figure 2 .

Figure F2 : Misaligned-target simulation under the main

(N,K,r)=(10,4,1)

configuration, with

s_{\text{h}}=Wc_{\text{h}}

and

c_{V}\sim\mathcal{N}(0,0.5^{2})

. Rows and columns follow Figure 2 ; dotted lines show the first-order excess-risk benchmarks for Human and calibration within

\operatorname{col}(W)

Figure F3 : Robustness to heterogeneous position effects under the main

(N,K,r)=(10,4,1)

configuration with

n_{\text{h}}=800

and

n_{\text{l}}=20{,}000

. Data are generated with

b_{kij}=b_{k}+\delta_{kij}

, where the within-judge pair effects have standard deviation

\sigma_{\mathrm{pos}}

, while all methods retain a constant

b_{k}

working model. Panels report excess human risk, Kendall’s

\tau

, and

\operatorname{RMSE}(\widehat{b},b)

over

50

replications.

	Chatbot Arena	MT-Bench	PandaLM
Records (prompts with a human label)	7,738	1,199	945
Candidate models $N$	20	6	5
Observed pairs / possible	189 / 190	15 / 15	10 / 10
Human labels a / b / tie / both bad	2,771 / 2,659 / 814 / 1,494	432 / 410 / 357 / 0	398 / 444 / 103 / 0
Decisive human labels	5,430	842	842
LLM judgments, kept judges	321,668	49,172	39,324

Appendix

Table G1 : Dataset statistics after preprocessing. “Kept” refers to the

21

retained judges; a cell is a (judge, canonical pair) combination.

Dataset	Candidate models
Chatbot Arena ( $N=20$ )	RWKV-4-Raven-14B , alpaca-13b , chatglm-6b , claude-instant-v1 , claude-v1 , dolly-v2-12b , fastchat-t5-3b , gpt-3.5-turbo , gpt-4 , gpt4all-13b-snoozy , guanaco-33b , koala-13b , llama-13b , mpt-7b-chat , oasst-pythia-12b , palm-2 , stablelm-tuned-alpha-7b , vicuna-13b , vicuna-7b , wizardlm-13b
MT-Bench ( $N=6$ )	alpaca-13b , claude-v1 , gpt-3.5-turbo , gpt-4 , llama-13b , vicuna-13b-v1.2
PandaLM ( $N=5$ )	bloom-7b , cerebras-gpt-6.7B , llama-7b , opt-7b , pythia-6.9b

Appendix

Table G2 : Candidate models treated as items after preprocessing.

Figure G4 : Real-data robustness sweeps on Chatbot Arena using five order-sensitive judges. Top: excess held-out log loss under (a) display imbalance, (b) injected pure-position judges, and (c) injected anti-consensus judges with scarce LLM data. Bottom: Kendall’s

\tau

under (d) the pure-position perturbation, (e) a balanced LLM-budget sweep, and (f) the anti-consensus perturbation. Means over up to

50

record splits with

\pm 1.96

Monte Carlo standard errors.

Figure G5 : Additional real-data diagnostics. (a) LLM-only fitted order effects of the

21

judges, with cell-clustered

95\%

intervals, against swap-inconsistency rate. (b) Rejection rate of the calibration-space likelihood-ratio test of

s_{\text{h}}=\alpha\mu

on Chatbot Arena; hollow markers denote budgets at which the unrestricted human-only MLE failed in some replications, and at the two smallest budgets it failed in most, so those points are not interpreted.

Figure G6 : Human-label efficiency and LLM-budget sensitivity. Top: excess held-out human log loss against the human budget

n_{\text{h}}

with the as-collected balanced judgments of all

21

judges; Human is off the chart at the smallest budgets, where its MLE does not exist. Bottom: the five order-sensitive judges’ balanced judgments subsampled to

n_{\text{l}}

rows at a fixed calibration sample (

n_{\text{h}}=1{,}000

80

60

); the dotted line is the candidate weight minimizing the test log loss. Means over up to

50

record splits with

\pm 1.96

Monte Carlo standard errors.

Figure G7 : Excess held-out human log loss against the human budget

n_{\text{h}}

, as in Figure G6 (top), on the small, large, and full judge panels (columns) across Chatbot Arena, MT-Bench, and PandaLM (rows). The small and large panels each contain six open-weight judges with disclosed parameter counts

\leq 9

B and

\geq 20

B per judge, respectively (

36.5

B versus

203

B total); full is the

21

-judge panel. All curves use up to the same

50

record splits, with

\pm 1.96

Monte Carlo standard errors; the

y

-axis is shared within each row.

Figure G8 : Complementary LLM-budget sweep for the small, large, and full judge panels of Figure G7 . The human calibration budget is fixed at

n_{\text{h}}=300

80

, and

60

on Chatbot Arena, MT-Bench, and PandaLM, respectively, while balanced LLM training judgments are subsampled uniformly to

n_{\text{l}}

rows. Curves compare DIAL-Ada, at the GACV-selected weight, with DIAL-Anc and Human. All curves use up to the same

50

record splits, with

\pm 1.96

Monte Carlo standard errors; the

y

-axis is shared within each row.

LLM-as-a-Judge

Large language model (LLM)-as-a-judge research uses language models to score, compare, and verify AI outputs as scalable alternatives to human evaluation.

See all 555 papers

A CaLLM Judge is a Better Judge: Mitigating Bias in LLM-as-a-judge
Oct 13, 2026Cesare Barbera, Luca Stradiotti, Lorenzo Cascioli +6Bias in LLM-as-a-Judge LLM-as-a-Judge
University of Pisa · University of Trento · KU Leuven +1
SenseJudge: Human-Centric Preference-Driven Judgment Framework
Jun 2, 2026Rui Li, Junfeng Liu, Xiangwen Kong +2LLM Evaluation LLM-as-a-Judge
State Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University · StepFun · Xi’an Jiaotong University
Accounting for Bias Enables Sustainable LLM Evaluation
Sep 25, 2026Harshita Katoch, David Antony Selby, Gerrit Großmann +1LLM Evaluation Bias in LLM-as-a-Judge
Department of Computer Science, University of Kaiserslautern–Landau (RPTU), Germany · Data Science and its Applications, German Research Centre for Artificial Intelligence (DFKI), Germany

Oct 13, 2026J/K move · Enter open · S save

A CaLLM Judge is a Better Judge: Mitigating Bias in LLM-as-a-judge

Cesare Barbera, Luca Stradiotti, Lorenzo Cascioli, Lorenzo Perini+5

University of Pisa · University of Trento · KU Leuven+1

Bias in LLM-as-a-Judge LLM-as-a-Judge Language Model Calibration Multicalibration

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used as judges to predict which of two candidate answers a human evaluator would prefer, providing a scalable alternative when human evaluation is too costly. Unfortunately, LLM judgments can exhibit systematic biases. Prior work has documented position, verbosity or model family of the candidate answers influencing the judge decision rather than their quality, while additional biases may remain undocumented.

\textit{Multicalibration}

provides a natural framework to address this problem: each bias corresponds to miscalibration on an identifiable group,

\textit{e.g.}

, family bias manifests as overconfidence in answers generated by models belonging to the same family of the judge. The challenge is to fix miscalibration simultaneously across many, potentially unknown groups. To this end, we introduce

\mathrm{CaLLM}

, a post hoc framework that multicalibrates an LLM judge, thereby bounding all such biases simultaneously. We prove that

\mathrm{CaLLM}

bounds the bias of every group whose error structure is captured by an available representation, and that augmenting it with explicit bias groups further reduces the residual bias. We then derive theoretical conditions under which

\mathrm{CaLLM}

improves the judge's accuracy, rather than trading it for calibration. Finally, we evaluate

\mathrm{CaLLM}

across pairwise preference benchmarks, showing that it reduces the multicalibration error on both specified and unspecified bias groups, while improving predictive performance of the judge.

Abstract

LLM-as-a-Judge

Explore similar work

A CaLLM Judge is a Better Judge: Mitigating Bias in LLM-as-a-judge

SenseJudge: Human-Centric Preference-Driven Judgment Framework

Accounting for Bias Enables Sustainable LLM Evaluation