Optimal Design for Active Preference Learning with Biased LLM Judges

Zhongman Du; Huiming Zhang; Haodong Zhu; Baochang Zhang

cs.LGSep 30, 2026

Optimal Design for Active Preference Learning with Biased LLM Judges

Authors: Zhongman Du, Huiming Zhang, Haodong Zhu, Baochang Zhang

Organizations: Beihang University · Zhongguancun Academy · Hangzhou Innovation Institute of Beihang University

Abstract

Learning from human preferences is central to large language model (LLM) alignment, but human preference annotation is costly. Active preference learning reduces this cost by selecting informative comparisons, and LLM judges can provide additional scalable feedback. However, the preferences of the judges may deviate from those of the target human population. Even after calibration on trusted reference data, active acquisition can shift the comparison distribution and expose residual judge bias. We therefore incorporate judge deviations into the acquisition design rather than relying on a separate calibration stage. Under joint estimation, comparisons that appear highly informative about the reward may also reflect judge bias and therefore provide less information about human preferences. To address this issue, we propose Nuisance-Adjusted Optimal Design (NAOD), a comparison-selection strategy that prioritizes policy-relevant target information after nuisance adjustment and uses the Frank-Wolfe algorithm for optimization. Theoretically, we establish a sharp conditional local asymptotic minimax lower bound on policy risk and construct an estimator that attains it. We further characterize the finite-sample cost of learning the nuisance representation and show that representation error can reverse an oracle design advantage. Finally, we validate these predictions experimentally and evaluate NAOD on Chatbot Arena data across 17 judges, 15 budget configurations, and 15 random cluster-level splits. NAOD reduces the mean regret of proxy policy by 29.1% relative to a matched target-information design, outperforms existing methods, and improves human-preference prediction on held-out data.

Figures & tables

Figure 1: Overview of the NAOD acquisition and estimation pipeline. Historical human–judge feedback is used to construct the frozen target and judge-deviation representations. Before current outcomes are observed, NAOD selects comparisons using nuisance-adjusted target information weighted by policy relevance. After selection, trusted human and judge feedback are combined in a joint estimator to update the target reward and induced policy.

Figure 2: Controlled tests of the theoretical predictions. Dashed references in panels (a) and (b) denote the corresponding theoretical predictions. Panel (c) reports the NAOD-minus-Target-info scaled-risk difference.

Method	Proxy regret $\downarrow$	Human CE $\downarrow$	Acc. $\uparrow$	Drop (%)	Wins
Initial	8.86	0.6752	57.82	77.53	13/15
Human-only	18.25	0.6829	56.84	89.09	15/15
Random	4.88	0.6727	58.22	59.18	15/15
Entropy	13.42	0.6784	57.10	85.17	15/15
D-opt	2.45	0.6703	58.19	18.87	12/15
PA D-opt	2.46	0.6703	58.19	18.93	12/15

Table 1: Overall Chatbot Arena results. Means over 17 judges, 15 budget regimes, and 15 random cluster-level splits. Regret is reported in

10^{-3}

units, CE in nats per pair, and accuracy in percent. The last two columns report NAOD’s relative regret reduction and paired split wins against each row. Bold denotes the best mean.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Convention
Current outcomes	Independent Bernoulli trusted and judge labels
Final estimator	Trusted preliminary + offset nuisance preliminary + one joint update
Target / nuisance radii	$2$ and $3$ , respectively, unless stated otherwise
Preliminary gap tolerance	$10^{-6}$ divided by fit sample size
Nuisance-Gram threshold	$10^{-10}$
Joint-information guard	$10^{-5}$

Appendix

Table 2: Common controlled-study protocol. Experiment-specific constructions and sample sizes are given in Sections E.2 – E.3 .

$n$	Design	$\Phi$	Mean $nD_{F}$	MC SE
240	NAOD	0.426	0.430	0.008
240	Target-info	1.139	1.164	0.021
240	Random	0.530	0.527	0.010
960	NAOD	0.426	0.441	0.008
960	Target-info	1.139	1.161	0.022
960	Random	0.530	0.518	0.010

Appendix

Table 3: Effective-information experiment underlying Figure 2 (a). Each row uses

6{,}000

independent repetitions.

\Phi

is evaluated using the realized integer counts.

$n$	$m/n$	Prediction	Mean $nD_{F}$	MC SE	Clip (%)
960	$1/4$	2.041	1.891	0.034	3.1
960	$1$	1.276	1.252	0.023	0.0
960	$4$	1.084	1.085	0.020	0.0
3840	$1/4$	2.041	1.993	0.037	0.0
3840	$1$	1.276	1.281	0.024	0.0
3840	$4$	1.084	1.116	0.020	0.0

Appendix

Table 4: Representation-learning cost underlying Figure 2 (b). Prediction is

\Phi_{0}+(n/m)C_{P}

. “Clip” is the percentage of historical fits for which the bounded rotation is active.

$m/n$	NAOD–Target-info	Outer SE	95% interval
$1/4$	$+1.588$	0.237	$[1.122,\,2.054]$
$1/2$	$+0.772$	0.200	$[0.378,\,1.165]$
$1$	$-0.039$	0.100	$[-0.235,\,0.157]$
$2$	$-0.446$	0.049	$[-0.541,\,-0.350]$

Appendix

Table 5: Design-ranking reversal underlying Figure 2 (c). Differences are paired over

300

independent historical realizations, each averaging eight current datasets. Intervals are pointwise two-sided

t_{299}

intervals.

Figure 3: Controlled diagnostics. (a) Directional omitted-error diagnostic. Dashed curves are local predictions from the frozen logit-influence kernel. Solid curves are empirical mean scaled policy risks. The two perturbation directions have identical Euclidean magnitudes. (b) Policy-anisotropy diagnostic. Dashed curves are information-based predictions. Solid curves are empirical mean scaled policy risks. The Unweighted rule changes only the acquisition weight and is evaluated under the same policy loss. Error bars show

1.96

MC SE over

6{,}000

independent datasets.

Fitted model	$r$	Nominal $\Phi$	Mean $nD_{F}$	MC SE
Underfit	0	0.391	27.562	0.064
Correct	1	0.686	0.699	0.010
Overfit	2	0.772	0.764	0.011
Saturated	6	4.000	4.015	0.059

Appendix

Table 6: Nuisance-rank ablation at

n=3840

. The Underfit nominal criterion is not a true-risk prediction because its fitted model omits the actual judge discrepancy.

$B$	Design	Nominal risk	Mean $D_{F}$	Pool SE
		$(\times 10^{-3})$	$(\times 10^{-3})$	$(\times 10^{-3})$
60	NAOD	6.108	6.146	0.081
60	Target-info	6.145	6.181	0.081
60	Random	8.945	8.851	0.117
180	NAOD	3.420	3.361	0.053
180	Target-info	3.430	3.373	0.051

Appendix

Table 7: Distinct finite-pool experiment. Risks are reported in

10^{-3}

units. Nominal risk uses the actual selected-subset information. SE is computed across

24

independent pool-level means, each averaging

300

current datasets.

Role	Clusters	Use
Upstream	$8{,}575$	Target representation, nuisance representation, and human reference
Historical initialization	$128$	Initial target and nuisance estimates
Policy support	$1{,}024$	Policy weighting covariates with no labels
Current human reservoir	$512$	Nested trusted budgets $H\leq 512$
Candidate	$24{,}061$	Judge acquisition with cluster capacity one
Test	$8{,}575$	Held-out human evaluation

Appendix

Table 8: Arena cluster partition within each repeated split. Roles are cluster-disjoint. Candidate clusters may contain multiple rows, but at most one row from a connected cluster can be acquired.

Component	Configuration
Embedding	Qwen3-Embedding-0.6B, last-token pooling, $L_{2}$ normalization
Maximum input	$2{,}048$ tokens with prompt cap $512$
Base reduction	Upstream-fitted whitened PCA, $128$ dimensions
Target representation	Judge-specific SVD span, dimension $d=2$
Target-head penalties	$32,128,512,2048$
Nuisance representation	Intercept and standardized learned residual, dimension $r=2$

Appendix

Table 9: Frozen Arena configuration. All acquisition methods use the same representation, initialization, current budgets, and final estimator unless required otherwise by the method definition.

Baseline	Proxy-regret gain	Human-CE gain	Accuracy gain
Initial	$+6.870\,[3.020,10.721]$	$+5.308\,[1.952,8.664]$	$+0.448\,[0.117,0.780]$
Human-only	$+16.259\,[8.849,23.668]$	$+12.981\,[7.214,18.748]$	$+1.426\,[0.331,2.520]$
Random	$+2.886\,[2.455,3.318]$	$+2.814\,[2.388,3.241]$	$+0.050\,[-0.069,0.169]$
Entropy	$+11.433\,[4.783,18.083]$	$+8.527\,[3.669,13.385]$	$+1.166\,[0.079,2.252]$
D-opt	$+0.463\,[0.093,0.833]$	$+0.385\,[0.011,0.759]$	$+0.075\,[0.028,0.121]$
PA D-opt	$+0.465\,[0.095,0.834]$	$+0.386\,[0.013,0.760]$	$+0.075\,[0.029,0.121]$

Appendix

Table 10: Overall paired Arena comparisons. Positive values favor NAOD. Proxy-regret and human-CE gains are in

10^{-3}

units. Accuracy gains are percentage points. Brackets give paired

95\%

t_{14}

intervals across the

15

repeated splits.

Figure 4: Budget dependence on Chatbot Arena. NAOD relative reduction in mean proxy policy regret across the complete

5\times 3

Arena budget grid. Positive values favor NAOD. Panel (a) compares against Target-info and panel (b) against D-opt. Cell values are ratios of macro-averaged mean regrets rather than averages of split-specific percentage reductions.

$H$	$B$	Target-info–NAOD	D-opt–NAOD
32	256	$+2.103\,[0.263,3.944]$	$+1.543\,[0.612,2.474]$
32	512	$+1.141\,[0.447,1.835]$	$+0.715\,[0.138,1.291]$
32	1024	$+0.408\,[0.143,0.673]$	$+0.160\,[-0.192,0.512]$
64	256	$+1.731\,[0.507,2.955]$	$+1.234\,[0.359,2.109]$
64	512	$+1.123\,[0.476,1.770]$	$+0.703\,[0.125,1.281]$
64	1024	$+0.471\,[0.184,0.758]$	$+0.234\,[-0.119,0.587]$

Appendix

Table 11: Paired proxy-regret gains across the Arena budget grid. Entries are baseline-minus-NAOD gains in

10^{-3}

units with pointwise paired

95\%

t_{14}

intervals.

Figure 5: Judge-level coupling and realized acquisition gain. The horizontal axis is Target-info target–nuisance coupling

\rho^{2}

and the vertical axis is the Target-info-minus-NAOD proxy-regret gain in

10^{-3}

units. Each point represents one judge after averaging over the

15

budget combinations and

15

repeated cluster-level splits. The Spearman rank correlation is

r_{s}=0.576

. Positive vertical values favor NAOD.

Judge	$\rho_{T}^{2}$	Policy gain
Athene-70B	0.570	$+0.428\,[-0.290,1.145]$
dolphin-2.1-mistral-7b	0.956	$+2.063\,[1.076,3.049]$
dolphin-2.5-mixtral-8x7b	0.960	$+2.819\,[1.248,4.390]$
Hermes-3-Llama-3.1-70B	0.610	$+0.366\,[-0.305,1.038]$
Meta-Llama-3-70B-Instruct	0.459	$-0.037\,[-0.358,0.285]$
Meta-Llama-3-8B-Instruct	0.716	$+0.733\,[0.069,1.397]$

Appendix

Table 12: Judge-level coupling and policy gain.

\rho_{T}^{2}

is the squared target–nuisance coupling under Target-info. Policy gain is Target-info minus NAOD in

10^{-3}

units with pointwise paired

95\%

t_{14}

intervals.

Explore similar work

Aug 31, 2026cs.CL

Low-Resource Preference Adaptation of LLMs via Activation-Based Label Propagation

Adapting large language models to user-specific preferences is often constrained by the cost of human annotation, making preference optimisation impractical in low-resource settings where preferences cannot be reliably labelled by LLMs themselves, e.g., due to cultural, subjective, or personalised contexts. In this paper, we investigate how language models encode preference information in their intermediate representations, finding that activations from chosen and rejected responses form distinct clusters across layers, even in pretrained models. Strikingly, this structure is strengthened by alignment on canonical datasets but erased when the target preferences differ from those the model was aligned on, suggesting aligned LLMs are poor judges for non-mainstream populations. Exploiting this structure, we propose training a lightweight linear probe on a few labelled preference pairs (

\leq

500) and using it to annotate large unlabelled datasets (50K+) for downstream preference optimisation. We systematically evaluate this approach across different datasets, preference optimisation methods and model scales and find that our method consistently outperforms direct training given the same annotation budget, and remains competitive against baselines trained on

50-100\times

more labelled data in the majority of our settings. Code is available at https://github.com/alessioGalatolo/activ-pref-probe.

Alessio Galatolo, Meriem Beloucif

Uppsala University

Sequential Preference Optimization Preference Alignment Learning Large Language Model Adaptation Label Propagation Human Annotations

Ask PDF

May 30, 2026cs.LG

CUPID in the Model Zoo: Online Matchmaking for Selecting Your Dream LLM

Users increasingly face the challenge of selecting an appropriate LLM for a given task from a rapidly growing pool of LLMs, each with distinct but often opaque latent properties. Compounding this challenge, users may lack the vocabulary or awareness to explicitly articulate the characteristics they value in an LLM's responses or deployment. We propose an interaction-efficient active learning framework in which a dueling bandit algorithm iteratively selects pairs of LLMs, collects user feedback about their responses, and updates its belief about the user's latent preferences. We introduce a novel belief-aware upper confidence bound strategy that balances exploration of the model pool with exploitation of inferred preferences, enabling efficient alignment between user needs and LLM capabilities under user-specified cost and time budgets. Through diverse experiments on LLMs and human studies, we experimentally verify that our model can efficiently match well-aligned LLMs to users at a lower cost.

Son Nguyen, Xinyuan Liu, Ransalu Senanayake

School of Computing and Augmented Intelligence, Arizona State University, Tempe, United States of America

Large Language Model Alignment Bandits Species

Ask PDF

Jun 17, 2026stat.ML

Quantifying and Auditing LLM Evaluation via Positive--Unlabeled Learning

Large Language Models (LLMs) are increasingly used as judges for scalable evaluation, yet such LLM--as--a--Judge systems exhibit systematic biases that are decoupled from semantic quality, most notably verbosity bias. Meanwhile, human supervision is costly and typically selective, yielding reliable positive judgments but leaving most outputs unlabelled and potentially mixed in quality. We formulate LLM evaluation under selective human supervision as a positive--unlabelled learning problem and propose a geometric auditing framework based on Partial Optimal Transport. By aligning a small set of human--verified positives with a reliable subset of unlabelled outputs in a fixed embedding space, our method identifies human--consistent preferences and corrects biased judges without retraining. Experiments demonstrate improved alignment with human preferences, increased robustness to presentation biases, and interpretable confidence estimates, offering a scalable and statistically grounded alternative to existing LLM--as--a--judge pipelines.

Zilong Zhang, Yi-Ting Hung, Lei Ding +1

Department of Mathematics and Statistics Georgia State University · Department of Statistics University of Manitoba

Large Language Model Evaluation Positive--Unlabeled Learning Position Bias Process-Level Supervision Quantifying

Ask PDF