Optimal Design for Active Preference Learning with Biased LLM Judges

Zhongman Du; Huiming Zhang; Haodong Zhu; Baochang Zhang

cs.LGSep 30, 2026

Optimal Design for Active Preference Learning with Biased LLM Judges

Authors: Zhongman Du, Huiming Zhang, Haodong Zhu, Baochang Zhang

Organizations: Beihang University · Zhongguancun Academy · Hangzhou Innovation Institute of Beihang University

Abstract

Learning from human preferences is central to large language model (LLM) alignment, but human preference annotation is costly. Active preference learning reduces this cost by selecting informative comparisons, and LLM judges can provide additional scalable feedback. However, the preferences of the judges may deviate from those of the target human population. Even after calibration on trusted reference data, active acquisition can shift the comparison distribution and expose residual judge bias. We therefore incorporate judge deviations into the acquisition design rather than relying on a separate calibration stage. Under joint estimation, comparisons that appear highly informative about the reward may also reflect judge bias and therefore provide less information about human preferences. To address this issue, we propose Nuisance-Adjusted Optimal Design (NAOD), a comparison-selection strategy that prioritizes policy-relevant target information after nuisance adjustment and uses the Frank-Wolfe algorithm for optimization. Theoretically, we establish a sharp conditional local asymptotic minimax lower bound on policy risk and construct an estimator that attains it. We further characterize the finite-sample cost of learning the nuisance representation and show that representation error can reverse an oracle design advantage. Finally, we validate these predictions experimentally and evaluate NAOD on Chatbot Arena data across 17 judges, 15 budget configurations, and 15 random cluster-level splits. NAOD reduces the mean regret of proxy policy by 29.1% relative to a matched target-information design, outperforms existing methods, and improves human-preference prediction on held-out data.

Figures & tables

Figure 1: Overview of the NAOD acquisition and estimation pipeline. Historical human–judge feedback is used to construct the frozen target and judge-deviation representations. Before current outcomes are observed, NAOD selects comparisons using nuisance-adjusted target information weighted by policy relevance. After selection, trusted human and judge feedback are combined in a joint estimator to update the target reward and induced policy.

Figure 2: Controlled tests of the theoretical predictions. Dashed references in panels (a) and (b) denote the corresponding theoretical predictions. Panel (c) reports the NAOD-minus-Target-info scaled-risk difference.

Method	Proxy regret $\downarrow$	Human CE $\downarrow$	Acc. $\uparrow$	Drop (%)	Wins
Initial	8.86	0.6752	57.82	77.53	13/15
Human-only	18.25	0.6829	56.84	89.09	15/15
Random	4.88	0.6727	58.22	59.18	15/15
Entropy	13.42	0.6784	57.10	85.17	15/15
D-opt	2.45	0.6703	58.19	18.87	12/15
PA D-opt	2.46	0.6703	58.19	18.93	12/15

Table 1: Overall Chatbot Arena results. Means over 17 judges, 15 budget regimes, and 15 random cluster-level splits. Regret is reported in

10^{-3}

units, CE in nats per pair, and accuracy in percent. The last two columns report NAOD’s relative regret reduction and paired split wins against each row. Bold denotes the best mean.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Component	Convention
Current outcomes	Independent Bernoulli trusted and judge labels
Final estimator	Trusted preliminary + offset nuisance preliminary + one joint update
Target / nuisance radii	$2$ and $3$ , respectively, unless stated otherwise
Preliminary gap tolerance	$10^{-6}$ divided by fit sample size
Nuisance-Gram threshold	$10^{-10}$
Joint-information guard	$10^{-5}$

Appendix

Table 2: Common controlled-study protocol. Experiment-specific constructions and sample sizes are given in Sections E.2 – E.3 .

$n$	Design	$\Phi$	Mean $nD_{F}$	MC SE
240	NAOD	0.426	0.430	0.008
240	Target-info	1.139	1.164	0.021
240	Random	0.530	0.527	0.010
960	NAOD	0.426	0.441	0.008
960	Target-info	1.139	1.161	0.022
960	Random	0.530	0.518	0.010

Appendix

Table 3: Effective-information experiment underlying Figure 2 (a). Each row uses

6{,}000

independent repetitions.

\Phi

is evaluated using the realized integer counts.

$n$	$m/n$	Prediction	Mean $nD_{F}$	MC SE	Clip (%)
960	$1/4$	2.041	1.891	0.034	3.1
960	$1$	1.276	1.252	0.023	0.0
960	$4$	1.084	1.085	0.020	0.0
3840	$1/4$	2.041	1.993	0.037	0.0
3840	$1$	1.276	1.281	0.024	0.0
3840	$4$	1.084	1.116	0.020	0.0

Appendix

Table 4: Representation-learning cost underlying Figure 2 (b). Prediction is

\Phi_{0}+(n/m)C_{P}

. “Clip” is the percentage of historical fits for which the bounded rotation is active.

$m/n$	NAOD–Target-info	Outer SE	95% interval
$1/4$	$+1.588$	0.237	$[1.122,\,2.054]$
$1/2$	$+0.772$	0.200	$[0.378,\,1.165]$
$1$	$-0.039$	0.100	$[-0.235,\,0.157]$
$2$	$-0.446$	0.049	$[-0.541,\,-0.350]$

Appendix

Table 5: Design-ranking reversal underlying Figure 2 (c). Differences are paired over

300

independent historical realizations, each averaging eight current datasets. Intervals are pointwise two-sided

t_{299}

intervals.

Figure 3: Controlled diagnostics. (a) Directional omitted-error diagnostic. Dashed curves are local predictions from the frozen logit-influence kernel. Solid curves are empirical mean scaled policy risks. The two perturbation directions have identical Euclidean magnitudes. (b) Policy-anisotropy diagnostic. Dashed curves are information-based predictions. Solid curves are empirical mean scaled policy risks. The Unweighted rule changes only the acquisition weight and is evaluated under the same policy loss. Error bars show

1.96

MC SE over

6{,}000

independent datasets.

Fitted model	$r$	Nominal $\Phi$	Mean $nD_{F}$	MC SE
Underfit	0	0.391	27.562	0.064
Correct	1	0.686	0.699	0.010
Overfit	2	0.772	0.764	0.011
Saturated	6	4.000	4.015	0.059

Appendix

Table 6: Nuisance-rank ablation at

n=3840

. The Underfit nominal criterion is not a true-risk prediction because its fitted model omits the actual judge discrepancy.

$B$	Design	Nominal risk	Mean $D_{F}$	Pool SE
		$(\times 10^{-3})$	$(\times 10^{-3})$	$(\times 10^{-3})$
60	NAOD	6.108	6.146	0.081
60	Target-info	6.145	6.181	0.081
60	Random	8.945	8.851	0.117
180	NAOD	3.420	3.361	0.053
180	Target-info	3.430	3.373	0.051

Appendix

Table 7: Distinct finite-pool experiment. Risks are reported in

10^{-3}

units. Nominal risk uses the actual selected-subset information. SE is computed across

24

independent pool-level means, each averaging

300

current datasets.

Role	Clusters	Use
Upstream	$8{,}575$	Target representation, nuisance representation, and human reference
Historical initialization	$128$	Initial target and nuisance estimates
Policy support	$1{,}024$	Policy weighting covariates with no labels
Current human reservoir	$512$	Nested trusted budgets $H\leq 512$
Candidate	$24{,}061$	Judge acquisition with cluster capacity one
Test	$8{,}575$	Held-out human evaluation

Appendix

Table 8: Arena cluster partition within each repeated split. Roles are cluster-disjoint. Candidate clusters may contain multiple rows, but at most one row from a connected cluster can be acquired.

Component	Configuration
Embedding	Qwen3-Embedding-0.6B, last-token pooling, $L_{2}$ normalization
Maximum input	$2{,}048$ tokens with prompt cap $512$
Base reduction	Upstream-fitted whitened PCA, $128$ dimensions
Target representation	Judge-specific SVD span, dimension $d=2$
Target-head penalties	$32,128,512,2048$
Nuisance representation	Intercept and standardized learned residual, dimension $r=2$

Appendix

Table 9: Frozen Arena configuration. All acquisition methods use the same representation, initialization, current budgets, and final estimator unless required otherwise by the method definition.

Baseline	Proxy-regret gain	Human-CE gain	Accuracy gain
Initial	$+6.870\,[3.020,10.721]$	$+5.308\,[1.952,8.664]$	$+0.448\,[0.117,0.780]$
Human-only	$+16.259\,[8.849,23.668]$	$+12.981\,[7.214,18.748]$	$+1.426\,[0.331,2.520]$
Random	$+2.886\,[2.455,3.318]$	$+2.814\,[2.388,3.241]$	$+0.050\,[-0.069,0.169]$
Entropy	$+11.433\,[4.783,18.083]$	$+8.527\,[3.669,13.385]$	$+1.166\,[0.079,2.252]$
D-opt	$+0.463\,[0.093,0.833]$	$+0.385\,[0.011,0.759]$	$+0.075\,[0.028,0.121]$
PA D-opt	$+0.465\,[0.095,0.834]$	$+0.386\,[0.013,0.760]$	$+0.075\,[0.029,0.121]$

Appendix

Table 10: Overall paired Arena comparisons. Positive values favor NAOD. Proxy-regret and human-CE gains are in

10^{-3}

units. Accuracy gains are percentage points. Brackets give paired

95\%

t_{14}

intervals across the

15

repeated splits.

Figure 4: Budget dependence on Chatbot Arena. NAOD relative reduction in mean proxy policy regret across the complete

5\times 3

Arena budget grid. Positive values favor NAOD. Panel (a) compares against Target-info and panel (b) against D-opt. Cell values are ratios of macro-averaged mean regrets rather than averages of split-specific percentage reductions.

$H$	$B$	Target-info–NAOD	D-opt–NAOD
32	256	$+2.103\,[0.263,3.944]$	$+1.543\,[0.612,2.474]$
32	512	$+1.141\,[0.447,1.835]$	$+0.715\,[0.138,1.291]$
32	1024	$+0.408\,[0.143,0.673]$	$+0.160\,[-0.192,0.512]$
64	256	$+1.731\,[0.507,2.955]$	$+1.234\,[0.359,2.109]$
64	512	$+1.123\,[0.476,1.770]$	$+0.703\,[0.125,1.281]$
64	1024	$+0.471\,[0.184,0.758]$	$+0.234\,[-0.119,0.587]$

Appendix

Table 11: Paired proxy-regret gains across the Arena budget grid. Entries are baseline-minus-NAOD gains in

10^{-3}

units with pointwise paired

95\%

t_{14}

intervals.

Figure 5: Judge-level coupling and realized acquisition gain. The horizontal axis is Target-info target–nuisance coupling

\rho^{2}

and the vertical axis is the Target-info-minus-NAOD proxy-regret gain in

10^{-3}

units. Each point represents one judge after averaging over the

15

budget combinations and

15

repeated cluster-level splits. The Spearman rank correlation is

r_{s}=0.576

. Positive vertical values favor NAOD.

Judge	$\rho_{T}^{2}$	Policy gain
Athene-70B	0.570	$+0.428\,[-0.290,1.145]$
dolphin-2.1-mistral-7b	0.956	$+2.063\,[1.076,3.049]$
dolphin-2.5-mixtral-8x7b	0.960	$+2.819\,[1.248,4.390]$
Hermes-3-Llama-3.1-70B	0.610	$+0.366\,[-0.305,1.038]$
Meta-Llama-3-70B-Instruct	0.459	$-0.037\,[-0.358,0.285]$
Meta-Llama-3-8B-Instruct	0.716	$+0.733\,[0.069,1.397]$

Appendix

Table 12: Judge-level coupling and policy gain.

\rho_{T}^{2}

is the squared target–nuisance coupling under Target-info. Policy gain is Target-info minus NAOD in

10^{-3}

units with pointwise paired

95\%

t_{14}

intervals.

Explore similar work

Low-Resource Preference Adaptation of LLMs via Activation-Based Label Propagation
Aug 31, 2026Alessio Galatolo, Meriem BeloucifSequential Preference Optimization Preference Alignment Learning
Uppsala University
CUPID in the Model Zoo: Online Matchmaking for Selecting Your Dream LLM
May 30, 2026Son Nguyen, Xinyuan Liu, Ransalu SenanayakeLarge Language Model Alignment Bandits
School of Computing and Augmented Intelligence, Arizona State University, Tempe, United States of America
Quantifying and Auditing LLM Evaluation via Positive--Unlabeled Learning
Jun 17, 2026Zilong Zhang, Yi-Ting Hung, Lei Ding +1Large Language Model Evaluation Positive--Unlabeled Learning
Department of Mathematics and Statistics Georgia State University · Department of Statistics University of Manitoba

Aug 31, 2026, cs.CLJ/K move · Enter open · S save

Low-Resource Preference Adaptation of LLMs via Activation-Based Label Propagation

Alessio Galatolo, Meriem Beloucif

Uppsala University

Sequential Preference Optimization Preference Alignment Learning Large Language Model Adaptation Label Propagation Human Annotations

Open paper AskPDF arXiv

Adapting large language models to user-specific preferences is often constrained by the cost of human annotation, making preference optimisation impractical in low-resource settings where preferences cannot be reliably labelled by LLMs themselves, e.g., due to cultural, subjective, or personalised contexts. In this paper, we investigate how language models encode preference information in their intermediate representations, finding that activations from chosen and rejected responses form distinct clusters across layers, even in pretrained models. Strikingly, this structure is strengthened by alignment on canonical datasets but erased when the target preferences differ from those the model was aligned on, suggesting aligned LLMs are poor judges for non-mainstream populations. Exploiting this structure, we propose training a lightweight linear probe on a few labelled preference pairs (

\leq

500) and using it to annotate large unlabelled datasets (50K+) for downstream preference optimisation. We systematically evaluate this approach across different datasets, preference optimisation methods and model scales and find that our method consistently outperforms direct training given the same annotation budget, and remains competitive against baselines trained on

50-100\times

more labelled data in the majority of our settings. Code is available at https://github.com/alessioGalatolo/activ-pref-probe.

Abstract

Explore similar work

Low-Resource Preference Adaptation of LLMs via Activation-Based Label Propagation

CUPID in the Model Zoo: Online Matchmaking for Selecting Your Dream LLM

Quantifying and Auditing LLM Evaluation via Positive--Unlabeled Learning