Towards Optimality in Active Preference Learning

Yao Zhao; Kwang-Sung Jun

cs.LGFeb 2, 2026

Towards Optimality in Active Preference Learning

Authors: Yao Zhao, Kwang-Sung Jun

Organizations: Department of Computer Science, University of Arizona, Tucson, USA.

Abstract

Aligning large language models (LLMs) depends on high-quality datasets of human preference labels, which are costly to collect. Although active learning has been studied to improve sample efficiency relative to passive collection, many existing approaches adopt classical experimental design criteria such as G- or D-optimality. These objectives are not tailored to the structure of preference learning, leaving open the design of problem-specific algorithms. In this work, we identify a simple intuition specific to preference learning that calls into question the suitability of these existing design objectives. Motivated by this insight, we propose two active learning algorithms. The first provides the first instance-dependent label complexity guarantee for this setting, and the second is a simple, practical greedy method. We show the optimality of our instance-dependent term up to a constant depending on the logit bound, compared with a lower-bound characteristic. We evaluate our algorithms on real-world preference datasets and synthetic data observing improved sample efficiency compared to existing methods.

Figures & tables

Panel of Figure 1 — Figure 1: Intuition for our proposed method. An arm’s sign is considered confidently determined (green) if its confidence interval does not overlap with zero, and uncertain (red) otherwise. The numbers indicate the widths of the confidence intervals.

\lambda_{0}=\arg\min_{\lambda\in\Delta(\mathcal{Z})}\;\max_{z\in\mathcal{Z}}\|z\|^{2}_{(\sum_{z\in\mathcal{Z}}\lambda_{z}zz^{\top})^{-1}}

Algorithm 1 Warm-up procedure

Panel of Figure 2 — Figure 2: Illustration of remaining uncertainty (in olive). The remaining uncertainty is positive only when a confidence interval contains the decision boundary (zero), indicating that the arm’s sign is not yet confidently determined.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Preference classification accuracy on Anthropic HH, averaged over $50$ independent runs.

Panel of Figure 4 — Figure 4: Best-of- $N$ response selection on Anthropic HH with $N=8$ , evaluated by two reward-model judges. Each bar shows the percentage improvement of our method over the indicated baseline in judge-score gain above a uniformly random response from the same candidate pool. Scores are averaged over $1000$ held-out prompts and $50$ runs. Positive values favor our method.

Figure 5: Synthetic preference learning with

n=100

arms and

d=15

. Bars show the percentage of arms satisfying the empirical confidence-width criterion, averaged over

50

runs.

Active Preference Learning

See all 16 papers

Optimal Design for Active Preference Learning with Biased LLM Judges
Sep 30, 2026Zhongman Du, Huiming Zhang, Haodong Zhu +1Pairwise Preference Learning LLM Alignment
Beihang University · Zhongguancun Academy · Hangzhou Innovation Institute of Beihang University
Low-Resource Preference Adaptation of LLMs via Activation-Based Label Propagation
Aug 31, 2026Alessio Galatolo, Meriem BeloucifPairwise Preference Learning LLM Alignment
Uppsala University
ActiveUltraFeedback: Efficient Preference Data Generation using Active Learning
Mar 10, 2026Davit Melikidze, Marian Schneider, Jessica Lam +4Data Selection Active Learning
ETH Zurich · Institute of Neuroinformatics, University of Zurich and ETH Zurich · ETH AI Center

Sep 30, 2026J/K move · Enter open · S save

Optimal Design for Active Preference Learning with Biased LLM Judges

Zhongman Du, Huiming Zhang, Haodong Zhu, Baochang Zhang

Beihang University · Zhongguancun Academy · Hangzhou Innovation Institute of Beihang University

Pairwise Preference Learning LLM Alignment Active Learning Active Preference Learning

Open paper AskPDF arXiv

Learning from human preferences is central to large language model (LLM) alignment, but human preference annotation is costly. Active preference learning reduces this cost by selecting informative comparisons, and LLM judges can provide additional scalable feedback. However, the preferences of the judges may deviate from those of the target human population. Even after calibration on trusted reference data, active acquisition can shift the comparison distribution and expose residual judge bias. We therefore incorporate judge deviations into the acquisition design rather than relying on a separate calibration stage. Under joint estimation, comparisons that appear highly informative about the reward may also reflect judge bias and therefore provide less information about human preferences. To address this issue, we propose Nuisance-Adjusted Optimal Design (NAOD), a comparison-selection strategy that prioritizes policy-relevant target information after nuisance adjustment and uses the Frank-Wolfe algorithm for optimization. Theoretically, we establish a sharp conditional local asymptotic minimax lower bound on policy risk and construct an estimator that attains it. We further characterize the finite-sample cost of learning the nuisance representation and show that representation error can reverse an oracle design advantage. Finally, we validate these predictions experimentally and evaluate NAOD on Chatbot Arena data across 17 judges, 15 budget configurations, and 15 random cluster-level splits. NAOD reduces the mean regret of proxy policy by 29.1% relative to a matched target-information design, outperforms existing methods, and improves human-preference prediction on held-out data.

Abstract

Active Preference Learning

Explore similar work

Optimal Design for Active Preference Learning with Biased LLM Judges

Low-Resource Preference Adaptation of LLMs via Activation-Based Label Propagation

ActiveUltraFeedback: Efficient Preference Data Generation using Active Learning