Active Preference Learning over Latent Preference Archetypes for Many-Objective Bayesian Optimization

Manisha Dubey; Sebastiaan De Peuter; Wanrong Wang; Samuel Kaski

cs.LGMar 30, 2026

Active Preference Learning over Latent Preference Archetypes for Many-Objective Bayesian Optimization

Authors: Manisha Dubey, Sebastiaan De Peuter, Wanrong Wang, Samuel Kaski

Organizations: Department of Computer Science, University of Manchester, UK · Centre for AI in Assistive Autonomy, University of Edinburgh, Scotland, UK · Department of Computer Science, Aalto University, Espoo, Finland · Informatics Institute, University of Amsterdam, Amsterdam, Netherlands ·

Abstract

Preference-based many-objective Bayesian optimization typically assumes that all pairwise comparisons arise from a single latent utility function, despite real decision makers often exhibiting multiple latent preference archetypes across contexts. We propose an active preference learning framework for many-objective Bayesian optimization that infers latent preference archetypes from pairwise comparisons, enabling both the identification of the active trade-off strategy and the refinement of its associated preferences. Our framework represents preferences as a Dirichlet-process mixture of latent archetypes and introduces mixture-aware information-theoretic query strategies that separately target archetype identification and within-archetype refinement through a hybrid acquisition policy. Experiments on synthetic benchmarks and real-world chemical process design case-study consistently outperforms state-of-the-art preference-based Bayesian optimization methods while recovering interpretable latent preference structure beyond conventional single-utility models. The proposed mixture-aware diagnostics further quantify archetype recovery and preference calibration, providing insights that are not captured by optimization performance alone.

Figures & tables

Figure 1: Overview of the proposed active preference learning framework for many-objective Bayesian optimization. Pairwise comparisons are used to infer latent preference archetypes and their mixture weights(left). The inferred posterior guides Gaussian-process-based many-objective Bayesian optimization through a mixture-aware acquisition function and iterative preference feedback from decision maker (centre). Active query selection chooses informative intra-mode, inter-mode, and hybrid comparisons to jointly refine utility estimation and latent preference archetype discovery (right).

Panel of Figure 2 — Figure 2: (Top) Mixture-aware policies reduce regret faster and achieve lower final regret than unimodal and random-scalarization baselines, with Hybrid performing best overall. This is visible from the steeper early decline and lower terminal curves of Inter and Hybrid across DTLZ2, WFG, and chemical task, while Clusterless and scalarization methods plateau higher with greater variability. Curves show mean simple regret over five independent runs; vertical bars denote $\pm$ one standard error.(Bottom)Hybrid most accurately and stably recovers true archetypes, showing the largest early drop and lowest final error with the narrowest band; Inter drops quickly but plateaus, Intra refines slowly with higher variance, and Clusterless remains flat. We report mean aligned L1 error per outer iteration (band = per-true min–max; lower is better). Dataset: DTLZ2, persistent context.

Panel of Figure 3 — Figure 3: Mixture weight trajectory for chemical process design process. (left) Hybrid quickly identifies and stabilizes near the correct mode proportions. (right) Grey: reference Pareto front; coloured: evaluated solutions; stars: final best-utility solution. Hybrid converges to the preferred Pareto region.

Panel of Figure 4 — Figure 4: Robustness Analysis for WFG

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$L$	number of objectives
$K$	truncation level (max modes)
$\ell$	objective index
$k$	preference mode index
$\mathbf{w}_{k}$	weight vector for mode $k$
$\eta_{k}$	mixture weight of mode $k$

Appendix

Table 1: Notation summary.

Panel of Figure 5 — Figure 5: Mixture weight trajectory for PET production process. Clusterless (left) collapses to a dominant component, while Hybrid (right) quickly identifies and stabilizes near the correct mode proportions.

Panel of Figure 6 — Figure 6: Pareto coverage in objective space. Grey points-reference Pareto front; colored points - evaluated solutions; stars - final best-utility solution under the true mixture preference. Efficient trade-offs lie along the grey frontier. Hybrid achieves broader frontier coverage and identifies final solutions closest to the mixture-preferred Pareto region.

Panel of Figure 7 — Figure 7: $L_{1}$ error trajectories per inferred component for a persistent user on DTLZ. After Hungarian alignment (lower is better), Inter queries prevent mode collapse and rapidly reduce error for at least one archetype, while Clusterless largely collapses, with only one component improving and others remaining high

Panel of Figure 8 — Figure 8: Mixture-weight calibration for persistent user for DTLZ dataset. Inter queries best match the true mixture, while random collapses to one mode and intra largely ignores $\eta$ . Blue bars are inferred $\hat{\eta}$ , orange bars are projected true $\eta^{*}$ via Hungarian alignment. (a) Random: blue mass on one index and orange-only bars → mode collapse. (b) Intra: blue focuses on one component → poor allocation across modes. (c) Inter: clear blue–orange alignment → best identification and calibration across modes

Panel of Figure 9 — Figure 9: Homogenous Oracle ( $K^{*}=1$ )

Panel of Figure 10 — Figure 10: Utility Misspecification

Panel of Figure 11 — Figure 11: Lambda Sensitivity

Panel of Figure 12 — Figure 12: Scalability with increasing preference heterogeneity

Panel of Figure 13 — Figure 13: Sensitivity to misspecified preference modes

Active Preference Learning

See all 16 papers

Local Preferential Bayesian Optimization
Jun 1, 2026Johanna Menn, Miriam Kober, Paul Brunzema +2Pairwise Preference Learning Pairwise Preference Evaluation
Institute for Data Science in Mechanical Engineering, RWTH Aachen University, Aachen, Germany · Department of Clinical Research, University of Bern, Bern, Switzerland · Center for Reproducible Science and Research Synthesis, University of Zurich, Zurich, Switzerland +1
Towards Optimality in Active Preference Learning
Feb 2, 2026Yao Zhao, Kwang-Sung JunActive Preference Learning Active Learning
Department of Computer Science, University of Arizona, Tucson, USA.
In-Context Preference-Conditioned Pareto Set Learning from Limited Offline Data
Oct 15, 2026Yiming Yao, Liang Zhao, Qingfu ZhangMulti-Objective Optimization Pareto Optimization
City University of Hong Kong

Jun 1, 2026J/K move · Enter open · S save

Local Preferential Bayesian Optimization

Johanna Menn, Miriam Kober, Paul Brunzema, David Stenger+1

Institute for Data Science in Mechanical Engineering, RWTH Aachen University, Aachen, Germany · Department of Clinical Research, University of Bern, Bern, Switzerland · Center for Reproducible Science and Research Synthesis, University of Zurich, Zurich, Switzerland+1

Pairwise Preference Learning Pairwise Preference Evaluation Bayesian Optimization Local Search Safe Bayesian Optimization Trust-Region Optimization

Open paper AskPDF arXiv

Bayesian optimization (BO) is a popular and effective approach for tuning expensive, noisy experiments, but requires the formulation of an explicit objective function. Preferential BO (PBO) removes this requirement by learning from pairwise human feedback, yet existing methods struggle to efficiently optimize beyond low- and medium-dimensional problems due to their global search approaches. We address this limitation by developing a family of local PBO methods that transfer key ideas from high-dimensional BO to the preferential setting. In particular, we introduce local PBO methods which adapt trust-region and derivative-informed local search to pairwise preference feedback, where the latter exploits first- and second-order derivatives of the Laplace-approximated GP posterior. Our benchmark on GP sample paths, standard optimization benchmark functions, and policy-search tasks shows that local PBO methods are especially effective in high-dimensional and complex landscapes with steep optima. Compared with global preference-based baselines, they can substantially reduce cumulative regret, making them particularly useful for real-world preference-based optimization tasks such as policy search.

Abstract

Active Preference Learning

Explore similar work

Local Preferential Bayesian Optimization

Towards Optimality in Active Preference Learning

In-Context Preference-Conditioned Pareto Set Learning from Limited Offline Data