Active Preference Learning over Latent Preference Archetypes for Many-Objective Bayesian Optimization
Authors: Manisha Dubey, Sebastiaan De Peuter, Wanrong Wang, Samuel Kaski
Organizations: Department of Computer Science, University of Manchester, UK · Centre for AI in Assistive Autonomy, University of Edinburgh, Scotland, UK · Department of Computer Science, Aalto University, Espoo, Finland · Informatics Institute, University of Amsterdam, Amsterdam, Netherlands · ELLIS Institute Finland, Helsinki, Finland
Preference-based many-objective Bayesian optimization typically assumes that all pairwise comparisons arise from a single latent utility function, despite real decision makers often exhibiting multiple latent preference archetypes across contexts. We propose an active preference learning framework for many-objective Bayesian optimization that infers latent preference archetypes from pairwise comparisons, enabling both the identification of the active trade-off strategy and the refinement of its associated preferences. Our framework represents preferences as a Dirichlet-process mixture of latent archetypes and introduces mixture-aware information-theoretic query strategies that separately target archetype identification and within-archetype refinement through a hybrid acquisition policy. Experiments on synthetic benchmarks and real-world chemical process design case-study consistently outperforms state-of-the-art preference-based Bayesian optimization methods while recovering interpretable latent preference structure beyond conventional single-utility models. The proposed mixture-aware diagnostics further quantify archetype recovery and preference calibration, providing insights that are not captured by optimization performance alone.
Figures & tables
Figure 1: Overview of the proposed active preference learning framework for many-objective Bayesian optimization. Pairwise comparisons are used to infer latent preference archetypes and their mixture weights(left). The inferred posterior guides Gaussian-process-based many-objective Bayesian optimization through a mixture-aware acquisition function and iterative preference feedback from decision maker (centre). Active query selection chooses informative intra-mode, inter-mode, and hybrid comparisons to jointly refine utility estimation and latent preference archetype discovery (right).
Figure 2: (Top) Mixture-aware policies reduce regret faster and achieve lower final regret than unimodal and random-scalarization baselines, with Hybrid performing best overall. This is visible from the steeper early decline and lower terminal curves of Inter and Hybrid across DTLZ2, WFG, and chemical task, while Clusterless and scalarization methods plateau higher with greater variability. Curves show mean simple regret over five independent runs; vertical bars denote ± one standard error.(Bottom)Hybrid most accurately and stably recovers true archetypes, showing the largest early drop and lowest final error with the narrowest band; Inter drops quickly but plateaus, Intra refines slowly with higher variance, and Clusterless remains flat. We report mean aligned L1 error per outer iteration (band = per-true min–max; lower is better). Dataset: DTLZ2, persistent context.
Figure 3: Mixture weight trajectory for chemical process design process. (left) Hybrid quickly identifies and stabilizes near the correct mode proportions. (right) Grey: reference Pareto front; coloured: evaluated solutions; stars: final best-utility solution. Hybrid converges to the preferred Pareto region.
Figure 4: Robustness Analysis for WFG
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
Symbol
Meaning
L
number of objectives
K
truncation level (max modes)
ℓ
objective index
k
preference mode index
wk
weight vector for mode k
ηk
mixture weight of mode k
Appendix
Table 1: Notation summary.
Figure 5: Mixture weight trajectory for PET production process. Clusterless (left) collapses to a dominant component, while Hybrid (right) quickly identifies and stabilizes near the correct mode proportions.
Figure 6: Pareto coverage in objective space. Grey points-reference Pareto front; colored points - evaluated solutions; stars - final best-utility solution under the true mixture preference. Efficient trade-offs lie along the grey frontier. Hybrid achieves broader frontier coverage and identifies final solutions closest to the mixture-preferred Pareto region.
Figure 7: L1 error trajectories per inferred component for a persistent user on DTLZ. After Hungarian alignment (lower is better), Inter queries prevent mode collapse and rapidly reduce error for at least one archetype, while Clusterless largely collapses, with only one component improving and others remaining high
Figure 8: Mixture-weight calibration for persistent user for DTLZ dataset. Inter queries best match the true mixture, while random collapses to one mode and intra largely ignores η . Blue bars are inferred η^ , orange bars are projected true η∗ via Hungarian alignment. (a) Random: blue mass on one index and orange-only bars → mode collapse. (b) Intra: blue focuses on one component → poor allocation across modes. (c) Inter: clear blue–orange alignment → best identification and calibration across modes
Figure 9: Homogenous Oracle ( K∗=1 )
Figure 10: Utility Misspecification
Figure 11: Lambda Sensitivity
Figure 12: Scalability with increasing preference heterogeneity
Figure 13: Sensitivity to misspecified preference modes
Bayesian optimization (BO) is a popular and effective approach for tuning expensive, noisy experiments, but requires the formulation of an explicit objective function. Preferential BO (PBO) removes this requirement by learning from pairwise human feedback, yet existing methods struggle to efficiently optimize beyond low- and medium-dimensional problems due to their global search approaches. We address this limitation by developing a family of local PBO methods that transfer key ideas from high-dimensional BO to the preferential setting. In particular, we introduce local PBO methods which adapt trust-region and derivative-informed local search to pairwise preference feedback, where the latter exploits first- and second-order derivatives of the Laplace-approximated GP posterior. Our benchmark on GP sample paths, standard optimization benchmark functions, and policy-search tasks shows that local PBO methods are especially effective in high-dimensional and complex landscapes with steep optima. Compared with global preference-based baselines, they can substantially reduce cumulative regret, making them particularly useful for real-world preference-based optimization tasks such as policy search.
Johanna Menn, Miriam Kober, Paul Brunzema +2
Institute for Data Science in Mechanical Engineering, RWTH Aachen University, Aachen, Germany · Department of Clinical Research, University of Bern, Bern, Switzerland · Center for Reproducible Science and Research Synthesis, University of Zurich, Zurich, Switzerland +1
Aligning large language models (LLMs) depends on high-quality datasets of human preference labels, which are costly to collect. Although active learning has been studied to improve sample efficiency relative to passive collection, many existing approaches adopt classical experimental design criteria such as G- or D-optimality. These objectives are not tailored to the structure of preference learning, leaving open the design of problem-specific algorithms. In this work, we identify a simple intuition specific to preference learning that calls into question the suitability of these existing design objectives. Motivated by this insight, we propose two active learning algorithms. The first provides the first instance-dependent label complexity guarantee for this setting, and the second is a simple, practical greedy method. We show the optimality of our instance-dependent term up to a constant depending on the logit bound, compared with a lower-bound characteristic. We evaluate our algorithms on real-world preference datasets and synthetic data observing improved sample efficiency compared to existing methods.
Yao Zhao, Kwang-Sung Jun
Department of Computer Science, University of Arizona, Tucson, USA.
Multi-objective optimization involves diverse preferences over conflicting objectives. Rather than returning a finite set of trade-off solutions, preference-conditioned Pareto set learning seeks to map a specified preference directly to a corresponding Pareto optimal solution. Yet in many real-world scenarios, optimization must proceed offline without further objective evaluations, leaving only a limited static dataset that sparsely covers the decision and objective spaces. Existing methods either optimize separately for each preference or train a problem-specific Pareto solution model, requiring repeated search across preferences or repeated training across problems. We introduce in-context preference-conditioned Pareto set learning (ICPSL), a framework that amortizes both preference-wise solution search and problem-specific model construction. ICPSL pretrains a foundation-style hypernetwork on a large and diverse set of synthetic prior data. Given the offline observations of an unseen problem, the pretrained hypernetwork learns to generate a problem-specific, preference-conditioned Pareto solution model in a single forward pass. The generated model can then be reused to predict solutions for arbitrary preferences without problem-specific training or preference-wise search. Experiments on synthetic and real-world problems show that ICPSL produces high-quality Pareto solutions while providing substantial speedups in problem-specific solution-model construction.