Adapting Nonstationary Multi-output Gaussian Processes to Bayesian Optimization

Zikai Xie

cs.LGSep 26, 2026

Adapting Nonstationary Multi-output Gaussian Processes to Bayesian Optimization

Authors: Zikai Xie

Organizations: State Key Laboratory of Precision and Intelligent Chemistry University of Science and Technology of China Hefei, Anhui 230000, China

Abstract

Multi-objective Bayesian optimization (MOBO) commonly relies on independent Gaussian processes (GPs) with stationary kernels, limiting its ability to represent nonstationary structure and share information between objectives. However, expressive nonstationary GPs do not necessarily make reliable BO decisions. We study this mismatch for the multi-output low-rank nonstationary (MO-LRN) GP: strong training fit can coexist with large off-design errors and optimistic acquisition predictions. We introduce MOLRN-BO, which combines a regularized shared-spectral surrogate with objective-specific residuals, prequential mean correction and tempered covariance scaling, and Pareto-local qLogEHVI optimization with periodic global search. Experiments on 12 deterministic bi-objective benchmarks show that MOLRN-BO substantially improves upon the original MO-LRN and achieves the best average problem ranks for final normalized hypervolume and normalized inverted generational distance among nine evaluated algorithms. It also achieves the strongest adverse-tail performance while remaining competitive with the leading baselines in anytime optimization. Ablation studies further show that the shared spectral construction improves off-design prediction, the local--global decision policy improves optimization performance, and hierarchical calibration reduces systematic candidate bias. These results demonstrate that nonstationary multi-output surrogates can deliver strong and robust MOBO performance when their structure and use are explicitly adapted to the demands of sequential optimization.

Figures & tables

Figure 1: Development diagnostics for the original MO-LRN surrogate on WFG4. (a) One-dimensional posterior slice at

n=20

, obtained by varying

x_{9}

while fixing the remaining coordinates at their training-data medians. Solid curves show posterior means, shaded regions show

\pm 2

posterior standard deviations, and dashed curves show the true objectives. (b) Strictly prequential standardized prediction errors,

|y-\mu|/\sigma_{\mathrm{pred}}

, at acquisition-selected candidates over three trials. Predictions are evaluated before each candidate is added to the training set; the dashed line at 1.645 denotes the nominal two-sided 90% predictive limit. (c) Training, global, and Pareto-region RMSE together with top-HVI optimism for

Q=2

and

Q=4

MO-LRN models fitted to identical observations at

n=110

. Bars denote means and points denote 15 fits from three data trajectories and five restarts.

Method	Final nHV	Final nIGD	HV-regret AUC	nIGD AUC
MOLRN-BO	2.25	2.17	2.42	2.83
qLogNParEGO	2.42	3.00	2.17	2.75
qLogEHVI	3.42	4.33	3.33	3.83
mobopt	4.67	4.25	4.92	4.00
JES	5.50	4.83	5.50	4.75
Original MO-LRN	6.42	5.33	6.33	5.58

Table 1: Mean problem ranks over 12 benchmarks using all valid completed trials. Lower rank is better. Bold and underlined entries denote the best and second-best method for each metric, respectively.

Panel of Figure 2 — Figure 2: Per-problem optimization trajectories on the 12 benchmarks. Solid curves show across-trial means, and shaded regions indicate one sample standard deviation. Both metrics are lower-is-better.

(a) Fixed-design surrogate	Training RMSE	Global RMSE	Pareto-region RMSE	Top-HVI optimism
Output-specific, $Q=4$	0.018	1.154	1.101	1.213
Output-specific, $Q=2$	0.062	0.816	0.792	0.498
Shared-spectrum, $Q=2$	0.063	0.746	0.729	0.087
(b) Search / bias	Final nHV $\uparrow$	Final nIGD $\downarrow$	HV-regret AUC $\downarrow$	nIGD AUC $\downarrow$
Full-domain / pooled	0.459	0.255	0.633	0.349
Local–global / independent	0.590	0.201	0.570	0.323

Table 2: Controlled diagnostics and ablations over 12 benchmarks. Values are means over available paired problem–seed observations, except candidate bias, reported as the median problem-level absolute bias. Panel (a) disables all RBF residuals.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Failure mode	Pilot experiment and sample	Key result
Capacity–decision mismatch	WFG4 at $n=110$ ; three data trajectories and five model fits per trajectory; $Q=4$ and $Q=2$ fitted to identical data	$Q=4$ vs. $Q=2$ : train RMSE $0.0022$ vs. $0.0082$ ; global RMSE $0.9518$ vs. $0.7730$ ; Pareto-region RMSE $0.9372$ vs. $0.8530$ .
Boundary-saturating spectra	Original $Q=4$ MO-LRN on WFG4; 300 sequential fits with $f_{\max}=8$	At least one learned frequency was within $5\%$ of the imposed bound in $296/300$ fits ( $98.7\%$ ); the mean per-fit maximum frequency was $7.96$ , and normalized roughness averaged $59.6$ .
Vanishing spectral overlap	Original $Q=4$ MO-LRN on ten-dimensional WFG4; spectral snapshots at $n=20,50,80,$ and $110$	Fractions of mixtures with $R_{q,12}<10^{-12}$ were $47\%,95\%,100\%,$ and $100\%$ , respectively.
Biased and overconfident predictions	Original $Q=4$ MO-LRN on WFG4; 300 strictly prequential predictions at acquisition-selected points	Normalized RMSE was $3.90$ , while the empirical coverage of nominal $90\%$ posterior intervals was only $3.7\%$ .
Extrapolative acquisition search	WFG4; three paired pilot trials and 300 acquisition decisions per policy	Full-domain vs. local–global search: nearest-data distance $0.741$ vs. $0.497$ ; boundary-coordinate fraction $24.5\%$ vs. $16.7\%$ .

Appendix

Table 3: Preliminary BO-oriented diagnostics on the ten-dimensional WFG4 development problem. These descriptive pilot results motivate the design of MOLRN-BO; broader controlled evaluations are presented in Section 4.3.

Method	nHV-q10	nIGD-q90	HV-regret-AUC-q90	nIGD-AUC-q90	Final nIGD +	Final $I_{\epsilon+}$
MOLRN-BO	2.08	2.08	2.00	2.83	2.25	3.17
qLogNParEGO	2.83	3.50	3.00	3.33	2.50	2.83
qLogEHVI	4.08	4.83	3.58	3.92	4.00	4.67
mobopt	5.12	4.33	5.54	4.67	4.58	3.67
JES	5.21	4.58	5.21	4.25	4.75	4.42
Original MO-LRN	6.12	5.08	6.29	5.75	5.83	5.33

Appendix

Table 4: Mean problem ranks for adverse-tail and additional Pareto-quality metrics over 12 benchmarks using all valid trials. Lower is better. Tail quantiles are computed across seeds within each problem. Bold and underlined entries denote the best and second-best ranks, respectively.

Explore similar work

Sep 26, 2026

Learning What to Evaluate: Correlation-Aware Decoupling for Multiobjective Bayesian Optimization

Multiobjective Bayesian optimization (MOBO) with Gaussian process (GP) surrogates is a sample efficient approach to solving multiobjective optimization problems. In MOBO, a Bayesian decision theoretic acquisition function guides the adaptive selection of new candidate inputs, on which objectives and constraints are evaluated to update the surrogate model sequentially. Existing approaches maintain independent GP models for the objectives and constraints, with new observations evaluating all objectives and constraints in a coupled fashion. However, the objectives and constraints often contain inherent correlations which, if exploited, can enable

{decoupled}

evaluations where only a subset of them are evaluated at each round. We present a new approach that leverages a multitask GP model to jointly learn all objectives and constraints, and propose a total correlation metric that enables identifying an

{optimal}

subset of objectives and constraints to be evaluated at every round, even under uniform evaluation costs. Theoretically, we show that our acquisition policy is asymptotically consistent despite decoupling and that our proposed decoupled subset selection rule maximizes the expected posterior entropy reduction about unevaluated tasks under mild conditions. Empirically, we show that our approach outperforms coupled and decoupled baselines in the state of the art.

Ashwin Renganathan, Peter Bachman

Aerospace Engineering Pennsylvania State University University Park, PA 16802 · Institute of Computational and Data Sciences (ICDS) Pennsylvania State University University Park, PA 16802

Bayesian Optimization Multi-Objective Optimization Gaussian Processes

Ask PDF

Jun 5, 2026

Accelerating Multi-Objective Bayesian Optimisation via Predictive-Gradient Catalysts

This paper presents a general acceleration mechanism for multi-objective Bayesian optimisation (MOBO) that leverages Gaussian process predictive gradients as auxiliary signals. Rather than replacing existing Pareto-compliant acquisition functions, the proposed approach augments them with local stationarity information derived from surrogate-derived gradients, enabling faster convergence toward the global Pareto set under limited evaluation budgets. Two catalyst instantiations are investigated: an adaptive Multiple-Gradient Descent Algorithm-Based Catalyst (MGDA) and a predefined-weight variant that enables focused exploration when budgets are tight. Experiments on the DTLZ benchmark suite (using 2 objectives and 10 decision variables) show that predictive gradient catalysis can deliver significant acceleration compared to other acquisition functions (EHVI, AugTch, tMPoI, SAF) when surrogates are accurate, particularly for stationary problems.

Alma Rahat, Tinkle Chugh, Jonathan Fieldsend +1

Loughborough University, Loughborough, LE11 3TU, U.K. · University of Exeter, Exeter EX4 4QD, U.K. · The University of Manchester, Manchester M15 6PB, U.K.

Surrogate-Assisted Optimization Bayesian Optimization

Ask PDF

Apr 17, 2026

Multi-Objective Bayesian Optimization via Adaptive \varepsilon-Constraints Decomposition

Multi-objective Bayesian optimization (MOBO) provides a principled framework for optimizing multiple expensive black-box functions. However, existing MOBO methods often struggle with coverage, scalability, and handling constraints and preferences. In this work we propose STAGE-BO, Sequential Targeting Adaptive Gap-Filling

\varepsilon

-Constraint Bayesian Optimization: by analyzing the coverage of the surrogate Pareto front, our method identifies the Pareto front point with the largest uncovered gap, and uses its coordinates to define adaptive constraints in

\varepsilon

-constraint method, which transforms the problem into a sequence of inequality-constrained subproblems, efficiently solved via constrained expected improvement acquisition. Our approach provides uniform Pareto coverage without hypervolume computation and naturally handles constraints and preferences. Experiments on synthetic and real-world benchmarks demonstrate superior coverage and competitive hypervolume performance against state-of-the-art baselines. Our code implementation can be found at https://github.com/YangYaohong1/STAGE-BO.

Yaohong Yang, Sammie Katt, Samuel Kaski

Department of Computer Science, Aalto University, Espoo, Finland · ELLIS Institute Finland · Department of Computer Science, University of Manchester, Manchester, United Kingdom

Bayesian Optimization Multi-Objective Optimization

Ask PDF