Adapting Nonstationary Multi-output Gaussian Processes to Bayesian Optimization

Zikai Xie

cs.LGSep 26, 2026

Adapting Nonstationary Multi-output Gaussian Processes to Bayesian Optimization

Authors: Zikai Xie

Organizations: State Key Laboratory of Precision and Intelligent Chemistry University of Science and Technology of China Hefei, Anhui 230000, China

Abstract

Multi-objective Bayesian optimization (MOBO) commonly relies on independent Gaussian processes (GPs) with stationary kernels, limiting its ability to represent nonstationary structure and share information between objectives. However, expressive nonstationary GPs do not necessarily make reliable BO decisions. We study this mismatch for the multi-output low-rank nonstationary (MO-LRN) GP: strong training fit can coexist with large off-design errors and optimistic acquisition predictions. We introduce MOLRN-BO, which combines a regularized shared-spectral surrogate with objective-specific residuals, prequential mean correction and tempered covariance scaling, and Pareto-local qLogEHVI optimization with periodic global search. Experiments on 12 deterministic bi-objective benchmarks show that MOLRN-BO substantially improves upon the original MO-LRN and achieves the best average problem ranks for final normalized hypervolume and normalized inverted generational distance among nine evaluated algorithms. It also achieves the strongest adverse-tail performance while remaining competitive with the leading baselines in anytime optimization. Ablation studies further show that the shared spectral construction improves off-design prediction, the local--global decision policy improves optimization performance, and hierarchical calibration reduces systematic candidate bias. These results demonstrate that nonstationary multi-output surrogates can deliver strong and robust MOBO performance when their structure and use are explicitly adapted to the demands of sequential optimization.

Figures & tables

Figure 1: Development diagnostics for the original MO-LRN surrogate on WFG4. (a) One-dimensional posterior slice at

n=20

, obtained by varying

x_{9}

while fixing the remaining coordinates at their training-data medians. Solid curves show posterior means, shaded regions show

\pm 2

posterior standard deviations, and dashed curves show the true objectives. (b) Strictly prequential standardized prediction errors,

|y-\mu|/\sigma_{\mathrm{pred}}

, at acquisition-selected candidates over three trials. Predictions are evaluated before each candidate is added to the training set; the dashed line at 1.645 denotes the nominal two-sided 90% predictive limit. (c) Training, global, and Pareto-region RMSE together with top-HVI optimism for

Q=2

and

Q=4

MO-LRN models fitted to identical observations at

n=110

. Bars denote means and points denote 15 fits from three data trajectories and five restarts.

Method	Final nHV	Final nIGD	HV-regret AUC	nIGD AUC
MOLRN-BO	2.25	2.17	2.42	2.83
qLogNParEGO	2.42	3.00	2.17	2.75
qLogEHVI	3.42	4.33	3.33	3.83
mobopt	4.67	4.25	4.92	4.00
JES	5.50	4.83	5.50	4.75
Original MO-LRN	6.42	5.33	6.33	5.58

Table 1: Mean problem ranks over 12 benchmarks using all valid completed trials. Lower rank is better. Bold and underlined entries denote the best and second-best method for each metric, respectively.

Panel of Figure 2 — Figure 2: Per-problem optimization trajectories on the 12 benchmarks. Solid curves show across-trial means, and shaded regions indicate one sample standard deviation. Both metrics are lower-is-better.

(a) Fixed-design surrogate	Training RMSE	Global RMSE	Pareto-region RMSE	Top-HVI optimism
Output-specific, $Q=4$	0.018	1.154	1.101	1.213
Output-specific, $Q=2$	0.062	0.816	0.792	0.498
Shared-spectrum, $Q=2$	0.063	0.746	0.729	0.087
(b) Search / bias	Final nHV $\uparrow$	Final nIGD $\downarrow$	HV-regret AUC $\downarrow$	nIGD AUC $\downarrow$
Full-domain / pooled	0.459	0.255	0.633	0.349
Local–global / independent	0.590	0.201	0.570	0.323

Table 2: Controlled diagnostics and ablations over 12 benchmarks. Values are means over available paired problem–seed observations, except candidate bias, reported as the median problem-level absolute bias. Panel (a) disables all RBF residuals.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Failure mode	Pilot experiment and sample	Key result
Capacity–decision mismatch	WFG4 at $n=110$ ; three data trajectories and five model fits per trajectory; $Q=4$ and $Q=2$ fitted to identical data	$Q=4$ vs. $Q=2$ : train RMSE $0.0022$ vs. $0.0082$ ; global RMSE $0.9518$ vs. $0.7730$ ; Pareto-region RMSE $0.9372$ vs. $0.8530$ .
Boundary-saturating spectra	Original $Q=4$ MO-LRN on WFG4; 300 sequential fits with $f_{\max}=8$	At least one learned frequency was within $5\%$ of the imposed bound in $296/300$ fits ( $98.7\%$ ); the mean per-fit maximum frequency was $7.96$ , and normalized roughness averaged $59.6$ .
Vanishing spectral overlap	Original $Q=4$ MO-LRN on ten-dimensional WFG4; spectral snapshots at $n=20,50,80,$ and $110$	Fractions of mixtures with $R_{q,12}<10^{-12}$ were $47\%,95\%,100\%,$ and $100\%$ , respectively.
Biased and overconfident predictions	Original $Q=4$ MO-LRN on WFG4; 300 strictly prequential predictions at acquisition-selected points	Normalized RMSE was $3.90$ , while the empirical coverage of nominal $90\%$ posterior intervals was only $3.7\%$ .
Extrapolative acquisition search	WFG4; three paired pilot trials and 300 acquisition decisions per policy	Full-domain vs. local–global search: nearest-data distance $0.741$ vs. $0.497$ ; boundary-coordinate fraction $24.5\%$ vs. $16.7\%$ .

Appendix

Table 3: Preliminary BO-oriented diagnostics on the ten-dimensional WFG4 development problem. These descriptive pilot results motivate the design of MOLRN-BO; broader controlled evaluations are presented in Section 4.3.

Method	nHV-q10	nIGD-q90	HV-regret-AUC-q90	nIGD-AUC-q90	Final nIGD +	Final $I_{\epsilon+}$
MOLRN-BO	2.08	2.08	2.00	2.83	2.25	3.17
qLogNParEGO	2.83	3.50	3.00	3.33	2.50	2.83
qLogEHVI	4.08	4.83	3.58	3.92	4.00	4.67
mobopt	5.12	4.33	5.54	4.67	4.58	3.67
JES	5.21	4.58	5.21	4.25	4.75	4.42
Original MO-LRN	6.12	5.08	6.29	5.75	5.83	5.33

Appendix

Table 4: Mean problem ranks for adverse-tail and additional Pareto-quality metrics over 12 benchmarks using all valid trials. Lower is better. Tail quantiles are computed across seeds within each problem. Bold and underlined entries denote the best and second-best ranks, respectively.

Explore similar work

Learning What to Evaluate: Correlation-Aware Decoupling for Multiobjective Bayesian Optimization
Sep 26, 2026Ashwin Renganathan, Peter BachmanBayesian Optimization Multi-Objective Optimization
Aerospace Engineering Pennsylvania State University University Park, PA 16802 · Institute of Computational and Data Sciences (ICDS) Pennsylvania State University University Park, PA 16802
Accelerating Multi-Objective Bayesian Optimisation via Predictive-Gradient Catalysts
Jun 5, 2026Alma Rahat, Tinkle Chugh, Jonathan Fieldsend +1Surrogate-Assisted Optimization Bayesian Optimization
Loughborough University, Loughborough, LE11 3TU, U.K. · University of Exeter, Exeter EX4 4QD, U.K. · The University of Manchester, Manchester M15 6PB, U.K.
Multi-Objective Bayesian Optimization via Adaptive \varepsilon-Constraints Decomposition
Apr 17, 2026Yaohong Yang, Sammie Katt, Samuel KaskiBayesian Optimization Multi-Objective Optimization
Department of Computer Science, Aalto University, Espoo, Finland · ELLIS Institute Finland · Department of Computer Science, University of Manchester, Manchester, United Kingdom

Sep 26, 2026J/K move · Enter open · S save

Learning What to Evaluate: Correlation-Aware Decoupling for Multiobjective Bayesian Optimization

Ashwin Renganathan, Peter Bachman

Aerospace Engineering Pennsylvania State University University Park, PA 16802 · Institute of Computational and Data Sciences (ICDS) Pennsylvania State University University Park, PA 16802

Bayesian Optimization Multi-Objective Optimization Gaussian Processes

Open paper AskPDF arXiv

Multiobjective Bayesian optimization (MOBO) with Gaussian process (GP) surrogates is a sample efficient approach to solving multiobjective optimization problems. In MOBO, a Bayesian decision theoretic acquisition function guides the adaptive selection of new candidate inputs, on which objectives and constraints are evaluated to update the surrogate model sequentially. Existing approaches maintain independent GP models for the objectives and constraints, with new observations evaluating all objectives and constraints in a coupled fashion. However, the objectives and constraints often contain inherent correlations which, if exploited, can enable

{decoupled}

evaluations where only a subset of them are evaluated at each round. We present a new approach that leverages a multitask GP model to jointly learn all objectives and constraints, and propose a total correlation metric that enables identifying an

{optimal}

subset of objectives and constraints to be evaluated at every round, even under uniform evaluation costs. Theoretically, we show that our acquisition policy is asymptotically consistent despite decoupling and that our proposed decoupled subset selection rule maximizes the expected posterior entropy reduction about unevaluated tasks under mild conditions. Empirically, we show that our approach outperforms coupled and decoupled baselines in the state of the art.

Abstract

Explore similar work

Learning What to Evaluate: Correlation-Aware Decoupling for Multiobjective Bayesian Optimization

Accelerating Multi-Objective Bayesian Optimisation via Predictive-Gradient Catalysts

Multi-Objective Bayesian Optimization via Adaptive \varepsilon-Constraints Decomposition