Outlier-Robust Multi-Output Gaussian Processes

Joshua Rooijakkers; Leiv Rønneberg; François-Xavier Briol; Jeremias Knoblauch; Matias Altamirano

stat.MLOct 30, 2025

Outlier-Robust Multi-Output Gaussian Processes

Authors: Joshua Rooijakkers, Leiv Rønneberg, François-Xavier Briol, Jeremias Knoblauch, Matias Altamirano

Organizations: Econometric Institute, Erasmus University Rotterdam, The Netherlands · Department of Mathematics, University of Oslo, Norway · Department of Statistical Science, University College London, United Kingdom

Abstract

Multi-output Gaussian process (MOGP) regression allows modelling dependencies among multiple correlated response variables. Similarly to standard Gaussian processes (GPs), MOGPs are sensitive to model misspecification and outliers, which can distort predictions within individual outputs. In the multi-output case, this situation can be further exacerbated as anomalous observations in one response can adversely influence predictions for other correlated outputs. To handle this situation, we propose R-MOGP, an MOGP that is provably robust to outliers in any individual response. Our approach preserves conjugacy, achieving robustness without sacrificing computational efficiency, and integrates naturally with sparse variational inference for scalability. Empirically, we show that R-MOGP achieves performance comparable to existing state-of-the-art robust MOGPs at a fraction of their computational costs.

Figures & tables

Figure 1: Navitoclax dose-response with outlier. We show the viability of two cancer cell lines exposed to varying doses of the drug Navitoclax.

Figure 2: Synthetic imputation experiment with focused outliers. We consider a regression problem (

T=2

) with structured missingness in

y_{1}

and outliers in

y_{2}

. The MOGP predictives are sensitive to outliers, with errors propagating across outputs, whereas R-MOGP is robust in both outputs.

Outliers	Method	RMSE	NLPD
✓	MOGP	$0.20\pm 0.03$	$-0.37\pm 0.04$
	R-MOGP ( $w_{\textup{RCGP}}$ )	$0.18\pm 0.04$	$-0.44\pm 0.07$
	R-MOGP ( $w_{\textup{RMOGP}}$ )	$0.11\pm 0.01$	$-0.67\pm 0.08$
✗	MOGP	$0.093\pm 0.004$	$-0.94\pm 0.04$
	R-MOGP ( $w_{\textup{RCGP}}$ )	$0.093\pm 0.004$	$-0.94\pm 0.04$
	R-MOGP ( $w_{\textup{RMOGP}}$ )	$0.093\pm 0.004$	$-0.95\pm 0.04$

Table 1: Average RMSE and NLPD and standard deviation over 20 seeds under well-specified and outlier-contaminated settings.

Figure 3: Trade-off between average total runtime and NLPD across several outlier scenarios and methods.

Figure 5: Average runtime in seconds (left) and RMSE (right) across 10 runs on Electricity Load forecasting.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Synthetic imputation experiment using R-MOGPs based on the Linear Model of Coregionalisation (LMC) and Process Convolution (PC) frameworks.

Figure 7: Synthetic imputation experiment for MOGP, R-MOGP,

t

-MOGP, and L-MOGP.

Figure 8: Coverage plots for MOGP and R-MOGP in the synthetic imputation experiment with

N=1000

, with and without outliers. Coverage is the proportion of uncontaminated observations that fall within predictive intervals with nominal coverage

q

Figure 9: Posterior mean of R-MOGP (left), R-MOGP without shrinkage term (middle) and R-MOGP without noise term (right) for varying values of

\epsilon_{2}

on the synthetic imputation experiment.

Outliers	Method	Unweighted		Weighted		Squared
		RMSE	NLPD	RMSE	NLPD	RMSE	NLPD
	MOGP	$0.20\pm 0.03$	$-0.37\pm 0.04$	$0.18\pm 0.03$	$-0.48\pm 0.05$	$0.18\pm 0.03$	$-0.52\pm 0.07$
✓	R-MOGP ( $w_{\textup{RCGP}}$ )	$0.19\pm 0.03$	$-0.38\pm 0.04$	$0.19\pm 0.04$	$-0.40\pm 0.04$	$0.18\pm 0.04$	$-0.44\pm 0.07$
	R-MOGP ( $w_{\textup{RMOGP}}$ )	$0.18\pm 0.04$	$-0.38\pm 0.04$	$0.13\pm 0.03$	$-0.53\pm 0.06$	$0.11\pm 0.01$	$-0.67\pm 0.08$
	MOGP	$0.093\pm 0.003$	$-0.95\pm 0.04$	$0.093\pm 0.004$	$-0.95\pm 0.04$	$0.093\pm 0.004$	$-0.95\pm 0.04$
✗	R-MOGP ( $w_{\textup{RCGP}}$ )	$0.093\pm 0.003$	$-0.95\pm 0.03$	$0.093\pm 0.004$	$-0.94\pm 0.04$	$0.093\pm 0.004$	$-0.94\pm 0.04$

Appendix

Table 3: Comparison of MOGP and R-MOGP (

w_{\textup{RCGP}}

) and R-MOGP (

w_{\textup{RMOGP}}

) under different LOO-CV weighting schemes (unweighted, weighted, squared weights). The results show that R-MOGP (

w_{\textup{RMOGP}}

) combined with a squared weighted hyperparameter optimisation objective is most robust to outliers.

Figure 10: Effect of the weight function on the robustness of R-MOGP. R-MOGP (

w_{\scriptscriptstyle\textup{RMOGP}}

) uses the weight function described in this paper, while the R-MOGP (

w_{\scriptscriptstyle\textup{RCGP}}

) use the original RCGP weight with

\gamma_{t}(\mathbf{x},\mathbf{y})=m_{t}(\mathbf{x})=\frac{1}{N}\sum^{N}_{i=1}y_{i,t}

. Among all methods, only R-MOGP (

w_{\scriptscriptstyle\textup{RMOGP}}

) is robust to the contamination.

Panel of Figure 11 — Figure 11: Performance of MOGP and R-MOGP as the number of outliers in the observed vector increases. We generate $N=80$ data points from an ICM across $T=10$ outputs. We then contaminate $\epsilon_{s}=10\%$ of the observations with multivariate outliers for $s\in\{1,\dots,S\}$ with $S\leq T$ increasing incrementally. The prior mean is $m_{t}(\mathbf{x})=\frac{1}{N}\sum_{i=1}^{N}y_{i,t}$ . As the percentage of outliers grows, the performance of the MOGP drops substantially faster than that of the R-MOGP.

Outliers	Method	RMSE	NLPD
None	MOGP	0.12 (0.01)	$-0.92$ (0.08)
	R-MOGP	0.12 (0.01)	$-0.88$ (0.07)
	MLP	0.10 (0.004)	–
	BNN	0.21 (0.01)	0.18 (0.13)
	XGBoost	0.083 (0.01)	–
Uniform	MOGP	2.27 (0.29)	1.26 (0.06)

Appendix

Table 4: RMSE and NLPD for MOGP, R-MOGP, and baseline machine learning methods on the Energy Efficiency dataset under different outlier settings.

Figure 12: Navitoclax dose-response analysis with outlier in separate plots for MOGP, R-MOGP,

t

-MOGP and L-MOGP.

Method	RMSE	NLPD	Runtime (s)
MOGP (no outliers)	0.066 (0.012)	$-1.23$ (0.11)	0.048
MOGP	0.170 (0.020)	$-0.21$ (0.05)	0.049
R-MOGP	0.081 (0.011)	$-1.00$ (0.07)	0.040
$t$ -MOGP	0.067 (0.013)	$-1.06$ (0.06)	3.820
L-MOGP	0.068 (0.013)	$-0.65$ (0.73)	1.060

Appendix

Table 5: Predictive performance and runtime of MOGP variants on the Navitoclax experiment.

Figure 13: Robustness of MOGP and R-MOGP in MBS trades. The outlier at 4 pm in the 5.5% MBS distorts the MOGP predictions for that output and propagates errors to the 6% MBS. In contrast, R-MOGP is robust.

Gaussian Processes

Gaussian Processes are probabilistic models of functions that use kernels to make predictions with explicit uncertainty.

See all 121 papers

Aug 17, 2026

Variational Outlier-Robust Gaussian Process Regression with Generative Modeling

Outliers can substantially distort Gaussian process regression (GPR) due to its conventional Gaussian observation likelihood, leading to inaccurate model learning and prediction. To address this limitation, this article introduces a generative GPR model that captures observation-specific contamination and adaptively mitigates the influence of outliers. Subsequently, a variational generalized expectation-maximization procedure is used to learn the latent variables and GPR model parameters. Experiments on synthetic and real datasets under different contamination settings demonstrate that the proposed method remains competitive with-and in several cases outperforms-robust GPR baselines in prediction accuracy. Moreover, the proposed method shares the cubic computational scaling of the compared GPR methods.

Arslan Majal, Aamir Hussain Chughtai

Zunitech, LLC, Houston, USA · Institute of Data Science, University of Engineering and Technology, Lahore, Pakistan

Gaussian Process Regression Robust Regression Generative Modeling

Ask PDF

May 6, 2026

Transformed Latent Variable Multi-Output Gaussian Processes

Multi-Output Gaussian Processes (MOGPs) provide a principled probabilistic framework for modelling correlated outputs but face scalability bottlenecks when applied to datasets with high-dimensional output spaces. To maintain tractability, existing methods typically resort to restrictive assumptions, such as employing low-rank or sum-of-separable kernels, which can limit expressiveness. We propose the Transformed Latent Variable MOGP (T-LVMOGP), a novel framework that scales MOGPs to a massive number of outputs while preserving the capacity to capture meaningful inter-output dependencies. T-LVMOGP constructs a flexible multi-output deep kernel by mapping inputs and output-specific latent variables into an embedding space using a Lipschitz-regularised neural network. Combined with stochastic variational inference, our model effectively scales to high-dimensional output settings. Across diverse benchmarks, including climate modelling with over 10,000 outputs and zero-inflated spatial transcriptomics data, T-LVMOGP outperforms baselines in both predictive accuracy and computational efficiency.

Xiaoyu Jiang, Xinxing Shi, Sokratia Georgaka +2

Department of Computer Science, University of Manchester, Manchester, UK · Faculty of Biology, Medicine and Health, University of Manchester, Manchester, UK

Deep Kernel Learning Latent Variable Models Gaussian Processes Multi-Output Gaussian Processes Kernel Methods

Ask PDF

Sep 26, 2026

Adapting Nonstationary Multi-output Gaussian Processes to Bayesian Optimization

Multi-objective Bayesian optimization (MOBO) commonly relies on independent Gaussian processes (GPs) with stationary kernels, limiting its ability to represent nonstationary structure and share information between objectives. However, expressive nonstationary GPs do not necessarily make reliable BO decisions. We study this mismatch for the multi-output low-rank nonstationary (MO-LRN) GP: strong training fit can coexist with large off-design errors and optimistic acquisition predictions. We introduce MOLRN-BO, which combines a regularized shared-spectral surrogate with objective-specific residuals, prequential mean correction and tempered covariance scaling, and Pareto-local qLogEHVI optimization with periodic global search. Experiments on 12 deterministic bi-objective benchmarks show that MOLRN-BO substantially improves upon the original MO-LRN and achieves the best average problem ranks for final normalized hypervolume and normalized inverted generational distance among nine evaluated algorithms. It also achieves the strongest adverse-tail performance while remaining competitive with the leading baselines in anytime optimization. Ablation studies further show that the shared spectral construction improves off-design prediction, the local--global decision policy improves optimization performance, and hierarchical calibration reduces systematic candidate bias. These results demonstrate that nonstationary multi-output surrogates can deliver strong and robust MOBO performance when their structure and use are explicitly adapted to the demands of sequential optimization.

Zikai Xie

State Key Laboratory of Precision and Intelligent Chemistry University of Science and Technology of China Hefei, Anhui 230000, China

Multi-Output Gaussian Processes Bayesian Optimization Multi-Objective Optimization Gaussian Processes Gaussian Process Regression

Ask PDF

Abstract

Gaussian Processes

Explore similar work

Variational Outlier-Robust Gaussian Process Regression with Generative Modeling

Transformed Latent Variable Multi-Output Gaussian Processes

Adapting Nonstationary Multi-output Gaussian Processes to Bayesian Optimization