Predictively Oriented Gaussian Process Posteriors

Callum Lau; Jeremias Knoblauch; Louis Sharrock

stat.MLOct 2, 2026

Predictively Oriented Gaussian Process Posteriors

Authors: Callum Lau, Jeremias Knoblauch, Louis Sharrock

Organizations: University College London

Abstract

Gaussian Processes (GPs) are a powerful tool for modelling and quantifying uncertainty in functional relationships. However, they require practitioners to make a number of design decisions, such as the choice of the kernel and the observation model. Suboptimal choices can produce misspecified models that do not capture the underlying data generating process. We introduce Predictively Oriented Gaussian Processes (PrO-GPs), which treat predictive uncertainty as the primary inferential target and provide a robust alternative to standard GPs. Although direct computation of a PrO posterior for nonparametric models is intractable, we derive a reduced formulation and practical sampling scheme for efficient computation. Through synthetic and real data experiments, we show that PrO-GPs produce better calibrated predictive distributions under model misspecification compared to standard GP approaches.

Figures & tables

Figure 1: Bayesian GP and PrO-GP posterior predictive on multimodal data . The GP posterior is centred around a single compromise function, with its predictive mean lying between the two modes. The PrO-GP instead recovers a mixture of functions, assigning predictive mass to both branches.

Model	Description
gp	Bayesian GP fit with marginal log-likelihood maximisation
pro-gp	PrO-GP with $\sigma^{2}$ estimated as in Section 3.5 . $\gamma$ is identical to the gp .
svgp	Sparse Variational GP.
ppgpr	svgp optimised with PPGPR.
i-pro-gp S/P	pro-gp with inducing point approximation. Inducing point locations and $\gamma$ are identical to the (S) svgp or the (P) ppgpr model.

Table 1: Model variants

Figure 2: Left : Illustrative gp and pro-gp predictive densities on four synthetic regression tasks. Right: Mean NLPD (lower is better) and 95% credible interval over increasing training set size, randomised over twenty DGPs.

Sensors	Graph gp	Graph pro-gp	$\Delta_{\text{NLPD}}$
200	1.13 $\pm$ 0.02	1.00 $\pm$ 0.02	0.13
225	1.13 $\pm$ 0.02	0.98 $\pm$ 0.03	0.15
250	1.12 $\pm$ 0.03	0.93 $\pm$ 0.04	0.19
275	1.11 $\pm$ 0.03	0.89 $\pm$ 0.04	0.22

Table 2: Graph regression NLPD (

\pm 1

SE).

Figure 3: Graph gp and Graph pro-gp posterior predictive standard deviations for the road network. Circles are sensor nodes, black dots are training nodes. Zoomed in predictions are in the bottom left.

Exact		concreteslump $n=103$	autos $n=159$	machine $n=209$	autompg $n=392$	housing $n=506$	stock $n=536$	energy $n=768$	concrete $n=1030$
	gp	-1.30 (0.17)	0.50 (0.10)	0.61 (0.05)	0.44 (0.07)	0.33 (0.06)	0.28 (0.03)	-1.68 (0.01)	0.18 (0.03)
	pro-gp	-1.65 (0.08)	0.41 (0.07)	0.61 (0.05)	0.30 (0.06)	0.25 (0.06)	0.28 (0.03)	-1.85 (0.02)	0.03 (0.03)
Sparse		wine $n=1599$	skillcraft $n=3338$	abalone $n=4177$	parkinsons $n=5875$	airquality $n=6941$	elevators $n=16599$	kin40k $n=40000$	protein $n=45730$
	svgp	0.61 (0.02)	0.98 (0.01)	1.00 (0.02)	-2.01 (0.01)	0.09 (0.02)	0.45 (0.00)	-0.21 (0.00)	1.06 (0.00)
	i-pro-gp s	0.51 (0.03)	0.95 (0.01)	0.81 (0.02)	-2.63 (0.01)	-0.28 (0.02)	0.32 (0.00)	-0.61 (0.00)	0.64 (0.00)
	ppgpr	0.59 (0.03)	1.07 (0.02)	0.93 (0.03)	-2.24 (0.03)	-0.26 (0.03)	0.35 (0.01)	-0.59 (0.00)	0.81 (0.00)

Table 3: Mean NLPD (

\pm 1

SE) across 16 UCI regression datasets with 10-fold random splits. Exact : gp and pro-gp evaluated on datasets with no inducing point approximation. Sparse : svgp , ppgpr , i-pro-gp s and i-pro-gp p evaluated on datasets with inducing point approximation. Recall here that i-pro-gp s and i-pro-gp p are inducing point approximations of pro-gp with kernel-hyperparameters and inducing point locations given by those of svgp and ppgpr , respectively. See Table 1 for full details.

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Graph gp and Graph pro-gp posterior inference on the San Jose road network. Zoomed in predictions are in the bottom left of each plot. Circles represent sensor nodes. Black dots represent training nodes. Top left: Graph gp predictive mean. Top right: Graph gp predictive standard deviation. Bottom left: Graph pro-gp predictive mean. Bottom right: Graph pro-gp predictive standard deviation

Figure 5: Diagnostic plots of whitened GP residuals for the machine, stock, and concreteslump datasets. Left: normal Q–Q plots with 95% pointwise reference bands. Centre: residual histograms overlaid with the standard normal density. Right: residuals against training index, with reference lines at 0 and ±2 standard deviations

Figure 6: Held-out NLPD on 16 UCI regression datasets. Blue: omgp as a function of the number of mixture components $M$ , with kernel-hyperparameters and inducing points fixed to the gp or svgp fit. Green line, solid: pro-gp or i-pro-gp s with $K=50$ particles. Orange line, dashed: gp or svgp . Error bars and shaded bands show 1 standard error.

Figure 7: i-pro-gp runtime analysis. Left: Cost per iteration for i-pro-gp and svgp with batch size of 512. Right: i-pro-gp average training score vs Replica–Gibbs iteration

Explore similar work

Information-Based Calibration of Uncertainty Quantification in Product-of-Experts Gaussian Process Models
Aug 29, 2026Yean Hoon Ong, Paolo Barucca, Wei Pan +1Uncertainty Quantification Gaussian Processes
University College London, London, UK · The University of Manchester, Manchester, UK
Goal-Oriented Lower-Tail Calibration of Gaussian Processes for Bayesian Optimization
May 19, 2026Aurélien Pion, Emmanuel VazquezPost-Hoc Calibration Bayesian Optimization
Transvalor S.A., Biot, France · Univ. Paris-Saclay, CNRS, CentraleSup´elec, L2S, Gif-sur-Yvette, France
GGMPs: Generalized Gaussian Mixture Processes
Mar 11, 2026Vardaan Tekriwal, Mark D. Risser, Hengrui Luo +1Conditional Distribution Estimation Gaussian Processes
Applied Math & Computational Research Division, Lawrence Berkeley National Laboratory, Berkeley, CA · Department of Electrical Engineering & Computer Sciences, University of California, Berkeley, CA · Engineering Mathematics & Statistics Division, University of California, Berkeley, CA +3

Aug 29, 2026J/K move · Enter open · S save

Information-Based Calibration of Uncertainty Quantification in Product-of-Experts Gaussian Process Models

Yean Hoon Ong, Paolo Barucca, Wei Pan, Jun Wang

University College London, London, UK · The University of Manchester, Manchester, UK

Uncertainty Quantification Gaussian Processes Gaussian Process Regression Uncertainty Calibration

Open paper AskPDF arXiv

Gaussian process (GP) regression with a single global GP (GP-glo) incurs cubic computational cost, limiting scalability to large datasets. Product-of-experts GP models (GP-pro), which combine local GP models to capture global correlations, alleviate this computational burden. However, training local experts on disjoint data subsets can lead to overestimated posterior variances. We propose GP-pro-c, a product-of-experts GP model that calibrates these variances using an information-based method. The method exploits the monotonicity and submodularity of information gain in GPs to define a calibration ratio that reduces the posterior variance of individual local GP models. We evaluate GP-pro-c using negative log-likelihood (NLL), root mean squared error (RMSE), and expected normalised calibration error (ENCE). Experiments on four synthetic functions and six regression datasets show that GP-pro-c achieves average reductions of 2.3% in NLL and 12.0% in ENCE compared with the uncalibrated GP-pro model. The proposed method mitigates posterior variance overestimation while maintaining predictive accuracy and reducing computational complexity. GP-pro-c provides a promising approach for uncertainty estimation in scalable GP models and may serve as a useful surrogate model for Bayesian optimisation with high-dimensional and large-scale data.

Abstract

Explore similar work

Information-Based Calibration of Uncertainty Quantification in Product-of-Experts Gaussian Process Models

Goal-Oriented Lower-Tail Calibration of Gaussian Processes for Bayesian Optimization

GGMPs: Generalized Gaussian Mixture Processes