Authors: Gianluca Audone, Francesco Marchetti, Emma Perracchione, Milvia Rossini
Organizations: Faculty of Engineering and Physical Sciences, University of Southampton, United Kindom · Dipartimento di Matematica “Tullio Levi-Civita”, Universit`a di Padova, Italy · Dipartimento di Scienze Matematiche Giuseppe Luigi Lagrange, Politecnico di Torino, Italy · Universit`a degli Studi di Milano-Bicocca, Dipartimento di Matematica e Applicazioni, Italy
Classical Gaussian processes and Kriging models are commonly based on stationary kernels, whereby correlations between observations depend exclusively on the relative distance between scattered data. While this assumption ensures analytical tractability, it limits the ability of Gaussian processes to represent heterogeneous correlation structures. In this work, we investigate variably scaled kernels as an effective tool for constructing non-stationary Gaussian processes by explicitly modifying the correlation structure of the data. Through a scaling function, variably scaled kernels alter the correlations between data and enable the modeling of targets exhibiting abrupt changes or discontinuities. We analyse the resulting predictive uncertainty via the variably scaled kernel power function and clarify the relationship between variably scaled kernels-based constructions and classical non-stationary kernels. Numerical experiments demonstrate that variably scaled kernels-based Gaussian processes yield improved reconstruction accuracy and provide uncertainty estimates that reflect the underlying structure of the data
Figures & tables
Figure 1: Stationary case (left) and VSK approach (right). The training data consist of N=6 equispaced nodes.
Figure 2: Realization of the GPs using the prior (left) and posterior (right) covariance: stationary case (top) and VSK approach (bottom).
Figure 3: Training matrices in the stationary (left) and VSK (right) case.
Standard
VSK
RMSE
9.1728×10−1
8.4409×10−1
Maximum error
2.6058×100
2.3776×100
Average posterior std.
2.3022×10−1
2.8207×10−1
Maximum posterior std.
3.7459×10−1
8.7057×10−1
Table 1: Comparison between the stationary Matérn C2 kernel and the VSK kernel with fixed hyperparameters ℓ=1 , σf=8 , σn=0, and N=6 uniform nodes.
Figure 4: Approximation errors (left) and uncertainty (right) varying N from 10 to 790 Halton nodes.
Figure 5: Stationary case (left) and VSK approach (right). The training data consist of N=27 Halton nodes.
Figure 6: Stationary case (left) and VSK approach (right). The training data consist of N=81 Halton nodes.
Metric
Standard
VSK
Length scale ℓ
0.0650
0.0627
Signal variance σf
1.4827
1.4956
Noise variance σn
0.3238
0.2428
RMSE
5.3569×10−1
3.6767×10−1
Max error
3.0195×100
1.7015×100
Average std
4.6966×10−1
4.0266×10−1
Table 2: Comparison between stationary kernel and VSK for N=27 training Halton points (MLE-tuned hyperparameters).
Metric
Standard
VSK
Length scale ℓ
0.0614
0.0694
Signal variance σf
1.6096
1.6613
Noise variance σn
0.3269
0.3321
RMSE
2.4512×10−1
1.3972×10−1
Max error
2.0480×100
4.1026×10−1
Average std
3.8423×10−1
3.8782×10−1
Table 3: Comparison between stationary kernel and VSK for N=81 training Halton points (MLE-tuned hyperparameters).
Figure 7: Approximation errors (left) and uncertainty (right) varying KVSK from 0 (stationary) to 12 .
Figure 8: Approximation of the Weierstrass function, with confidence intervals. Top: the target function (left) and the stationary reconstruction (right). Bottom: VSK reconstructions with KVSK=2 (left), KVSK=4 (center) and KVSK=12 (right).
Figure 9: Approximation errors (left) and uncertainty (right) varying N from 11 to 791 Halton nodes.
Figure 10: Stationary case (left) and VSK approach (right). The training data consist of N=21 uniform nodes.
Figure 11: Stationary case (left) and VSK approach (right). The training data consist of N=20 uniform nodes.
Figure 12: Training matrices in the stationary (left) and VSK (right) case: with N=21 (top) and N=20 (bottom) uniform nodes.
Figure 13: Stationary case (left), VSK approach (center) and a corresponding Gibbs kernel (right). The training data consist of N=9 Chebyshev nodes.
Figure 14: Basis functions centered at x′=0 : stationary case (left), VSK approach (center) and a corresponding Gibbs kernel (right).
Kernels measure similarity or correlation in tasks such as regression and classification. The Gaussian kernel, other names of which include squared exponential and radial basis function kernel, is one of the most popular in Gaussian process regression. We argue that the Gaussian kernel is best avoided and should never be used as a default. The argument rests on two results demonstrating that the Gaussian kernel is extremely brittle. First, the Gaussian kernel gives rise to a conditional variance that is unrealistically small. If the variance is used to quantify predictive uncertainty, catastrophic overconfidence is almost inevitable. Second, a small variance goes hand in hand with numerical ill-conditioning, so that to use the Gaussian kernel in practice requires tricks such as nugget terms that effectively modify the underlying regression or classification model. These problems are caused by the unnatural smoothness of the Gaussian kernel, a fact we are far from the first to take notice of. The problem is not the Gaussian form itself but the analyticity of the kernel: Our argument is more broadly that analytic kernels are best avoided. For stationary kernels analyticity is essentially equivalent to an exponential decay of the spectral density.
Toni Karvonen, Chris J. Oates
School of Engineering Sciences Lappeenranta–Lahti University of Technology LUT Yliopistonkatu 34, 53850 Lappeenranta, Finland · School of Mathematics, Statistics & Physics Newcastle University Newcastle upon Tyne, NE1 7RU, United Kingdom
We introduce a scalable Gaussian process (GP) framework with deep product kernels for data-driven learning of parametrized spatio-temporal fields over fixed or parameter-dependent domains. The proposed framework learns a continuous representation, enabling predictions at arbitrary spatio-temporal coordinates, independent of the training data resolution. We leverage Kronecker matrix algebra to formulate a computationally efficient training procedure with complexity that scales nearly linearly with the total number of spatio-temporal grid points. A key feature of our approach is the efficient computation of the posterior variance at essentially the same computational cost as the posterior mean (exactly for Cartesian grids and via rigorous bounds for unstructured grids), thereby enabling scalable uncertainty quantification. Numerical studies on a range of benchmark problems demonstrate that the proposed method achieves accuracy competitive with operator learning methods such as Fourier neural operators and deep operator networks. On the one-dimensional unsteady Burgers' equation, our method surpasses the accuracy of projection-based reduced-order models. These results establish the proposed framework as an effective tool for data-driven surrogate modeling, particularly when uncertainty estimates are required for downstream tasks.
Srinath Dama, Prasanth B. Nair
University of Toronto Institute for Aerospace Studies, 4925 Dufferin Street, Toronto, M3H 5T6, Ontario, Canada
Gaussian processes constitute a cornerstone of probabilistic machine learning, yet scaling them to large datasets typically forces a trade-off between computational efficiency and model fidelity. This work bridges this gap by presenting an adaptive multi-resolution Gaussian process framework that is both scalable and exact. Our key innovation is constructing a naturally data-sparse covariance matrix with adaptive multi-resolution basis functions. These basis functions are directly anchored to samples, eliminating the need for auxiliary points. By shrinking the support domains of multi-resolution basis, the matrix block sizes are limited, guaranteeing sparsity. The inverse of the data-sparse covariance matrix is computed exactly and efficiently via the sparse Cholesky inverse algorithm. To further improve predictive uncertainties, we construct an augmented basis function. Theoretical analysis and numerical experiments demonstrate that our model achieves exact inference with O(nlog2n) training cost and O(logdn) prediction cost, establishing a principled framework for scalable and high-fidelity Gaussian process regression.
Yanchuang Cao, Jun Liu, Tengchao Yu +1
Institute of Applied Physics and Computational Mathematics, Beijing 100094, China