Low-rank Low Precision Training of Gaussian Processes
Authors: Moritz Poguntke, Eda Oktay, Yuanzhe Xi, Erin Carson, Martin Stoll
Organizations: Department of Mathematics, TU Chemnitz, Chemnitz, Germany · Max Planck Institute for the Dynamics of Complex Technical Systems, Magdeburg, Germany · Department of Mathematics, Emory University, USA · Department of Numerical Mathematics, Faculty of Mathematics and Physics, Charles University
Training Gaussian processes (GPs) at large scale is computationally challenging because evaluating the log-marginal likelihood and its gradients involves large, dense kernel matrices. We develop a scalable low-rank GP training approach that combines AnchorNet landmark selection with a fully independent training conditional (FITC) correction. The resulting objective and its analytic gradients can be evaluated using low-rank factors, diagonal quantities, and small dense matrices, without forming the full kernel matrix or its derivatives. The complete training procedure is implemented in single precision and evaluated on nine large-scale regression datasets. Across these problems, the method reaches the reference predictive accuracy while substantially reducing training time, although the required approximation rank varies across datasets. We also study the numerical behavior of the full-kernel predictive solve and show that single precision can become limiting for ill-conditioned problems. Finally, we investigate further reducing selected matrix operations to fp16 or TF32. These lower-precision variants provide only modest additional speedups and can become unreliable when the reduced system is poorly conditioned.
Figures & tables
Figure 1: A one-dimensional GP before and after conditioning on training data.
RMSE
time [s]
dataset
(n,d)
r
ANGP
Maddox
ANGP
Maddox
speed-up
PoleTele
(13.5K, 26)
100
0.1075 ± 0.0085
0.117 ± 0.004
2.2
58
26.6×
Elevators
(14.9K, 18)
400
0.3607 ± 0.0037
0.364 ± 0.004
3.4
58
17.4×
Bike
(15.6K, 17)
100
0.0707 ± 0.0071
0.074 ± 0.003
2.4
62
26.4×
Kin40K
(36K, 8)
1600
0.0981 ± 0.0018
0.099 ± 0.001
17.4
66
3.8×
Protein
(41.2K, 9)
1600
0.5406 ± 0.0068
0.550 ± 0.006
20.5
71
3.5×
Table 1: Fit time comparison with Maddox [ 15 , Table 1] .
PoleTele
Elevators
Bike
Kin40K
Protein
3DRoad ∗
Song
Buzz ∗
HouseElectric ∗
r
(0.117)
(0.364)
(0.074)
(0.099)
(0.550)
(0.194)
(0.761)
(0.300)
(0.052)
50
0.1251
0.3806
0.0924
0.2663
0.7535
0.6319
0.7740
0.2672
0.0551
100
0.1075
0.3771
0.0707
0.2135
0.7086
0.5389
0.7710
0.2630
0.0549
200
0.1015
0.3667
0.0497
0.1764
0.6765
0.4944
0.7685
0.2655
0.0537
250
0.1000
0.3649
0.0480
0.1743
0.6539
0.4828
0.7677
0.2642
0.0534
400
0.1011
0.3607
0.0406
0.1371
0.6151
0.3991
0.7661
0.2648
0.0526
Table 2: Test RMSE from the posterior mean solve against rank. The last row reports the speedup for hyperparameter fitting at the smallest rank that achieves at least the same accuracy as Maddox .
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 2: Residuals (top) and test RMSE (bottom) of fp32 CG with a rank- 200 AnchorNet preconditioner for Buzz with rank 250 .
Figure 3: True residual of CG for different preconditioner ranks for HouseElectric in fp32, at model rank 500 (left) and 3000 (right).
Figure 4: True residual (top) and test RMSE (bottom) of the fp64 solves on HouseElectric (left) and Buzz (right). Dotted lines show the low-rank predictive mean at the same rank, in the same color.
Gaussian processes constitute a cornerstone of probabilistic machine learning, yet scaling them to large datasets typically forces a trade-off between computational efficiency and model fidelity. This work bridges this gap by presenting an adaptive multi-resolution Gaussian process framework that is both scalable and exact. Our key innovation is constructing a naturally data-sparse covariance matrix with adaptive multi-resolution basis functions. These basis functions are directly anchored to samples, eliminating the need for auxiliary points. By shrinking the support domains of multi-resolution basis, the matrix block sizes are limited, guaranteeing sparsity. The inverse of the data-sparse covariance matrix is computed exactly and efficiently via the sparse Cholesky inverse algorithm. To further improve predictive uncertainties, we construct an augmented basis function. Theoretical analysis and numerical experiments demonstrate that our model achieves exact inference with O(nlog2n) training cost and O(logdn) prediction cost, establishing a principled framework for scalable and high-fidelity Gaussian process regression.
Yanchuang Cao, Jun Liu, Tengchao Yu +1
Institute of Applied Physics and Computational Mathematics, Beijing 100094, China
We present a theoretically grounded Gaussian process framework that leverages neural feature maps to construct expressive kernels. We show that the learned feature map can be interpreted as an optimal low-rank approximation to a Gram matrix derived from an implied RKHS, from which we establish consistency of the GP posterior. We further analyse the spectral properties of the induced kernels and introduce product feature-map kernels to address oversmoothing. This simple yet powerful approach enables fast, scalable, and accurate exact GP inference with minimal upfront work. The flexibility of kernel design supports seamless application to both regression and classification tasks across diverse data modalities, including tabular inputs and structured domains such as images. On benchmark datasets, this approach surpasses pre-existing methods in terms of accuracy and training and prediction efficiency.
This paper presents GpFlash, a Gaussian process library that is focused on fast training and posterior evaluations, minimal memory usage, and scalability to high-dimensional problems. GpFlash was compared to the popular GP libraries scikit-learn, GPy, GPflow, GPyTorch, and SMT on a MacBook Pro and the supercomputer Trillium. GpFlash was found to use less memory and to be faster for training and posterior evaluations relative to the other GP libraries for all cases considered. For example, when using \nx=104 training points in a 10-dimensional space GpFlash required 9.6 to 35 times less memory and was 2.6 to 54 times faster than the other GP libraries for the evaluation of the marginal log likelihood and its gradient. The memory reduction was achieved by only using one matrix of size \nx×\nx to store different symmetric matrices in its lower and upper triangular entries. The computational speedup was the result of designing algorithms to access memory efficiently, using BLAS and LAPACK methods extensively, and factoring kernel equations so that expensive exponential calculations could be reused. GpFlash was more than 440 times faster than all of the other GP libraries considered when evaluating the posterior mean together with its gradient for the case with 104 training points and 10 dimensions. GpFlash is available at https://github.com/marchildon/gpflash.
Andre Marchildon
Applied Scientist at Amazon Prime Air, Seattle, USA