Low-rank Low Precision Training of Gaussian Processes
Organizations: Department of Mathematics, TU Chemnitz, Chemnitz, Germany · Max Planck Institute for the Dynamics of Complex Technical Systems, Magdeburg, Germany · Department of Mathematics, Emory University, USA · Department of Numerical Mathematics, Faculty of Mathematics and Physics, Charles University
Abstract
Training Gaussian processes (GPs) at large scale is computationally challenging because evaluating the log-marginal likelihood and its gradients involves large, dense kernel matrices. We develop a scalable low-rank GP training approach that combines AnchorNet landmark selection with a fully independent training conditional (FITC) correction. The resulting objective and its analytic gradients can be evaluated using low-rank factors, diagonal quantities, and small dense matrices, without forming the full kernel matrix or its derivatives. The complete training procedure is implemented in single precision and evaluated on nine large-scale regression datasets. Across these problems, the method reaches the reference predictive accuracy while substantially reducing training time, although the required approximation rank varies across datasets. We also study the numerical behavior of the full-kernel predictive solve and show that single precision can become limiting for ill-conditioned problems. Finally, we investigate further reducing selected matrix operations to fp16 or TF32. These lower-precision variants provide only modest additional speedups and can become unreliable when the reduced system is poorly conditioned.
Figures & tables
| RMSE | time [s] | ||||||
|---|---|---|---|---|---|---|---|
| dataset | ANGP | Maddox | ANGP | Maddox | speed-up | ||
| PoleTele | (13.5K, 26) | 100 | 0.1075 0.0085 | 0.117 0.004 | 2.2 | 58 | |
| Elevators | (14.9K, 18) | 400 | 0.3607 0.0037 | 0.364 0.004 | 3.4 | 58 | |
| Bike | (15.6K, 17) | 100 | 0.0707 0.0071 | 0.074 0.003 | 2.4 | 62 | |
| Kin40K | (36K, 8) | 1600 | 0.0981 0.0018 | 0.099 0.001 | 17.4 | 66 | |
| Protein | (41.2K, 9) | 1600 | 0.5406 0.0068 | 0.550 0.006 | 20.5 | 71 | |
| PoleTele | Elevators | Bike | Kin40K | Protein | 3DRoad ∗ | Song | Buzz ∗ | HouseElectric ∗ | |
| (0.117) | (0.364) | (0.074) | (0.099) | (0.550) | (0.194) | (0.761) | (0.300) | (0.052) | |
| 50 | 0.1251 | 0.3806 | 0.0924 | 0.2663 | 0.7535 | 0.6319 | 0.7740 | 0.2672 | 0.0551 |
| 100 | 0.1075 | 0.3771 | 0.0707 | 0.2135 | 0.7086 | 0.5389 | 0.7710 | 0.2630 | 0.0549 |
| 200 | 0.1015 | 0.3667 | 0.0497 | 0.1764 | 0.6765 | 0.4944 | 0.7685 | 0.2655 | 0.0537 |
| 250 | 0.1000 | 0.3649 | 0.0480 | 0.1743 | 0.6539 | 0.4828 | 0.7677 | 0.2642 | 0.0534 |
| 400 | 0.1011 | 0.3607 | 0.0406 | 0.1371 | 0.6151 | 0.3991 | 0.7661 | 0.2648 | 0.0526 |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
Gaussian Processes
Gaussian Processes are probabilistic models of functions that use kernels to make predictions with explicit uncertainty.
See all 127 papers