Geometry-Aware Block Selection for Sparse Variational Gaussian Processes

Matteo Hoxha

stat.MLOct 20, 2026

Geometry-Aware Block Selection for Sparse Variational Gaussian Processes

Authors: Matteo Hoxha

Organizations: Nanyang Technological University, Singapore

Abstract

Sparse variational Gaussian processes (SVGPs) are a widely used, principled approach to scalable Gaussian process modelling. Their success stems from their theoretically interpretable structure: they provide a lower bound on the marginal likelihood of the full GP while allowing scal- able stochastic optimization. Recent work has explored modifications of the posterior conditional distribution and has shown that such extensions can improve upon this well-established framework. In particular, introducing a block-diagonal structured scal- ing matrix leads to a tighter lower bound on the GP marginal likelihood. We investigate whether selecting blocks that tighten the variational objective improves test RMSE and NLL at a fixed inducing-point budget. Since the scaling matrix contributes to the variational lower bound through an upper bound on the log-determinant of the full residual covariance matrix, our work makes the following contributions. First, we introduce a simple, scalable, geometry-aware algorithm, with explicit computational complexity and low computational overhead, that recovers a large fraction of this log- determinant contribution by selecting an appropriate partition of the data. Second, we show how to train these models using a doubly stochastic optimization framework. Finally, we investigate how sparse GP methods based on alternative conditional posterior structures can be used to learn better hyperparameters and inducing points for SVGP.

Figures & tables

Figure 1: The same fitted Snelson residual under random blocks (left) and selected blocks (right), using five inducing points. Stars mark inducing points; colours denote block membership. Blocks have $b=40$ points, and the log-det scale on the right compares the resulting penalties with the exact one.

Figure 2: Two-dimensional RBF example of Theorem 1 . The ordering

\pi

(bottom) returns each coloured group as one block; stars are inducing points, dashed lines cell boundaries.

Figure 3: Snelson: training bounds (top; dashed: E-SGPR); predictive fits with the block-diagonal predictor for T-SGPR, (R)BT-SGPR and (S)BT-SGPR and fitted hyperparameters (bottom).

Method	RMSE	NLL	Search (ms)
SGPR	$0.229(2)$	$-0.041(7)$	—
T-SGPR	$0.205(2)$	$-0.146(7)$	—
(R)BT-SGPR	$0.197(2)$	$-0.179(7)$	—
(S)BT-SGPR	$0.185(2)$	$-0.240(7)$	6.07
(METIS)BT-SGPR †	$0.179(2)$	$-0.296(6)$	1034.92
Exact GP †	$0.151(2)$	$-0.618(14)$	—

Table 1: RBF–Kin40k:

M=512

N=5000

, 2000 test points and

b=500

. Test RMSE/NLL use the block-diagonal predictor of Eq. ( 14 ); mean (SE in parentheses) over ten seeds. Search is the partition-build time in ms. Bold: best non-dense method; † dense reference.

Figure 4: RBF on Pol, Superconductivity, Kin40k and Protein. Left three columns: RMSE, NLL and negative final objective per datum of the block-diagonal predictors (Section 2 ; solid); right two columns: RMSE and NLL of the sparse SVGP after 300-epoch transplant refinement (dashed). Means and one standard error over ten seeds; lower is better. The objectives are method-specific stochastic estimates, not a common marginal likelihood. Amplitude-RBF follows on the next page; the full campaign is in Appendix E .

Figure 5: (Continued.) Amplitude-RBF on Pol, Superconductivity, Kin40k and Protein, with the same layout.

Figure 6: Negative stochastic objective during training on RBF–Pol,

M=128

, ten seeds. Separate likelihood and block batches converge better than a single shared batch for (S)BT-SVGP.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Test RMSE (left) and NLL (right) of the split-batch diagnostic of Figure 6 , with the block-diagonal predictor at the saved parameters; dashed: shared batch. Mean and one standard error over ten seeds.

	Direct sparse		Refined sparse
Donor	RMSE	NLL	RMSE	NLL
SGPR	.229(2)	$-$ .041(7)	.229(2)	$-$ .041(7)
T-SGPR	.205(2)	$-$ .122(7)	.207(2)	$-$ .119(7)
(R)BT-SGPR	.197(2)	$-$ .144(7)	.200(2)	$-$ .142(7)
(S)BT-SGPR	.185(2)	$-$ .158(7)	.190(2)	$-$ .164(7)

Appendix

Table 2: Sparse SGPR refinement of the same

M=512

donors. Kernel and noise parameters are fixed; inducing locations are optimized. Direct sparsification drops the covariance correction without refinement, leaving the predictive mean and RMSE unchanged. Mean (SE in the last decimal places), ten seeds; bold: lowest mean in each column.

Kernel	Dataset	$M$	Current	Common selected rule	Common random rule
RBF	Pol	64	-.059(5)	-.036(5)	-.017(4)
RBF	Pol	128	-.073(6)	-.045(6)	-.014(3)
RBF	Pol	256	-.078(4)	-.047(4)	-.011(4)
RBF	Pol	512	-.084(3)	-.051(4)	-.013(2)
RBF	Supercond.	64	-.033(2)	-.023(2)	-.008(3)
RBF	Supercond.	128	-.035(2)	-.026(2)	-.005(3)

Appendix

Table 3: Prediction-partition control on the configurations of Figure 4 : test NLL of (S)BT-SVGP minus that of (R)BT-SVGP (negative favours (S)BT-SVGP), mean (SE) over ten paired seeds. Current: each donor with its own

\mathcal{P}_{\mathrm{pred}}

, as in Figure 4 . Common selected rule: both donors with

\mathcal{P}_{\mathrm{pred}}

from Algorithm 1 at their saved parameters. Common random rule: both donors with a uniformly random partition into blocks of 500. The common selected rule applies Algorithm 1 separately at each donor’s saved parameters; block memberships need not coincide. Under the common random rule the donors do not share an index permutation: (R)BT-SVGP keeps the random partition it was trained with, and (S)BT-SVGP receives an independent uniformly random permutation in each paired seed. Donor parameters are frozen; RMSE is unchanged.

Method	Block diag. NLL	Dense NLL
SGPR	-.041(7)	-.041(7)
T-SGPR	-.146(7)	-.160(8)
(R)BT-SGPR	-.179(7)	-.213(8)
(S)BT-SGPR	-.240(7)	-.294(7)
(METIS)BT-SGPR †	-.296(6)	-.336(7)
Exact GP †	—	-.618(14)

Appendix

Table 4: RBF–Kin40k,

M=512

: test NLL (mean (SE) over ten paired seeds; lower is better). Block diag. is the predictor in the main results; dense uses the full

\mathbf{D}_{\mathbf{ff}}

in Eq. ( 7 ). Bold marks the best of the four principal methods; † denotes comparators.

Figure 8: Pol test NLL for the block-diagonal predictor used in the main results (dashed) and the dense variational predictor (solid), paired at the same saved parameters. Mean and one standard error over ten paired seeds.

Figure 9: RBF SVGP campaign: Elevators, Traffic, Electricity, 3Droad. Left: block-diagonal predictor and negative final objective (solid); right: sparse predictor after the transplant (dashed). Mean and one standard error over ten seeds.

Figure 10: Matérn-5/2 SVGP campaign: Pol, Superconductivity, Kin40k, Protein. Left: block-diagonal predictor and negative final objective (solid); right: sparse predictor after the transplant (dashed). Mean and one standard error over ten seeds.

Figure 11: Matérn-5/2 SVGP campaign: Elevators, Traffic, Electricity, 3Droad. Left: block-diagonal predictor and negative final objective (solid); right: sparse predictor after the transplant (dashed). Mean and one standard error over ten seeds.

Figure 12: Amplitude-RBF SVGP campaign: Elevators, Traffic, Electricity, 3Droad. Left: block-diagonal predictor and negative final objective (solid); right: sparse predictor after the transplant (dashed). Mean and one standard error over ten seeds.

Figure 13: Gibbs SVGP campaign: Pol, Superconductivity, Kin40k, Protein. Left: block-diagonal predictor and negative final objective (solid); right: sparse predictor after the transplant (dashed). Mean and one standard error over ten seeds.

Figure 14: Gibbs SVGP campaign: Elevators, Traffic, Electricity, 3Droad. Left: block-diagonal predictor and negative final objective (solid); right: sparse predictor after the transplant (dashed). Mean and one standard error over ten seeds.

Sparse Gaussian Processes

See all 10 papers

Inverse-Free Sparse Variational Gaussian Processes
Apr 1, 2026Stefano Cortinovis, Laurence Aitchison, Stefanos Eleftheriadis +1Sparse Gaussian Processes Gaussian Processes
University of Oxford · University of Bristol
Triply-Scalable Equivariant Gaussian Process Modeling
Sep 17, 2026Tim Steinert, David GinsbourgerSparse Gaussian Processes Gaussian Processes
Institute of Mathematical Statistics and Actuarial Science, University of Bern, Bern, Switzerland
Amortized Structured Stochastic Variational Inference for Gaussian Process Latent Variable Models
Oct 2, 2026Maksym Tretiakov, Sarah Filippi, Vincent Fortuin +3Latent Variable Models Amortized Inference
LMU Munich · Munich Center for Machine Learning (MCML) · Imperial College London +3

Apr 1, 2026J/K move · Enter open · S save

Inverse-Free Sparse Variational Gaussian Processes

Stefano Cortinovis, Laurence Aitchison, Stefanos Eleftheriadis, Mark van der Wilk

University of Oxford · University of Bristol

Sparse Gaussian Processes Gaussian Processes Variational Inference Gaussian Process Regression Natural Gradient Methods

Open paper AskPDF arXiv

Gaussian processes (GPs) offer appealing properties but are costly to train at scale. Sparse variational GP (SVGP) approximations reduce cost yet still rely on Cholesky decompositions of kernel matrices, ill-suited to low-precision, massively parallel hardware. While one can construct valid variational bounds that rely only on matrix multiplications (matmuls) via an auxiliary matrix parameter, optimising them with off-the-shelf first-order methods is challenging. We make the inverse-free approach practical by proposing a better-conditioned bound and deriving a matmul-only natural-gradient update for the auxiliary parameter, markedly improving stability and convergence. We further provide simple heuristics, such as step-size schedules and stopping criteria, that make the overall optimisation routine fit seamlessly into existing workflows. Across regression and classification benchmarks, we demonstrate that our method 1) serves as a drop-in replacement in SVGP-based models (e.g., deep GPs), 2) recovers similar performance to traditional methods, and 3) can be faster than baselines when well tuned.

Abstract

Sparse Gaussian Processes

Explore similar work

Inverse-Free Sparse Variational Gaussian Processes

Triply-Scalable Equivariant Gaussian Process Modeling

Amortized Structured Stochastic Variational Inference for Gaussian Process Latent Variable Models