stat.MLOct 7, 2026

Derivative Gaussian Processes on a Two-Direction Budget

Authors: Hyunseok Seung, Matthias Katzfuss

Organizations: University of Wisconsin–Madison

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Gradient observations promise more accurate Gaussian process (GP) surrogates, but the cost of incorporating them has long stood in the way of realizing that promise. We propose a derivative GP with a budget of just two directions per observed gradient. One direction focuses on each gradient's direct contribution to target prediction, while the other aggregates its indirect contributions through correlations with the conditioning function values. Within a Vecchia approximation, where each prediction conditions on mm nearby inputs in dd dimensions, this construction represents their mdmd gradient coordinates using at most 2m2m directional derivatives, giving O(m3)\mathcal{O}(m^3) dense factorization cost per prediction target. For general conditioning sets, we bound the posterior approximation error relative to using full gradients and characterize when the error is small or the approximation is exact. In simulations, our method matches the accuracy of a leading exact gradient-reduction method at equal conditioning set size. Because its cost grows much more slowly with that size, it can use conditioning sets well beyond the memory limit of the exact method, reaching lower prediction error with a small fraction of the time and memory. Notably, our method can exploit gradient observations while requiring less computation time or memory than function-only GP baselines.

Figures & tables

Figure 1: Posterior fidelity under different directional-derivative choices as mm increases for a Matérn- 5/25/2 GP in d=50d=50 . We compare LITE with methods using uij\mathbf{u}_{ij} or uij\mathbf{u}_{ij} paired with a random direction in col⁡(Di)\operatorname{col}(\mathbf{D}_{i}) . KL divergence between the TERA and approximate posteriors is averaged over prediction inputs, with all methods using the same conditioning inputs.
Figure 2: Relative compound symmetry (CS) error (solid) and expected KL divergence from the full-gradient posterior to LITE ’s (dashed) as dd increases. We use an isotropic Matérn- 5/25/2 kernel, n=1024n=1024 uniform inputs in [0,1]d[0,1]^{d} , 200200 targets, maximin ordering, and m=30m=30 nearest preceding inputs.
 Factor sizeTimeMemory
Full gradm(d+1)m(d+1)O(m3d3)\mathcal{O}(m^{3}d^{3})O(m2d2)\mathcal{O}(m^{2}d^{2})
TERAm2+mm^{2}+mO(dm2+m6)\mathcal{O}(dm^{2}+m^{6})O(dm2+m4)\mathcal{O}(dm^{2}+m^{4})
LITE3m3mO(dm2+m3)\mathcal{O}(dm^{2}+m^{3})O(dm+m2)\mathcal{O}(dm+m^{2})
Table 1: Per-target cost of one Vecchia conditional factor with mm conditioning inputs in dd dimensions. TERA’s entries assume m≤dm\leq d ; when m>dm>d , the full-gradient factor is smaller.
Figure 3: RMSE under the assumed GP against total prediction time for 256256 targets and peak GPU memory (top). Wall-clock time and peak GPU memory of one training step against mm on the Buckyball-catcher dataset (bottom). Point labels in the top panels indicate mm . Prediction uses B=32B=32 , while the bottom panels use B=32B=32 and B=1,024B=1{,}024 . TERA exceeds the 8080 GB GPU memory for m≥160m\geq 160 in prediction; missing TERA points in the bottom panels exceed the GPU memory.
Figure 4: Test RMSE per atom, wall-clock time, and peak GPU memory on six MD22 molecular datasets. LITE achieves the lowest runtime and memory usage on every benchmark, running 2.82.8 – 3.5×3.5\times faster than TERA while using only 13.313.3 – 30.2%30.2\% of its memory. Its RMSE is higher than TERA’s but lower than that of the other methods. Standard GP and DSoftKI run out of memory on an 80 GB GPU when applied to DHA, whereas LITE requires only 1.061.06 GB. Results are averaged over five independent runs, with error bars indicating standard errors.
Figure 5: Simple regret on Ackley ( 500D500D , left), Levy ( 800D800D , center), and best observed value on LassoDNA ( 180D180D , right) versus iteration. With only two directions per gradient, LITE substantially outperforms the function-only baselines, VBO and TuRBO-1, on all three tasks. It achieves the lowest final regret on Levy and a final value comparable to TERA on LassoDNA, while TERA attains lower regret on Ackley. Curves average five independent runs, with shaded bands indicating standard errors.
Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

SettingValue
GP prediction
Training/test inputs100,000100,000 / 256256
Input dimensiond=500d=500
Input designSobol in [0,1]d[0,1]^{d}
KernelARD Matérn- 5/25/2
Signal variance11
Appendix
Table 2: Settings for conditioning set and batch size scaling.
SettingValue
Train/test split90/10
Seeds1,27,42,86,99
Kernelisotropic SE
Initial lengthscale1.0
Initial signal variance1.0
Initial value-noise variance10−310^{-3}
Appendix
Table 3: Settings for MD22 regression.
BenchmarksAckley ( d=500d=500 ), Levy ( d=800d=800 ), LassoDNA ( d=180d=180 )
Domain[0,1]d[0,1]^{d}
Iterations / initial design100100 iterations / 3030 Sobol points
Seeds1, 27, 42, 86, 991,\ 27,\ 42,\ 86,\ 99
Noise parameters10−310^{-3} , fixed on synthetic tasks; initialized at 10−310^{-3} and learned on LassoDNA
Appendix
Table 4: Settings for BO synthetic objectives and LassoDNA benchmark.
Figure 6: Test RMSE versus conditioning set size on six MD22 datasets, with all methods using the covariance parameters learned by TERA at m=20m=20 after one epoch. Parameters remain fixed throughout prediction. Increasing mm provides little improvement for LITE and the function-only Vecchia GP, and LITE ’s RMSE remains higher than TERA’s at matched mm .
Figure 7: Test RMSE versus conditioning set size on six MD22 datasets, using the same TERA estimates as in Figure 6 . Solid curves keep these parameters fixed; the dashed curve trains LITE for one additional epoch at each mm , starting independently from the same estimates. Further training improves LITE on four datasets, changes little on AT-AT-CG-CG, and worsens prediction on Buckyball-catcher.
Figure 8: Test RMSE versus training step on six MD22 datasets. Both methods use m=20m=20 , training and prediction batch sizes of 3232 , and five training epochs. Additional updates do not uniformly improve predictive accuracy, with RMSE increasing after early minima on some datasets. Curves average five independent runs.
Figure 9: Wall-clock time and peak GPU memory on Ackley ( 500D500D , left), Levy ( 800D800D , center), and LassoDNA ( 180D180D , right). Time includes objective evaluations and gradient computation where used, including hypergradient computation on LassoDNA. Bars average five independent runs, with error bars indicating standard errors.

Explore similar work

CardsList

  1. Scalable Derivative Gaussian Processes via Exact Gradient Reduction

    Jun 1, 2026Hyunseok Seung, Matthias KatzfussGaussian ProcessSurrogate Gradient

    Department of Statistics · University of Wisconsin–Madison · Madison, WI, 53706

  2. Online Gradient Computation for Warping Gaussian Process Transformations

    Sep 15, 2026Emilio Ruiz-Moreno, Konstantinos Slavakis, Baltasar Beferull-LozanoGaussian ProcessWasserstein Gradient Flows

    SIGIPRO Department, Simula Metropolitan Center for Digital Engineering, Oslo, Norway · SURE-AI Center, Simula Research Laboratory, Oslo, Norway · Institute of Science Tokyo, Yokohama, Japan

  3. Conditioning Gaussian Processes on Almost Anything

    May 20, 2026Henry Moss, Lachlan Astfalck, Thomas Cowperthwaite +5Gaussian ProcessDiffusion Models

    Lancaster University · University of New South Wales · University of Cambridge +1