Continuous-depth Deep Gaussian Processes

Ying Li; Sanyou Wu; Zi Yang; Qiaochu Xu; Yuhao Liu; Zhidi Lin; Michael Minyi Zhang; Petar Djuric

stat.MLOct 22, 2026

Continuous-depth Deep Gaussian Processes

Authors: Ying Li, Sanyou Wu, Zi Yang, Qiaochu Xu, Yuhao Liu, Zhidi Lin, Michael Minyi Zhang, Petar Djuric,

Organizations: National University of Singapore · Columbia University · The University of Hong Kong · Capital One ·

Abstract

Deep Gaussian processes (DGPs) are appealing Bayesian regression models, but their standard discrete-depth parameterization often becomes harder to optimize as depth grows. We argue that depth is better treated as a latent continuous evolution than as a long discrete stack. We therefore propose a continuous-depth formulation in which the latent representation follows a stochastic differential equation with a depth-indexed drift family equipped with a GP-based prior. This reframes the core design choice as the prior placed on that family, rather than adopting a purely neural continuous-depth parameterization as in neural ODE or neural SDE models. We study two instantiations of this idea: CDGP, a direct GP prior over the state--depth drift, and FlowDGP, a flow-evolved prior that captures richer depth dependence at lower training cost. To motivate the redesign, we analyze how the single-sample DGP objective can become increasingly sensitive under repeated layer composition, while residual DGP is a useful first repair that still remains a finite discrete construction. We also give pathwise sensitivity analysis for the continuous-depth formulation. Experiments on synthetic, benchmark, and large-scale regression tasks show that residual DGP improves over plain DGP, while CDGP and FlowDGP provide a stronger overall story across the regimes we study.

Figures & tables

Figure 1: Toy regression results. Top: test root mean squared error (RMSE), negative log-likelihood (NLL), and mean absolute error (MAE) versus depth

L

for standard DGP and residual DGP. Middle: CDGP and FlowDGP versus

J

at fixed

T=1

, with the dashed line showing the two-layer residual DGP reference. Bottom: CDGP and FlowDGP versus

T

at fixed

J=3

. Shaded regions denote one standard deviation across random seeds.

		Dataset
Metric	Method	heavy_tailed	heteroscedastic	nonstationary	sarcos_small	uci_bike_small	uci_energy_small
RMSE $\downarrow$	FlowDGP	$\underline{0.235\pm 0.003}$	$\mathbf{0.819\pm 0.011}$	$0.377\pm 0.123$	$\underline{0.323\pm 0.028}$	$\mathbf{0.934\pm 0.121}$	$0.804\pm 0.009$
	CDGP	$\mathbf{0.231\pm 0.005}$	$0.856\pm 0.077$	$0.634\pm 0.008$	$0.332\pm 0.004$	$1.122\pm 0.012$	$\underline{0.802\pm 0.001}$
	Heteroskedastic GP	$0.306\pm 0.002$	$\underline{0.823\pm 0.006}$	$0.654\pm 0.008$	$0.345\pm 0.010$	$1.152\pm 0.006$	$\mathbf{0.797\pm 0.001}$
	Residual DGP	$0.311\pm 0.005$	$\underline{0.823\pm 0.003}$	$0.676\pm 0.056$	$0.347\pm 0.006$	$1.126\pm 0.005$	$\underline{0.802\pm 0.001}$
	Sparse GP	$0.309\pm 0.001$	$1.023\pm 0.008$	$0.756\pm 0.006$	$1.051\pm 0.003$	$1.162\pm 0.004$	$\underline{0.802\pm 0.002}$

Table 1: Regression results on benchmark datasets. We report mean

\pm

std over multiple random seeds. Bold indicates the best result; underline indicates the second best. Training time is in seconds. Rows highlighted in light blue correspond to our methods. Method abbreviations: DKL = deep kernel learning, TDGP = thin DGP, TGP = transformed GP, NODE = neural ODE, and NSDE = neural SDE.

Figure 2: Results on full-size uci_bike (top), uci_energy (middle), and sarcos (bottom) datasets comparing FlowDGP, deep kernel learning (DKL), neural ordinary differential equations (NODE), and neural stochastic differential equations (NSDE). We report root mean squared error (RMSE), negative log-likelihood (NLL), and continuous ranked probability score (CRPS) averaged over random seeds; error bars denote one standard deviation. For readability, the NLL values of DKL on sarcos are scaled by

0.1

(first) and

1/3

(second) to avoid distorting the

y

-axis scale.

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	$D_{x}$	$N_{\mathrm{train}}$	$N_{\mathrm{test}}$	Primary challenge
heteroscedastic	$10$	$300$	$500$	input-dependent noise
nonstationary	$10$	$300$	$500$	two-regime lengthscale change
heavy_tailed	$10$	$300$	$500$	Student- $t$ ( $\nu\!=\!3$ ) outliers
sarcos_small	$21$	$300$	$500$	high-dimensional nonlinear dynamics
uci_bike_small	$12$	$300$	$500$	mixed diurnal/seasonal regimes
uci_energy_small	$29$	$300$	$500$	skewed, spiky real sensor data

Appendix

Table 2: Summary of evaluation datasets. The first block lists the three synthetic benchmarks, the second block the subsampled real-world datasets used in Section 5.2 , and the third block their full-size counterparts used in Section 5.3 . All tasks are scalar regression with

D_{y}=1

Method	Search space
SparseGP	lr $\in\{0.01,0.02,0.03,0.05\}$
HeteroskedasticGP	lr $\in\{0.01,0.02,0.03,0.05\}$
DKL	grid_size $\in\{80,100,120\}$
	num_dims $\in\{2,3\}$
	lr $\in\{0.01,0.02,0.03,0.05\}$
TDGP	m_v $\in\{20,30\}$

Appendix

Table 3: Grid-searched hyperparameters for the benchmark methods. For each method and dataset, we select the configuration with the lowest validation NLL.

Scope	Setting
Defaults	num_inducing $=30$ (whenever applicable)
	epochs $=2500$
	optimiser $=$ Adam
DKL	epochs $=500$
CDGP	$\sigma=0.15$
	$T=1.0$ (unless otherwise specified)

Appendix

Table 4: Fixed hyperparameter settings for the toy and benchmark experiments. Method-specific entries override the defaults.

Method	RMSE	NLL	CRPS
DKL	$0.8090\pm 0.0022$	$1.2078\pm 0.0023$	$0.4243\pm 0.0012$
NODE	$0.8096\pm 0.0018$	$1.2088\pm 0.0025$	$0.4252\pm 0.0018$
NSDE	$0.8075\pm 0.0022$	$1.2074\pm 0.0033$	$\mathbf{0.4227\pm 0.0016}$
FlowDGP	$\mathbf{0.8052\pm 0.0015}$	$\mathbf{1.2027\pm 0.0023}$	$0.4234\pm 0.0016$

Appendix

Table 5: YearPredictionMSD results, mean

\pm

standard deviation over random seeds. Lower is better; bold marks the lowest mean.

Drift	$\mu$	$S(1)$	$S(4)$	$S(8)$	$L_{\theta}/\mu$
Contractive	0.5	0.637	1.002	1.023	2.000
Contractive	1.0	0.521	0.672	0.673	1.000
Contractive	2.0	0.368	0.401	0.401	0.500
Non-contractive	—	1.136	6.870	25.542	—

Appendix

Table 6: Controlled SDE illustration of Corollary 4.2 .

Figure 3: Learned drift evolution and latent trajectories on the toy dataset. Top: drift posterior means versus $t$ at fixed states $h$ for CDGP and fixed inputs $x$ for FlowDGP. Bottom: sampled latent trajectories (thin lines) and their means (thick lines), colored by input.

Deep Gaussian Processes

See all 10 papers

Efficient Deep Gaussian Process Surrogates via Mini-Batch Latent MAP Estimation
Oct 27, 2026Chun-Yi Chang, Chih-Li SungDeep Gaussian Processes Surrogate Modeling
Michigan State University
How Deep Are Deep GPs, Really? A Sharp Threshold and a Non-Gaussian Limit for Compositional GPs
Jun 6, 2026Mark Kozdoba, Shie MannorDeep Gaussian Processes Gaussian Processes
Technion, IIT · NVIDIA
Onsager-Machlup Posterior Transport for Deep Gaussian Processes
May 22, 2026Jian Xu, Delu Zeng, John Paisley +1Deep Gaussian Processes Gaussian Processes
RIKEN iTHEMS · RIKEN AIP · South China University of Technology +1

Oct 27, 2026J/K move · Enter open · S save

Efficient Deep Gaussian Process Surrogates via Mini-Batch Latent MAP Estimation

Chun-Yi Chang, Chih-Li Sung

Michigan State University

Deep Gaussian Processes Surrogate Modeling MAP Estimation

Open paper AskPDF arXiv

Deep Gaussian processes (DGPs) are flexible surrogate models for computer experiments with nonstationary response surfaces, but inference over their latent representations remains computationally challenging. Existing methods typically rely on variational inference or Markov chain Monte Carlo (MCMC), which impose variational-family assumptions or require costly posterior sampling. We propose the latent maximum-a-posteriori deep Gaussian process (LaMP-DGP), which uses gradient-based optimization to obtain MAP estimates of the latent representations and enables efficient probabilistic prediction through linked-GP calculations. To scale LaMP-DGP to large datasets, we develop a mini-batch optimization algorithm and, under a polynomial mini-batch bias-decay condition, establish a stationarity bound for latent-state optimization that motivates a growing batch-size schedule. Across numerical experiments and real-data analyses, LaMP-DGP provides consistently competitive predictive performance relative to doubly stochastic variational inference, stochastic gradient Hamiltonian Monte Carlo, and fully Bayesian Vecchia DGP inference. Computationally, LaMP-DGP is substantially faster than the Vecchia MCMC approach while remaining broadly competitive with the stochastic-gradient alternatives. These results position LaMP-DGP as a competitive and computationally efficient alternative for DGP inference. An R package, "LaMPDGP", is available for implementing the proposed method.

Abstract

Deep Gaussian Processes

Explore similar work

Efficient Deep Gaussian Process Surrogates via Mini-Batch Latent MAP Estimation

How Deep Are Deep GPs, Really? A Sharp Threshold and a Non-Gaussian Limit for Compositional GPs

Onsager-Machlup Posterior Transport for Deep Gaussian Processes