Sharp Asymptotic Theory of Maximum Likelihood Estimation for Gaussian Processes with an RBF Kernel

Ameer Qaqish; Didong Li

math.STOct 7, 2026

Sharp Asymptotic Theory of Maximum Likelihood Estimation for Gaussian Processes with an RBF Kernel

Authors: Ameer Qaqish, Didong Li

Organizations: Department of Biostatistics, University of North Carolina at Chapel Hill

Abstract

Gaussian processes (GPs) are widely used across machine learning, spatial statistics, time-series analysis, optimization, Bayesian statistics, and scientific applications. A central component of a GP model is its kernel, which is typically specified through a parametric family. Among the most widely used choices is the radial basis function (RBF), also known as the squared exponential or Gaussian kernel, owing to its simple form, smoothness, and flexibility. In practice, the kernel parameters are routinely estimated by the maximum likelihood estimators (MLEs), as implemented by standard GP software. Despite this widespread use, the asymptotic behavior of the MLEs remains poorly understood under fixed-domain asymptotics, even for the RBF kernel. The main difficulty arises from the increasingly strong dependence among densely sampled observations and the nonlinear dependence of the covariance matrix on the kernel parameters. In this paper, we address this gap by providing, to the best of our knowledge, the first complete asymptotic characterization of the joint MLE of the spatial variance, lengthscale, and nugget variance under fixed-domain asymptotics. We establish consistency, derive convergence rates for all three parameters, prove joint asymptotic normality, and show that these rates are minimax optimal.

Figures & tables

Figure 1: Simulation results for

p=1

on the log scale, with

\theta_{0}=(1,0.25,0.01)

and

1000

replicates per

n

. Top: RMSE (filled circles), robust standard deviation

\mathrm{IQR}/1.349

(open squares), both with bootstrap

95\%

intervals, and the exact asymptotic standard deviation

[\mathcal{I}_{n}(\theta_{0})^{-1}]_{jj}^{1/2}/\theta_{0j}

(grey line). The grey region indicates sample sizes beyond the Monte Carlo range, where repeated MLE computation is computationally infeasible but the Fisher-information can still be evaluated. Dashed lines show the theoretical rates in Theorem 3.1 (ii). Bottom: normal Q–Q plots of the standardized errors, with a

95\%

band for exact normality; triangles mark points outside the frame.

Figure 2: Simulation results for

p=2

, with the layout and graphical elements as in Figure 1 .

Figure 3: Simulation results for

p=3

, with the layout and graphical elements as in Figure 1 .

		RMSE	$\mathrm{IQR}/1.349$	Asymptotic SD	Theory
$\log\sigma^{2}$	$p=1$	$-0.49$	$-0.49$	$-0.35$	$-0.5$
	$p=2$	$-0.75$	$-0.72$	$-0.74$	$-1$
	$p=3$	$-1.34$	$-1.48$	$-1.32$	$-1.5$
$\log l$	$p=1$	$-2.05$	$-1.81$	$-1.66$	$-1.5$
	$p=2$	$-2.32$	$-2.24$	$-2.22$	$-2$
	$p=3$	$-3.11$	$-3.11$	$-3.09$	$-2.5$

Table 1: Least-squares slopes of

\log(\text{error})

against

\log b_{n}

for

\log\sigma^{2}

and

\log l

, and against

\log n

for

\log\tau^{2}

, over the nine simulated sample sizes (

10^{2}\lesssim n\leq 10^{6}

). The last column gives the theoretical rate shown in Theorem 3.1 .

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Figure A1: Simulation results for

p=1

on the original parameter scale, with the layout and graphical elements as in Figure 1 .

Figure A2: Simulation results for

p=2

on the original parameter scale, with the layout and graphical elements as in Figure 1 .

Figure A3: Simulation results for

p=3

on the original parameter scale, with the layout and graphical elements as in Figure 1 .

Explore similar work

Regularized Variational and Spectral Log-Density-Ratio Estimation in the Gaussian Location Model
Jul 2, 2026Francis BachDensity Ratio Estimation Covariance
SIERRA · Inria - Ecole Normale Sup´erieure PSL Research University
Data eccentricity, asymptotics of Gaussian RBF reproducing kernel Hilbert space, and kernel PCA
Jul 23, 2026Sergio A. AlvarezPrincipal Component Analysis Kernel Hilbert Spaces
Department of Computer Science, Boston College, Chestnut Hill, MA 02467 USA
Polylogarithmic Sparsity of Randomly Reweighted NPMLEs for Gaussian Mixtures
Oct 1, 2026Hansheng JiangMaximum Likelihood Gaussian Mixture Models
University of Toronto

Jul 2, 2026, cs.LGJ/K move · Enter open · S save

Regularized Variational and Spectral Log-Density-Ratio Estimation in the Gaussian Location Model

Francis Bach

SIERRA · Inria - Ecole Normale Sup´erieure PSL Research University

Density Ratio Estimation Covariance Spectral

Open paper AskPDF arXiv

We study ridge-regularized log-density-ratio estimation in the Gaussian location model with a common covariance matrix. By affine invariance, the model is written as q

\sim

N(0, I), p

\sim

Δ

, I), with linear features, where

Δ

is a mean vector. The variational estimator is the empirical Kullback-Leibler (KL) log-normalized fit with a squared L2-penalty on its nonconstant coefficient, and the spectral estimator recently introduced in [1] replaces a single variational problem by a continuum of ridge-regularized least-squares problems. We derive high-dimensional deterministic asymptotic equivalents when the numbers of observations and dimension tend to infinity with fixed ratios. The regularized variational limit is characterized by a scalar entropy minimization problem derived from the convex-Gaussian-min-max theorem (CGMT), while the regularized spectral limit follows from deterministic equivalents for resolvents of weighted sums of two independent Gaussian sample covariance matrices. We use these formulas to compare population risks, with experiments focused on fixed-signal aspect-ratio sweeps and optimized regularization. Our conclusion is that with many observations, under the criteria and asymptotic regimes analyzed here, the well-specified variational estimator has the smaller risk, while with fewer observations, the spectral estimator is favored because its covariance-based construction has lower variance. We also study how a nuclear penalty can be used and partially analyzed to perform feature learning.

Abstract

Explore similar work

Regularized Variational and Spectral Log-Density-Ratio Estimation in the Gaussian Location Model

Data eccentricity, asymptotics of Gaussian RBF reproducing kernel Hilbert space, and kernel PCA

Polylogarithmic Sparsity of Randomly Reweighted NPMLEs for Gaussian Mixtures