Amortized Bandwidth Learning for Kernel Density Estimation under Logarithmic Score

Junyi Liang; Hailiang Du

cs.LGAug 20, 2026

Amortized Bandwidth Learning for Kernel Density Estimation under Logarithmic Score

Authors: Junyi Liang, Hailiang Du

Organizations: School of Mathematics, East China University of Science and Technology, 200237 Shanghai, China · Department of Mathematical Sciences; Institute of Hazard, Risk and Resilience, Durham University, DH1 3LE Durham, United Kingdom · Data Science Institute; Global School of Sustainability, The London School of Economics and Political Science, WC2A 2AE London, United Kingdom

Abstract

Kernel density estimation converts finite samples into probability densities, but its performance depends critically on bandwidth selection. Classical selectors prescribe the sample-to-bandwidth rule analytically or asymptotically, or solve a new optimization for each sample. An amortized framework is proposed that instead learns this mapping across a distribution of density-estimation tasks by optimizing the logarithmic score. A truncated-and-renormalized bounded-support formulation enables stable learning across heterogeneous tasks, while affine standardization allows a selector trained on a single reference interval to transfer across bounded intervals. Experiments under Gaussian sampling, a multi-family benchmark, and randomized Gaussian-mixture training show that the amortized selector consistently and substantially outperforms Silverman's rule, the Sheather--Jones selector, and least-squares cross-validation, with especially large gains in small and heterogeneous samples. Finite Gaussian mixtures provide a generic training mechanism supported by their $L^1$ approximation property. Selectors trained in this way generalize strongly across different density structures, allowing the same trained selector to be applied directly to finite samples from unknown densities without specifying or fitting a distributional family. This combination of broad applicability and strong empirical performance makes the framework attractive for a wide range of applications in which finite samples or ensembles must be converted into continuous probability densities.

Figures & tables

Fig 1: Empirical logarithmic score versus sample size under Gaussian sampling. Scores are averaged over $N_{\mathrm{test}}=30000$ independent samples at each $n$ ; shaded bands show 90% bootstrap intervals, and the black dashed line denotes the score of the true $\mathcal{N}(0,1)$ density. The horizontal axis is linear for $n\leq 20$ and logarithmically compressed thereafter, while the vertical axis is linear up to $4.5$ bits and logarithmically compressed above.

Family	Density $g(x)$	Parameters
Gaussian	$g(x)=\phi_{\sigma}(x-\mu)$	$\sigma\in[0.13,0.61]$ .
Laplace	$g(x)=(2b)^{-1}\exp\{-\|x-\mu\|/b\}$	$b\in[0.06,0.43]$ .
Student- $t$	$g(x)=s^{-1}t_{\nu}\{(x-\mu)/s\}$	$\nu\sim LU[3,30]$ ; $s\in[0.02,0.59]$ .
Gamma	$\begin{array}[]{@{}l@{}}y=k\theta+\varepsilon(x-\mu),\\[-2.0pt] g(x)=\gamma_{k,\theta}(y)\mathbf{1}_{\{y>0\}}.\end{array}$	$k\sim LU[0.5,10]$ ; $\varepsilon\in\{-1,1\}$ equiprobably; $\theta\in[0.04,1.4]$ .
Beta	$\begin{array}[]{@{}l@{}}z=\dfrac{a}{a+b}+\dfrac{x-\mu}{s},\\[-2.0pt] g(x)=s^{-1}\beta_{a,b}(z)\mathbf{1}_{\{0<z<1\}}.\end{array}$	Four equiprobable regimes: $a,b\stackrel{{\scriptstyle\mathrm{iid}}}{{\sim}}U[0.4,0.9]$ , $U[1.2,5]$ , or $U[0.8,1.2]$ ; or $a$ and $b$ are drawn, in random order, from $U[0.4,0.9]$ and $U[1.2,5]$ . $s\in[0.7,8.3]$ .
Logistic	$g(x)=\frac{\exp\{-(x-\mu)/s\}}{s[1+\exp\{-(x-\mu)/s\}]^{2}}$	$s\in[0.06,0.34]$ .

Table 1: Densities and parameter specifications for the bounded multi-family benchmark.

Fig 2: Empirical logarithmic score versus sample size under bounded multi-family sampling on $[-1,1]$ . Scores are averaged equally across the ten families using $N_{\mathrm{test}}=3000$ independent tasks per family at each sample size. Shaded bands show 90% bootstrap intervals, and the black line denotes the corresponding score of the true target densities.

	Number of detected modes
Number of components $K$	1	2	3	4	5	6	7	8	$\geq 9$
8	17.0%	36.2%	27.2%	13.7%	4.8%	1.0%	0.2%	0.0%	0.0%
16	14.4%	32.0%	26.8%	15.2%	7.3%	3.1%	0.9%	0.2%	0.0%
32	13.5%	30.0%	26.3%	15.5%	8.7%	4.0%	1.2%	0.6%	0.2%

Table 2: Empirical distributions of detected mode counts under the bounded GMM generators. Entries are percentages from

5000

independently generated densities for each

K

Fig 3: Empirical logarithmic score versus sample size under bounded $K=32$ GMM sampling on $[-1,1]$ . Scores are averaged over $N_{\mathrm{test}}=30000$ independent GMM tasks at each sample size. GMM $K=8$ , GMM $K=16$ , and GMM $K=32$ denote amortized selectors trained using the respective GMM generators, while Multi-family denotes the selector trained on the bounded multi-family task distribution. Shaded bands show $90\%$ bootstrap intervals, and the black line denotes the corresponding score of the true target densities.

Fig 4: Empirical logarithmic score versus sample size under bounded multi-family sampling on $[-1,1]$ . Scores are averaged equally across the ten families using $N_{\mathrm{test}}=3000$ independent tasks per family at each sample size. GMM $K=8$ , $16$ , and $32$ denote selectors trained using the corresponding GMM generators, while Multi-family denotes the multi-family-trained selector. Shaded bands show $90\%$ bootstrap intervals, and the black line denotes the corresponding score of the true target densities.

Density Estimation

Density estimation learns probability distributions from data to measure likelihood, detect anomalies, quantify uncertainty, and generate samples.

See all 66 papers

May 13, 2026

Adaptive Kernel Density Estimation with Pre-training

Density estimation in high-dimensional settings is an important and challenging statistical problem.Traditional methods based on kernel smoothing are inefficient in high dimensions due to the difficulties in specifying appropriate location-adaptive kernels. In this work, we introduce pre-training, a key idea behind many cutting-edge AI technologies, to the context of non-parametric density estimation. By establishing a pre-trained neural network that can recommend an appropriate location-adaptive kernel for each sample point, efficient density estimation with adaptive kernels is achieved in high dimensions. A wide range of numerical experiments show that this strategy is highly effective for improving density-estimation accuracy, when the target distribution is close to the distribution family for pre-training. When the target distribution is substantially different from the pre-training distribution family, the benefit from the proposed pre-training strategy may be diluted, but can be reactivated by an additional fine-tuning procedure.

Ruitong Zhang, Ke Deng

Department of Statistics and Data Science Tsinghua University Beijing, China

Kernel Density Estimation Density Estimation

Ask PDF

Sep 14, 2026

Learning under Target Shift: Optimal Density Ratio Estimation and Importance-Weighted Regression

We study density ratio estimation and importance-weighted regression under target shift with continuous outputs. Under target shift, the conditional distribution of the inputs given the outputs remains invariant across the training and test distributions, while the output marginal distribution may change. Although this problem has been extensively studied for discrete outputs, the continuous setting is substantially less understood: the importance weights are determined by an unknown density ratio function, for which existing estimation methods lack explicit finite-sample convergence rates. We propose a spectral regularization method in a reproducing kernel Hilbert space (RKHS) for estimating the continuous density ratio from labeled training samples and unlabeled test inputs. Under a source condition with regularity parameter

ι>0

, we establish high-probability finite-sample guarantees and show that the estimator achieves the capacity-independent minimax-optimal RKHS-norm rate

O(n_η^{-ι/(2ι+2)})

. We then incorporate the estimated density ratio into importance-weighted regression and characterize the propagation of density-ratio estimation error to the final predictor. When sufficiently many samples are available for density ratio estimation, the resulting regression estimator attains the minimax-optimal rates of standard kernel regression. These results establish a finite-sample theory for continuous density ratio estimation and importance-weighted learning under target shift.

Ren-Rui Liu, Zheng-Chu Guo

School of Mathematical Sciences, Zhejiang University, Hangzhou 310058, China

Kernel Regression Distribution Shift Density Ratio Estimation

Ask PDF

Sep 3, 2026

Geometry-Aware Graph Construction via Adaptive Spectral Bandwidth Control

Kernelized graph methods - spectral clustering, diffusion maps, and sparse kernel -regression graphs - that use Gaussian kernels depend on the choice of Gaussian bandwidth sigma, which governs the spectral character of the local kernel operator. When sigma is too small, the kernel overestimates local complexity and treats each sample as an independent direction; when sigma is too large, the kernel collapses multiple directions together, the condition number diverges, and all geometric discrimination is lost. We propose a choice of scale to make the spectral complexity of the kernel consistent with the intrinsic complexity of the underlying manifold. We propose a per-node bandwidth criterion that operationalizes this principle by jointly matching the kernel's effective rank to the local intrinsic dimension estimated via minimum spanning tree, anchoring the search in the manifold-consistent log-log scaling regime. We evaluate SSL embeddings from six encoders on CIFAR-100, showing that adaptive bandwidth consistently improves leave-one-out (LOO) classification and label propagation (LP) accuracy over fixed-bandwidth methods and competing adaptive methods.

Ecem Bozkurt, Antonio Ortega

Department of Electrical and Computer Engineering University of Southern California, Los Angeles, CA, USA

Graph Structure Learning Graph-Based Semi-Supervised Learning Kernel Methods Spectral Methods

Ask PDF

Abstract

Density Estimation

Explore similar work

Adaptive Kernel Density Estimation with Pre-training

Learning under Target Shift: Optimal Density Ratio Estimation and Importance-Weighted Regression

Geometry-Aware Graph Construction via Adaptive Spectral Bandwidth Control