Worst-Case Optimal $\sqrtε$-Rate Stability of KL Divergence under Perturbations of Gaussian Distributions

Jialu Pan; Yufeng Zhang; Nan Hu; Zhenbang Chen; Ji Wang; Keqin Li

cs.LGApr 13, 2026

Worst-Case Optimal $\sqrtε$ -Rate Stability of KL Divergence under Perturbations of Gaussian Distributions

Authors: Jialu Pan, Yufeng Zhang, Nan Hu, Zhenbang Chen, Ji Wang, Keqin Li

Organizations: College of Computer Science and Electronic Engineering, Hunan University, Changsha, China. · National University of Defense Technology, China. · Department of Computer Science, State University of New York at New Paltz, 1 Hawk Drive, New Paltz, NY 12561, USA.

Abstract

We establish a stability bound for the Kullback-Leibler (KL) divergence when its Gaussian reference distribution is perturbed. Existing relaxed triangle inequalities of KL divergence between Gaussian distributions require all three distributions to be Gaussian, which limits their use in applications such as out-of-distribution detection with flow-based generative models. Let $\mathcal{N}_1$ and $\mathcal{N}_2$ be $d$ -dimensional Gaussian distributions, and let $P$ be any $d$ -dimensional distribution satisfying $\mathbb{E}_P[\|X\|^2]<\infty$ and $\mathrm{KL}(P\Vert \mathcal{N}_1)<\infty$ . If $\mathrm{KL}(\mathcal{N}_1\Vert\mathcal{N}_2)\leqε$ for $0<ε<1/12$ , we give an upper bound of $\left|\mathrm{KL}(P\Vert\mathcal{N}_1)-\mathrm{KL}(P\Vert\mathcal{N}_2)\right|$ . Under suitable uniform boundedness conditions, the bound is $\frac{\sqrt{6dε}}{2}+K\sqrtε+o(\sqrtε)$ as $ε\rightarrow 0$ for fixed $d$ , where $K$ depends on the parameters of $\mathcal{N}_1$ and $\mathcal{N}_2$ , and on $\mathbb{E}_P[\|X\|]$ and $\mathbb{E}_P[\|X\|^2]$ . Furthermore, we prove that the $\sqrtε$ dependence is worst-case optimal. The result extends stability guarantees restricted to Gaussian triples to comparisons involving an arbitrary distribution and two nearby Gaussian references. Experiments on synthetic distributions and latent representations from flow-based models support the predicted stability behavior.

Figures & tables

Fig. 1: KL divergence-based analysis of OOD detection for the Gaussian case where

Q_{Z}

can be approximated by a Gaussian distribution [ 13 ] .

Panel of Figure 2 — Fig. 2: Proof roadmap of Theorem 1 .

Fig. 3: Stability of KL divergence $\mathrm{KL}(P\|\mathcal{N}_{1})$ under Gaussian perturbation. Arrows represent KL divergences.

Perturbation	$\epsilon$	$\widehat{\Delta}_{KL}$	$\widehat{B}(\epsilon)$	$\mathrm{TR}$	$\widehat{K^{\prime}}$	$K^{\prime}$	MC s.e.
mean-plus	0.08	-0.08	4.66	0.02	-0.28	13.18	0.0016
mean-minus	0.08	-0.08	4.66	0.02	-0.29	13.18	0.0016
sigma-expand	0.07	0.83	3.69	0.22	3.03	9.65	0.0007
sigma-shrink	0.07	-1.30	3.61	0.36	-4.98	11.30	0.0010
both-plus	0.06	0.60	3.45	0.17	2.55	11.53	0.0009
both-minus	0.08	-0.95	4.52	0.21	-3.32	12.84	0.0017

TABLE I: Results of the one-dimensional synthetic experiment at

\alpha=1

. MC s.e. denotes the Monte Carlo standard error in estimating

\widehat{\Delta}_{KL}

Perturbation	Description
mean-x	shift rightward along the $x$ -axis
mean-diag	shift diagonally downward and rightward
cov-x-expand	expand along the $x$ -axis
cov-y-shrink	contract along the $y$ -axis
cov-diag-expand	expand along a diagonal direction
cov-rot-anisotropic	apply rotated anisotropic expansion and contraction

TABLE II: Perturbations to

\mathcal{N}_{1}=\mathcal{N}(\mathbf{0},\mathbf{I}_{2})

Perturbation	$\epsilon$	$\widehat{\Delta}_{KL}$	$\widehat{B}(\epsilon)$	$\mathrm{TR}$	$\widehat{K^{\prime}}$	$K^{\prime}$	MC s.e.
mean-x	0.08	-0.14	5.78	0.02	-0.50	16.97	0.0015
mean-diag	0.08	-0.10	5.70	0.02	-0.34	16.94	0.0005
cov-x-expand	0.03	0.49	3.50	0.14	2.94	18.15	0.0006
cov-y-shrink	0.02	-0.28	3.24	0.09	-1.79	18.14	0.0006
cov-diag-expand	0.03	0.77	3.50	0.22	4.68	18.15	0.0008
cov-rot-anisotropic	0.03	0.72	4.61	0.16	3.95	21.37	0.0007

TABLE III: Results of the two-dimensional synthetic experiment at

\alpha=1

. MC s.e. is the Monte Carlo standard error in estimating

\widehat{\Delta}_{KL}

ID	OOD	$\widehat{\Delta}_{KL}$	$\widehat{B}(\epsilon)$	$\mathrm{TR}$	$\widehat{K^{\prime}}$	$K^{\prime}$	$\widehat{\mathrm{KL}}_{\widehat{Q}}$
abs	crescent	0.0144	0.04	0.33	5.25	15.79	2.18
	gaussian	0.0088	0.03	0.26	3.23	12.44	1.10
	sine_wave	0.0142	0.04	0.32	5.18	15.92	2.03
crescent	abs	-0.0294	0.09	0.34	-4.02	11.71	1.53
	gaussian	-0.0315	0.09	0.36	-4.31	11.97	1.79
	sine_wave	-0.0258	0.09	0.30	-3.53	11.78	1.22

TABLE IV: Results on a two-dimensional NSF. All MC s.e. values are less than

10^{-4}

\widehat{\mathrm{KL}}_{\widehat{Q}}

\mathrm{KL}(\widehat{Q}_{Z}\|\mathcal{N}(\mathbf{0},\mathbf{I}_{2}))

Panel of Figure 8 — Fig. 4: Distributions of the tightness ratio $\mathrm{TR}$ and empirical coefficient $\widehat{K^{\prime}}$ across groups of Glow latent dimensions. Dimension groups satisfying $\mathbb{E}[\|Z\|^{2}]>50$ are excluded from the violin plots for readability.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. S1: Distribution of model log-likelihood. Glow model trained on CIFAR-10 assigns higher likelihoods to SVHN than CIFAR-10 and comparable likelihoods to SVHN with an adjusted contrast of factor 2.0 (SVHN-C(2.0)).

Fig. S2:

f(x)=x-\log x-1

Fig. S3: $P$ is a two-component truncated Gaussian mixture, $\mathcal{N}_{1}$ is the standard Gaussian distribution, and $\mathcal{N}_{2}$ is obtained by increasing the standard deviation of $\mathcal{N}_{1}$ .

Panel of Figure 12 — Fig. S4: KL behavior under one-dimensional Gaussian perturbations. (a) $\mathrm{KL}(\mathcal{N}_{1}\|\mathcal{N}_{2})$ versus the perturbation strength $\alpha$ . (b) $\mathrm{KL}(P\|\mathcal{N}_{1})-\mathrm{KL}(P\|\mathcal{N}_{2})$ versus $\sqrt{\mathrm{KL}(\mathcal{N}_{1}\|\mathcal{N}_{2})}$ . The curve of mean-plus almost overlaps with that of mean-minus .

Fig. S5: The estimated theoretical bound $\widehat{B}(\epsilon)$ in the one-dimensional synthetic experiment. Each marker corresponds to a value of $\alpha$ .

Fig. S6: $P$ is a two-component truncated Gaussian mixture, $\mathcal{N}_{1}$ is the standard Gaussian distribution, and $\mathcal{N}_{2}$ is obtained by perturbing $\mathcal{N}_{1}$ with covariance expansion along the $x$ -axis.

Panel of Figure 15 — Fig. S7: KL behavior under two-dimensional Gaussian perturbations. (a) $\mathrm{KL}(\mathcal{N}_{1}\|\mathcal{N}_{2})$ versus the perturbation strength $\alpha$ . The cov-x-expand and cov-diag-expand curves overlap. (b) $\widehat{\mathrm{KL}}(P\|\mathcal{N}_{1})-\widehat{\mathrm{KL}}(P\|\mathcal{N}_{2})$ versus $\sqrt{\mathrm{KL}(\mathcal{N}_{1}\|\mathcal{N}_{2})}$ . Each dot on a curve corresponds to one value of $\alpha$ .

Fig. S8: The estimated theoretical bound $\widehat{B}(\epsilon)$ in the two-dimensional synthetic experiment. Each marker corresponds to a value of $\alpha$ .

Fig. S9: Datasets used in the two-dimensional neural spline flow experiments.

Fig. S10: Neural spline flow model trained on the abs dataset. Left: density learned by the flow model; Right: samples generated by the flow-based model.

Fig. S11: Representations of ID ( abs ) and OOD datasets.

ID dataset	$p$ -value	$\mathrm{KL}(\widehat{P_{Z}}\\|\mathcal{N}(\mathbf{0},\mathbf{I}_{2}))$
abs	$0.47$	$7.48\times 10^{-6}$
crescent	$0.37$	$5.34\times 10^{-5}$
gaussian	$0.39$	$2.21\times 10^{-5}$
sine_wave	$0.39$	$1.84\times 10^{-5}$

Appendix

TABLE S1:

p

-value: Henze-Zirkler normality test results on the ID representations from the two-dimensional NSF models.

Dataset	$\mathrm{KL}(\widehat{P}_{Z}\\|\mathcal{N}(\mathbf{0},\mathbf{I}_{4}))<1/12$	$p>0.05$
CelebA32	98.6%	93.0%
SVHN	99.3%	94.4%
CIFAR-10	99.1%	91.1%

Appendix

TABLE S2: Proportions of dimension groups of ID representations.

Panel of Figure 22 — Fig. S12: Distributions of the tightness ratio $\mathrm{TR}$ and empirical coefficient $\widehat{K^{\prime}}$ across groups of Glow latent dimensions. Dimension groups satisfying $\mathbb{E}[\|Z\|^{2}]>50$ are excluded from the violin plots for readability.

ID	OOD	Proportion	Pearson’s $r$	$p$ -value
SVHN	ImageNet32	13.8%	-0.99	$<10^{-10}$
	LSUN	0.5%	-0.95	$<10^{-10}$
CIFAR-10	CelebA32	2.2%	1.00	$<10^{-10}$
	LSUN	7.5%	1.00	$<10^{-10}$

Appendix

TABLE S3: Proportions of dimension groups where

\mathbb{E}[\|Z\|^{2}]>50

. ID–OOD pairs with no such dimension groups are omitted. The Pearson correlation coefficient

r

between

\widehat{K^{\prime}}

and

\mathbb{E}[\|Z\|^{2}]

, together with the

p

-value.

Panel of Figure 24 — Fig. S13: Estimated $\mathbb{E}[\|Z\|^{2}]$ and the ratio $F_{3}/K^{\prime}$ on each dimension group when testing OOD datasets on Glow models. Groups with estimated second moments satisfying $\mathbb{E}[\|Z\|^{2}]>50$ are excluded from the violin plots for readability.

Panel of Figure 25 — Fig. S14: $\mathrm{KL}(\widehat{P}_{Z}^{(g)}\|\mathcal{N}(\mathbf{0},\mathbf{I}_{4}))$ for all groups of ID representations and $\mathrm{KL}(\widehat{Q}_{Z}^{(g)}\|\mathcal{N}(\mathbf{0},\mathbf{I}_{4}))$ for groups of OOD representations satisfying $\mathbb{E}[\|Z\|^{2}]>50$ . KL divergence values greater than $50$ are omitted to improve the scale of the visualization.

Kullback-Leibler Divergence

See all 27 papers

Jan 20, 2026

A Gaussian Perspective for Distributional Discrepancy in Generative Diffusion Models

This paper introduces an analytical approach to quantifying and optimizing the distributional discrepancy in generative diffusion models. For a multivariate Gaussian source, we explicitly derive the closed-form evolution trajectory and the resulting Kullback-Leibler (KL) divergence between the distributions of the source data and the reversely sampled data. Asymptotic analysis via the Euler-Maclaurin expansion characterizes the convergence behavior of this KL divergence, extracting its dominant term as an explicit functional of the noise schedule. Minimizing this dominant term via the calculus of variations yields a noise schedule described by a tangent law, inherently determined by the source covariance spectrum. We further prove that the Gaussian source exhibits an extremal property for the KL divergence among general source distributions with a given covariance. We also utilize the analytical KL divergence as a principled metric to identify efficient time discretization strategies for pretrained diffusion models, and demonstrate via experiments over diverse datasets that the identified strategies consistently outperform established baselines, particularly under constrained function evaluation budgets.

Qiang Sun, H. Vincent Poor, Wenyi Zhang

Department of Electronic Engineering and Information Science, University of Science and Technology of China, Hefei, China · Department of Electrical Engineering, Princeton University, Princeton, NJ, USA

Diffusion Sampling Kullback-Leibler Divergence Diffusion Model Inference Acceleration

Ask PDF

May 13, 2026

On the Generalization of Knowledge Distillation: An Information-Theoretic View

Knowledge distillation is widely used to improve generalization in practice, yet its theoretical understanding remains elusive. In the standard distillation setting, a teacher model provides soft predictions to guide the training of a student model. We model teacher and student training as coupled stochastic processes and introduce a distillation divergence, defined as the Kullback-Leibler divergence between these two stochastic kernels. Within this framework, we derive two generalization bounds for the student model relative to the teacher's generalization gap: an upper bound under a sub-Gaussian assumption via algorithmic stability, and a lower bound under a central condition with sharper dependence on the distillation divergence. We further develop a loss-sharpness-aware bound with an explicit tightness regime, showing that the teacher's local flatness can strictly tighten the bound. Additionally, in a linear Gaussian case study, the distillation divergence admits an interpretable decomposition into bias, variance, and rank-bottleneck costs, yielding practical guidance for distillation design.

Bingying Li, Haiyun He

Internet of Things Thrust, Information Hub The Hong Kong University of Science and Technology (Guangzhou) Guangzhou, China

Information-Theoretic Generalization Bounds Statistical Learning Theory Teacher-Student Learning Knowledge Distillation Generalization Bounds

Ask PDF

Oct 12, 2026

Optimal Transport PAC-Bayes Bounds

PAC-Bayesian generalization bounds are among the strongest distribution-dependent guarantees in statistical learning and are typically expressed in terms of the Kullback-Leibler (KL) divergence between a prior and posterior distribution. While analytically convenient, the KL divergence requires the posterior to be absolutely continuous w.r.t. the prior, imposing support constraints that can substantially restrict the choice of posterior. Moreover, the KL divergence does not account for the underlying geometry of the probability space. In this work, we derive new PAC-Bayesian bounds replacing the KL divergence with general optimal transport (OT) distances, including the total variation and Wasserstein-

p

distances.

Armin Beck, Peter Ochs, Jalal Fadili

Max Planck Institute for Informatics, Saarland Informatics Campus, Saarbrücken, Germany · Saarland University, Saarland Informatics Campus, Saarbrücken, Germany · ENSICAEN, CNRS GREYC, Caen, France

TV Distance Computation PAC-Bayesian Generalization Bounds PAC-Bayesian Theory

Ask PDF

Abstract

Kullback-Leibler Divergence

Explore similar work

A Gaussian Perspective for Distributional Discrepancy in Generative Diffusion Models

On the Generalization of Knowledge Distillation: An Information-Theoretic View

Optimal Transport PAC-Bayes Bounds