Worst-Case Optimal $\sqrtε$-Rate Stability of KL Divergence under Perturbations of Gaussian Distributions

Jialu Pan; Yufeng Zhang; Nan Hu; Zhenbang Chen; Ji Wang; Keqin Li

cs.LGApr 13, 2026

Worst-Case Optimal $\sqrtε$ -Rate Stability of KL Divergence under Perturbations of Gaussian Distributions

Authors: Jialu Pan, Yufeng Zhang, Nan Hu, Zhenbang Chen, Ji Wang, Keqin Li

Organizations: College of Computer Science and Electronic Engineering, Hunan University, Changsha, China. · National University of Defense Technology, China. · Department of Computer Science, State University of New York at New Paltz, 1 Hawk Drive, New Paltz, NY 12561, USA.

Abstract

We establish a stability bound for the Kullback-Leibler (KL) divergence when its Gaussian reference distribution is perturbed. Existing relaxed triangle inequalities of KL divergence between Gaussian distributions require all three distributions to be Gaussian, which limits their use in applications such as out-of-distribution detection with flow-based generative models. Let $\mathcal{N}_1$ and $\mathcal{N}_2$ be $d$ -dimensional Gaussian distributions, and let $P$ be any $d$ -dimensional distribution satisfying $\mathbb{E}_P[\|X\|^2]<\infty$ and $\mathrm{KL}(P\Vert \mathcal{N}_1)<\infty$ . If $\mathrm{KL}(\mathcal{N}_1\Vert\mathcal{N}_2)\leqε$ for $0<ε<1/12$ , we give an upper bound of $\left|\mathrm{KL}(P\Vert\mathcal{N}_1)-\mathrm{KL}(P\Vert\mathcal{N}_2)\right|$ . Under suitable uniform boundedness conditions, the bound is $\frac{\sqrt{6dε}}{2}+K\sqrtε+o(\sqrtε)$ as $ε\rightarrow 0$ for fixed $d$ , where $K$ depends on the parameters of $\mathcal{N}_1$ and $\mathcal{N}_2$ , and on $\mathbb{E}_P[\|X\|]$ and $\mathbb{E}_P[\|X\|^2]$ . Furthermore, we prove that the $\sqrtε$ dependence is worst-case optimal. The result extends stability guarantees restricted to Gaussian triples to comparisons involving an arbitrary distribution and two nearby Gaussian references. Experiments on synthetic distributions and latent representations from flow-based models support the predicted stability behavior.

Figures & tables

Fig. 1: KL divergence-based analysis of OOD detection for the Gaussian case where

Q_{Z}

can be approximated by a Gaussian distribution [ 13 ] .

Panel of Figure 2 — Fig. 2: Proof roadmap of Theorem 1 .

Fig. 3: Stability of KL divergence $\mathrm{KL}(P\|\mathcal{N}_{1})$ under Gaussian perturbation. Arrows represent KL divergences.

Perturbation	$\epsilon$	$\widehat{\Delta}_{KL}$	$\widehat{B}(\epsilon)$	$\mathrm{TR}$	$\widehat{K^{\prime}}$	$K^{\prime}$	MC s.e.
mean-plus	0.08	-0.08	4.66	0.02	-0.28	13.18	0.0016
mean-minus	0.08	-0.08	4.66	0.02	-0.29	13.18	0.0016
sigma-expand	0.07	0.83	3.69	0.22	3.03	9.65	0.0007
sigma-shrink	0.07	-1.30	3.61	0.36	-4.98	11.30	0.0010
both-plus	0.06	0.60	3.45	0.17	2.55	11.53	0.0009
both-minus	0.08	-0.95	4.52	0.21	-3.32	12.84	0.0017

TABLE I: Results of the one-dimensional synthetic experiment at

\alpha=1

. MC s.e. denotes the Monte Carlo standard error in estimating

\widehat{\Delta}_{KL}

Perturbation	Description
mean-x	shift rightward along the $x$ -axis
mean-diag	shift diagonally downward and rightward
cov-x-expand	expand along the $x$ -axis
cov-y-shrink	contract along the $y$ -axis
cov-diag-expand	expand along a diagonal direction
cov-rot-anisotropic	apply rotated anisotropic expansion and contraction

TABLE II: Perturbations to

\mathcal{N}_{1}=\mathcal{N}(\mathbf{0},\mathbf{I}_{2})

Perturbation	$\epsilon$	$\widehat{\Delta}_{KL}$	$\widehat{B}(\epsilon)$	$\mathrm{TR}$	$\widehat{K^{\prime}}$	$K^{\prime}$	MC s.e.
mean-x	0.08	-0.14	5.78	0.02	-0.50	16.97	0.0015
mean-diag	0.08	-0.10	5.70	0.02	-0.34	16.94	0.0005
cov-x-expand	0.03	0.49	3.50	0.14	2.94	18.15	0.0006
cov-y-shrink	0.02	-0.28	3.24	0.09	-1.79	18.14	0.0006
cov-diag-expand	0.03	0.77	3.50	0.22	4.68	18.15	0.0008
cov-rot-anisotropic	0.03	0.72	4.61	0.16	3.95	21.37	0.0007

TABLE III: Results of the two-dimensional synthetic experiment at

\alpha=1

. MC s.e. is the Monte Carlo standard error in estimating

\widehat{\Delta}_{KL}

ID	OOD	$\widehat{\Delta}_{KL}$	$\widehat{B}(\epsilon)$	$\mathrm{TR}$	$\widehat{K^{\prime}}$	$K^{\prime}$	$\widehat{\mathrm{KL}}_{\widehat{Q}}$
abs	crescent	0.0144	0.04	0.33	5.25	15.79	2.18
	gaussian	0.0088	0.03	0.26	3.23	12.44	1.10
	sine_wave	0.0142	0.04	0.32	5.18	15.92	2.03
crescent	abs	-0.0294	0.09	0.34	-4.02	11.71	1.53
	gaussian	-0.0315	0.09	0.36	-4.31	11.97	1.79
	sine_wave	-0.0258	0.09	0.30	-3.53	11.78	1.22

TABLE IV: Results on a two-dimensional NSF. All MC s.e. values are less than

10^{-4}

\widehat{\mathrm{KL}}_{\widehat{Q}}

\mathrm{KL}(\widehat{Q}_{Z}\|\mathcal{N}(\mathbf{0},\mathbf{I}_{2}))

Panel of Figure 8 — Fig. 4: Distributions of the tightness ratio $\mathrm{TR}$ and empirical coefficient $\widehat{K^{\prime}}$ across groups of Glow latent dimensions. Dimension groups satisfying $\mathbb{E}[\|Z\|^{2}]>50$ are excluded from the violin plots for readability.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Fig. S1: Distribution of model log-likelihood. Glow model trained on CIFAR-10 assigns higher likelihoods to SVHN than CIFAR-10 and comparable likelihoods to SVHN with an adjusted contrast of factor 2.0 (SVHN-C(2.0)).

Fig. S2:

f(x)=x-\log x-1

Fig. S3: $P$ is a two-component truncated Gaussian mixture, $\mathcal{N}_{1}$ is the standard Gaussian distribution, and $\mathcal{N}_{2}$ is obtained by increasing the standard deviation of $\mathcal{N}_{1}$ .

Panel of Figure 12 — Fig. S4: KL behavior under one-dimensional Gaussian perturbations. (a) $\mathrm{KL}(\mathcal{N}_{1}\|\mathcal{N}_{2})$ versus the perturbation strength $\alpha$ . (b) $\mathrm{KL}(P\|\mathcal{N}_{1})-\mathrm{KL}(P\|\mathcal{N}_{2})$ versus $\sqrt{\mathrm{KL}(\mathcal{N}_{1}\|\mathcal{N}_{2})}$ . The curve of mean-plus almost overlaps with that of mean-minus .

Fig. S5: The estimated theoretical bound $\widehat{B}(\epsilon)$ in the one-dimensional synthetic experiment. Each marker corresponds to a value of $\alpha$ .

Fig. S6: $P$ is a two-component truncated Gaussian mixture, $\mathcal{N}_{1}$ is the standard Gaussian distribution, and $\mathcal{N}_{2}$ is obtained by perturbing $\mathcal{N}_{1}$ with covariance expansion along the $x$ -axis.

Panel of Figure 15 — Fig. S7: KL behavior under two-dimensional Gaussian perturbations. (a) $\mathrm{KL}(\mathcal{N}_{1}\|\mathcal{N}_{2})$ versus the perturbation strength $\alpha$ . The cov-x-expand and cov-diag-expand curves overlap. (b) $\widehat{\mathrm{KL}}(P\|\mathcal{N}_{1})-\widehat{\mathrm{KL}}(P\|\mathcal{N}_{2})$ versus $\sqrt{\mathrm{KL}(\mathcal{N}_{1}\|\mathcal{N}_{2})}$ . Each dot on a curve corresponds to one value of $\alpha$ .

Fig. S8: The estimated theoretical bound $\widehat{B}(\epsilon)$ in the two-dimensional synthetic experiment. Each marker corresponds to a value of $\alpha$ .

Fig. S9: Datasets used in the two-dimensional neural spline flow experiments.

Fig. S10: Neural spline flow model trained on the abs dataset. Left: density learned by the flow model; Right: samples generated by the flow-based model.

Fig. S11: Representations of ID ( abs ) and OOD datasets.

ID dataset	$p$ -value	$\mathrm{KL}(\widehat{P_{Z}}\\|\mathcal{N}(\mathbf{0},\mathbf{I}_{2}))$
abs	$0.47$	$7.48\times 10^{-6}$
crescent	$0.37$	$5.34\times 10^{-5}$
gaussian	$0.39$	$2.21\times 10^{-5}$
sine_wave	$0.39$	$1.84\times 10^{-5}$

Appendix

TABLE S1:

p

-value: Henze-Zirkler normality test results on the ID representations from the two-dimensional NSF models.

Dataset	$\mathrm{KL}(\widehat{P}_{Z}\\|\mathcal{N}(\mathbf{0},\mathbf{I}_{4}))<1/12$	$p>0.05$
CelebA32	98.6%	93.0%
SVHN	99.3%	94.4%
CIFAR-10	99.1%	91.1%

Appendix

TABLE S2: Proportions of dimension groups of ID representations.

Panel of Figure 22 — Fig. S12: Distributions of the tightness ratio $\mathrm{TR}$ and empirical coefficient $\widehat{K^{\prime}}$ across groups of Glow latent dimensions. Dimension groups satisfying $\mathbb{E}[\|Z\|^{2}]>50$ are excluded from the violin plots for readability.

ID	OOD	Proportion	Pearson’s $r$	$p$ -value
SVHN	ImageNet32	13.8%	-0.99	$<10^{-10}$
	LSUN	0.5%	-0.95	$<10^{-10}$
CIFAR-10	CelebA32	2.2%	1.00	$<10^{-10}$
	LSUN	7.5%	1.00	$<10^{-10}$

Appendix

TABLE S3: Proportions of dimension groups where

\mathbb{E}[\|Z\|^{2}]>50

. ID–OOD pairs with no such dimension groups are omitted. The Pearson correlation coefficient

r

between

\widehat{K^{\prime}}

and

\mathbb{E}[\|Z\|^{2}]

, together with the

p

-value.

Panel of Figure 24 — Fig. S13: Estimated $\mathbb{E}[\|Z\|^{2}]$ and the ratio $F_{3}/K^{\prime}$ on each dimension group when testing OOD datasets on Glow models. Groups with estimated second moments satisfying $\mathbb{E}[\|Z\|^{2}]>50$ are excluded from the violin plots for readability.

Panel of Figure 25 — Fig. S14: $\mathrm{KL}(\widehat{P}_{Z}^{(g)}\|\mathcal{N}(\mathbf{0},\mathbf{I}_{4}))$ for all groups of ID representations and $\mathrm{KL}(\widehat{Q}_{Z}^{(g)}\|\mathcal{N}(\mathbf{0},\mathbf{I}_{4}))$ for groups of OOD representations satisfying $\mathbb{E}[\|Z\|^{2}]>50$ . KL divergence values greater than $50$ are omitted to improve the scale of the visualization.

Kullback-Leibler Divergence

See all 27 papers

A Gaussian Perspective for Distributional Discrepancy in Generative Diffusion Models
Jan 20, 2026Qiang Sun, H. Vincent Poor, Wenyi ZhangDiffusion Sampling Kullback-Leibler Divergence
Department of Electronic Engineering and Information Science, University of Science and Technology of China, Hefei, China · Department of Electrical Engineering, Princeton University, Princeton, NJ, USA
On the Generalization of Knowledge Distillation: An Information-Theoretic View
May 13, 2026Bingying Li, Haiyun HeInformation-Theoretic Generalization Bounds Statistical Learning Theory
Internet of Things Thrust, Information Hub The Hong Kong University of Science and Technology (Guangzhou) Guangzhou, China
Optimal Transport PAC-Bayes Bounds
Oct 12, 2026Armin Beck, Peter Ochs, Jalal FadiliTV Distance Computation PAC-Bayesian Generalization Bounds
Max Planck Institute for Informatics, Saarland Informatics Campus, Saarbrücken, Germany · Saarland University, Saarland Informatics Campus, Saarbrücken, Germany · ENSICAEN, CNRS GREYC, Caen, France

Jan 20, 2026J/K move · Enter open · S save

A Gaussian Perspective for Distributional Discrepancy in Generative Diffusion Models

Qiang Sun, H. Vincent Poor, Wenyi Zhang

Department of Electronic Engineering and Information Science, University of Science and Technology of China, Hefei, China · Department of Electrical Engineering, Princeton University, Princeton, NJ, USA

Diffusion Sampling Kullback-Leibler Divergence Diffusion Model Inference Acceleration

Open paper AskPDF arXiv

This paper introduces an analytical approach to quantifying and optimizing the distributional discrepancy in generative diffusion models. For a multivariate Gaussian source, we explicitly derive the closed-form evolution trajectory and the resulting Kullback-Leibler (KL) divergence between the distributions of the source data and the reversely sampled data. Asymptotic analysis via the Euler-Maclaurin expansion characterizes the convergence behavior of this KL divergence, extracting its dominant term as an explicit functional of the noise schedule. Minimizing this dominant term via the calculus of variations yields a noise schedule described by a tangent law, inherently determined by the source covariance spectrum. We further prove that the Gaussian source exhibits an extremal property for the KL divergence among general source distributions with a given covariance. We also utilize the analytical KL divergence as a principled metric to identify efficient time discretization strategies for pretrained diffusion models, and demonstrate via experiments over diverse datasets that the identified strategies consistently outperform established baselines, particularly under constrained function evaluation budgets.

Abstract

Kullback-Leibler Divergence

Explore similar work

A Gaussian Perspective for Distributional Discrepancy in Generative Diffusion Models

On the Generalization of Knowledge Distillation: An Information-Theoretic View

Optimal Transport PAC-Bayes Bounds