Differentially Private Distribution Release of Gaussian Mixture Models via KL-Divergence Minimization

Hang Liu; Anna Scaglione; Sean Peisert

cs.ITJun 4, 2025

Differentially Private Distribution Release of Gaussian Mixture Models via KL-Divergence Minimization

Authors: Hang Liu, Anna Scaglione, Sean Peisert

Organizations: State Key Laboratory of Internet of Things for Smart City and the Department of Electrical and Computer Engineering, University of Macau, Macao S.A.R. · Department of Electrical and Computer Engineering, Cornell Tech, Cornell University, New York, NY, 10044 USA. · Computing Sciences Research, Lawrence Berkeley National Laboratory, Berkeley, CA 94720 USA.

Abstract

Gaussian Mixture Models (GMMs) are widely used statistical models for representing multi-modal data distributions, with numerous applications in data mining, pattern recognition, data simulation, and machine learning. However, recent research has shown that releasing GMM parameters poses significant privacy risks, potentially exposing sensitive information about the underlying data. In this paper, we address the challenge of releasing GMM parameters while ensuring differential privacy (DP) guarantees. Specifically, we focus on the privacy protection of mixture weights, component means, and covariance matrices. We propose to use Kullback-Leibler (KL) divergence as a utility metric to assess the accuracy of the released GMM, as it captures the joint impact of noise perturbation on all the model parameters. We develop a KL-divergence-driven optimization framework for differentially private GMM release that jointly designs privacy-budget allocation and parameter perturbations. Through theoretical analysis, we quantify the effects of privacy budget allocation and perturbation statistics on the DP guarantee, and derive a tractable expression for evaluating KL divergence. We formulate and solve an optimization problem to minimize the KL divergence between the released and fitted non-private models, subject to a given $(ε, δ)$ -DP constraint. Extensive experiments on both synthetic and real-world datasets demonstrate that our approach achieves strong privacy guarantees while maintaining high utility.

Figures & tables

	( 8 )	$\displaystyle={\sum_{k=1}^{K}\mathbb{E}\!\left[\widetilde{\boldsymbol{\pi}}_{k}\ln\frac{\widetilde{\boldsymbol{\pi}}_{k}}{\boldsymbol{\pi}_{k}}\right]+\sum_{k=1}^{K}\mathbb{E}\!\left[\widetilde{\boldsymbol{\pi}}_{k}\,\mathrm{KL}\!\left(\mathcal{N}(\widetilde{\boldsymbol{\mu}}_{k},\widetilde{\boldsymbol{\Sigma}}_{k})\,\hbox{$ \left\|\vbox to0.0pt{}\right. $}\,\mathcal{N}(\boldsymbol{\mu}_{k},\boldsymbol{\Sigma}_{k})\right)\right]}$
		$\displaystyle{\color[rgb]{0,0,0}=\sum_{\widetilde{\boldsymbol{\pi}}\in\mathcal{S}}{f(\widetilde{\boldsymbol{\pi}}\|\boldsymbol{\pi}(\mathcal{D}))}\underbrace{\sum_{k=1}^{K}\tilde{\pi}_{k}\left(\ln\frac{\tilde{\pi}_{k}}{\pi_{k}}+\frac{1}{2}\left(\frac{d+1}{{\gamma_{k}}}\mathop{\rm tr}\nolimits(\boldsymbol{\Sigma}_{k}^{-1})+\mathop{\rm tr}\nolimits(\boldsymbol{\Sigma}_{k}^{-1}{\boldsymbol{\Gamma}_{k}}^{-1})-\mathbb{E}_{{\bf W}_{k}}\log\left\|{\bf I}+\boldsymbol{\Sigma}_{k}^{-1/2}{\bf W}_{k}\boldsymbol{\Sigma}_{k}^{-1/2}\right\|\right)\right)}_{\triangleq g(\widetilde{\boldsymbol{\pi}},\{\gamma_{k},\boldsymbol{\Gamma}_{k}^{-1}\}_{\forall k})}.}$		(13)

Table 1

Figure 1: Average KL divergence and confidence interval for the proposed method across

200

Monte Carlo trials. The privacy level is represented by the value of

\epsilon

Figure 2: Expected KL divergence of the proposed algorithm versus the algorithm optimization iteration with

\epsilon=1

Figure 3: Running time (in seconds) of the proposed algorithm and the baselines versus the data size

N

Figure 4: KL divergence versus the privacy level in terms of the value of

\epsilon

Figure 5: KL divergence versus the data size

N

with

\epsilon=1

Figure 6: KL divergence versus the number of classes

K

, where the total data size is fixed to

N=1000

Figure 7: KL divergence versus the dimension of the data points

d

with

K=5

and

N=1000

Figure 8: KL divergence versus the data dimension

d\in[10,70]

Figure 9: KL divergence versus the value of

\epsilon

for releasing the GMM fitted to the Iris dataset.

Figure 10: Marginal probability density of GMMs and empirical histograms of the true values fitted to the Iris dataset for the first feature, i.e., the first entry of the data points $\{{\bf x}_{n}\}_{n=1}^{N}$ . Each subplot displays the conditional density for one class.

Figure 11: KL divergence versus the value of

\epsilon

for releasing the GMM fitted to the MNIST training data using the GMM classifier.

Figure 12: Test accuracy versus the value of

\epsilon

over the MNIST testing data using the GMM classifier.

Figure 13: Test accuracy over the MNIST testing data using the MLP model trained on synthetic data.

Data class	# of data points
Class $1$	$687$
Class $2$	$428$
Class $3$	$102$
Class $4$	$147$
Class $5$	$14$
Class $6$	$31$

Table I : The number of data points in each class of the AMI data.

Figure 14: KL divergence versus the value of

\epsilon

for releasing the GMM fitting to the AMI load profile data.

Figure 15: Entry-wise marginal probability density of GMMs fitted to the AMI load profile data for each class. The subplots in each column correspond to the marginal density of one class. In these plots, the gray bars represent the empirical histograms of the true values, the blue curves represent the marginal density of the GMM computed using the proposed DP method, and the red curves represent the marginal density of the non-private GMMs computed using (2).

Figure 16: KL divergence versus the clipping threshold

B

for record-level DP preservation.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

$\displaystyle\eqref{eq08}=$	$\displaystyle\mathbb{E}\left[\sum_{k=1}^{K}\widetilde{p}(y=k)\int\widetilde{p}({\bf x}\|y=k)\ln\frac{\widetilde{p}(y=k)\widetilde{p}({\bf x}\|y=k)}{p(y=k)p({\bf x}\|y=k)}d{\bf x}\right]$
$\displaystyle=$	$\displaystyle\mathbb{E}_{\widetilde{\boldsymbol{\pi}}\|\boldsymbol{\pi}(\mathcal{D})}\left[\sum_{k=1}^{K}\widetilde{p}(y=k)\left(\ln\frac{\widetilde{p}(y=k)}{p(y=k)}+\mathbb{E}_{\{{\bf w}_{k},{\bf W}_{k}\}_{\forall k}}\left[\int\widetilde{p}({\bf x}\|y=k)\ln\frac{\widetilde{p}({\bf x}\|y=k)}{p({\bf x}\|y=k)}d{\bf x}\right]\right)\right]$
$\displaystyle=$	$\displaystyle\sum_{\widetilde{\boldsymbol{\pi}}\in\mathcal{S}}f(\widetilde{\boldsymbol{\pi}}\|\boldsymbol{\pi}(\mathcal{D}))\sum_{k=1}^{K}\tilde{\pi}_{k}\left(\ln\frac{\tilde{\pi}_{k}}{\pi_{k}}+\mathbb{E}_{\{{\bf w}_{k},{\bf W}_{k}\}_{\forall k}}\left[\int\mathcal{N}(\widetilde{\boldsymbol{\mu}}_{k},\widetilde{\boldsymbol{\Sigma}}_{k})\ln\frac{\mathcal{N}(\widetilde{\boldsymbol{\mu}}_{k},\widetilde{\boldsymbol{\Sigma}}_{k})}{\mathcal{N}(\boldsymbol{\mu}_{k},\boldsymbol{\Sigma}_{k})}d{\bf x}\right]\right).$	(36)

Appendix

Table 19

Privacy-Preserving ML

Privacy-preserving machine learning develops methods to learn, share, and run models while limiting exposure of sensitive data.

See all 432 papers

Jun 16, 2026

Differential Privacy of Gaussian Process Posterior Sampling

We study the privacy of releasing functional posterior sample paths from a Gaussian process (GP) when the entire training set including covariates and responses is private. Unlike standard differential-privacy (DP) mechanisms that inject external noise, posterior sampling is intrinsically random and we show that this randomness provides useful privacy guarantees. We derive Rényi-DP guarantees separating privacy leakage through the posterior mean from a distinct channel induced by the data-dependent posterior covariance. The analysis identifies effective ridge regularisation and covariance scale as the principal privacy-controlling quantities and yields sharper guarantees in several regimes of practical interest as well as extensions to repeated and adaptive releases. Membership inference attacks confirm the predicted dependence on regularisation, covariance scale and the number of released paths. Utility experiments on downstream posterior sampling tasks identify noisy observation regimes where privacy-compatible regularisation preserves useful samples. Finally we identify large-data asymptotic regime in which the privacy parameter and posterior mean-square risk vanish simultaneously, yielding privacy for free. Together, these results provide a comprehensive characterisation of privacy and utility of GP posterior sampling.

Tomasz Maciazek

School of Mathematics, University of Bristol

Gaussian Processes Privacy-Preserving ML Differential Privacy

Ask PDF

Jul 31, 2026

Differentially Private Nonparametric Modal Learning with Applications to Regression and Clustering

Density modes provide a localized and interpretable summary of multimodal distributions, but their estimation under rigorous differential privacy constraints remains largely unexplored. We study differentially private recovery of density modes for multivariate distributions under local smoothness, curvature, and separation conditions. We propose DP-GRAMS, a mean-shift inspired method that performs noisy ascent on a differentially private score estimator. Assuming the density belongs locally to a Hölder class with smoothness parameter

β> 2

, our score estimator uses bias-reducing higher-order kernels, and then enforces privacy in the gradient ascent steps via gradient clipping and calibrated Gaussian noise. A private initialization scheme combines a density-aware utility with a suppression rule and, with

k\asymp M\log n

draws over a public

h_{\mathrm{DAP}}

-grid and suppression radius

ρ_{\mathrm{init}}\asymp (\log n)^{-1/d}

, achieves high-probability coverage of the modal basins by successively suppressing selected local neighborhoods in competitive regions, while correlated noise across multiple starts enables joint release under a single

(\varepsilon,δ)

-differential privacy guarantee. We prove that all population modes are recovered with high probability and establish asymptotic error rates of the form

O\!\left((\tfrac{\log n}{n})^{\frac{2(β-1)}{d+2β}}\right) + O\!\left((\tfrac{\mathrm{polylog}(n,δ)}{n^2\varepsilon^2})^{\frac{β-1}{d+β}}\right)

. We also provide minimax lower bounds for private mode estimation, and show that our estimators are nearly optimal, up to a logarithmic factor in the MSE. We present two natural extensions: DP-PMS, a private modal-regression method, and DP-GRAMS-C, a clustering pipeline. Extensive experiments on synthetic and real data demonstrate favorable privacy-utility trade-offs relative to common baselines.

Arkajyoti Bhattacharjee, Arnab Auddy

Department of Statistics The Ohio State University

Clustering Differential Privacy

Ask PDF

Jun 3, 2026

Revisiting Privacy Amplification by Subsampling in Selective Release DPSGD

Machine learning's reliance on sensitive data necessitates privacy-preserving techniques like Differentially Private Stochastic Gradient Descent (DPSGD). However, DPSGD suffers from substantial utility degradation and slow convergence due to gradient clipping and noise injection. Prior works have attempted to improve DPSGD from various perspectives; notably, the Differentially Private Selective Update and Release (DPSUR) algorithm has achieved remarkable model utility. However, the privacy accounting in DPSUR overlooks the variation in sampling probability introduced by the selective release mechanism, which compromises the rigor of its privacy guarantees. To address these limitations, we re-evaluate the privacy analysis of the selective release mechanism and propose a novel algorithm: Differentially Private Selective Release based on Clipped Gradients (DPSR-CG). Through a rigorous, newly derived privacy analysis and extensive experiments on multiple datasets (MNIST, CIFAR-10, IMDB, and FMNIST), we demonstrate that our DPSR-CG mechanism maintains strict privacy guarantees while achieving exceptional model performance.

Xiaobo Huang, Fang Xie

Guangdong Provincial Key Laboratory of IRADS, Beijing · Normal-Hong Kong Baptist University · Zhuhai, China

Differentially Private Stochastic Gradient Descent Differential Privacy

Ask PDF

Abstract

Privacy-Preserving ML

Explore similar work

Differential Privacy of Gaussian Process Posterior Sampling

Differentially Private Nonparametric Modal Learning with Applications to Regression and Clustering

Revisiting Privacy Amplification by Subsampling in Selective Release DPSGD