Differentially Private Distribution Release of Gaussian Mixture Models via KL-Divergence Minimization

Hang Liu; Anna Scaglione; Sean Peisert

cs.ITJun 4, 2025

Differentially Private Distribution Release of Gaussian Mixture Models via KL-Divergence Minimization

Authors: Hang Liu, Anna Scaglione, Sean Peisert

Organizations: State Key Laboratory of Internet of Things for Smart City and the Department of Electrical and Computer Engineering, University of Macau, Macao S.A.R. · Department of Electrical and Computer Engineering, Cornell Tech, Cornell University, New York, NY, 10044 USA. · Computing Sciences Research, Lawrence Berkeley National Laboratory, Berkeley, CA 94720 USA.

Abstract

Gaussian Mixture Models (GMMs) are widely used statistical models for representing multi-modal data distributions, with numerous applications in data mining, pattern recognition, data simulation, and machine learning. However, recent research has shown that releasing GMM parameters poses significant privacy risks, potentially exposing sensitive information about the underlying data. In this paper, we address the challenge of releasing GMM parameters while ensuring differential privacy (DP) guarantees. Specifically, we focus on the privacy protection of mixture weights, component means, and covariance matrices. We propose to use Kullback-Leibler (KL) divergence as a utility metric to assess the accuracy of the released GMM, as it captures the joint impact of noise perturbation on all the model parameters. We develop a KL-divergence-driven optimization framework for differentially private GMM release that jointly designs privacy-budget allocation and parameter perturbations. Through theoretical analysis, we quantify the effects of privacy budget allocation and perturbation statistics on the DP guarantee, and derive a tractable expression for evaluating KL divergence. We formulate and solve an optimization problem to minimize the KL divergence between the released and fitted non-private models, subject to a given $(ε, δ)$ -DP constraint. Extensive experiments on both synthetic and real-world datasets demonstrate that our approach achieves strong privacy guarantees while maintaining high utility.

Figures & tables

	( 8 )	$\displaystyle={\sum_{k=1}^{K}\mathbb{E}\!\left[\widetilde{\boldsymbol{\pi}}_{k}\ln\frac{\widetilde{\boldsymbol{\pi}}_{k}}{\boldsymbol{\pi}_{k}}\right]+\sum_{k=1}^{K}\mathbb{E}\!\left[\widetilde{\boldsymbol{\pi}}_{k}\,\mathrm{KL}\!\left(\mathcal{N}(\widetilde{\boldsymbol{\mu}}_{k},\widetilde{\boldsymbol{\Sigma}}_{k})\,\hbox{$ \left\|\vbox to0.0pt{}\right. $}\,\mathcal{N}(\boldsymbol{\mu}_{k},\boldsymbol{\Sigma}_{k})\right)\right]}$
		$\displaystyle{\color[rgb]{0,0,0}=\sum_{\widetilde{\boldsymbol{\pi}}\in\mathcal{S}}{f(\widetilde{\boldsymbol{\pi}}\|\boldsymbol{\pi}(\mathcal{D}))}\underbrace{\sum_{k=1}^{K}\tilde{\pi}_{k}\left(\ln\frac{\tilde{\pi}_{k}}{\pi_{k}}+\frac{1}{2}\left(\frac{d+1}{{\gamma_{k}}}\mathop{\rm tr}\nolimits(\boldsymbol{\Sigma}_{k}^{-1})+\mathop{\rm tr}\nolimits(\boldsymbol{\Sigma}_{k}^{-1}{\boldsymbol{\Gamma}_{k}}^{-1})-\mathbb{E}_{{\bf W}_{k}}\log\left\|{\bf I}+\boldsymbol{\Sigma}_{k}^{-1/2}{\bf W}_{k}\boldsymbol{\Sigma}_{k}^{-1/2}\right\|\right)\right)}_{\triangleq g(\widetilde{\boldsymbol{\pi}},\{\gamma_{k},\boldsymbol{\Gamma}_{k}^{-1}\}_{\forall k})}.}$		(13)

Table 1

Figure 1: Average KL divergence and confidence interval for the proposed method across

200

Monte Carlo trials. The privacy level is represented by the value of

\epsilon

Figure 2: Expected KL divergence of the proposed algorithm versus the algorithm optimization iteration with

\epsilon=1

Figure 3: Running time (in seconds) of the proposed algorithm and the baselines versus the data size

N

Figure 4: KL divergence versus the privacy level in terms of the value of

\epsilon

Figure 5: KL divergence versus the data size

N

with

\epsilon=1

Figure 6: KL divergence versus the number of classes

K

, where the total data size is fixed to

N=1000

Figure 7: KL divergence versus the dimension of the data points

d

with

K=5

and

N=1000

Figure 8: KL divergence versus the data dimension

d\in[10,70]

Figure 9: KL divergence versus the value of

\epsilon

for releasing the GMM fitted to the Iris dataset.

Figure 10: Marginal probability density of GMMs and empirical histograms of the true values fitted to the Iris dataset for the first feature, i.e., the first entry of the data points $\{{\bf x}_{n}\}_{n=1}^{N}$ . Each subplot displays the conditional density for one class.

Figure 11: KL divergence versus the value of

\epsilon

for releasing the GMM fitted to the MNIST training data using the GMM classifier.

Figure 12: Test accuracy versus the value of

\epsilon

over the MNIST testing data using the GMM classifier.

Figure 13: Test accuracy over the MNIST testing data using the MLP model trained on synthetic data.

Data class	# of data points
Class $1$	$687$
Class $2$	$428$
Class $3$	$102$
Class $4$	$147$
Class $5$	$14$
Class $6$	$31$

Table I : The number of data points in each class of the AMI data.

Figure 14: KL divergence versus the value of

\epsilon

for releasing the GMM fitting to the AMI load profile data.

Figure 15: Entry-wise marginal probability density of GMMs fitted to the AMI load profile data for each class. The subplots in each column correspond to the marginal density of one class. In these plots, the gray bars represent the empirical histograms of the true values, the blue curves represent the marginal density of the GMM computed using the proposed DP method, and the red curves represent the marginal density of the non-private GMMs computed using (2).

Figure 16: KL divergence versus the clipping threshold

B

for record-level DP preservation.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

$\displaystyle\eqref{eq08}=$	$\displaystyle\mathbb{E}\left[\sum_{k=1}^{K}\widetilde{p}(y=k)\int\widetilde{p}({\bf x}\|y=k)\ln\frac{\widetilde{p}(y=k)\widetilde{p}({\bf x}\|y=k)}{p(y=k)p({\bf x}\|y=k)}d{\bf x}\right]$
$\displaystyle=$	$\displaystyle\mathbb{E}_{\widetilde{\boldsymbol{\pi}}\|\boldsymbol{\pi}(\mathcal{D})}\left[\sum_{k=1}^{K}\widetilde{p}(y=k)\left(\ln\frac{\widetilde{p}(y=k)}{p(y=k)}+\mathbb{E}_{\{{\bf w}_{k},{\bf W}_{k}\}_{\forall k}}\left[\int\widetilde{p}({\bf x}\|y=k)\ln\frac{\widetilde{p}({\bf x}\|y=k)}{p({\bf x}\|y=k)}d{\bf x}\right]\right)\right]$
$\displaystyle=$	$\displaystyle\sum_{\widetilde{\boldsymbol{\pi}}\in\mathcal{S}}f(\widetilde{\boldsymbol{\pi}}\|\boldsymbol{\pi}(\mathcal{D}))\sum_{k=1}^{K}\tilde{\pi}_{k}\left(\ln\frac{\tilde{\pi}_{k}}{\pi_{k}}+\mathbb{E}_{\{{\bf w}_{k},{\bf W}_{k}\}_{\forall k}}\left[\int\mathcal{N}(\widetilde{\boldsymbol{\mu}}_{k},\widetilde{\boldsymbol{\Sigma}}_{k})\ln\frac{\mathcal{N}(\widetilde{\boldsymbol{\mu}}_{k},\widetilde{\boldsymbol{\Sigma}}_{k})}{\mathcal{N}(\boldsymbol{\mu}_{k},\boldsymbol{\Sigma}_{k})}d{\bf x}\right]\right).$	(36)

Appendix

Table 19

Privacy-Preserving ML

Privacy-preserving machine learning develops methods to learn, share, and run models while limiting exposure of sensitive data.

See all 432 papers

Differential Privacy of Gaussian Process Posterior Sampling
Jun 16, 2026Tomasz MaciazekGaussian Processes Privacy-Preserving ML
School of Mathematics, University of Bristol
Differentially Private Nonparametric Modal Learning with Applications to Regression and Clustering
Jul 31, 2026Arkajyoti Bhattacharjee, Arnab AuddyClustering Differential Privacy
Department of Statistics The Ohio State University
Revisiting Privacy Amplification by Subsampling in Selective Release DPSGD
Jun 3, 2026Xiaobo Huang, Fang XieDifferentially Private Stochastic Gradient Descent Differential Privacy
Guangdong Provincial Key Laboratory of IRADS, Beijing · Normal-Hong Kong Baptist University · Zhuhai, China

Jun 16, 2026J/K move · Enter open · S save

Differential Privacy of Gaussian Process Posterior Sampling

Tomasz Maciazek

School of Mathematics, University of Bristol

Gaussian Processes Privacy-Preserving ML Differential Privacy

Open paper AskPDF arXiv

We study the privacy of releasing functional posterior sample paths from a Gaussian process (GP) when the entire training set including covariates and responses is private. Unlike standard differential-privacy (DP) mechanisms that inject external noise, posterior sampling is intrinsically random and we show that this randomness provides useful privacy guarantees. We derive Rényi-DP guarantees separating privacy leakage through the posterior mean from a distinct channel induced by the data-dependent posterior covariance. The analysis identifies effective ridge regularisation and covariance scale as the principal privacy-controlling quantities and yields sharper guarantees in several regimes of practical interest as well as extensions to repeated and adaptive releases. Membership inference attacks confirm the predicted dependence on regularisation, covariance scale and the number of released paths. Utility experiments on downstream posterior sampling tasks identify noisy observation regimes where privacy-compatible regularisation preserves useful samples. Finally we identify large-data asymptotic regime in which the privacy parameter and posterior mean-square risk vanish simultaneously, yielding privacy for free. Together, these results provide a comprehensive characterisation of privacy and utility of GP posterior sampling.

Abstract

Privacy-Preserving ML

Explore similar work

Differential Privacy of Gaussian Process Posterior Sampling

Differentially Private Nonparametric Modal Learning with Applications to Regression and Clustering

Revisiting Privacy Amplification by Subsampling in Selective Release DPSGD