Noise-Aware Differentially Private Variational Inference

Talal Alrawajfeh; Joonas Jälkö; Antti Honkela

stat.MLOct 25, 2024

Noise-Aware Differentially Private Variational Inference

Authors: Talal Alrawajfeh, Joonas Jälkö, Antti Honkela

Organizations: University of Helsinki

Abstract

Differential privacy (DP) provides robust privacy guarantees for statistical inference, but this can lead to unreliable results and biases in downstream applications. While several noise-aware approaches have been proposed which integrate DP perturbation into the inference, they are limited to specific types of simple probabilistic models. In this work, we propose a novel method for noise-aware approximate Bayesian inference based on stochastic gradient variational inference which can also be applied to high-dimensional and non-conjugate models. We also propose a more accurate evaluation method for noise-aware posteriors. Empirically, our inference method has similar performance to existing methods in the domain where they are applicable. Outside this domain, we obtain accurate coverages on high-dimensional Bayesian linear regression and well-calibrated predictive probabilities on Bayesian logistic regression with the UCI Adult dataset.

Figures & tables

Figure 1: An example of the linear model of the perturbed gradients

\kappa\mathbf{A}(\boldsymbol{\phi}_{t}-\boldsymbol{\phi}^{*})

(black line) based on M1 .

Figure 2: The first row in the figure shows the TARP coverages for the exponential families experiment for NA-DPVI (NUTS), Bernstein & Sheldon’s method ( Bernstein and Sheldon, 2018 ) , last iterate DPVI, and DPVIm ( Jälkö et al., 2023 ) . The second row shows the error for the coverages (

C(\alpha)-(1-\alpha)

). The solid lines show the average performance over

20

independent TARP repetitions and the error bars show the corresponding std. The parameters for NA-DPVI are

\delta=10^{-5}

N=5000

\kappa=0.1

and

T=10^{4}

Method	Gamma-Exponential	Beta-Bernoulli	Dirichlet-Categorical
NA-DPVI (NUTS)	0.023 $\pm$ 0.008	0.016 $\pm$ 0.006	0.020 $\pm$ 0.004
Bernstein & Sheldon	0.034 $\pm$ 0.011	0.018 $\pm$ 0.006	0.017 $\pm$ 0.007
Naive (last iterate)	0.232 $\pm$ 0.003	0.273 $\pm$ 0.007	0.355 $\pm$ 0.009
DPVIm	0.038 $\pm$ 0.005	0.044 $\pm$ 0.004	0.251 $\pm$ 0.011

Table 1: The RMSE errors corresponding to the exponential family coverages in Fig. 2 . Average RMSE

\pm

std.

\delta=10^{-5}

in all experiments.

Figure 3: The first row in the figure shows the TARP coverages for the 10D Bayesian linear regression experiment for NA-DPVI (NUTS, Laplace), last iterate DPVI, DPVIm ( Jälkö et al., 2023 ) , and Gibbs-SS-Noisy ( Bernstein and Sheldon, 2019 ) . The second row shows the error for the coverages (

C(\alpha)-(1-\alpha)

). The solid lines show the average performance over

20

independent TARP repetitions and the error bars show the corresponding std. The parameters for NA-DPVI are

\delta=10^{-5}

N=5000

\kappa=0.1

and

T=10^{4}

Figure 4: The first row in the figure shows the predictive calibration for the UCI Adult logistic regression experiment for NA-DPVI (NUTS), NA-DPVI (Laplace), last iterate DPVI, and DPVIm ( Jälkö et al., 2023 ) . The second row shows the calibration error (Fraction of Positives - Predicted Probability). The solid lines show the average performance over

20

independent repetitions, and the error bars show the corresponding std. The parameters for NA-DPVI are

\delta=10^{-5}

\kappa=0.1

and

T=10^{4}

Method	$\epsilon=0.1$	$\epsilon=0.3$	$\epsilon=1.0$
NA-DPVI (NUTS)	36 $\pm$ 11	35 $\pm$ 11	27 $\pm$ 9
NA-DPVI (Laplace)	78 $\pm$ 11	98 $\pm$ 6	60 $\pm$ 7
Naive (last iterate)	512 $\pm$ 3	307 $\pm$ 3	360 $\pm$ 3
Bernstein & Sheldon	301 $\pm$ 1	299 $\pm$ 1	323 $\pm$ 3
DPVIm	584 $\pm$ 0	584 $\pm$ 0	584 $\pm$ 0

Table 2: The RMSE errors corresponding to the Bayesian linear regression coverages in Fig. 3 . Average RMSE

\pm

std, with all the values scaled by

10^{3}

\delta=10^{-5}

in all experiments.

Method	$\epsilon=0.1$	$\epsilon=0.3$	$\epsilon=1.0$
NA-DPVI (NUTS)	61 $\pm$ 31	26 $\pm$ 5	24 $\pm$ 7
NA-DPVI (Laplace)	84 $\pm$ 43	44 $\pm$ 11	46 $\pm$ 14
Naive (last iterate)	120 $\pm$ 65	67 $\pm$ 24	54 $\pm$ 17
DPVIm	101 $\pm$ 52	65 $\pm$ 25	38 $\pm$ 14

Table 3: The RMSE errors corresponding to the logistic regression calibration in Fig. 4 . Average RMSE

\pm

std, with all the values scaled by

10^{3}

\delta=10^{-5}

in all experiments.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Published location	Original	Corrected
p. 3 §2.3, Def. 2; before (6)	$S\subset\operatorname{im}(A)$	$S\subset\operatorname{Range}(\mathcal{A})$
p. 4 §3.1, (14), RHS	$\mathbf{1}_{\widetilde{\mathcal{C}}_{\alpha}(\xi)}(\theta)p(\theta,\xi)$	$\mathbb{E}_{\theta,\xi\sim p(\theta,\xi)}[\mathbf{1}_{\widetilde{\mathcal{C}}_{\alpha}(\xi)}(\theta)]$
p. 4 §3.3, (16), (17)	$i\in\mathcal{B}_{t}$	$i\in\mathcal{B}_{t+1}$
p. 5 Thm. 1; after (26)	$e_{\rm approx}^{2}=e_{\rm sub}^{2}+(\kappa e_{\rm tay}r^{*})^{2}$	$e_{\rm approx}^{2}=2e_{\rm sub}^{2}+2(\kappa e_{\rm tay}r^{*})^{2}$
p. 6 §4.1, Hessian paragraph	Positive Hessian; ELBO sign implicit	Defines the minimized loss as the negative ELBO
p. 8 §4.4, (29), integrand	$p(y=1\mid\theta)$	$p(y=1\mid\theta,x)$

Appendix

Table 8

Published location	Original	Corrected
p. 12 App. B; before (A3)	Squared-norm bounds omit factors of 2 ( $\\|a+b\\|^{2}\leq\\|a\\|^{2}+\\|b\\|^{2}$ )	Coefficient 2 on both terms ( $\\|a+b\\|^{2}\leq 2\\|a\\|^{2}+2\\|b\\|^{2}$ )
p. 12 App. C; after (A5)	Marginalizes out $\widehat{\phi}^{*}_{\rm map}$	Marginalizes out $V$ , retaining the marginal distribution of $\phi^{*}$ .
p. 13 (A9) and following prose	$\max_{i}\operatorname{Var}[\hat{a}_{i}]$	$\max_{i}\sqrt{\operatorname{Var}[\hat{a}_{i}]}$
p. 14 App. D, (A11)	$e_{\rm sub}^{2}+\kappa^{2}G^{2}+\sigma_{\rm DP}^{2}C^{2}d$	$2e_{\rm sub}^{2}+2\kappa^{2}G^{2}+\sigma_{\rm DP}^{2}C^{2}d$
p. 14 App. E, opening	$\mathcal{L}(\phi^{*};D)=0$	$\nabla\mathcal{L}(\phi^{*};D)=0$
p. 15 App. E, (A16), lines 1–3	$\widetilde{g}_{t}^{(i)}$	$\widetilde{g}_{t+1}^{(i)}$

Appendix

Table 9

Explore similar work

Jul 26, 2026

DP-IVON-Gradsq: Differentially Private Squared-Gradient Improved Variational Online Newton

Differential privacy provides formal privacy guarantees for training neural networks on sensitive data, while Bayesian deep learning offers a principled framework for uncertainty-aware prediction. Combining these two objectives remains challenging, as privacy noise can interact with the stochasticity introduced by Bayesian posterior sampling. In this work, we investigate differentially private variational Bayesian learning through the Improved Variational Online Newton (IVON) optimizer. We introduce DP-IVON-Gradsq, a private variant of IVON. The proposed method constructs its curvature estimate from the privatized gradient using a noise-corrected squared-gradient estimator, reducing the direct interaction between posterior-sampling noise and privacy noise while preserving the Adam-like computational efficiency of IVON. We evaluate DP-IVON-Gradsq on CIFAR-10 against the standard private optimizers DP-SGD and DP-Adam over a range of privacy budgets. The results show that DP-IVON-Gradsq is competitive under weak-to-moderate privacy constraints, i.e., large-to-moderate values of

\varepsilon

, while degrading under strong privacy. Code is available at https://github.com/NourJamoussi/DP-IVON-Gradsq.git.

Nour Jamoussi, Ikram Dridi, Giuseppe Serra +1

EURECOM, Sophia Antipolis, France · Concordia University, Montreal, Canada · University of Granada, Granada, Spain

Bayesian Neural Networks Privacy-Preserving ML Differential Privacy Variational Inference

Ask PDF

Mar 2, 2026

Noise-Calibrated Inference from Differentially Private Sufficient Statistics in Exponential Families

Many differentially private (DP) data release systems either output DP synthetic data and leave analysts to perform inference as usual, which can lead to severe miscalibration, or output a DP point estimate without a principled way to do uncertainty quantification. This paper develops a clean and tractable middle ground for exponential families: release only DP sufficient statistics, then perform noise-calibrated likelihood-based inference and optional parametric synthetic data generation as post-processing. Our contributions are: (1) a general recipe for approximate-DP release of clipped sufficient statistics under the Gaussian mechanism; (2) asymptotic normality, explicit variance inflation, and valid Wald-style confidence intervals for the plug-in DP MLE; (3) a noise-aware likelihood correction that is first-order equivalent to the plug-in but supports bootstrap-based intervals; and (4) a matching minimax lower bound showing the privacy distortion rate is unavoidable. The resulting theory yields concrete design rules and a practical pipeline for releasing DP synthetic data with principled uncertainty quantification, validated on three exponential families and real census data.

Amir Asiaee, Samhita Pal

Department of Biostatistics, Vanderbilt University Medical Center, Nashville, TN, USA

Differential Privacy Confidence Region Estimation

Ask PDF

Jun 19, 2026

Enhancing Differentially Private Mechanisms via Empirical Bayes

Differential privacy (DP) has become the gold standard for ensuring the privacy protection of machine learning and statistical algorithms in recent decades. A plethora of algorithms and methods have been developed to enhance the utility of DP algorithms while maintaining the same level of DP. However, these are often overly complex or computationally ineffective. We propose a novel approach focusing on denoising the output of the simple additive Gaussian mechanism by adopting the idea of empirical Bayes estimation. We highlight that the empirical Bayes approach can reduce the mean-squared error solely by taking the output of the Gaussian mechanism as input. Our numerical studies show that this simple yet powerful approach can be applied to improve upon various statistical problems, including histogram release, principal component analysis, and linear regression, often outperforming existing private algorithms.

Minwoo Kim, Junyong Park, Sungkyu Jung

Department of Statistics Seoul National University Seoul, 08826, South Korea · Department of Statistics and Institute for Data Innovation in Science Seoul National University Seoul, 08826, South Korea

Empirical Bayes Differential Privacy

Ask PDF