Noise-Aware Differentially Private Variational Inference

Talal Alrawajfeh; Joonas Jälkö; Antti Honkela

stat.MLOct 25, 2024

Noise-Aware Differentially Private Variational Inference

Authors: Talal Alrawajfeh, Joonas Jälkö, Antti Honkela

Organizations: University of Helsinki

Abstract

Differential privacy (DP) provides robust privacy guarantees for statistical inference, but this can lead to unreliable results and biases in downstream applications. While several noise-aware approaches have been proposed which integrate DP perturbation into the inference, they are limited to specific types of simple probabilistic models. In this work, we propose a novel method for noise-aware approximate Bayesian inference based on stochastic gradient variational inference which can also be applied to high-dimensional and non-conjugate models. We also propose a more accurate evaluation method for noise-aware posteriors. Empirically, our inference method has similar performance to existing methods in the domain where they are applicable. Outside this domain, we obtain accurate coverages on high-dimensional Bayesian linear regression and well-calibrated predictive probabilities on Bayesian logistic regression with the UCI Adult dataset.

Figures & tables

Figure 1: An example of the linear model of the perturbed gradients

\kappa\mathbf{A}(\boldsymbol{\phi}_{t}-\boldsymbol{\phi}^{*})

(black line) based on M1 .

Figure 2: The first row in the figure shows the TARP coverages for the exponential families experiment for NA-DPVI (NUTS), Bernstein & Sheldon’s method ( Bernstein and Sheldon, 2018 ) , last iterate DPVI, and DPVIm ( Jälkö et al., 2023 ) . The second row shows the error for the coverages (

C(\alpha)-(1-\alpha)

). The solid lines show the average performance over

20

independent TARP repetitions and the error bars show the corresponding std. The parameters for NA-DPVI are

\delta=10^{-5}

N=5000

\kappa=0.1

and

T=10^{4}

Method	Gamma-Exponential	Beta-Bernoulli	Dirichlet-Categorical
NA-DPVI (NUTS)	0.023 $\pm$ 0.008	0.016 $\pm$ 0.006	0.020 $\pm$ 0.004
Bernstein & Sheldon	0.034 $\pm$ 0.011	0.018 $\pm$ 0.006	0.017 $\pm$ 0.007
Naive (last iterate)	0.232 $\pm$ 0.003	0.273 $\pm$ 0.007	0.355 $\pm$ 0.009
DPVIm	0.038 $\pm$ 0.005	0.044 $\pm$ 0.004	0.251 $\pm$ 0.011

Table 1: The RMSE errors corresponding to the exponential family coverages in Fig. 2 . Average RMSE

\pm

std.

\delta=10^{-5}

in all experiments.

Figure 3: The first row in the figure shows the TARP coverages for the 10D Bayesian linear regression experiment for NA-DPVI (NUTS, Laplace), last iterate DPVI, DPVIm ( Jälkö et al., 2023 ) , and Gibbs-SS-Noisy ( Bernstein and Sheldon, 2019 ) . The second row shows the error for the coverages (

C(\alpha)-(1-\alpha)

). The solid lines show the average performance over

20

independent TARP repetitions and the error bars show the corresponding std. The parameters for NA-DPVI are

\delta=10^{-5}

N=5000

\kappa=0.1

and

T=10^{4}

Figure 4: The first row in the figure shows the predictive calibration for the UCI Adult logistic regression experiment for NA-DPVI (NUTS), NA-DPVI (Laplace), last iterate DPVI, and DPVIm ( Jälkö et al., 2023 ) . The second row shows the calibration error (Fraction of Positives - Predicted Probability). The solid lines show the average performance over

20

independent repetitions, and the error bars show the corresponding std. The parameters for NA-DPVI are

\delta=10^{-5}

\kappa=0.1

and

T=10^{4}

Method	$\epsilon=0.1$	$\epsilon=0.3$	$\epsilon=1.0$
NA-DPVI (NUTS)	36 $\pm$ 11	35 $\pm$ 11	27 $\pm$ 9
NA-DPVI (Laplace)	78 $\pm$ 11	98 $\pm$ 6	60 $\pm$ 7
Naive (last iterate)	512 $\pm$ 3	307 $\pm$ 3	360 $\pm$ 3
Bernstein & Sheldon	301 $\pm$ 1	299 $\pm$ 1	323 $\pm$ 3
DPVIm	584 $\pm$ 0	584 $\pm$ 0	584 $\pm$ 0

Table 2: The RMSE errors corresponding to the Bayesian linear regression coverages in Fig. 3 . Average RMSE

\pm

std, with all the values scaled by

10^{3}

\delta=10^{-5}

in all experiments.

Method	$\epsilon=0.1$	$\epsilon=0.3$	$\epsilon=1.0$
NA-DPVI (NUTS)	61 $\pm$ 31	26 $\pm$ 5	24 $\pm$ 7
NA-DPVI (Laplace)	84 $\pm$ 43	44 $\pm$ 11	46 $\pm$ 14
Naive (last iterate)	120 $\pm$ 65	67 $\pm$ 24	54 $\pm$ 17
DPVIm	101 $\pm$ 52	65 $\pm$ 25	38 $\pm$ 14

Table 3: The RMSE errors corresponding to the logistic regression calibration in Fig. 4 . Average RMSE

\pm

std, with all the values scaled by

10^{3}

\delta=10^{-5}

in all experiments.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Published location	Original	Corrected
p. 3 §2.3, Def. 2; before (6)	$S\subset\operatorname{im}(A)$	$S\subset\operatorname{Range}(\mathcal{A})$
p. 4 §3.1, (14), RHS	$\mathbf{1}_{\widetilde{\mathcal{C}}_{\alpha}(\xi)}(\theta)p(\theta,\xi)$	$\mathbb{E}_{\theta,\xi\sim p(\theta,\xi)}[\mathbf{1}_{\widetilde{\mathcal{C}}_{\alpha}(\xi)}(\theta)]$
p. 4 §3.3, (16), (17)	$i\in\mathcal{B}_{t}$	$i\in\mathcal{B}_{t+1}$
p. 5 Thm. 1; after (26)	$e_{\rm approx}^{2}=e_{\rm sub}^{2}+(\kappa e_{\rm tay}r^{*})^{2}$	$e_{\rm approx}^{2}=2e_{\rm sub}^{2}+2(\kappa e_{\rm tay}r^{*})^{2}$
p. 6 §4.1, Hessian paragraph	Positive Hessian; ELBO sign implicit	Defines the minimized loss as the negative ELBO
p. 8 §4.4, (29), integrand	$p(y=1\mid\theta)$	$p(y=1\mid\theta,x)$

Appendix

Table 8

Published location	Original	Corrected
p. 12 App. B; before (A3)	Squared-norm bounds omit factors of 2 ( $\\|a+b\\|^{2}\leq\\|a\\|^{2}+\\|b\\|^{2}$ )	Coefficient 2 on both terms ( $\\|a+b\\|^{2}\leq 2\\|a\\|^{2}+2\\|b\\|^{2}$ )
p. 12 App. C; after (A5)	Marginalizes out $\widehat{\phi}^{*}_{\rm map}$	Marginalizes out $V$ , retaining the marginal distribution of $\phi^{*}$ .
p. 13 (A9) and following prose	$\max_{i}\operatorname{Var}[\hat{a}_{i}]$	$\max_{i}\sqrt{\operatorname{Var}[\hat{a}_{i}]}$
p. 14 App. D, (A11)	$e_{\rm sub}^{2}+\kappa^{2}G^{2}+\sigma_{\rm DP}^{2}C^{2}d$	$2e_{\rm sub}^{2}+2\kappa^{2}G^{2}+\sigma_{\rm DP}^{2}C^{2}d$
p. 14 App. E, opening	$\mathcal{L}(\phi^{*};D)=0$	$\nabla\mathcal{L}(\phi^{*};D)=0$
p. 15 App. E, (A16), lines 1–3	$\widetilde{g}_{t}^{(i)}$	$\widetilde{g}_{t+1}^{(i)}$

Appendix

Table 9

Explore similar work

DP-IVON-Gradsq: Differentially Private Squared-Gradient Improved Variational Online Newton
Jul 26, 2026Nour Jamoussi, Ikram Dridi, Giuseppe Serra +1Bayesian Neural Networks Privacy-Preserving ML
EURECOM, Sophia Antipolis, France · Concordia University, Montreal, Canada · University of Granada, Granada, Spain
Noise-Calibrated Inference from Differentially Private Sufficient Statistics in Exponential Families
Mar 2, 2026Amir Asiaee, Samhita PalDifferential Privacy Confidence Region Estimation
Department of Biostatistics, Vanderbilt University Medical Center, Nashville, TN, USA
Enhancing Differentially Private Mechanisms via Empirical Bayes
Jun 19, 2026Minwoo Kim, Junyong Park, Sungkyu JungEmpirical Bayes Differential Privacy
Department of Statistics Seoul National University Seoul, 08826, South Korea · Department of Statistics and Institute for Data Innovation in Science Seoul National University Seoul, 08826, South Korea

Jul 26, 2026J/K move · Enter open · S save

DP-IVON-Gradsq: Differentially Private Squared-Gradient Improved Variational Online Newton

Nour Jamoussi, Ikram Dridi, Giuseppe Serra, Marios Kountouris

EURECOM, Sophia Antipolis, France · Concordia University, Montreal, Canada · University of Granada, Granada, Spain

Bayesian Neural Networks Privacy-Preserving ML Differential Privacy Variational Inference

Open paper AskPDF arXiv

Differential privacy provides formal privacy guarantees for training neural networks on sensitive data, while Bayesian deep learning offers a principled framework for uncertainty-aware prediction. Combining these two objectives remains challenging, as privacy noise can interact with the stochasticity introduced by Bayesian posterior sampling. In this work, we investigate differentially private variational Bayesian learning through the Improved Variational Online Newton (IVON) optimizer. We introduce DP-IVON-Gradsq, a private variant of IVON. The proposed method constructs its curvature estimate from the privatized gradient using a noise-corrected squared-gradient estimator, reducing the direct interaction between posterior-sampling noise and privacy noise while preserving the Adam-like computational efficiency of IVON. We evaluate DP-IVON-Gradsq on CIFAR-10 against the standard private optimizers DP-SGD and DP-Adam over a range of privacy budgets. The results show that DP-IVON-Gradsq is competitive under weak-to-moderate privacy constraints, i.e., large-to-moderate values of

\varepsilon

, while degrading under strong privacy. Code is available at https://github.com/NourJamoussi/DP-IVON-Gradsq.git.

Abstract

Explore similar work

DP-IVON-Gradsq: Differentially Private Squared-Gradient Improved Variational Online Newton

Noise-Calibrated Inference from Differentially Private Sufficient Statistics in Exponential Families

Enhancing Differentially Private Mechanisms via Empirical Bayes