Evaluating tight Differential Privacy (DP) parameters for DP matrix factorization with privacy amplification by sampling requires analyzing high-dimensional, correlated Gaussian mixtures. Because integrating these mixture distributions to find the exact privacy failure probability (δ) is analytically intractable, these mechanisms are typically analyzed using Monte Carlo accounting. However, standard sampling suffers from prohibitive variance when estimating small δ associated with strict privacy guarantees, which in turn requires a large sample complexity, making Monte Carlo accounting inefficient. We demonstrate that Monte Carlo accounting can be made much more efficient via importance sampling. We give a shifted importance sampling proposal distribution parameterized by a stretch factor α>1. We prove a closed-form bound B(α)<1 on the support of the importance-weighted estimator that holds identically for both directions of the hockey-stick divergence, show that B is log-convex so that the optimal α is a one-dimensional root-finding problem, and show that the resulting estimator seamlessly plugs into the KL-Hoeffding verification protocol used by past work, with a sample complexity reduced by exactly a factor of 1/B(α∗).
Figures & tables
Figure 1: Samples needed to certify (ϵ,δ) -DP with ϵ=10 for balls-in-bins sampling with T∈{10,100,1000} modes over a single epoch, as a function of δ (swept via the noise multiplier), for a banded C whose column is the normalized geometric sequence (1,21,41,81) . Dashed: standard Monte Carlo ( B=1 ). Solid: importance sampling with the support bound Bκ,ϵ(α∗) . Each point is estimated to 10% relative standard error.
T
σ
δ^
B
Bδ^/v^
nest
ncert
10
0.32
5.1⋅10−3
1.1⋅10−1
6.8
3.1⋅102
3.4⋅102
10
0.44
2.0⋅10−6
7.8⋅10−4
56
6.7⋅102
1.6⋅104
10
0.57
1.1⋅10−10
1.3⋅10−6
918
1.3⋅103
8.9⋅105
100
0.27
9.3⋅10−3
3.4⋅10−1
7.1
5.1⋅102
5.3⋅102
100
0.40
5.9⋅10−7
4.7⋅10−3
356
2.2⋅103
3.7⋅105
100
0.49
1.1⋅10−10
8.6⋅10−5
12,209
6.2⋅103
5.7⋅107
Table 1: Empirical variance of the importance-weighted samples for the configurations of Fig. 1 , at the largest, an intermediate, and the smallest δ of each curve. Bδ^/v^ is the factor by which the worst-case variance implied by the support bound exceeds the observed variance; nest=100v^/δ^2 is the number of samples needed to estimate δ^ to 10% relative standard error, and ncert the number of samples to certify it ( τ=2 , failure probability δ ).
Figure 2: Samples needed to certify δ for the optimized banded configuration ( m=T=7000 , 256 bands, ϵ=10 ). Curves as in Fig. 1 ; the vertical line marks the target δ=10−9 .