We consider differentially private (DP) optimization in the federated setting, where a DP estimate of the feature covariance matrix serves as a Hessian approximation for generalized linear models. Existing central DP methods for covariance estimation have communication cost that grows quadratically in the data dimension, while existing baselines with linear communication cost are only locally private and degrade as the number of clients increases. As a remedy, we propose a federated DP-PCA framework that achieves linear communication cost while providing central DP guarantees under secure aggregation. It consists of a DP rank selection step, a DP power method, and a noisy release of the covariance projected onto the selected low-dimensional subspace. We give error bounds for this covariance estimate and for the resulting low-communication preconditioned DP gradient descent. Empirically, our proposed DP optimization method matches a centralized baseline with quadratic communication cost at a fraction of the communication.
Figures & tables
Figure 1: Singular values of the MNIST feature matrix X∈Rp×n versus the square roots of the eigenvalues of the DP estimate C for projection dimensions r∈{20,50,80} .
Figure 2: Comparison of our method against the central DP-PCA method of Dwork et al. (2014) on UCI HAR ( p=561 ) with target subspace dimension k∈{10,20,40} . The y -axis shows the relative Frobenius error of the rank- k covariance approximation, the x -axis ε for δ=10−5 . The noisy power method communicates Tpk+k(k+1)/2 scalars in total (from 16885 to 68140 for T=3 ), whereas Analyze Gauss communicates the upper triangle of the covariance, p(p+1)/2=157641 .
Figure 3: CIFAR-10 classification with frozen ViT-B/16 features for ε=2 ( δ=10−5 ). Left: classification accuracy as a function of the total number of communication rounds; each point corresponds to independent runs with that number of rounds. Our DP low-rank preconditioner achieves accuracy comparable to that of vanilla DP-FC. Right: classification accuracy as a function of total communication (number of transmitted scalars) for ε=2 ( δ=10−5 ). Our method achieves the best accuracy–communication trade-off across the entire communication range.
Figure 4: FashionMNIST with ResNet-34 features ( p=512 ), ε=2 ( δ=10−5 ). Left: accuracy vs. rounds. Right: accuracy vs. total communication. Setup as in Figure 3 .
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Comparison against the federated DP-PCA method of Grammenos et al. (2020) . The privacy–utility trade-off of Grammenos et al. (2020) degrades as the number of clients grows.
Figure 6: Heatmaps of the WISDM covariance matrix when ε=0.16 and δ=10−5 . Left: the true (non-private) covariance. Center: the rank- 20 approximation produced by our method. Right: the rank- 20 approximation produced by the Analyze Gauss baseline ( Dwork et al., 2014 ) .
Figure 7: The original heatmap of the covariance matrix for UCI HAR dataset features, heatmap obtained using our method with ε=0.16 and heatmap obtained using the baseline Analyze Gauss with ε=0.16 and rank 20.
Figure 8: Comparison of our method against the central DP-PCA method of Dwork et al. (2014) on SVHN ( p=3072 ), with target subspace dimension k∈{10,20,40} . The vertical axis reports the relative Frobenius error of the rank- k covariance approximation, the horizontal axis ε for δ=10−5 . The results are means of 5 independent runs.
Figure 9: CIFAR-10 total communication rounds vs. accuracy.
Figure 10: CIFAR-100 total communication rounds vs. accuracy.
Figure 11: FashionMNIST total communication rounds vs. accuracy.
Figure 12: CIFAR-10 total transmitted parameters vs. accuracy.
Figure 13: CIFAR-100 total transmitted parameters vs. accuracy.
Figure 14: FashionMNIST total transmitted parameters vs. accuracy.
M(m)←∑j=1nmxj(m)(xj(m))⊤
Appendix
Algorithm 2 Federated DP Power Method with Server-Side QR-Orthogonalization
Yk(m)←Vk⊤(n1∑i=1nmxi(m)xi(m)⊤)Vk∈Rk×k
Appendix
Algorithm 3 Federated Private Estimation of the Covariance based on Gaussian mechanism ( Dwork et al., 2014 )
G∈Rp×r,Gij∼N(0,1/r)
Appendix
Algorithm 4 Private Rank Selection via Random Projection & Gaussian mechanism
Differentially private federated learning trains shared models across data-holding clients while limiting what updates reveal about individual records or users.
Department of Computer Science, City University of Hong Kong · Department of Electrical and Computer Engineering, University of British Columbia · School of Computer Science and Engineering, Hebrew University of Jerusalem