We consider differentially private (DP) optimization in the federated setting, where a DP estimate of the feature covariance matrix serves as a Hessian approximation for generalized linear models. Existing central DP methods for covariance estimation have communication cost that grows quadratically in the data dimension, while existing baselines with linear communication cost are only locally private and degrade as the number of clients increases. As a remedy, we propose a federated DP-PCA framework that achieves linear communication cost while providing central DP guarantees under secure aggregation. It consists of a DP rank selection step, a DP power method, and a noisy release of the covariance projected onto the selected low-dimensional subspace. We give error bounds for this covariance estimate and for the resulting low-communication preconditioned DP gradient descent. Empirically, our proposed DP optimization method matches a centralized baseline with quadratic communication cost at a fraction of the communication.
Figures & tables
Figure 1: Singular values of the MNIST feature matrix X∈Rp×n versus the square roots of the eigenvalues of the DP estimate C for projection dimensions r∈{20,50,80} .
Figure 2: Comparison of our method against the central DP-PCA method of Dwork et al. (2014) on UCI HAR ( p=561 ) with target subspace dimension k∈{10,20,40} . The y -axis shows the relative Frobenius error of the rank- k covariance approximation, the x -axis ε for δ=10−5 . The noisy power method communicates Tpk+k(k+1)/2 scalars in total (from 16885 to 68140 for T=3 ), whereas Analyze Gauss communicates the upper triangle of the covariance, p(p+1)/2=157641 .
Figure 3: CIFAR-10 classification with frozen ViT-B/16 features for ε=2 ( δ=10−5 ). Left: classification accuracy as a function of the total number of communication rounds; each point corresponds to independent runs with that number of rounds. Our DP low-rank preconditioner achieves accuracy comparable to that of vanilla DP-FC. Right: classification accuracy as a function of total communication (number of transmitted scalars) for ε=2 ( δ=10−5 ). Our method achieves the best accuracy–communication trade-off across the entire communication range.
Figure 4: FashionMNIST with ResNet-34 features ( p=512 ), ε=2 ( δ=10−5 ). Left: accuracy vs. rounds. Right: accuracy vs. total communication. Setup as in Figure 3 .
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Comparison against the federated DP-PCA method of Grammenos et al. (2020) . The privacy–utility trade-off of Grammenos et al. (2020) degrades as the number of clients grows.
Figure 6: Heatmaps of the WISDM covariance matrix when ε=0.16 and δ=10−5 . Left: the true (non-private) covariance. Center: the rank- 20 approximation produced by our method. Right: the rank- 20 approximation produced by the Analyze Gauss baseline ( Dwork et al., 2014 ) .
Figure 7: The original heatmap of the covariance matrix for UCI HAR dataset features, heatmap obtained using our method with ε=0.16 and heatmap obtained using the baseline Analyze Gauss with ε=0.16 and rank 20.
Figure 8: Comparison of our method against the central DP-PCA method of Dwork et al. (2014) on SVHN ( p=3072 ), with target subspace dimension k∈{10,20,40} . The vertical axis reports the relative Frobenius error of the rank- k covariance approximation, the horizontal axis ε for δ=10−5 . The results are means of 5 independent runs.
Figure 9: CIFAR-10 total communication rounds vs. accuracy.
Figure 10: CIFAR-100 total communication rounds vs. accuracy.
Figure 11: FashionMNIST total communication rounds vs. accuracy.
Figure 12: CIFAR-10 total transmitted parameters vs. accuracy.
Figure 13: CIFAR-100 total transmitted parameters vs. accuracy.
Figure 14: FashionMNIST total transmitted parameters vs. accuracy.
M(m)←∑j=1nmxj(m)(xj(m))⊤
Appendix
Algorithm 2 Federated DP Power Method with Server-Side QR-Orthogonalization
Yk(m)←Vk⊤(n1∑i=1nmxi(m)xi(m)⊤)Vk∈Rk×k
Appendix
Algorithm 3 Federated Private Estimation of the Covariance based on Gaussian mechanism ( Dwork et al., 2014 )
G∈Rp×r,Gij∼N(0,1/r)
Appendix
Algorithm 4 Private Rank Selection via Random Projection & Gaussian mechanism
Differentially private federated learning trains shared models across data-holding clients while limiting what updates reveal about individual records or users.
Federated Learning is a leading framework for training ML and AI models collaboratively across numerous user devices or databases. We study the trade-offs among estimation accuracy, privacy constraints, and communication cost for differentially private (DP) federated M estimation. The two standard methods in the literature are FedAvg, which may suffer from high federation bias, and FedSGD, which can incur high communication cost. Aimed at improving accuracy at a reduced communication cost, we propose FedHybrid, which uses FedSGD starting with an improved initialization by the FedAvg estimator. We propose FedNewton, which averages local Newton iterations to reduce bias in FedAvg, achieving an estimation accuracy comparable to FedSGD with much fewer communication rounds when the number of clients grows sufficiently slowly. We establish finite sample upper bounds on the mean-squared error rates of the DP versions of these estimators as functions of the number of clients, local sample sizes, privacy budget, and number of iterations. We further derive a minimax lower bound on the MSE of any iterative private federated procedure that provides a benchmark to assess the optimality gap of these methods. We numerically evaluate our methods for training a logistic regression and a neural network on the computer vision datasets MNIST and CIFAR-10.
Arnab Auddy, Xiangni Peng, Subhadeep Paul
Department of Statistics · The Ohio State University
Differentially private optimization suffers from a fundamental geometric mismatch: deep networks have highly anisotropic loss landscapes, yet DP-SGD injects isotropic noise. Second-order preconditioning can resolve this, but estimating curvature typically requires private data (consuming privacy budget) or public data (introducing distribution shift). We show that the Fisher Information Matrix decouples into architectural sensitivity, recoverable via synthetic noise, and input correlations, approximable from modality-specific frequency statistics. We propose DP-KFC, which constructs KFAC preconditioners by probing networks with structured synthetic noise, requiring neither private nor public data. Empirically, DP-KFC consistently outperforms DP-SGD and adaptive baselines across diverse modalities in strong privacy regimes (ε≤3). DP-KFC matches private-data preconditioners while public-data variants degrade by up to 4.8%, showing that curvature can be estimated without consuming privacy budget or introducing distribution shift. This enables privacy-preserving learning in specialized domains (e.g., medical applications) where regulatory constraints make data scarce.
Marc Molina Van den Bosch, Riccardo Taiello, Albert Sund Aillet +3
CERN, Geneva, Switzerland · BCN MedTech, Department of Engineering, Universitat Pompeu Fabra, Barcelona, Spain · ICREA, Barcelona, Spain
Record-level differential privacy exposes a structural misalignment in personalized federated learning when client-specific variation is low-dimensional while training repeatedly releases high-dimensional updates. In this paper, we address this misalignment by releasing a private client context once and confining repeated adaptation to a fixed coefficient space. Beyond dimensionality reduction, the factorized generator induces an adaptive optimization geometry that reshapes noisy updates, and controlled ablations show that most of its private-training gain is retained by radial evolution. To further reduce the communication cost, we realize the Gaussian mechanism for coefficient updates directly through variable-length quantization with finite expected code length, so that the quantization error itself serves as the required privacy perturbation rather than extra distortion. Across MNIST and CIFAR-10, our design matches or outperforms full-model private adaptation across privacy budgets and client heterogeneity, while reducing protected uplink by a factor of 2.67 at ε=16 on CIFAR-10 with comparable future-client accuracy.
Yilin Xu, Chun Hei Michael Shiu, Chih Wei Ling +1
Department of Computer Science, City University of Hong Kong · Department of Electrical and Computer Engineering, University of British Columbia · School of Computer Science and Engineering, Hebrew University of Jerusalem