Tackling Privacy Heterogeneity in Differentially Private Federated Learning

Ruichen Xu; Ying-Jun Angela Zhang; Jianwei Huang

cs.LGFeb 26, 2026

Tackling Privacy Heterogeneity in Differentially Private Federated Learning

Authors: Ruichen Xu, Ying-Jun Angela Zhang, Jianwei Huang

Organizations: Department of Information Engineering, The Chinese University of Hong Kong, Hong Kong · School of Science and Engineering, Shenzhen Institute of Artificial Intelligence and Robotics for Society, Shenzhen Key Laboratory of Crowd Intelligence Empowered Low-Carbon Energy Network, and CSIJRI Joint Research Centre on Smart Energy Storage, The Chinese University of Hong Kong, Shenzhen, Guangdong, 518172, P.R. China

Abstract

Differentially private federated learning (DP-FL) enables clients to collaboratively train machine learning models while preserving the privacy of their local data. However, most existing DP-FL approaches assume that all clients share a uniform privacy budget, an assumption that does not hold in real-world scenarios where privacy requirements vary widely. This privacy heterogeneity poses a significant challenge: conventional client selection strategies, which typically rely on data quantity, cannot distinguish between clients providing high-quality updates and those introducing substantial noise due to strict privacy constraints. To address this gap, we present the first systematic study of privacy-aware client selection in DP-FL. We establish a theoretical foundation by deriving a convergence analysis that quantifies the impact of privacy heterogeneity on training error. Building on this analysis, we propose a privacy-aware client selection strategy, formulated as a convex optimization problem, that adaptively adjusts selection probabilities to minimize training error. Extensive experiments on benchmark datasets demonstrate that our approach achieves up to a 10% improvement in test accuracy on CIFAR-10 compared to existing baselines under heterogeneous privacy budgets. These results highlight the importance of incorporating privacy heterogeneity into client selection for practical and effective federated learning.

Figures & tables

Fig. 1 : Illustration of differentially private federated learning.

Ref.	Client Selection Method	DP
[ 4 ]	Unbiased selection	✓
[ 5 ]	Unbiased selection	✓
[ 6 ]	Unbiased selection	✓
[ 7 ]	Unbiased selection	✓
[ 8 ]	Importance sampling based selection	✗
UCB-CS [ 9 ]	Bandit based selection	✗

TABLE I : Comparison of related works.

Symbol	Meaning
$p^{\text{s}}_{k}$	Selection probability of client $k$
$p^{\text{u}}_{k}$	Unbiased selection probability of client $k$
$\epsilon_{k}$	Privacy budget of client $k$
$\sigma_{k}^{2}$	Noise variance of client $k$
$\mathcal{M}_{k}$	Local dataset of client $k$
$r_{k}$	Subsampling ratio of client $k$

TABLE II : Summary of important symbols.

Fig. 2 : Workflow of the privacy-aware client selection.

Fig. 3 : Selection probability of 100 clients with heterogeneous dataset sizes and privacy budgets ( $\epsilon$ ). Subsampling ratio $r$ is fixed to 0.1 for all clients.

Panel of Figure 6 — Fig. 4 : Test accuracy on CIFAR-10.

Mechanism	Non-IID Degree	Test Accuracy
Privacy-aware	$s=100$ (IID)	68.71%
	$s=70$	65.12%
	$s=30$	68.50%
	$s=0$	67.63%
Unbiased	$s=100$ (IID)	28.02%
	$s=70$	14.69%

TABLE III : Test accuracy performance (MNIST).

Mechanism	Non-IID Degree	Test Accuracy
Privacy-aware	$s=100$ (IID)	53.53%
	$s=70$	55.26%
	$s=30$	54.80%
	$s=0$	48.94%
Unbiased	$s=100$ (IID)	14.68%
	$s=70$	16.47%

TABLE IV : Test accuracy performance (FashionMNIST).

Fig. 5 : Test accuracy versus batch size.

Fig. 6 : Test accuracy versus privacy budget distributions. "P- $i$ - $j$ ", "U- $i$ - $j$ ", and "B- $i$ - $j$ " refer to the proposed, unbiased and biased client selection approaches with privacy budgets following Gaussian mixture distribution $0.3\mathcal{N}(i,0.04)+0.7\mathcal{N}(j,1)$ .

	$\displaystyle\sqrt{\mathbb{E}\left[F(\boldsymbol{w}_{T+1})\!-\!F^{*}\right]}\leq\mathchoice{\frac{1}{2}}{1/2}{1/2}{1/2}\left[\mathchoice{\frac{\beta T}{\gamma+T}}{\beta T/\gamma+T}{\beta T/\gamma+T}{\beta T/\gamma+T}\sqrt{2L}G^{\text{select,clip}}C\right]\!+\!\mathchoice{\frac{\sqrt{2}}{2}}{\sqrt{2}/2}{\sqrt{2}/2}{\sqrt{2}/2}C\sqrt{\!\!\mathchoice{\frac{\beta^{2}T^{2}L}{(\gamma\!+\!T)^{2}}}{\beta^{2}T^{2}L/(\gamma\!+\!T)^{2}}{\beta^{2}T^{2}L/(\gamma\!+\!T)^{2}}{\beta^{2}T^{2}L/(\gamma\!+\!T)^{2}}(1\!+\!\mathchoice{\frac{1}{\Gamma}}{1/\Gamma}{1/\Gamma}{1/\Gamma})(G^{\text{select,clip}})^{2}\!\!+\!\!\mathchoice{\frac{\beta^{2}T^{2}\!L}{(\gamma\!+\!T)^{2}}}{\beta^{2}T^{2}\!L/(\gamma\!+\!T)^{2}}{\beta^{2}T^{2}\!L/(\gamma\!+\!T)^{2}}{\beta^{2}T^{2}\!L/(\gamma\!+\!T)^{2}}(9\Gamma\!+\!1\!)G^{\text{DP}}}$	(32)
	$\displaystyle+\sqrt{\mathchoice{\frac{1}{\gamma+T}}{1/\gamma+T}{1/\gamma+T}{1/\gamma+T}}\sqrt{\mathchoice{\frac{9}{2}}{9/2}{9/2}{9/2}\!L\Gamma\gamma\!\left\\|\boldsymbol{w}_{1}\!\!-\!\boldsymbol{w}^{\text{b},}\!\right\\|^{2}_{2}\!\!+\!\!\mathchoice{\frac{9\Gamma\!+\!1}{2}}{9\Gamma\!+\!1/2}{9\Gamma\!+\!1/2}{9\Gamma\!+\!1/2}\mathchoice{\frac{\beta^{2}T\!L}{\gamma\!+\!T}}{\beta^{2}T\!L/\gamma\!+\!T}{\beta^{2}T\!L/\gamma\!+\!T}{\beta^{2}T\!L/\gamma\!+\!T}C^{2}\!+\!18\Gamma\beta TL\mathchoice{\frac{B_{1}}{\mu}}{B_{1}/\mu}{B_{1}/\mu}{B_{1}/\mu}G^{\text{b,clip}}C\!+\!\gamma\!\left(F(\boldsymbol{w}_{1})\!-\!F^{}\!\right)}$
$\displaystyle\leq$	$\displaystyle\mathchoice{\frac{1}{2}}{1/2}{1/2}{1/2}\mathchoice{\frac{\sqrt{2L}\beta TC}{\gamma+T}}{\sqrt{2L}\beta TC/\gamma+T}{\sqrt{2L}\beta TC/\gamma+T}{\sqrt{2L}\beta TC/\gamma+T}\left[G^{\text{select,clip}}\!+\!\!\sqrt{\!\left(1\!+\!\mathchoice{\frac{1}{\Gamma}}{1/\Gamma}{1/\Gamma}{1/\Gamma}\right)\!(G^{\text{select,clip}})^{2}\!\!+\!(9\Gamma+1)G^{\text{DP}}}\right]+\!\sqrt{\!18\Gamma\mathchoice{\frac{\beta TL}{\gamma+T}}{\beta TL/\gamma+T}{\beta TL/\gamma+T}{\beta TL/\gamma+T}\mathchoice{\frac{B_{1}}{\mu}}{B_{1}/\mu}{B_{1}/\mu}{B_{1}/\mu}G^{\text{b,clip}}C}$
	$\displaystyle+\!\sqrt{\mathchoice{\frac{1}{2}}{1/2}{1/2}{1/2}\mathchoice{\frac{9L\Gamma}{\gamma\!+\!T}}{9L\Gamma/\gamma\!+\!T}{9L\Gamma/\gamma\!+\!T}{9L\Gamma/\gamma\!+\!T}\gamma\left\\|\boldsymbol{w}_{1}\!-\!\boldsymbol{w}^{\text{b},}\right\\|^{2}_{2}\!+\!\mathchoice{\frac{9\Gamma+1}{2}}{9\Gamma+1/2}{9\Gamma+1/2}{9\Gamma+1/2}\mathchoice{\frac{\beta^{2}TL}{(\gamma\!+\!T)^{2}}}{\beta^{2}TL/(\gamma\!+\!T)^{2}}{\beta^{2}TL/(\gamma\!+\!T)^{2}}{\beta^{2}TL/(\gamma\!+\!T)^{2}}C^{2}\!+\mathchoice{\frac{\gamma}{\gamma+T}}{\gamma/\gamma+T}{\gamma/\gamma+T}{\gamma/\gamma+T}\left(F(\boldsymbol{w}_{1})-F^{}\right)}.$

Table 11

Differentially Private Federated Learning

Differentially private federated learning trains shared models across data-holding clients while limiting what updates reveal about individual records or users.

See all 59 papers

Heterogeneity-Aware Client Selection Methodology For Efficient Federated Learning
Feb 24, 2026Nihal Balivada, Shrey Gupta, Shashank Shreedhar Bhatt +1Client Selection in Federated Learning Heterogeneous Federated Learning
University of Oregon, US · Boston College, US · Microsoft Research, India
A Privacy-Preserving Federated Learning Method for Heterogeneous Clients
Nov 8, 2025Yusaku Negoya, Feifei Cui, Zilong Zhang +3Privacy-Preserving Federated Learning Differentially Private Federated Learning
Department of Computer and Network Engineering, The University of Electro-Communications, Tokyo, Japan · School of Computer Science and Technology, Hainan University, Haikou 570228, China. · Department of Electrical and Computer Engineering, University of Houston, Houston, TX, 77204, USA +1
An Adaptive Differentially Private Federated Learning Framework
Feb 6, 2026Jin Wang, Hui Ma, Yajun Zhang +4Non-IID Federated Learning Differentially Private Federated Learning
Xinjiang Key Laboratory of Intelligent Computing and Smart Applications, School of Software, Xinjiang University, Urumqi 830017, China · College of Geography and Remote Sensing Sciences, Xinjiang University, Urumqi, China · Jilin University, Changchun 130012, China

Feb 24, 2026J/K move · Enter open · S save

Heterogeneity-Aware Client Selection Methodology For Efficient Federated Learning

Nihal Balivada, Shrey Gupta, Shashank Shreedhar Bhatt, Suyash Gupta

University of Oregon, US · Boston College, US · Microsoft Research, India

Client Selection in Federated Learning Heterogeneous Federated Learning Non-IID Federated Learning

Open paper AskPDF arXiv

Federated Learning (FL) enables a distributed client-server architecture where multiple clients collaboratively train a global Machine Learning (ML) model without sharing sensitive local data. However, FL often results in lower accuracy than traditional ML algorithms due to statistical heterogeneity across clients. Prior works attempt to address this by using model updates, such as loss and bias, from client models to select participants that can improve the global model's accuracy. However, these updates neither accurately represent a client's heterogeneity nor are their selection methods deterministic. We mitigate these limitations by introducing Terraform, a novel client selection methodology that uses gradient updates and a deterministic selection algorithm to select heterogeneous clients for retraining. This bi-pronged approach allows Terraform to achieve up to 47 percent higher accuracy over prior works. We further demonstrate its efficiency through comprehensive ablation studies and training time analyses, providing strong justification for the robustness of Terraform.

Abstract

Differentially Private Federated Learning

Explore similar work

Heterogeneity-Aware Client Selection Methodology For Efficient Federated Learning

A Privacy-Preserving Federated Learning Method for Heterogeneous Clients

An Adaptive Differentially Private Federated Learning Framework