Data-Driven Laplacian-Constrained Personalized Federated Learning

Samuel Erickson; Mikael Johansson

cs.LGMay 12, 2025

Data-Driven Laplacian-Constrained Personalized Federated Learning

Authors: Samuel Erickson, Mikael Johansson

Organizations: Division of Decision and Control Systems, KTH Royal Institute of Technology, Stockholm, Sweden

Abstract

Personalization addresses one of the defining challenges in federated learning: statistical heterogeneity among clients. We show that the dissimilarity between the ideal personalized models of two clients can be upper bounded by the dissimilarity of their distributions. Motivated by this, we propose Karula, a data-driven regularization method that constrains pairwise model dissimilarities between clients according to their distributional differences. Distributional similarity is measured using a surrogate for the 1-Wasserstein distance adapted for the federated setting, allowing the method to adapt to highly complex interrelations between clients that approaches based on hard clustering may fail to recover. We propose an approximate projected stochastic gradient algorithm to solve the resulting constrained optimization problem, and prove that it converges at a rate of $O(1/K)$ to a neighborhood of a stationary point.

Figures & tables

\alpha_{i}^{k+1}=\begin{cases}\theta_{i}^{k}&\text{if }i\in S_{k},\\ \alpha_{i}^{k}&\text{otherwise}.\end{cases}

Algorithm 1 Karula

	FEMNIST			CIFAR10 $\alpha=0.1$			CIFAR10 $\alpha=0.5$			CIFAR100 $\alpha=0.1$			CIFAR100 $\alpha=0.5$
Method	10th	50th	90th	10th	50th	90th	10th	50th	90th	10th	50th	90th	10th	50th	90th
FedAvg	71.61	83.07	91.83	44.44	63.78	78.72	59.61	70.45	79.02	31.48	38.90	46.83	31.65	37.40	42.37
pFedMe	46.69	60.80	74.92	59.65	81.78	98.11	43.10	58.47	73.32	30.81	43.38	51.50	25.92	32.41	39.28
IFCA	71.08	82.18	92.12	46.63	65.67	85.07	60.08	69.17	76.61	31.34	39.35	48.49	32.43	36.17	41.67
FedSoft	40.61	63.47	76.33	59.42	82.85	96.84	43.09	60.74	73.47	39.80	49.49	57.84	12.92	36.96	45.27
pFedGraph	64.65	80.66	88.78	21.01	49.75	71.27	41.85	57.59	69.43	23.71	31.80	41.20	29.31	36.19	42.61

Table 1: Average 10th, 50th, and 90th percentile client test accuracies on FEMNIST and CIFAR datasets over three random seeds. The results including standard deviations are given in Table 2 in Appendix A .

Panel of Figure 1 — Figure 1: Validation accuracy for different values of the threshold parameter, with individually tuned step sizes.

Panel of Figure 2 — Figure 2: Karula ’s mean validation accuracy (and standard devation error bars) on the FEMNIST dataset for (a) different reference distributions and (b) different reference dataset sizes $N_{0}$ .

Panel of Figure 3 — Figure 3: Samples of FEMNIST handwritten digits from (a) client chosen at random, (b) its nearest neighbor, and (c) its furthest neighbor.

Panel of Figure 4 — Figure 4: Label densities in CIFAR10 ( $\alpha=0.1$ ) of (a) client chosen at random, (b) its nearest neighbor, and (c) its furthest neighbor.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Percentile	FEMNIST	CIFAR10 $\alpha=0.1$	CIFAR10 $\alpha=0.5$	CIFAR100 $\alpha=0.1$	CIFAR100 $\alpha=0.5$
FedAvg	10th	$71.61\pm 2.22\%$	$44.44\pm 16.84\%$	$59.61\pm 2.42\%$	$31.48\pm 2.19\%$	$31.65\pm 1.34\%$
	50th	$83.07\pm 0.85\%$	$63.78\pm 8.69\%$	$70.45\pm 1.20\%$	$38.90\pm 0.95\%$	$37.40\pm 3.46\%$
	90th	$91.83\pm 0.35\%$	$78.72\pm 1.92\%$	$79.02\pm 1.28\%$	$46.83\pm 1.09\%$	$42.37\pm 3.82\%$
pFedMe	10th	$46.69\pm 5.01\%$	$59.65\pm 4.39\%$	$43.10\pm 1.28\%$	$30.81\pm 0.78\%$	$25.92\pm 1.93\%$
	50th	$60.80\pm 5.13\%$	$81.78\pm 2.21\%$	$58.47\pm 1.84\%$	$43.38\pm 1.06\%$	$32.41\pm 0.46\%$
	90th	$74.92\pm 5.24\%$	$98.11\pm 0.74\%$	$73.32\pm 2.77\%$	$51.50\pm 2.70\%$	$39.28\pm 1.87\%$

Appendix

Table 2: Average 10th, 50th, and 90th percentile client test accuracies on FEMNIST and CIFAR datasets over three random seeds with

\pm

standard deviations.

Method	Learning rate	Tuned hyperparameter	Other hyperparameters
FedAvg	0.25	—	n_local_epochs=1
pFedMe	0.0625	moreau_param=10.0	beta=1.0 ; inner_steps=5 ; mu=0.001 ; n_local_epochs=1
IFCA	0.125	k_clusters=3	aggregation=samples ; max_init_attempts=100 ; min_cluster_fraction=0.0 ; n_local_epochs=1 ; variant=full
FedSoft	0.25	reg_weight=0.01	k_clusters=4 ; n_local_epochs=1 ; refresh_every=2 ; smoothing=0.0001
pFedGraph	0.25	reg_param=0.1	alpha=0.8 ; n_local_epochs=1
FedAMP	0.125	lambda_1=1.0	momentum=0.9 ; n_local_epochs=1 ; self_weight=0.3 ; similarity_scale=10.0 ; weight_decay=1e-05

Appendix

Table 3: Method hyperparameters for FEMNIST.

Method	Learning rate	Tuned hyperparameter	Other hyperparameters
FedAvg	0.015625	—	n_local_epochs=1
pFedMe	0.015625	moreau_param=10.0	beta=1.0 ; inner_steps=5 ; mu=0.001 ; n_local_epochs=1
IFCA	0.0078125	k_clusters=3	aggregation=uniform ; max_init_attempts=100 ; min_cluster_fraction=0.0 ; n_local_epochs=1 ; variant=full
FedSoft	0.0078125	reg_weight=1.0	k_clusters=4 ; n_local_epochs=1 ; refresh_every=2 ; smoothing=0.0001
pFedGraph	0.03125	reg_param=1.0	alpha=0.8 ; n_local_epochs=1
FedAMP	0.015625	lambda_1=0.01	momentum=0.9 ; n_local_epochs=1 ; self_weight=0.3 ; similarity_scale=10.0 ; weight_decay=1e-05

Appendix

Table 4: Method hyperparameters for CIFAR10

\alpha=0.1

Method	Learning rate	Tuned hyperparameter	Other hyperparameters
FedAvg	0.015625	—	n_local_epochs=1
pFedMe	0.015625	moreau_param=10.0	beta=1.0 ; inner_steps=5 ; mu=0.001 ; n_local_epochs=1
IFCA	0.0078125	k_clusters=2	aggregation=uniform ; max_init_attempts=100 ; min_cluster_fraction=0.0 ; n_local_epochs=1 ; variant=full
FedSoft	0.0078125	reg_weight=1.0	k_clusters=4 ; n_local_epochs=1 ; refresh_every=2 ; smoothing=0.0001
pFedGraph	0.03125	reg_param=0.1	alpha=0.8 ; n_local_epochs=1
FedAMP	0.0078125	lambda_1=1.0	momentum=0.9 ; n_local_epochs=1 ; self_weight=0.3 ; similarity_scale=10.0 ; weight_decay=1e-05

Appendix

Table 5: Method hyperparameters for CIFAR10

\alpha=0.5

Method	Learning rate	Tuned hyperparameter	Other hyperparameters
FedAvg	0.015625	—	n_local_epochs=1
pFedMe	0.125	moreau_param=1.0	beta=1.0 ; inner_steps=5 ; mu=0.001 ; n_local_epochs=1
IFCA	0.03125	k_clusters=8	aggregation=uniform ; max_init_attempts=100 ; min_cluster_fraction=0.0 ; n_local_epochs=1 ; variant=full
FedSoft	0.03125	reg_weight=0.01	k_clusters=4 ; n_local_epochs=1 ; refresh_every=2 ; smoothing=0.0001
pFedGraph	0.03125	reg_param=0.1	alpha=0.8 ; n_local_epochs=1
FedAMP	0.00390625	lambda_1=0.1	momentum=0.9 ; n_local_epochs=1 ; self_weight=0.3 ; similarity_scale=10.0 ; weight_decay=1e-05

Appendix

Table 6: Method hyperparameters for CIFAR100

\alpha=0.1

Method	Learning rate	Tuned hyperparameter	Other hyperparameters
FedAvg	0.0078125	—	n_local_epochs=1
pFedMe	0.125	moreau_param=10.0	beta=1.0 ; inner_steps=5 ; mu=0.001 ; n_local_epochs=1
IFCA	0.125	k_clusters=2	aggregation=uniform ; max_init_attempts=100 ; min_cluster_fraction=0.0 ; n_local_epochs=1 ; variant=full
FedSoft	0.125	reg_weight=1.0	k_clusters=4 ; n_local_epochs=1 ; refresh_every=2 ; smoothing=0.0001
pFedGraph	0.125	reg_param=1.0	alpha=0.8 ; n_local_epochs=1
FedAMP	0.03125	lambda_1=1.0	momentum=0.9 ; n_local_epochs=1 ; self_weight=0.3 ; similarity_scale=10.0 ; weight_decay=1e-05

Appendix

Table 7: Method hyperparameters for CIFAR100

\alpha=0.5

Layer	Operation	Output Shape
Input	Grayscale image	$1\times 28\times 28$
Conv1	Conv2d(1, 32, kernel=3, pad=1) + ReLU	$32\times 28\times 28$
Pool1	MaxPool2d(2,2)	$32\times 14\times 14$
Conv2	Conv2d(32, 64, kernel=3, pad=1) + ReLU	$64\times 14\times 14$
Pool2	MaxPool2d(2,2)	$64\times 7\times 7$
Flatten	Vectorization	$64\cdot 7\cdot 7$

Appendix

Table 8: Architecture of the CNN used for FEMNIST classification.

Layer	Operation	Output Shape
Input	RGB image	$3\times 32\times 32$
Conv1	Conv2d(3, 32, kernel=5, padding=2) + BatchNorm + ReLU	$32\times 32\times 32$
Pool1	MaxPool2d(2,2)	$32\times 16\times 16$
Conv2	Conv2d(32, 64, kernel=5, padding=2) + BatchNorm + ReLU	$64\times 16\times 16$
Pool2	MaxPool2d(2,2)	$64\times 8\times 8$
Flatten	Vectorization	$64\cdot 8\cdot 8$

Appendix

Table 9: Architecture of the CNN used for CIFAR-

K

classification.

Federated Learning

Federated learning trains shared models across data holders without requiring them to pool their raw data.

See all 943 papers

Few-for-Many Personalized Federated Learning
Mar 12, 2026Ping Guo, Tiantian Zhang, Xi Lin +3Federated Learning Heterogeneous Federated Learning
City University of Hong Kong · CityU Shenzhen Research Institute · Hong Kong Metropolitan University +3
A Locally Penalized Cross-Estimate Federated Method with Guarantees for Constrained Personalized Learning
Mar 20, 2026Mohammadjavad Ebrahimi, Daniel Burbano, Farzad YousefianFederated Learning Heterogeneous Federated Learning
Department of Industrial and Systems Engineering, Rutgers University, USA. · Department of Electrical and Computer Engineering, Rutgers University, USA.
FedLoRD: Personalized Federated Learning based on Low-Rank Structures of Model Parameter Differences
Oct 13, 2026Tsutahiro Fukuhara, Junya Hara, Hiroshi Higashi +1Federated Learning Non-IID Federated Learning
Graduate School of Engineering, The University of Osaka, Osaka, Japan · Faculty of Engineering Science, Kansai University, Osaka, Japan

Mar 12, 2026J/K move · Enter open · S save

Few-for-Many Personalized Federated Learning

Ping Guo, Tiantian Zhang, Xi Lin, Xiang Li+2

City University of Hong Kong · CityU Shenzhen Research Institute · Hong Kong Metropolitan University+3

Federated Learning Heterogeneous Federated Learning Multi-Objective Optimization

Open paper AskPDF arXiv

Personalized Federated Learning (PFL) aims to train customized models for clients with highly heterogeneous data distributions while preserving data privacy. Existing approaches often rely on heuristics like clustering or model interpolation, which lack principled mechanisms for balancing heterogeneous client objectives. Serving

M

clients with distinct data distributions is inherently a multi-objective optimization problem, where achieving optimal personalization ideally requires

M

distinct models on the Pareto front. However, maintaining

M

separate models poses significant scalability challenges in federated settings with hundreds or thousands of clients. To address this challenge, we reformulate PFL as a few-for-many optimization problem that maintains only

K

shared server models (

K \ll M

) to collectively serve all

M

clients. We prove that this framework achieves near-optimal personalization: the approximation error diminishes as

K

increases and each client's model converges to each client's optimum as data grows. Building on this reformulation, we propose FedFew, a practical algorithm that jointly optimizes the

K

server models through efficient gradient-based updates. Unlike clustering-based approaches that require manual client partitioning or interpolation-based methods that demand careful hyperparameter tuning, FedFew automatically discovers the optimal model diversity through its optimization process. Experiments across vision, NLP, and real-world medical imaging datasets demonstrate that FedFew, with just 3 models, consistently outperforms other state-of-the-art approaches. Code is available at https://github.com/pgg3/FedFew.

Abstract

Federated Learning

Explore similar work

Few-for-Many Personalized Federated Learning

A Locally Penalized Cross-Estimate Federated Method with Guarantees for Constrained Personalized Learning

FedLoRD: Personalized Federated Learning based on Low-Rank Structures of Model Parameter Differences