FedDAG: Clustered Federated Learning via Global Data and Gradient Integration for Heterogeneous Environments

Anik Pramanik; Murat Kantarcioglu; Vincent Oria; Shantanu Sharma

cs.LGFeb 26, 2026

FedDAG: Clustered Federated Learning via Global Data and Gradient Integration for Heterogeneous Environments

Authors: Anik Pramanik, Murat Kantarcioglu, Vincent Oria, Shantanu Sharma

Organizations: New Jersey Institute of Technology, USA · Virginia Tech, USA

Abstract

Federated Learning (FL) enables a group of clients to collaboratively train a model without sharing individual data, but its performance drops when client data are heterogeneous. Clustered FL tackles this by grouping similar clients. However, existing clustered FL approaches rely solely on either data similarity or gradient similarity; however, this results in an incomplete assessment of client similarities. Prior clustered FL approaches also restrict knowledge and representation sharing to clients within the same cluster. This prevents cluster models from benefiting from the diverse client population across clusters. To address these limitations, FedDAG introduces a clustered FL framework, FedDAG, that employs a weighted, class-wise similarity metric that integrates both data and gradient information, providing a more holistic measure of similarity during clustering. In addition, FedDAG adopts a dual-encoder architecture for cluster models, comprising a primary encoder trained on its own clients' data and a secondary encoder refined using gradients from complementary clusters. This enables cross-cluster feature transfer while preserving cluster-specific specialization. Experiments on diverse benchmarks and data heterogeneity settings show that FedDAG consistently outperforms state-of-the-art clustered FL baselines in accuracy.

Figures & tables

Algorithm	Technique	CIFAR-10	FMNIST
PACFL	Data (D)	90.45 $\pm$ 0.30	94.41 $\pm$ 0.31
CFL	Gradient (G)	72.80 $\pm$ 0.66	86.97 $\pm$ 0.23
IFCA	Gradient (G)	89.68 $\pm$ 0.17	94.03 $\pm$ 0.09
FedDAG (Ours)	D + G + Global Feature Sharing	94.53 $\pm$ 0.12	96.82 $\pm$ 0.18

Table 1: Accuracy (%) of FedDAG vs. clustering baselines under non-IID label skew (20%) and quantity shift (Dirichlet

\alpha^{\prime}{=}1

Figure 1: Overview of FedDAG . Clients compute principal vectors and gradients to build an adjacency matrix and a graph indicating which clusters can supply features for cross-cluster sharing. Training proceeds in two phases: (1) the primary encoder and classifier are trained on each cluster’s local data; (2) the secondary encoder of a requesting cluster is trained on source cluster’s data

Algorithm	CIFAR-10	FMNIST	SVHN
FedAvg	42.87 $\pm$ 0.36	42.68 $\pm$ 0.49	37.93 $\pm$ 0.39
FedBR	62.41 $\pm$ 0.28	82.81 $\pm$ 0.17	80.12 $\pm$ 0.22
FedSoft	64.34 $\pm$ 0.38	75.89 $\pm$ 0.15	76.35 $\pm$ 0.40
PACFL	59.82 $\pm$ 0.22	78.42 $\pm$ 0.35	78.82 $\pm$ 0.12
CFL	61.48 $\pm$ 0.15	82.73 $\pm$ 0.23	79.15 $\pm$ 0.36
CFL-GP	66.74 $\pm$ 0.28	84.71 $\pm$ 0.13	82.38 $\pm$ 0.13

Table 2: Exp 5: Performance comparison for concept shift across datasets.

Algorithm	CIFAR-10	FMNIST	SVHN	CIFAR-100
FedDAG †	88.79 $\pm$ 0.20	92.61 $\pm$ 0.31	91.95 $\pm$ 0.25	70.28 $\pm$ 0.38
FedDAG ∗	88.67 $\pm$ 0.18	92.75 $\pm$ 0.22	91.87 $\pm$ 0.26	70.37 $\pm$ 0.33
FedDAG	90.76 $\pm$ 0.12	93.82 $\pm$ 0.20	93.91 $\pm$ 0.23	72.84 $\pm$ 0.30

Table 3: Exp 3: Ablation study of cross-cluster representation sharing under 20% label skew (Dirichlet

\alpha^{\prime}=0.25

), comparing FedDAG , FedDAG † (dual encoder w/o GRS), and FedDAG ∗ (single encoder).

Panel of Figure 2 — Figure 2: Exp 2: Clustering score vs cluster $\alpha$ and number of clusters for finding optimal clustering.

	20% Label Skew				30% Label Skew
Algorithm	CIFAR-10	FMNIST	SVHN	CIFAR-100	CIFAR-10	FMNIST	SVHN	CIFAR-100
FedAvg	42.02 $\pm$ 1.17	53.11 $\pm$ 0.31	69.79 $\pm$ 0.51	47.16 $\pm$ 0.91	54.24 $\pm$ 0.08	72.86 $\pm$ 0.40	64.15 $\pm$ 0.64	50.99 $\pm$ 1.35
FedProx	43.98 $\pm$ 0.17	53.61 $\pm$ 0.20	74.75 $\pm$ 0.27	50.56 $\pm$ 0.70	54.99 $\pm$ 0.20	68.22 $\pm$ 0.16	64.80 $\pm$ 0.25	48.66 $\pm$ 0.80
PerFedAvg	81.09 $\pm$ 0.35	86.51 $\pm$ 0.19	89.20 $\pm$ 0.05	65.59 $\pm$ 0.02	77.45 $\pm$ 0.24	89.77 $\pm$ 0.15	88.23 $\pm$ 0.31	57.38 $\pm$ 0.10
FedMix	77.94 $\pm$ 0.26	83.55 $\pm$ 0.31	83.12 $\pm$ 0.29	60.33 $\pm$ 0.24	76.90 $\pm$ 0.33	81.96 $\pm$ 0.27	82.21 $\pm$ 0.34	53.55 $\pm$ 0.30
FedBR	81.62 $\pm$ 0.28	85.32 $\pm$ 0.23	84.05 $\pm$ 0.30	61.61 $\pm$ 0.37	81.48 $\pm$ 0.37	84.12 $\pm$ 0.25	84.78 $\pm$ 0.38	56.32 $\pm$ 0.33

Table 4: Exp 1: Performance comparison for Data Distribution I with a high degree of quantity shift (Dirichlet

\alpha^{\prime}

= 0.25)

	$\alpha^{\prime}=0.25$			$\alpha^{\prime}=1.0$
Algorithm	CIFAR-10	FMNIST	SVHN	CIFAR-10	FMNIST	SVHN
FedAvg	66.48 $\pm$ 0.21	47.26 $\pm$ 0.28	46.13 $\pm$ 0.48	41.78 $\pm$ 0.73	85.48 $\pm$ 0.36	81.89 $\pm$ 0.31
FedSoft	71.08 $\pm$ 0.26	83.75 $\pm$ 0.26	85.67 $\pm$ 0.19	73.83 $\pm$ 0.42	87.85 $\pm$ 0.31	85.92 $\pm$ 0.13
PACFL	73.91 $\pm$ 0.43	85.93 $\pm$ 0.12	87.23 $\pm$ 0.20	80.52 $\pm$ 0.15	93.31 $\pm$ 0.28	92.17 $\pm$ 0.23
CFL	67.46 $\pm$ 0.12	85.18 $\pm$ 0.17	85.19 $\pm$ 0.25	78.94 $\pm$ 0.18	83.16 $\pm$ 0.26	82.75 $\pm$ 0.28
CFL-GP	73.84 $\pm$ 0.28	86.43 $\pm$ 0.14	88.04 $\pm$ 0.19	83.57 $\pm$ 0.15	92.21 $\pm$ 0.23	91.67 $\pm$ 0.19

Table 5: Exp 5: Performance comparison under LDA skew (

\alpha^{\prime}=0.25

and

\alpha^{\prime}=1.0

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

$\bm{t_{g}}$	CIFAR-10	SVHN	FMNIST
1	80.81 $\pm$ 0.59	84.82 $\pm$ 0.24	93.18 $\pm$ 0.11
2	83.34 $\pm$ 0.52	90.05 $\pm$ 0.16	93.18 $\pm$ 0.11
3	83.34 $\pm$ 0.52	90.05 $\pm$ 0.16	93.18 $\pm$ 0.11

Appendix

Table 6: Test accuracy for different values of local training rounds

t_{g}

(with 10 local steps per round) across datasets, evaluated under Data Distribution I with 30% class skew and Dirichlet

\alpha^{\prime}=1

$\bm{\delta}$	CIFAR-10	SVHN	FMNIST
0.2	87.51 $\pm$ 0.18	91.74 $\pm$ 0.08	91.79 $\pm$ 0.08
0.4	87.51 $\pm$ 0.18	91.74 $\pm$ 0.08	92.21 $\pm$ 0.09
0.6	87.95 $\pm$ 0.13	91.91 $\pm$ 0.13	92.21 $\pm$ 0.09
0.8	87.95 $\pm$ 0.13	91.91 $\pm$ 0.13	92.21 $\pm$ 0.09
1.0	87.95 $\pm$ 0.13	91.91 $\pm$ 0.13	92.21 $\pm$ 0.09

Appendix

Table 7: Accuracy metrics for various values of weight range

\delta

under Data Distribution I (Dirichlet

\alpha^{\prime}=0.25

, 20% class skew).

$\bm{k}$	CIFAR-10	SVHN	FMNIST
1	90.85 $\pm$ 0.13	97.09 $\pm$ 0.08	97.71 $\pm$ 0.05
2	91.02 $\pm$ 0.12	97.19 $\pm$ 0.04	98.36 $\pm$ 0.10
3	90.95 $\pm$ 0.16	97.01 $\pm$ 0.07	98.57 $\pm$ 0.08
4	90.96 $\pm$ 0.21	96.78 $\pm$ 0.12	98.41 $\pm$ 0.11

Appendix

Table 8: Accuracy for different values of top-

k

retained in the complementarity graph

H

under Data Distribution I (Dirichlet

\alpha^{\prime}=1

, 30% class skew).

Sparsity (entries kept)	CIFAR-10	SVHN	FMNIST
0.1%	$82.52\pm 0.60$	$89.23\pm 0.25$	$92.81\pm 0.14$
0.5%	$83.34\pm 0.52$	$90.05\pm 0.16$	$93.18\pm 0.11$
1%	$83.34\pm 0.52$	$90.05\pm 0.16$	$93.18\pm 0.11$
2%	$83.34\pm 0.52$	$90.05\pm 0.16$	$93.18\pm 0.11$
20%	$83.34\pm 0.52$	$90.05\pm 0.16$	$93.18\pm 0.11$

Appendix

Table 9: Test accuracy (%) as a function of the gradient sparsification ratio with only gradient similarity and singel encoder (fraction of coordinates retained in

\tilde{\Delta}^{i}

) under Data Distribution I (

30\%

label skew,

\alpha^{\prime}=1

t_{g}=2

	20% Label Skew				30% Label Skew
Algorithm	CIFAR-10	FMNIST	SVHN	CIFAR-100	CIFAR-10	FMNIST	SVHN	CIFAR-100
FedAvg	46.20 $\pm$ 0.97	57.12 $\pm$ 0.30	74.61 $\pm$ 0.36	51.34 $\pm$ 0.78	57.48 $\pm$ 0.17	77.17 $\pm$ 0.24	68.34 $\pm$ 0.45	53.13 $\pm$ 1.46
FedProx	46.77 $\pm$ 0.14	56.81 $\pm$ 0.16	77.23 $\pm$ 0.45	53.38 $\pm$ 0.86	57.80 $\pm$ 0.23	73.87 $\pm$ 0.25	69.65 $\pm$ 0.19	53.97 $\pm$ 0.85
PerFedAvg	84.68 $\pm$ 0.19	91.18 $\pm$ 0.21	92.34 $\pm$ 0.13	69.43 $\pm$ 0.22	82.83 $\pm$ 0.14	94.74 $\pm$ 0.17	91.48 $\pm$ 0.29	60.70 $\pm$ 0.30
FedSoft	77.42 $\pm$ 0.21	87.64 $\pm$ 0.35	90.48 $\pm$ 0.24	65.98 $\pm$ 0.37	76.94 $\pm$ 0.38	89.56 $\pm$ 0.37	84.86 $\pm$ 0.45	56.61 $\pm$ 0.31
PACFL	90.45 $\pm$ 0.30	94.41 $\pm$ 0.31	94.96 $\pm$ 0.12	70.35 $\pm$ 0.36	87.01 $\pm$ 0.38	97.28 $\pm$ 0.24	94.36 $\pm$ 0.19	63.91 $\pm$ 0.76

Appendix

Table 10: Exp 1: Performance comparison for Data Distribution I with 20% and 30% non-IID label skew under low quantity shift (Dirichlet

\alpha^{\prime}=1

Panel of Figure 3 — Figure 3: Exp 6: Accuracy vs. number of comm. rounds, Data Distribution I, non-IID (30%), $\alpha^{\prime}$ =1.

Algorithm	CIFAR-10-C	Tiny ImageNet -C	PACS	Office-Caltech-10
FedAvg	30.73 $\pm$ 0.36	18.43 $\pm$ 0.43	38.47 $\pm$ 0.28	46.12 $\pm$ 0.31
PerFedAvg	60.39 $\pm$ 0.13	25.54 $\pm$ 0.31	70.82 $\pm$ 0.22	67.34 $\pm$ 0.23
FBLG	60.25 $\pm$ 0.30	29.91 $\pm$ 0.23	72.38 $\pm$ 0.37	68.44 $\pm$ 0.14
FedBR	61.09 $\pm$ 0.32	30.83 $\pm$ 0.25	73.83 $\pm$ 0.32	71.37 $\pm$ 0.18
FedMix	60.11 $\pm$ 0.14	30.71 $\pm$ 0.24	73.25 $\pm$ 0.41	70.56 $\pm$ 0.23
PACFL	63.62 $\pm$ 0.22	33.53 $\pm$ 0.38	76.63 $\pm$ 0.19	73.31 $\pm$ 0.14

Appendix

Table 11: Exp 8: Performance comparison of various SOTA algorithms and FedDAG under combined feature skew and label skew (Data Distribution IV). Feature skew arises from corruption types on CIFAR-10-C and Tiny ImageNet -C, and from domains on PACS and Office-Caltech-10.

Dataset	FedAvg	PACFL	CFL	FedGWC	IFCA	FedDAG
Google Landmarks	$36.53\pm 0.24$	$54.74\pm 0.21$	$45.29\pm 0.28$	$51.51\pm 0.31$	$51.97\pm 0.16$	$\mathbf{58.23\pm 0.15}$

Appendix

Table 12: Performance on the large-scale real-world Google Landmarks dataset.

Algorithm	CIFAR-10	FMNIST	SVHN
FedAvg	55.38 $\pm$ 0.15	74.93 $\pm$ 0.22	66.86 $\pm$ 0.28
PerFedAvg	80.92 $\pm$ 0.10	92.62 $\pm$ 0.17	90.00 $\pm$ 0.15
FedSoft	74.98 $\pm$ 0.27	87.45 $\pm$ 0.22	83.59 $\pm$ 0.12
PACFL	85.33 $\pm$ 0.15	95.17 $\pm$ 0.24	92.76 $\pm$ 0.08
CFL	69.97 $\pm$ 0.11	83.64 $\pm$ 0.13	78.94 $\pm$ 0.17
FedGWC	84.30 $\pm$ 0.13	94.53 $\pm$ 0.10	91.56 $\pm$ 0.06

Appendix

Table 13: Test accuracy of newcomer clients, Data Distribution I with 30% label skew and

\alpha^{\prime}=1

Method	CIFAR-10	FMNIST	SVHN
FedDAG ∗ -Grad (gradient-only)	$83.79\pm 0.50$	$91.26\pm 0.38$	$89.05\pm 0.19$
FedDAG ∗ -Data (data-only)	$85.03\pm 0.33$	$91.82\pm 0.25$	$89.52\pm 0.21$
FedDAG ∗ -Data+Grad (combined)	$\mathbf{86.95\pm 0.21}$	$\mathbf{92.18\pm 0.15}$	$\mathbf{90.97\pm 0.13}$

Appendix

Table 14: Ablation on combining data and gradient under Data Distribution I (label skew 30%, Dirichlet

\alpha^{\prime}=0.25

Method	CIFAR-10	FMNIST	SVHN
FedDAG ∗ -Grad (gradient-only)	$64.52\pm 0.45$	$83.88\pm 0.32$	$81.17\pm 0.21$
FedDAG ∗ -Data (data-only)	$67.10\pm 0.23$	$85.74\pm 0.28$	$83.15\pm 0.14$
FedDAG ∗ -Data+Grad (combined)	$\mathbf{67.79\pm 0.27}$	$\mathbf{86.03\pm 0.21}$	$\mathbf{83.73\pm 0.19}$

Appendix

Table 15: Ablation on similarity design under Data Distribution II (concept shift).

Algorithm	CIFAR-10	FMNIST	SVHN
FedDAG ∗ (single encoder)	67.79 $\pm$ 0.27	86.03 $\pm$ 0.21	83.73 $\pm$ 0.19
FedDAG † (dual encoder, no sharing)	67.51 $\pm$ 0.22	85.91 $\pm$ 0.15	83.65 $\pm$ 0.24
FedDAG (dual encoder + sharing)	69.13 $\pm$ 0.23	88.79 $\pm$ 0.19	85.06 $\pm$ 0.26

Appendix

Table 16: Ablation of cross-cluster representation sharing under concept shift (Data Distribution II in § 5 ), comparing the single-encoder baseline ( FedDAG ∗ ), a dual-encoder variant without cross-cluster sharing ( FedDAG † ), and full FedDAG .

Panel of Figure 4 — Figure 4: Effect of the federated-aware clustering loss on the adaptive clustering mechanism under Data Distribution I ( $30\%$ label skew, Dirichlet $\alpha^{\prime}=0.25$ ) for CIFAR-10 and SVHN.

Encoder Initialization Method	CIFAR-10	FMNIST	SVHN
Main (warm-start) (see § 4 )	$90.76\pm 0.12$	$93.82\pm 0.20$	$93.91\pm 0.23$
Random init+regularizer (§ A.11 )	$90.93\pm 0.16$	$93.76\pm 0.17$	$93.86\pm 0.10$

Appendix

Table 17: Comparison of dual-encoder initialization strategies in FedDAG (Data Distribution I).

Schedule (Primary : Secondary)	CIFAR-10	SVHN
Full FedDAG ( both )	89.87 $\pm$ 0.19	92.65 $\pm$ 0.11
1 : 1	80.34 $\pm$ 0.44	86.26 $\pm$ 0.32
5 : 1	87.65 $\pm$ 0.28	91.31 $\pm$ 0.22
10 : 1	87.25 $\pm$ 0.31	91.46 $\pm$ 0.20
15 : 1	86.54 $\pm$ 0.35	90.87 $\pm$ 0.27

Appendix

Table 18: Accuracy of FedDAG under different primary:secondary training scheduling patterns (Data Distribution I, 80 comm. rounds). “Full FedDAG ” updates both encoders every round.

$\alpha$	#clusters	$\mathcal{L}_{\mathbb{C}}$	#clusters	$\mathcal{L}_{\mathbb{C}}$
	CIFAR-10		SVHN
1.000	1	0.770	1	0.675
0.950	1	0.770	1	0.675
0.900	2	1.000	1	0.675
0.850	3	1.000	1	0.675
0.800	3	1.000	2	0.866

Appendix

Table 19: Number of clusters and clustering loss

\mathcal{L}_{\mathbb{C}}

as a function of the distance threshold

\alpha

for CIFAR-10 and SVHN in the inherent high-cluster (i.e., 11) distribution setting.

Panel of Figure 5 — Figure 5: Behavior of FedDAG ’s adaptive clustering mechanism in a setting with inherent high number of clusters (e.g,. $>10$ ) ground-truth distributions.

Clustered Federated Learning

Clustered federated learning groups clients with similar data or tasks to improve collaboration and personalization without sharing raw data.

See all 43 papers

FMCL: Class-Aware Client Clustering with Foundation Model Representations for Heterogeneous Federated Learning
Apr 30, 2026Mahad Ali, Laura J. BrattainNon-IID Federated Learning Clustered Federated Learning
University of Central Florida, Orlando FL 32816, USA
FedVG: Gradient-Guided Aggregation for Enhanced Federated Learning
Feb 24, 2026Alina Devkota, Jacob Thrasher, Donald Adjeroh +2Federated Learning Federated Learning Aggregation
West Virginia University · University of Aberdeen · Fogsphere (Redev.AI)
FedMTFI: Feature Importance Based Optimized Multi Teacher Knowledge Distillation in Heterogeneous Federated Learning Environment
Jun 1, 2026Nazmus Shakib Shadin, Aaron Cummings, Xinyue Zhang +1Non-IID Federated Learning Multi-Teacher Knowledge Distillation
Department of Computer Science, Kennesaw State University, Marietta, GA, 30060 USA

Apr 30, 2026J/K move · Enter open · S save

FMCL: Class-Aware Client Clustering with Foundation Model Representations for Heterogeneous Federated Learning

Mahad Ali, Laura J. Brattain

University of Central Florida, Orlando FL 32816, USA

Non-IID Federated Learning Clustered Federated Learning Federated Learning Heterogeneous Federated Learning

Open paper AskPDF arXiv

Federated Learning (FL) enables collaborative model training across distributed clients without sharing raw data, yet its performance deteriorates under statistical heterogeneity. Clustered Federated Learning addresses this challenge by grouping similar clients and training separate models per cluster. However, existing clustering strategies often rely on raw data statistics, model parameters, or heuristic similarity measures that fail to capture class-level semantic structure across heterogeneous domains and frequently require iterative coordination. We propose FMCL, a one-shot, class-aware client clustering framework that leverages foundation model representations to construct semantic client signatures. Using a frozen foundation model, FMCL computes class-level embedding prototypes for each client and measures similarity via cosine distance between their class-aware representations. Clustering is performed once prior to training, introducing no additional communication during federated optimization and remaining agnostic to the downstream model architecture. Extensive experiments across heterogeneous benchmarks demonstrate that FMCL improves federated performance and yields more stable clustering behavior compared to existing clustering-based methods under non-identically distributed data partitioning.

Abstract

Clustered Federated Learning

Explore similar work

FMCL: Class-Aware Client Clustering with Foundation Model Representations for Heterogeneous Federated Learning

FedVG: Gradient-Guided Aggregation for Enhanced Federated Learning

FedMTFI: Feature Importance Based Optimized Multi Teacher Knowledge Distillation in Heterogeneous Federated Learning Environment