Data Heterogeneity

Data heterogeneity, the variability in data distributions across different sources in federated learning, significantly hinders model accuracy and convergence. Current research focuses on mitigating this challenge through various techniques, including personalized model architectures (e.g., using adaptors or subnetworks), robust aggregation methods (e.g., weighted averaging based on client performance or data characteristics), and innovative training strategies (e.g., warmup phases, loss decomposition, and adversarial training). Addressing data heterogeneity is crucial for advancing federated learning's applicability in diverse real-world scenarios, particularly in healthcare, industrial IoT, and other domains with decentralized, privacy-sensitive data.

Papers

January 31, 2023

Distributed sequential federated learning
Z. F. Wang, X. Y. Zhang, Y-c I. Chang
LeArning Abstract Data Driven Data Heterogeneity Averaging Algorithm COVID 19 Data

December 24, 2022

When Do Curricula Work in Federated Learning?
Saeed Vahidian, Sreevatsank Kadaveru, Woonjoon Baek, Weijia Wang, Vyacheslav Kungurtsev, Chen Chen, Mubarak Shah, Bill Lin
Data Heterogeneity Sequential Learning Client Selection Local SGD Biased Gradient

December 21, 2022

Personalized Decentralized Multi-Task Learning Over Dynamic Communication Graphs
Matin Mortaheb, Sennur Ulukus
Data Heterogeneity Decentralized Learning Faster Convergence Decentralized Algorithm Fully Connected Communication Graph

December 17, 2022

Toward Data Heterogeneity of Federated Learning
Yuchuan Huang, Chen Hu
Machine Learning Data Heterogeneity Heterogeneous Data

December 16, 2022

Addressing Data Heterogeneity in Decentralized Learning via Topological Pre-processing
Waqwoya Abebe, Ali Jannesari
Heterogeneous Graph Data Heterogeneity Decentralized Learning Decentralized Federated Learning Peer to Peer

December 6, 2022

Tackling Data Heterogeneity in Federated Learning with Class Prototypes
Yutong Dai, Zeyuan Chen, Junnan Li, Shelby Heinecke, Lichao Sun, Ran Xu
Strong Generalization Class Imbalance Data Heterogeneity Global Model Local Model Class Prototype

December 5, 2022

On the effectiveness of partial variance reduction in federated learning with heterogeneous data
Bo Li, Mikkel N. Schmidt, Tommy S. Alstrøm, Sebastian U. Stich
LeArning Abstract Data Heterogeneity Heterogeneous Data Variance Reduction FedAvg Algorithm Gradient Update Client Drift

November 17, 2022

November 10, 2022

Robust Federated Learning against both Data Heterogeneity and Poisoning Attack via Aggregation Optimization
Yueqi Xie, Weizhong Zhang, Renjie Pi, Fangzhao Wu, Qifeng Chen, Xing Xie, Sunghun Kim
Poisoning Attack Data Heterogeneity Generalization Capacity Optimal Aggregation Server Side Aggregation Non IID Subspace Training

October 26, 2022

Addressing Heterogeneity in Federated Learning via Distributional Transformation
Haolin Yuan, Bo Hui, Yuchen Yang, Philippe Burlina, Neil Zhenqiang Gong, Yinzhi Cao
Deep Learning Model Data Heterogeneity Benchmark Datasets Heterogeneity Challenge Distribution Change

October 14, 2022

FedFM: Anchor-based Feature Matching for Data Heterogeneity in Federated Learning
Rui Ye, Zhenyang Ni, Chenxin Xu, Jianyu Wang, Siheng Chen, Yonina C. Eldar
Data Heterogeneity Feature Space

October 12, 2022

Find Your Friends: Personalized Federated Learning with the Right Collaborators
Yi Sui, Junfeng Wen, Yenson Lau, Brendan Leigh Ross, Jesse C. Cresswell
Data Heterogeneity Personalized Federated Learning Expectation Maximization Global Model First Benchmark Dataset

October 1, 2022

Towards Understanding and Mitigating Dimensional Collapse in Heterogeneous Federated Learning
Yujun Shi, Jian Liang, Wenqing Zhang, Vincent Y. F. Tan, Song Bai
Human Understanding Data Heterogeneity Heterogeneous Federated Learning Dimensional Collapse

September 30, 2022

September 21, 2022

FedFOR: Stateless Heterogeneous Federated Learning with First-Order Regularization
Junjiao Tian, James Seale Smith, Zsolt Kira
Data Heterogeneity Heterogeneous Federated Learning Faster Convergence Heterogeneous Setting

September 7, 2022

Modular Federated Learning
Kuo-Yun Liang, Abhishek Srinivasan, Juan Carlos Andresen
Data Heterogeneity Learning Framework Modular Network Personalized Federated

July 23, 2022

Handling Data Heterogeneity in Federated Learning via Knowledge Distillation and Fusion
Xu Zhou, Xinyu Lei, Cong Yang, Yichun Shi, Xiao Zhang, Jingwen Shi
Knowledge Distillation Hybrid Fusion Data Heterogeneity Data Free Knowledge Distillation Global Semantic

July 8, 2022

Tackling Data Heterogeneity: A New Unified Framework for Decentralized SGD with Sample-induced Topology
Yan Huang, Ying Sun, Zehan Zhu, Changzhi Yan, Jinming Xu
Unified Framework Empirical Risk Minimization Data Heterogeneity Convergence Analysis Variance Reduction Topological Metric Decentralized SGD Stochastic Gradient Based

Data Heterogeneity

Papers

Distributed sequential federated learning

When Do Curricula Work in Federated Learning?

Personalized Decentralized Multi-Task Learning Over Dynamic Communication Graphs

Toward Data Heterogeneity of Federated Learning

Addressing Data Heterogeneity in Decentralized Learning via Topological Pre-processing

Tackling Data Heterogeneity in Federated Learning with Class Prototypes

On the effectiveness of partial variance reduction in federated learning with heterogeneous data

FedSiam-DA: Dual-aggregated Federated Learning via Siamese Network under Non-IID Data

Personalized Federated Learning for Multi-task Fault Diagnosis of Rotating Machinery

Robust Federated Learning against both Data Heterogeneity and Poisoning Attack via Aggregation Optimization

Addressing Heterogeneity in Federated Learning via Distributional Transformation

FedFM: Anchor-based Feature Matching for Data Heterogeneity in Federated Learning

Find Your Friends: Personalized Federated Learning with the Right Collaborators

Towards Understanding and Mitigating Dimensional Collapse in Heterogeneous Federated Learning

Rethinking Data Heterogeneity in Federated Learning: Introducing a New Notion and Standard Benchmarks

Momentum Tracking: Momentum Acceleration for Decentralized Deep Learning on Heterogeneous Data

FedFOR: Stateless Heterogeneous Federated Learning with First-Order Regularization

Modular Federated Learning

Handling Data Heterogeneity in Federated Learning via Knowledge Distillation and Fusion

Tackling Data Heterogeneity: A New Unified Framework for Decentralized SGD with Sample-induced Topology