Federated Training

Federated training is a machine learning approach enabling collaborative model training across decentralized datasets without direct data sharing, prioritizing data privacy. Current research emphasizes addressing challenges like data heterogeneity (variations in data quality, format, and distribution across participating devices), computational heterogeneity (differences in device capabilities), and communication efficiency. This approach is significant for enabling large-scale model training on sensitive data in various domains, including healthcare, natural language processing, and mobile sensing, while mitigating privacy risks. Active research focuses on improving model architectures (e.g., graph neural networks, transformer models) and algorithms (e.g., federated averaging, personalized federated learning) to enhance efficiency and accuracy in these diverse settings.

Papers

November 5, 2024

Photon: Federated LLM Pre-Training
Lorenzo Sani, Alex Iacob, Zeyu Cao, Royson Lee, Bill Marino, Yan Gao, Dongqi Cai, Zexi Li, Wanru Zhao, Xinchi Qiu, Nicholas D. Lane
Large Language Model Decoder Only Large Language Model Single Photon Large Scale Training Federated Training Particle Photon

July 12, 2024

Multi-Modal Dataset Creation for Federated Learning with DICOM Structured Reports
Malte Tölle, Lukas Burger, Halvar Kelm, Florian André, Peter Bannas, Gerhard Diller, Norbert Frey, Philipp Garthe, Stefan Groß, Anja Hennemuth, Lars Kaderali, Nina Krüger, Andreas Leha, Simon Martin, Alexander Meyer, Eike Nagel, Stefan Orwat, Clemens Scherer, Moritz Seiffert, Jan Moritz Seliger, Stefan Simm, Tim Friede, Tim Seidler, Sandy Engelhardt
Multi Modal Inter Vendor Harmonization Federated Training Imaging and Communication in Medicine

May 23, 2024

Worldwide Federated Training of Language Models
Alex Iacob, Lorenzo Sani, Bill Marino, Preslav Aleksandrov, William F. Shen, Nicholas Donald Lane
Language Model Language Data Federated Training Federated Large Language Model Weak Model

November 16, 2023

Straggler-resilient Federated Learning: Tackling Computation Heterogeneity with Layer-wise Partial Model Training in Mobile Edge Network
Hongda Wu, Ping Wang, C V Aswartha Narayana
Heterogeneous Computing Mobile Edge Federated Training Weak Model Partial Model Training

September 18, 2023

FedGKD: Unleashing the Power of Collaboration in Federated Graph Neural Networks
Qiying Pan, Ruofan Wu, Tengfei Liu, Tianyi Zhang, Yifei Zhu, Weiqiang Wang
Graph Neural Network Real Power Graph Level Task Federated Training Federated Graph Neural Network

September 3, 2023

Federated Orthogonal Training: Mitigating Global Catastrophic Forgetting in Continual Federated Learning
Yavuz Faruk Bakman, Duygu Nur Yaldiz, Yahya H. Ezzeldin, Salman Avestimehr
Catastrophic Forgetting Federated Continual Learning State of the Art Continual Federated Training

August 25, 2023

REFT: Resource-Efficient Federated Training Framework for Heterogeneous and Resource-Constrained Environments
Humaid Ahmed Desai, Amr Hilal, Hoda Eldardiry
Edge Computing Resource Constrained Resource Constrained Device Federated Training

August 12, 2023

SLoRA: Federated Parameter Efficient Fine-Tuning of Language Models
Sara Babakniya, Ahmed Roushdy Elkordy, Yahya H. Ezzeldin, Qingfeng Liu, Kee-Bong Song, Mostafa El-Khamy, Salman Avestimehr
Language Model Fine Tuning Pre Trained Transformer Efficient Fine Tuning Federated Training

August 3, 2023

Federated Representation Learning for Automatic Speech Recognition
Guruprasad V Ramesh, Gopinath Chennupati, Milind Rao, Anit Kumar Sahu, Ariya Rastrow, Jasha Droppo
Automatic Speech Recognition Speech Data Federated Training Federated Representation Learning

July 18, 2023

Towards Federated Foundation Models: Scalable Dataset Pipelines for Group-Structured Learning
Zachary Charles, Nicole Mitchell, Krishna Pillutla, Michael Reneer, Zachary Garrett
Heterogeneous Datasets Aggregated Data Federated Training Federated Large Language Model Federated Foundation Model Group Learning

April 18, 2023

Federated Alternate Training (FAT): Leveraging Unannotated Data Silos in Federated Segmentation for Medical Imaging
Erum Mushtaq, Yavuz Faruk Bakman, Jie Ding, Salman Avestimehr
Federated Learning Medical Image Medical Imaging Federated Data Federated Training Data Silo Unified Federated Partially Labeled Segmentation

April 15, 2023

SalientGrads: Sparse Models for Communication Efficient and Data Aware Distributed Federated Training
Riyasat Ohib, Bishal Thapaliya, Pratyush Gaggenapalli, Jingyu Liu, Vince Calhoun, Sergey Plis
Sparse Model Communication Efficiency Sparse Training Federated Training Sparse Gradient

March 10, 2023

Zone-based Federated Learning for Mobile Sensing Data
Xiaopeng Jiang, Thinh On, NhatHai Phan, Hessamaldin Mohammadi, Vijaya Datta Mayyuri, An Chen, Ruoming Jin, Cristian Borcea
Sensor Data Federated Training

November 21, 2022

SIFU: Sequential Informed Federated Unlearning for Efficient and Provable Client Unlearning in Federated Optimization
Yann Fraboni, Martin Van Waerebeke, Kevin Scaman, Richard Vidal, Laetitia Kameni, Marco Lorenzi
Machine Unlearning Federated Optimization Federated Unlearning Federated Training

September 30, 2022

Federated Training of Dual Encoding Models on Small Non-IID Client Datasets
Raviteja Vemulapalli, Warren Richard Morningstar, Philip Andrew Mansfield, Hubert Eichner, Karan Singhal, Arash Afkanpour, Bradley Green
Representation Learning Loss Function Centralized Training Small Scale Federated Training Client Data

August 24, 2022

Towards Sparsified Federated Neuroimaging Models via Weight Pruning
Dimitris Stripelis, Umang Gupta, Nikhil Dhinagar, Greg Ver Steeg, Paul Thompson, José Luis Ambite
Neural Network Model Pruning Weight Pruning Federated Training Sparse Federated

May 23, 2022

Orchestra: Unsupervised Federated Learning via Globally Consistent Clustering
Ekdeep Singh Lubana, Chi Ian Tang, Fahim Kawsar, Robert P. Dick, Akhil Mathur
Federated Learning Generalization Capability Co Orchestration Federated Training Unsupervised Federated Learning Discriminative Clustering

April 11, 2022

Production federated keyword spotting via distillation, filtering, and joint federated-centralized training
Andrew Hard, Kurt Partridge, Neng Chen, Sean Augenstein, Aishanee Shah, Hyun Jin Park, Alex Park, Sara Ng, Jessica Nguyen, Ignacio Lopez Moreno, Rajiv Mathews, Françoise Beaufays
Low Temperature Distillation Device Training Online Filtering Production Incident Keyword Enrollment Federated Training Experience Metric

February 18, 2022

PerFED-GAN: Personalized Federated Learning via Generative Adversarial Networks
Xingjian Cao, Gang Sun, Hongfang Yu, Mohsen Guizani
Generative Adversarial Network Personalized Federated Learning C Gan Personalized Model Federated Training

January 28, 2022

FedGCN: Convergence-Communication Tradeoffs in Federated Training of Graph Convolutional Networks
Yuhang Yao, Weizhao Jin, Srivatsan Ravi, Carlee Joe-Wong
Semi Supervised Node Classification Federated Training Convergence Performance Federated Graph Neural Network

Federated Training

Papers

Photon: Federated LLM Pre-Training

Multi-Modal Dataset Creation for Federated Learning with DICOM Structured Reports

Worldwide Federated Training of Language Models

Straggler-resilient Federated Learning: Tackling Computation Heterogeneity with Layer-wise Partial Model Training in Mobile Edge Network

FedGKD: Unleashing the Power of Collaboration in Federated Graph Neural Networks

Federated Orthogonal Training: Mitigating Global Catastrophic Forgetting in Continual Federated Learning

REFT: Resource-Efficient Federated Training Framework for Heterogeneous and Resource-Constrained Environments

SLoRA: Federated Parameter Efficient Fine-Tuning of Language Models

Federated Representation Learning for Automatic Speech Recognition

Towards Federated Foundation Models: Scalable Dataset Pipelines for Group-Structured Learning

Federated Alternate Training (FAT): Leveraging Unannotated Data Silos in Federated Segmentation for Medical Imaging

SalientGrads: Sparse Models for Communication Efficient and Data Aware Distributed Federated Training

Zone-based Federated Learning for Mobile Sensing Data

SIFU: Sequential Informed Federated Unlearning for Efficient and Provable Client Unlearning in Federated Optimization

Federated Training of Dual Encoding Models on Small Non-IID Client Datasets

Towards Sparsified Federated Neuroimaging Models via Weight Pruning

Orchestra: Unsupervised Federated Learning via Globally Consistent Clustering

Production federated keyword spotting via distillation, filtering, and joint federated-centralized training

PerFED-GAN: Personalized Federated Learning via Generative Adversarial Networks

FedGCN: Convergence-Communication Tradeoffs in Federated Training of Graph Convolutional Networks