Partial Model Training

Partial model training addresses the challenges of training large models on resource-constrained devices, particularly within federated learning frameworks. Current research focuses on adapting model architectures (like CNNs, LSTMs, and Transformers) and algorithms (including federated averaging and asynchronous methods) to allow clients to train only portions of a model, based on their capabilities. This approach improves efficiency and participation rates in federated learning, enabling the training of larger and more complex models on diverse, heterogeneous networks of devices, with applications ranging from speech recognition to graph neural networks.

Papers

June 21, 2024

Embracing Federated Learning: Enabling Weak Client Participation via Partial Model Training
Sunwoo Lee, Tuo Zhang, Saurav Prakash, Yue Niu, Salman Avestimehr
Federated Learning Embracing CompAct Client Participation Partial Model Training

May 27, 2024

Federating Dynamic Models using Early-Exit Architectures for Automatic Speech Recognition on Heterogeneous Clients
Mohamed Nabih Ali, Alessio Brutti, Daniele Falavigna
Automatic Speech Recognition Dynamic Model Multiple Model Speech Recognition Model Speech Recording Early Exit Heterogeneous Client Partial Model Training

March 12, 2024

Graph Unlearning with Efficient Partial Retraining
Jiahao Zhang, Lin Wang, Shijie Wang, Wenqi Fan
Graph Neural Network Multi Layer Neural Network Graph Unlearning Partial Model Training

November 16, 2023

Straggler-resilient Federated Learning: Tackling Computation Heterogeneity with Layer-wise Partial Model Training in Mobile Edge Network
Hongda Wu, Ping Wang, C V Aswartha Narayana
Heterogeneous Computing Mobile Edge Federated Training Weak Model Partial Model Training

April 14, 2023

TimelyFL: Heterogeneity-aware Asynchronous Federated Learning with Adaptive Partial Training
Tuo Zhang, Lei Gao, Sunwoo Lee, Mi Zhang, Salman Avestimehr
Asynchronous Federated Learning Cross Device Federated Learning Slow Convergence Stale Gradient Partial Model Training

March 31, 2022

Scaling Language Model Size in Cross-Device Federated Learning
Jae Hun Ro, Theresa Breiner, Lara McConnaughey, Mingqing Chen, Ananda Theertha Suresh, Shankar Kumar, Rajiv Mathews
Language Model Neural Network Larger Language Model Efficient Transfer Learning Cross Device Federated Learning Partial Model Training

Partial Model Training

Papers

Embracing Federated Learning: Enabling Weak Client Participation via Partial Model Training

Federating Dynamic Models using Early-Exit Architectures for Automatic Speech Recognition on Heterogeneous Clients

Graph Unlearning with Efficient Partial Retraining

Straggler-resilient Federated Learning: Tackling Computation Heterogeneity with Layer-wise Partial Model Training in Mobile Edge Network

TimelyFL: Heterogeneity-aware Asynchronous Federated Learning with Adaptive Partial Training

Scaling Language Model Size in Cross-Device Federated Learning