Federated Tuning

Federated tuning focuses on efficiently fine-tuning large language models (LLMs) across decentralized datasets while preserving data privacy. Current research emphasizes reducing the substantial communication overhead inherent in this process, exploring techniques like gradient compression, low-rank adaptations (LoRA), and zeroth-order optimization methods. These advancements aim to improve the scalability and performance of federated learning for LLMs, enabling broader deployment of powerful language models while respecting user data privacy and mitigating the computational burden on individual devices. The resulting improvements in efficiency and privacy have significant implications for both NLP research and real-world applications.

Papers

November 18, 2024

FedCoLLM: A Parameter-Efficient Federated Co-tuning Framework for Large and Small Language Models
Tao Fan, Yan Kang, Guoqiang Ma, Lixin Fan, Kai Chen, Qiang Yang
Language Model Domain Knowledge Domain Specific Task Federated Tuning

September 10, 2024

Ferret: Federated Full-Parameter Tuning at Scale for Large Language Models
Yao Shu, Wenyang Hu, See-Kiong Ng, Bryan Kian Hsiang Low, Fei Richard Yu
Parameter Efficient Fine Tuning Visual Analogue Scale Global Aggregation Federated Tuning

May 22, 2024

CG-FedLLM: How to Compress Gradients in Federated Fune-tuning for Large Language Models
Huiwen Wu, Xiaohan Li, Deyi Zhang, Xiaogang Xu, Jiafei Wu, Puning Zhao, Zhe Liu
Gradient Compression Centralized Learning Federated Variational Federated Tuning CG FedLLM

April 20, 2024

Personalized Wireless Federated Learning for Large Language Models
Feibo Jiang, Li Dong, Siwei Tu, Yubo Peng, Kezhi Wang, Kun Yang, Cunhua Pan, Dusit Niyato
Federated Instruction Tuning Federated Tuning

February 18, 2024

Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resources
Jiamu Bai, Daoyuan Chen, Bingchen Qian, Liuyi Yao, Yaliang Li
Fine Tuning Federated Learning LLM Fine Tuning Client Collaboration Heterogeneous Task Federated Tuning

February 8, 2024

On the Convergence of Zeroth-Order Federated Tuning for Large Language Models
Zhenqing Ling, Daoyuan Chen, Liuyi Yao, Yaliang Li, Ying Shen
Large Language Model Early Stage Convergence Learning Rate Zeroth Order Optimization Federated Tuning

December 11, 2023

Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes
Zhen Qin, Daoyuan Chen, Bingchen Qian, Bolin Ding, Yaliang Li, Shuiguang Deng
Large Language Model Communication Cost Sub Byte Federated Tuning

December 20, 2022

When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methods
Zhuo Zhang, Yuanhang Yang, Yong Dai, Lizhen Qu, Zenglin Xu
Natural Language Processing Pre Trained Language Model Local Adaptivity Federated Large Language Model Federated Tuning

Federated Tuning

Papers

FedCoLLM: A Parameter-Efficient Federated Co-tuning Framework for Large and Small Language Models

Ferret: Federated Full-Parameter Tuning at Scale for Large Language Models

CG-FedLLM: How to Compress Gradients in Federated Fune-tuning for Large Language Models

Personalized Wireless Federated Learning for Large Language Models

Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resources

On the Convergence of Zeroth-Order Federated Tuning for Large Language Models

Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes

When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methods