cs.LGNov 6, 2026

FedLoVA: Value-Only Aggregation for Federated LoRA Fine-Tuning of Large Language Models

Authors: Ensieh Khazaei, Baturalp Buyukates, Dimitrios Hatzinakos

Organizations: University of Toronto · University of Birmingham

Abstract

Low-Rank Adaptation (LoRA) is a popular parameter-efficient fine-tuning (PEFT) method for fine-tuning of large language models (LLMs), especially in federated learning, due to its strong performance and communication efficiency. In practice, LoRA is applied to the query and value projections of transformer layers without considering the distinct roles of these components. In this work, we analyze the impact of these projections in federated fine-tuning of LLMs using LoRA and observe that the value projection converges significantly faster than the query projection. Motivated by this finding, we propose FedLoVA, a federated fine-tuning framework that trains LoRA adapters on both query and value projections locally while sharing only the value projection with the server. This approach reduces communication overhead while improving convergence efficiency. Experiments on natural language understanding, multilingual sentiment analysis, and generation tasks show that FedLoVA reduces the communication cost by up to 7.8×7.8\times compared to existing federated LLM fine-tuning baselines while achieving comparable or better performance. The code is available at: https://github.com/EnsiehKhazaei/FedLoVA.

Figures & tables

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. FedSmoothLoRA: Toward Smoother and Faster Convergence in Federated Low-Rank Adaptation

    May 28, 2026Zehao Wang, Guanglei Yang, Yihan Zeng +4Low-Rank AdaptationFederated Fine-Tuning

  2. CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregation

    Sep 27, 2026Yanan Ma, Qiyuan Chen, Zihan Fang +2Federated Fine-TuningFederated Learning