cs.AINov 5, 2026

Align Before Aggregation: Basis-Consistent Federated LoRA under Heterogeneous Ranks

Authors: Pengpeng Qiao, Yang Cao, Lingling Zhang, Guo Cheng, Junwei Chen, Manjiang Yu, Wei Yang Bryan Lim, Masatoshi Yoshikawa

Organizations: Institute of Science Tokyo · Capital Normal University · Beijing Institute of Technology · The University of Queensland · 

Abstract

Federated fine-tuning of large language models (LLMs) under heterogeneous client capacities is increasingly important, and low-rank adaptation (LoRA) with heterogeneous ranks makes it practical. Existing server-side aggregators compose local matrix updates ΔWiΔW_i from client-specific LoRA factors (Ai,Bi)(A_i,B_i) before aggregating them. Although these matrix updates lie in the same weight space Rm×n\mathbb{R}^{m\times n} and can be averaged directly, the LoRA factors that generate them may expressed client-specific local basis. In the typical compose--aggregate--factorize--dispatch loop, this basis inconsistency can make the rank-rir_i reference prefix dispatched to lower-rank clients unstable after server-side refactorization. In this work, we propose FedAbA, a basis-consistent federated LoRA aggregation framework for heterogeneous ranks. FedAbA first extracts each client's local basis, aligns it to the corresponding prefix of the global reference basis, and constructs a basis-consistent reconstruction of each local matrix update before aggregation. It then aggregates these reconstructions with weights that combine client data size and consistency score, followed by exact coefficient-space SVD refactorization and gauge fixing before rank-prefix dispatch. Our theoretical analysis clarifies the role of basis consistency and supports the proposed consistency-aware aggregation mechanism. Extensive experiments show that FedAbA outperforms representative federated LoRA baselines, with the largest gains under the strongest rank heterogeneity.

Figures & tables

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. FedPA-LoRA: Product-Aligned Framework for Mitigating Aggregation and Initialization Errors in Heterogeneous Federated LoRA

    Aug 15, 2026Juseok Jeon, Ramy E. Ali, Doyun Kwon +3Federated Fine-TuningFederated Learning

  2. CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregation

    Sep 27, 2026Yanan Ma, Qiyuan Chen, Zihan Fang +2Federated Fine-TuningFederated Learning

  3. Beyond Factor Aggregation: Gauge-Aware Low-Rank Server Representations for Federated LoRA

    May 7, 2026Jinqian Chen, Chang Liu, Jihua ZhuFederated Learning AggregationLow-Rank Adaptation