cs.LGNov 9, 2026

Stochastic Projections and Rank-Aware Learning Rates for Federated LoRA

Authors: Saber Malekmohammadi, Kevin Kuo, Virginia Smith, Golnoosh Farnadi

Organizations: Stanford University · Carnegie Mellon University · McGill University, MILA

Abstract

Fine-tuning large-scale foundation models across resource-constrained, distributed clients presents significant computational and communication challenges. While merging Low-Rank Adaptation (LoRA) with Federated Learning (FL) enables rank-based flexibility, the interdependence of the adaptation matrices (A\mathbf{A} and B\mathbf{B}) often complicates optimization. In this work, we theoretically demonstrate that LoRA fine-tuning with Gaussian initialization and a frozen A\mathbf{A} matrix acts as a stochastic approximation of full fine-tuning, where batch gradients are subject to a random, rank-dependent perturbation. Motivated by this insight, we propose freezing and periodically resampling A\mathbf{A} matrices at the client level. This strategy decouples the A\mathbf{A} and B\mathbf{B} gradients, diminishes LoRA low-rank constraints and reduces both computational and uplink communication overheads by 50%, allowing for the reallocation of resources toward higher adaptation ranks. Furthermore, we show that in this frozen-A\mathbf{A} regime, the adaptation rank rr directly governs the signal-to-noise ratio (SNR) of the perturbed gradients. Specifically, we find that the SNR increases linearly with rr, implying that clients' learning rates must be dynamically scaled with their adaptation ranks for higher utility. Building on these foundations, we introduce SPRA-LoRA (Stochastic Projection Rank-Aware LoRA), a federated rank-aware fine-tuning strategy that utilizes frozen, resampled A\mathbf{A} matrices and rank-dependent learning rates for clients. We also prove its convergence with heterogeneous client resources. Our experiments across benchmark datasets demonstrate that SPRA-LoRA consistently outperforms state-of-the-art baselines with half the overhead on clients.

Figures & tables

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Stabilized Fine-Tuning with LoRA in Federated Learning: Mitigating the Side Effect of Client Size and Rank via the Scaling Factor

    Mar 9, 2026Jiayu Huang, Xiaohu Wu, Tiantian He +1Federated Fine-TuningLow-Rank Adaptation

  2. FedSmoothLoRA: Toward Smoother and Faster Convergence in Federated Low-Rank Adaptation

    May 28, 2026Zehao Wang, Guanglei Yang, Yihan Zeng +4Low-Rank AdaptationFederated Fine-Tuning