Stochastic Projections and Rank-Aware Learning Rates for Federated LoRA
Organizations: Stanford University · Carnegie Mellon University · McGill University, MILA
Abstract
Fine-tuning large-scale foundation models across resource-constrained, distributed clients presents significant computational and communication challenges. While merging Low-Rank Adaptation (LoRA) with Federated Learning (FL) enables rank-based flexibility, the interdependence of the adaptation matrices ( and ) often complicates optimization. In this work, we theoretically demonstrate that LoRA fine-tuning with Gaussian initialization and a frozen matrix acts as a stochastic approximation of full fine-tuning, where batch gradients are subject to a random, rank-dependent perturbation. Motivated by this insight, we propose freezing and periodically resampling matrices at the client level. This strategy decouples the and gradients, diminishes LoRA low-rank constraints and reduces both computational and uplink communication overheads by 50%, allowing for the reallocation of resources toward higher adaptation ranks. Furthermore, we show that in this frozen- regime, the adaptation rank directly governs the signal-to-noise ratio (SNR) of the perturbed gradients. Specifically, we find that the SNR increases linearly with , implying that clients' learning rates must be dynamically scaled with their adaptation ranks for higher utility. Building on these foundations, we introduce SPRA-LoRA (Stochastic Projection Rank-Aware LoRA), a federated rank-aware fine-tuning strategy that utilizes frozen, resampled matrices and rank-dependent learning rates for clients. We also prove its convergence with heterogeneous client resources. Our experiments across benchmark datasets demonstrate that SPRA-LoRA consistently outperforms state-of-the-art baselines with half the overhead on clients.
Figures & tables
| algorithm | aggregation strategy | freezing or | resampling } | non-uniform learning rate |
|---|---|---|---|---|
| SLoRA [ 2023 ] | and separately | ✕ | ✕ | ✕ |
| HetLoRA [ 2024 ] | and separately | ✕ | ✕ | ✕ |
| FeDeRA [ 2024 ] | and separately | ✕ | ✕ | ✕ |
| FedIT [ 2024 ] | and separately | ✕ | ✕ | ✕ |
| FlexLoRA [ 2024 ] | } | ✕ | ✕ | ✕ |
| FLoRA [ 2024 ] | } | ✕ | every round | ✕ |
| Method | Run time | QNLI | MRPC | CoLA | MNLI | RTE | SST-2 | QQP | Avg |
|---|---|---|---|---|---|---|---|---|---|
| Base Pretrained Model | — | 49.46 | 31.62 | 30.87 | 35.45 | 52.71 | 49.08 | 63.18 | 44.62 |
| FFA [ 2024 ] | 0.90h | 83.23 ±0.60 | 80.15 ±1.07 | 77.66 ±0.89 | 73.22 ±0.95 | 66.64 ±1.78 | 91.01 ±0.25 | 79.62 ±0.70 | 78.79 ±0.38 |
| HetLoRA [ 2024 ] | 0.62h | 81.33 ±1.51 | 85.29 ±0.17 | 79.94 ±0.29 | 77.25 ±0.11 | 64.40 ±0.48 | 92.61 ±0.19 | 81.25 ±0.64 | 80.30 ±0.25 |
| FlexLoRA [ 2024 ] | 1.11h | 87.71 ±0.20 | 84.61 ±1.61 | 81.36 ±0.41 | 78.02 ±0.76 | 69.53 ±1.38 | 93.12 ±0.19 | 81.12 ±2.18 | 82.21 ±0.45 |
| FLoRA [ 2024 ] | 0.92h | 84.47 ±1.25 | 85.05 ±0.23 | 79.23 ±1.15 | 76.20 ±0.70 | 67.87 ±0.62 | 92.36 ±0.45 | 81.60 ±0.40 | 80.97 ±0.29 |
| FSLoRA [ 2025 ] | 0.84h | 76.02 ±2.29 | 83.43 ±0.64 | 79.31 ±0.63 | 60.40 ±2.88 | 65.06 ±1.87 | 92.00 ±0.13 | 79.13 ±0.96 | 76.48 ±0.62 |
| Period | BoolQ | WinoGrande | OBQA | ARC-e | ARC-c | SIQA | PIQA | HellaSwag | Avg |
|---|---|---|---|---|---|---|---|---|---|
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
| (clients) | 40 | 60 | 80 | ||
|---|---|---|---|---|---|
| (heterogeneity) | 0.4 | 0.8 | 2.0 | 0.8 | |
| Method | Run time | QNLI | MRPC | CoLA | MNLI | RTE | SST-2 | QQP | Avg |
|---|---|---|---|---|---|---|---|---|---|
| Base Pretrained Model | — | 49.46 | 31.62 | 30.87 | 35.45 | 52.71 | 49.08 | 63.18 | 44.62 |
| FFA [ 2024 ] | 0.82h | 81.28 ±0.70 | 77.80 ±1.70 | 77.66 ±0.33 | 70.19 ±1.15 | 66.14 ±0.82 | 90.48 ±0.19 | 73.14 ±3.35 | 76.67 ±0.58 |
| HetLoRA [ 2024 ] | 0.61h | 83.61 ±0.78 | 81.42 ±1.82 | 78.01 ±0.95 | 65.98 ±1.89 | 53.93 ±2.70 | 91.42 ±0.29 | 77.78 ±2.37 | 76.02 ±0.66 |
| FlexLoRA [ 2024 ] | 1.05h | 83.68 ±3.44 | 83.67 ±0.59 | 80.81 ±0.44 | 77.05 ±0.82 | 65.34 ±3.48 | 90.83 ±0.84 | 79.42 ±1.49 | 80.11 ±0.76 |
| FLoRA [ 2024 ] | 0.84h | 82.15 ±2.16 | 80.78 ±1.97 | 79.64 ±0.93 | 73.24 ±1.20 | 66.50 ±1.66 | 91.77 ±0.35 | 73.85 ±3.96 | 78.28 ±0.77 |
| FSLoRA [ 2025 ] | 0.78h | 79.48 ±1.47 | 78.72 ±1.32 | 74.27 ±0.89 | 58.83 ±2.35 | 56.17 ±3.95 | 91.88 ±0.47 | 76.80 ±1.02 | 73.74 ±0.74 |
| Method | Run time | QNLI | MRPC | CoLA | MNLI | RTE | SST-2 | QQP | Avg |
|---|---|---|---|---|---|---|---|---|---|
| Base Pretrained Model | — | 49.46 | 31.62 | 30.87 | 35.45 | 52.71 | 49.08 | 63.18 | 44.62 |
| FFA [ 2024 ] | 0.76h | 84.94 ±0.19 | 84.51 ±0.80 | 79.77 ±0.77 | 74.95 ±0.37 | 67.51 ±0.52 | 91.22 ±0.11 | 77.11 ±0.93 | 80.00 ±0.23 |
| HetLoRA [ 2024 ] | 0.59h | 86.18 ±0.48 | 83.53 ±1.44 | 81.72 ±0.29 | 78.44 ±0.12 | 70.18 ±0.54 | 92.39 ±0.15 | 77.09 ±0.46 | 81.36 ±0.24 |
| FlexLoRA [ 2024 ] | 1.02h | 87.31 ±0.34 | 86.42 ±0.33 | 82.26 ±0.21 | 80.20 ±0.09 | 73.86 ±0.80 | 92.98 ±0.11 | 80.35 ±0.70 | 83.34 ±0.17 |
| FLoRA [ 2024 ] | 0.83h | 86.20 ±0.60 | 84.95 ±0.80 | 81.84 ±0.30 | 78.50 ±0.23 | 71.48 ±0.59 | 92.75 ±0.23 | 79.78 ±0.76 | 82.21 ±0.21 |
| FSLoRA [ 2025 ] | 0.78h | 85.19 ±0.04 | 83.28 ±1.56 | 81.05 ±0.25 | 74.15 ±0.61 | 65.34 ±1.67 | 92.00 ±0.17 | 76.01 ±0.75 | 79.57 ±0.36 |
| Method | Run time | QNLI | MRPC | CoLA | MNLI | RTE | SST-2 | QQP | Avg |
|---|---|---|---|---|---|---|---|---|---|
| Base Pretrained Model | — | 49.46 | 31.62 | 30.87 | 35.45 | 52.71 | 49.08 | 63.18 | 44.62 |
| FFA [ 2024 ] | 1.26h | 85.15 ±0.43 | 83.48 ±0.28 | 79.06 ±0.37 | 74.04 ±0.58 | 63.83 ±1.54 | 91.49 ±0.28 | 79.95 ±0.79 | 79.57 ±0.28 |
| HetLoRA [ 2024 ] | 0.85h | 84.75 ±0.92 | 85.05 ±0.28 | 81.15 ±0.31 | 77.41 ±0.84 | 61.45 ±0.91 | 92.09 ±0.25 | 80.21 ±0.94 | 80.30 ±0.27 |
| FlexLoRA [ 2024 ] | 1.45h | 86.22 ±0.85 | 84.75 ±0.46 | 80.79 ±0.33 | 78.30 ±0.69 | 69.03 ±0.79 | 92.06 ±0.38 | 83.02 ±0.20 | 82.02 ±0.22 |
| FLoRA [ 2024 ] | 1.31h | 85.48 ±0.85 | 86.13 ±0.28 | 79.79 ±0.61 | 77.06 ±0.32 | 68.45 ±0.67 | 93.09 ±0.07 | 81.17 ±0.87 | 81.60 ±0.23 |
| FSLoRA [ 2025 ] | 1.23h | 81.31 ±2.57 | 84.31 ±0.77 | 80.04 ±0.25 | 67.99 ±2.77 | 62.60 ±2.12 | 91.56 ±0.24 | 79.56 ±0.75 | 78.20 ±0.64 |
| Method | Run time | QNLI | MRPC | CoLA | MNLI | RTE | SST-2 | QQP | Avg |
|---|---|---|---|---|---|---|---|---|---|
| Base Pretrained Model | — | 49.46 | 31.62 | 30.87 | 35.45 | 52.71 | 49.08 | 63.18 | 44.62 |
| FFA [ 2024 ] | 1.51h | 84.68 ±0.55 | 82.45 ±0.85 | 79.46 ±0.50 | 74.76 ±0.80 | 66.57 ±1.59 | 91.72 ±0.32 | 78.12 ±1.44 | 79.68 ±0.37 |
| HetLoRA [ 2024 ] | 1.08h | 86.10 ±0.33 | 84.07 ±0.58 | 80.71 ±0.35 | 77.74 ±0.48 | 58.77 ±2.02 | 92.27 ±0.11 | 79.46 ±1.06 | 79.87 ±0.35 |
| FlexLoRA [ 2024 ] | 1.54h | 87.36 ±0.20 | 85.73 ±0.40 | 80.59 ±1.01 | 78.88 ±0.23 | 70.47 ±1.99 | 92.98 ±0.25 | 81.66 ±1.15 | 82.52 ±0.37 |
| FLoRA [ 2024 ] | 1.48h | 85.87 ±0.63 | 84.85 ±0.66 | 80.36 ±0.23 | 76.45 ±0.39 | 68.52 ±1.30 | 92.61 ±0.23 | 80.30 ±1.23 | 81.28 ±0.30 |
| FSLoRA [ 2025 ] | 1.40h | 84.50 ±0.35 | 82.69 ±0.91 | 80.50 ±0.41 | 71.83 ±1.22 | 58.70 ±2.63 | 91.79 ±0.18 | 77.55 ±0.72 | 78.22 ±0.45 |
Federated Fine-Tuning
Federated fine-tuning adapts pretrained models across decentralized data holders without pooling their private data.
See all 127 papers