RBLA+: Efficient Rank-Specific LoRA Adapter Extraction for Federated Fine-Tuning with Heterogeneous Clients
Abstract
Federated fine-tuning with low-rank adaptation (LoRA) reduces communication and computational overhead by training and exchanging compact trainable LoRA factors. Clients may use different LoRA ranks to meet adapter communication budgets and accommodate different local adaptation needs. The server must process these differences during aggregation and construct adapters at each client's rank for local training. We distinguish pre-aggregation alignment, which handles factor dimensions and scaling conventions before aggregation and rank-specific LoRA adapter extraction, which constructs client-sized pairs from the aggregated global model. Two common approaches to rank-specific extraction are direct rank truncation and Singular Value Decomposition (SVD) of the dense update. Direct prefix truncation keeps the leading columns of one aggregated factor and the corresponding rows of the other. It is cheap but can increase reconstruction error. The dense SVD approach first forms the product of aggregated LoRA factors. It then decomposes this matrix and retains the leading singular components to construct adapters at the target rank. This gives the optimal rank-constrained approximation in Frobenius norm but incurs significant computational overhead. To address this trade-off, we propose RBLA+ as an extension of RBLA. It uses established compact QR/core-SVD decomposition to minimize Frobenius error for the extracted LoRA update without forming the dense update. It also accounts for client and server scaling conventions during upload and broadcast. Across vision and language tasks, RBLA+ improves accuracy over RBLA in multiple heterogeneous settings. In CPU benchmarks with synthetic adapters, RBLA+ has lower extraction time than the dense partial-SVD baseline. RBLA+ preserves each client's prescribed adapter size and per-round communication payload.
Figures & tables
| Category | Representative methods | Het. rank | Collab. | Matrix | Multi-rank | Compact | Main difference from SP+ |
| LoRA formulations and rank adaptation | LoRA, AdaLoRA, DyLoRA, QLoRA, LoRA+, DoRA, PiSSA, LoRA-GA [ 4 , 16 , 17 , 18 , 19 , 20 , 13 , 21 ] | ~ | ~ | ~ | Improves local rank allocation, memory, scaling, decomposition, or initialization, but does not aggregate independent client adapters or define a shared representation for client-specific extraction. | ||
| LoRA theory and matrix foundations | Intrinsic dimension, LoRA expressivity, LoRA+, RAC-LoRA, LoRA-One, truncated SVD [ 22 , 23 , 24 , 19 , 25 , 26 , 11 ] | ~ | ~ | ~ | Provides expressivity, convergence, and approximation results, including a federated RAC-LoRA extension. Rank-specific adapter extraction is not its primary focus. | ||
| Federated LoRA with compatible adapters | FedIT, pFedLoRA, FFA-LoRA, FedSA-LoRA, FLASC, LoRA-FAIR, RoLoRA [ 27 , 28 , 29 , 30 , 31 , 32 , 33 ] | ~ | ~ | Studies collaborative tuning, communication, personalization, or aggregation bias, but usually assumes compatible adapters and one broadcast rank. | |||
| Heterogeneous adapter-coordinate aggregation | HetLoRA, replication, RBLA [ 8 , 34 , 15 ] | ~ | Supports heterogeneous ranks through adapter-coordinate aggregation, but prefix-based extraction remains representation-dependent. | ||||
| Update-/subspace-aware federated LoRA | FLoRA, FlexLoRA, FLoRIST [ 14 , 9 , 35 ] | ~ | ~ | Constructs the global update from client matrices or subspaces, potentially requiring a dense update, growing intermediate rank, or unified-rank objective. | |||
| Distributed and split LoRA training | SplitLoRA, mLoRA [ 36 , 37 ] | ~ | ~ | Optimizes partitioning, pipelines, operators, and accelerator utilization, not heterogeneous aggregation or rank-specific adapter extraction. |
| Notation | Meaning |
| Output and input dimensions. | |
| Client rank and maximum client rank. | |
| Frozen pretrained weight matrix. | |
| Client ’s LoRA factors. | |
| Aggregated server LoRA factors. | |
| Spectrally ordered SP+ factors. |
| Dataset / Method Communication overhead | RBLA ( ) | RBLA+ ( ) | FlexLoRA ( ) | Zero Padding ( ) | Rep. Padding ( ) | FLoRIST ( ) | FLoRA ( ) |
| MNIST | |||||||
| FMNIST | |||||||
| KMNIST | |||||||
| QMNIST | |||||||
| CIFAR-10 | 38.33 | 79.12 | 78.84 | 78.39 | 45.61 | 72.18 | 77.25 |
| CINIC-10 | 31.54 | 69.65 | 69.33 | 68.06 | 26.31 | 60.10 | 68.51 |
| Dataset / Method Communication overhead | RBLA ( ) | RBLA+ ( ) | FlexLoRA ( ) | Zero Padding ( ) | Rep. Padding ( ) | FLoRIST ( ) | FLoRA ( ) |
| MNIST | |||||||
| FMNIST | |||||||
| KMNIST | |||||||
| QMNIST | |||||||
| CIFAR-10 | 57.63 | 78.95 | 78.53 | 78.22 | 41.08 | 71.27 | 78.29 |
| CINIC-10 | 42.10 | 68.89 | 68.33 | 68.46 | 34.03 | 61.25 | 68.61 |
| Method | CIFAR-10 | MNLI |
| RBLA | 75.17 | 56.17 |
| RBLA+ | 75.21 | 61.21 |
| FlexLoRA | 73.94 | 52.93 |
| Zero Padding | 72.34 | 56.43 |
| Rep. Padding | 74.54 | 56.40 |
| FLoRIST | 56.84 | 56.79 |
| Round | 0 | 4 | 5 | 7 | 24 | 25 | 35 | 55 |
| CIFAR-10 Conv. | 53 | 51 | 44 | 41 | 35 | 35 | 35 | 34 |
| CIFAR-10 FC | 13 | 13 | 11 | 10 | 9 | 9 | 9 | 9 |
| MNLI | 5 | 5 | 5 | 5 | 5 | 4 | 3 | 3 |
| Dataset | Without conversion | With conversion | (pp) |
| Correct rank assignment | |||
| CIFAR-10 | 78.30 | 79.12 | +0.82 |
| CINIC-10 | 69.13 | 69.65 | +0.52 |
| SVHN | 92.90 | 92.98 | +0.08 |
| AG News | 90.99 | 91.33 | +0.34 |
| IMDB | 82.25 | 84.32 | +2.07 |
| Method | Time (ms) | RSS (MiB) |
| MLP: 0.224M | ||
| RBLA | 0.4 | 5.5 |
| RBLA+ (QR-SVD) | 5.0 | 13.5 |
| Dense full SVD | 6.9 | 13.6 |
| Dense partial SVD | 88.3 | 16.4 |
| Implicit partial SVD | 103.2 | 16.3 |
| Extraction | Server computation | Additional working storage |
| RBLA (rank prefix) | ||
| RBLA+ (SP+) | ||
| Dense product and full SVD |
Federated Fine-Tuning
Federated fine-tuning adapts pretrained models across decentralized data holders without pooling their private data.
See all 127 papersCFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregation
CFLoRA, a federated LoRA scheme that partitions latent LoRA channels into two complementary sets in every communication round. By ensuring that columns and rows are complementary across factors, we eliminate bilinear terms in matrix multiplications, making federated aggregation exact. Crucially, our framework also supports clients with heterogeneous rank budgets. Convergence analysis validates CFLoRA achieves convergence rate of the original LoRA objective in homogeneous-rank cases. Extensive experiments with RoBERTa on the GLUE benchmark and with LLaMA-3.2-3B-Instruct on commonsense reasoning tasks demonstrate that CFLoRA achieves superior performance and training efficiency compared to state-of-the-art federated LoRA baselines.