FedLAFP: Low-Rank Aggregation Meets Full-Rank Personalization in Federated Fine-Tuning

Mengjun Yi; Huaian Gu; Yinghao Ai; Furao Shen; Jian Zhao

cs.AISep 29, 2026

FedLAFP: Low-Rank Aggregation Meets Full-Rank Personalization in Federated Fine-Tuning

Authors: Mengjun Yi, Huaian Gu, Yinghao Ai, Furao Shen, Jian Zhao

Organizations: State Key Laboratory for Novel Software Technology · School of Artificial Intelligence · School of Computer Science · School of Electronic Science and Engineering

Abstract

Federated parameter-efficient fine-tuning enables clients to adapt pre-trained models without sharing raw data or communicating the full model, but statistical heterogeneity makes a single global adapter insufficient for personalized prediction. Existing personalized methods typically use the same low-rank structure for both shared and private adaptation, overlooking their distinct requirements for aggregation and personalization. We propose FedLAFP, a role-aware framework that couples a compact, globally aggregated LoRA branch with a client-private, full-rank-capable RandLoRA branch. The shared branch provides an efficient interface for transferring common knowledge, whereas the private branch combines fixed random low-rank bases with learned scaling coefficients to provide expressive client-specific adaptation without additional communication. Client- and layer-specific mixing coefficients jointly fuse the two branches, and only the shared LoRA parameters are exchanged. A controlled linear study supports this role assignment: LoRA yields more aligned client updates and lower aggregation error, while RandLoRA more accurately recovers client-specific residuals. Experiments across four visual recognition benchmarks show that FedLAFP consistently outperforms local-only and federated LoRA baselines, achieving an average personalized accuracy of $86.93\%$ and exceeding the best baseline average by $1.30$ percentage points.

Figures & tables

Figure 1: Comparison of global-only adaptation, symmetric shared–private LoRA, and the proposed role-aware FedLAFP.

Figure 2: Controlled comparison of LoRA and RandLoRA under increasing client heterogeneity. Panels report (a) personalization error, (b) aggregation error, and (c) mean cross-client update similarity.

Figure 3: Overview of the FedLAFP framework. (a) The server broadcasts the shared LoRA parameters, and clients upload only their locally updated shared parameters for sample-weighted aggregation, while the private RandLoRA parameters remain local. (b) Within target layer $\ell$ of client $i$ , the output of the frozen linear transformation is augmented by the weighted outputs of the shared LoRA and private RandLoRA branches, using $\pi_{\ell,i}^{\mathrm{sh}}$ and $\pi_{\ell,i}^{\mathrm{pr}}$ , respectively. These client- and layer-specific fusion weights are derived from private logits and are never communicated.

Method	DTD	Oxford Pets	SUN397	UCF101	Average
Local-only LoRA	$67.42\pm 0.30$	$90.81\pm 0.17$	$78.97\pm 0.03$	$83.57\pm 0.22$	$80.19$
Local-only RandLoRA	$67.99\pm 0.36$	$91.11\pm 0.22$	$78.99\pm 0.07$	$83.70\pm 0.20$	$80.45$
FedLoRA [ICASSP’24]	$74.59\pm 0.41$	$95.47\pm 0.07$	$83.02\pm 0.04$	$87.81\pm 0.11$	$85.22$
FedRandLoRA	$75.00\pm 0.31$	$95.43\pm 0.07$	$83.89\pm 0.07$	$88.20\pm 0.09$	$85.63$
FFA-LoRA [ICLR’24]	$74.35\pm 0.36$	$95.48\pm 0.08$	$82.86\pm 0.02$	$87.86\pm 0.23$	$85.14$
FedEx-LoRA [ACL’25]	$74.47\pm 0.12$	$95.45\pm 0.09$	$82.95\pm 0.01$	$87.73\pm 0.03$	$85.15$

Table 1: Comparison with local-only and federated LoRA baselines on four visual recognition benchmarks. Accuracy (%) is reported as mean

\pm

standard deviation over three runs. The best result in each column is shown in bold.

Method	MRPC	SST-2	Average
FedLoRA [ICASSP’24]	72.30	93.81	83.06
FedALT [AAAI’26]	88.24	95.41	91.83
FedLAFP (Ours)	88.73	95.76	92.25

Table 2: Accuracy (%) on GLUE language tasks using RoBERTa-Large.

Figure 4: Role-assignment matrices on DTD and UCF101. Rows specify the shared adapter and columns specify the private adapter. Each cell reports accuracy (%); the smaller annotation and cell color encode the gain over LoRA/LoRA. Bold values denote the best assignment per dataset.

Scheme	DTD	Pets	SUN	UCF
Private $(0,1)$	$67.99$	$91.11$	$78.99$	$83.70$
Shared $(1,0)$	$74.59$	$95.47$	$83.02$	$87.81$
Equal $(0.5,0.5)$	$77.48$	$95.78$	$83.96$	$88.82$
Instance gate	$78.07$	$95.86$	$82.49$	$88.76$
Client–layer (Ours)	$\mathbf{78.62}$	$\mathbf{95.96}$	$\mathbf{84.00}$	$\mathbf{89.12}$

Table 3: Comparison of strategies for fusing the shared and private branches.

Figure 5: Hyperparameter sensitivity on DTD and UCF101. Each panel varies one hyperparameter while holding the remaining settings fixed. The horizontal axis shows the accuracy change in percentage points relative to the default configuration

(r_{L},r_{R},K,E)=(4,128,12,3)

; marker labels show absolute accuracy (%). The shaded row denotes the default value.

Figure 6: Accuracy gain of FedLAFP over FedLoRA under different Dirichlet concentration parameters. Positive values indicate an improvement over FedLoRA under the same client partition. The black curve shows the mean gain across the four datasets; larger

\alpha

corresponds to less heterogeneous client data.

Explore similar work

Breaking the Structural Identity: Personalized Federated LoRA Fine-tuning under Rank Heterogeneity
Sep 1, 2026Lei Wang, Jieming Bian, Letian Zhang +1Federated Learning Model Fine-Tuning
University of Florida · Gainesville, FL 32611 · Middle Tennessee State University +1
Amortizing Federated Adaptation: Hypernetwork Driven LoRA for Personalized Foundation Models
Jun 4, 2026Sunny Gupta, Shambhavi Shanker, Amit SethiWireless Foundation Models Hypernetwork
Indian Institute of Technology, Bombay
CF-LoRA: Decoupled Factor Aggregation and Adaptation-Aware Client Clustering for Federated LoRA Fine-Tuning
Sep 29, 2026Mengjun Yi, Langxing Yang, Suhan Guo +2Hip-Lora Parameter-Efficient Adaptation
State Key Laboratory for Novel Software Technology · School of Artificial Intelligence School of Electronic Science and Engineering Nanjing University

Sep 1, 2026, cs.LGJ/K move · Enter open · S save

Breaking the Structural Identity: Personalized Federated LoRA Fine-tuning under Rank Heterogeneity

Lei Wang, Jieming Bian, Letian Zhang, Jie Xu

University of Florida · Gainesville, FL 32611 · Middle Tennessee State University+1

Federated Learning Model Fine-Tuning Heterogeneity Subspace

Open paper AskPDF arXiv

Large Language Models (LLMs) have achieved remarkable success across diverse domains, but their adaptation to privacy-sensitive, distributed datasets remains a challenge. While Federated Learning (FL) combined with Low-Rank Adaptation (LoRA) provides a resource-efficient paradigm for collaborative fine-tuning, practical deployments are hindered by the dual challenges of resource heterogeneity and data heterogeneity. Existing rank-heterogeneous methods primarily focus on bridging dimension mismatches for aggregation but typically provide a unified global model for all clients sharing the same rank, failing to capture client-specific features in non-IID scenarios. In this paper, we propose FedRoRA (Federated Rank-wise Personalized LoRA), a novel framework that enables fine-grained personalization within rank-heterogeneous federations. FedRoRA decouples adaptation into shared global directions and personalized rank-wise magnitudes governed by learnable diagonal scales. On the server side, it extracts a global subspace via singular value decomposition (SVD) and redistributes client-specific initializations through a personalized projection and top-

k

selection mechanism. Extensive experiments on NLU and NLG benchmarks demonstrate that FedRoRA consistently outperforms state-of-the-art methods.

Abstract

Explore similar work

Breaking the Structural Identity: Personalized Federated LoRA Fine-tuning under Rank Heterogeneity

Amortizing Federated Adaptation: Hypernetwork Driven LoRA for Personalized Foundation Models

CF-LoRA: Decoupled Factor Aggregation and Adaptation-Aware Client Clustering for Federated LoRA Fine-Tuning