FedPA-LoRA: Product-Aligned Framework for Mitigating Aggregation and Initialization Errors in Heterogeneous Federated LoRA

Juseok Jeon; Ramy E. Ali; Doyun Kwon; Myungbeom Her; Jinhwi Kim; Jinhyun So

cs.AIAug 15, 2026

FedPA-LoRA: Product-Aligned Framework for Mitigating Aggregation and Initialization Errors in Heterogeneous Federated LoRA

Authors: Juseok Jeon, Ramy E. Ali, Doyun Kwon, Myungbeom Her, Jinhwi Kim, Jinhyun So

Organizations: DGIST Daegu, Republic of Korea · Samsung San Diego, USA

Abstract

Low-Rank Adaptation (LoRA) enables efficient federated fine-tuning of large language models, but its factorized parameterization creates a tension between accurate aggregation of local updates and continuity of locally optimized factors. Factor-wise aggregation incurs aggregation mismatch but better preserves factor continuity, whereas product-space reconstruction reduces this mismatch at the cost of greater factor-level initialization mismatch from newly reconstructed factors. We propose FedPA-LoRA, a product-aligned federated LoRA framework that jointly addresses these limitations and provably converges under both homogeneous and heterogeneous client ranks. Each client preserves its local factors across communication rounds and aligns its product toward a rank-specific global reference, maintaining local optimization continuity while promoting global consistency under data heterogeneity. The server aggregates heterogeneous-rank updates in the common product space and efficiently reconstructs a rank-constrained global adapter without forming the dense aggregate. This design supports client-specific computation and communication budgets. Experiments on natural language understanding and generation tasks show that FedPA-LoRA consistently outperforms representative baselines across varying levels of data heterogeneity and homogeneous- and heterogeneous-rank settings, with up to a $6.82$ percentage-point improvement in average GLUE accuracy under heterogeneous client ranks.

Figures & tables

Panel of Figure 1 — Figure 1: Aggregation error and relative factor-level initialization errors on MNLI with three clients. Panels (a)–(c) show the aggregation error and the relative initialization errors of $\bm{B}$ and $\bm{A}$ , respectively, for the query projection in layer 0 of RoBERTa-Large. The y-axes use logarithmic scales, with FedPA-LoRA’s zero initialization errors shown at the plotting floor.

Figure 2: Overview of FedPA-LoRA. Clients preserve local factors to avoid initialization mismatch and align their product $\bm{X}_{i}^{(t,k)}$ with a rank- $R_{i}$ global reference. The server reconstructs the optimal rank- $R_{g}$ adapter via reduced QR and core SVD without forming the dense aggregate.

Setting	Method	SST-2	QNLI	QQP	RTE	MNLI	Average
Homogeneous	FedIT	$0.956{\pm}0.000$	$0.891{\pm}0.046$	$0.861{\pm}0.001$	$0.696{\pm}0.018$	$0.858{\pm}0.001$	$0.8524$
	FlexLoRA	$0.956{\pm}0.001$	$0.642{\pm}0.100$	$0.848{\pm}0.010$	$0.742{\pm}0.061$	$0.858{\pm}0.002$	$0.8092$
	FFA-LoRA	$0.946{\pm}0.001$	$0.890{\pm}0.010$	$0.848{\pm}0.002$	$0.611{\pm}0.021$	$0.840{\pm}0.001$	$0.8270$
	RoLoRA	$0.957{\pm}0.002$	$0.882{\pm}0.046$	$0.845{\pm}0.024$	$0.794{\pm}0.022$	$0.852{\pm}0.002$	$0.8660$
	FedRot-LoRA	$0.954{\pm}0.002$	$0.908{\pm}0.004$	$0.849{\pm}0.002$	$0.714{\pm}0.038$	$0.858{\pm}0.002$	$0.8566$
	FedDPA-LoRA	$0.957{\pm}0.001$	$0.851{\pm}0.062$	$0.811{\pm}0.045$	$0.612{\pm}0.105$	$0.879{\pm}0.002$	$0.8220$

Table 1: GLUE accuracy under homogeneous- and heterogeneous-rank settings with Dirichlet concentration parameter

\beta=0.5

. The homogeneous setting uses

N=10

clients with rank

r=4

, whereas the heterogeneous setting uses

N=20

clients divided into three resource groups.

Method	$\bm{\beta}=\mathbf{100}$	$\bm{\beta}=\mathbf{1}$	$\bm{\beta}=\mathbf{0.5}$	$\bm{\beta}=\mathbf{0.1}$
FedIT	$0.873{\pm}0.002$	$0.859{\pm}0.001$	$0.869{\pm}0.001$	$0.858{\pm}0.004$
FlexLoRA	$0.868{\pm}0.001$	$0.877{\pm}0.002$	$0.869{\pm}0.002$	$0.859{\pm}0.011$
FFA-LoRA	$0.867{\pm}0.002$	$0.856{\pm}0.002$	$0.856{\pm}0.002$	$0.824{\pm}0.003$
RoLoRA	$0.867{\pm}0.008$	$0.676{\pm}0.252$	$0.862{\pm}0.000$	$0.822{\pm}0.008$
FedRot-LoRA	$0.874{\pm}0.001$	$0.868{\pm}0.002$	$0.868{\pm}0.002$	$0.857{\pm}0.005$
FedDPA-LoRA	$0.873{\pm}0.003$	$0.702{\pm}0.265$	$0.879{\pm}0.002$	$0.686{\pm}0.257$

Table 2: MNLI accuracy across data heterogeneity levels

\beta\in\{100,1,0.5,0.1\}

with

N=3

and LoRA rank

r=4

Method	$\bm{N}=\mathbf{3}$	$\bm{N}=\mathbf{10}$	$\bm{N}=\mathbf{50}$
FedIT	$0.869{\pm}0.001$	$0.858{\pm}0.001$	$0.877{\pm}0.001$
FlexLoRA	$0.869{\pm}0.002$	$0.858{\pm}0.002$	$0.740{\pm}0.175$
FFA-LoRA	$0.856{\pm}0.002$	$0.840{\pm}0.001$	$0.860{\pm}0.001$
RoLoRA	$0.862{\pm}0.000$	$0.852{\pm}0.002$	$0.882{\pm}0.002$
FedRot-LoRA	$0.868{\pm}0.002$	$0.858{\pm}0.002$	$0.872{\pm}0.001$
FedDPA-LoRA	$0.879{\pm}0.002$	$0.879{\pm}0.002$	$0.883{\pm}0.014$

Table 3: MNLI accuracy under different numbers of clients

N\in\{3,10,50\}

with LoRA rank

r=4

and Dirichlet concentration parameter

\beta=0.5

Figure 3: MNLI accuracy under different LoRA ranks

r\in\{2,4,8,16,24\}

with

\beta=0.5

and

N=3

Method	Local	Local	Regularization	Accuracy
Method	Head	Preservation	Regularization	Accuracy
FedIT	$\times$	$\times$	$\times$	$0.869{\pm}0.001$
FedIT	$\checkmark$	$\times$	$\times$	$0.869{\pm}0.000$
FlexLoRA	$\times$	$\times$	$\times$	$0.869{\pm}0.002$
FlexLoRA	$\checkmark$	$\times$	$\times$	$0.870{\pm}0.001$
Ours	$\times$	$\times$	$\bm{B}\bm{A}$	$0.867{\pm}0.003$

Table 4: Component ablation results on MNLI with

N=3

clients and Dirichlet concentration parameter

\beta=0.5

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

\mathcal{L}_{i}^{(t)}(\bm{B}_{i},\bm{A}_{i})=f_{i}\left(\bm{W}_{0}+\bm{B}_{i}\bm{A}_{i}\right)+\frac{\lambda}{2}\left\|\bm{B}_{i}\bm{A}_{i}-\bm{B}_{g,i}^{(t-1)}\bm{A}_{g,i}^{(t-1)}\right\|_{\mathrm{F}}^{2}.

Appendix

Algorithm 1 FedPA-LoRA

Experiment

Dataset

Clients (

N

)

Rank (

r

)

Data Distribution (

\beta

)

Optimal Learning Rate

\bm{\eta}

Optimal

\bm{\lambda}

FedIT

FlexLoRA

FFA- LoRA

RoLoRA

FedRot- LoRA

FedDPA- LoRA

FedSA- LoRA

FedPA- LoRA

FedRot- LoRA

FedPA- LoRA

Natural Language Understanding

SST-2

10

4

0.5

1e-2

2e-2

1e-2

5e-3

2e-2

0.6

1.0

QNLI

1e-2

2e-2

5e-3

1e-3

2e-2

0.2

2.0

QQP

1e-2

5e-3

2e-2

5e-3

2e-2

0.6

2.0

RTE

1e-2

2e-2

5e-3

1e-2

2e-2

1e-3

2e-2

0.4

1.0

MNLI

1e-2

5e-3

2e-2

5e-3

1e-2

2e-2

0.4

2.0

Appendix

Table 5: Selected hyperparameters for experiments under homogeneous-rank settings.

Experiment	Dataset	Clients ( $N$ )	Rank Configuration	Data Distribution ( $\beta$ )	Optimal Learning Rate $\bm{\eta}$					Optimal $\bm{\lambda}$
Experiment	Dataset	Clients ( $N$ )	Rank Configuration	Data Distribution ( $\beta$ )	FlexLoRA	HetLoRA	Ravan	Fed- PLoRA	FedPA- LoRA	FedPA- LoRA
Natural Language Understanding	SST-2	$20$	Other methods: $r_{i}\in\{2,4,16\}$ Ravan: $h_{i}\in\{1,2,8\}$ , $r_{h}=64$	$0.5$	1e-2	2e-2	2e-2	1e-2	1e-2	$2.0$
	QNLI				1e-2	1e-2	2e-2	1e-2	1e-2	$2.0$
	QQP				1e-2	1e-2	1e-2	5e-3	2e-2	$1.0$
	RTE				1e-2	1e-2	1e-2	1e-2	1e-2	$1.0$
	MNLI				5e-3	2e-2	1e-2	5e-3	2e-2	$1.0$

Appendix

Table 6: Selected hyperparameters for experiments under heterogeneous-rank settings.

Method	$\bm{r}=\mathbf{2}$	$\bm{r}=\mathbf{4}$	$\bm{r}=\mathbf{8}$	$\bm{r}=\mathbf{16}$	$\bm{r}=\mathbf{24}$
FedIT	$0.869{\pm}0.002$	$0.869{\pm}0.001$	$0.863{\pm}0.001$	$0.865{\pm}0.001$	$0.873{\pm}0.000$
FlexLoRA	$0.876{\pm}0.004$	$0.869{\pm}0.002$	$0.866{\pm}0.001$	$0.839{\pm}0.032$	$0.868{\pm}0.001$
FFA-LoRA	$0.837{\pm}0.013$	$0.856{\pm}0.002$	$0.845{\pm}0.002$	$0.868{\pm}0.001$	$0.871{\pm}0.002$
RoLoRA	$0.849{\pm}0.005$	$0.862{\pm}0.000$	$0.856{\pm}0.002$	$0.858{\pm}0.002$	$0.857{\pm}0.004$
FedRot-LoRA	$0.869{\pm}0.001$	$0.868{\pm}0.002$	$0.869{\pm}0.002$	$0.870{\pm}0.001$	$0.872{\pm}0.002$
FedDPA-LoRA	$0.871{\pm}0.001$	$0.879{\pm}0.002$	$0.874{\pm}0.005$	$0.877{\pm}0.001$	$0.868{\pm}0.012$

Appendix

Table 7: MNLI accuracy under different LoRA ranks

r\in\{2,4,8,16,24\}

with Dirichlet concentration parameter

\beta=0.5

and

N=3

clients.

Table 13

Method	Client Upload	Server Operation	Computation	Memory
FedIT	$\bm{B}_{i},\bm{A}_{i}$	Two-Factor Averaging	$\mathcal{O}(Ndr)$	$\mathcal{O}(Ndr)$
FedRot-LoRA	$\widetilde{\bm{B}}_{i},\widetilde{\bm{A}}_{i}$	Two-Factor Averaging	$\mathcal{O}(Ndr)$	$\mathcal{O}(Ndr)$
FedDPA-LoRA	$\bm{B}_{g,i},\bm{A}_{g,i}$	Two-Factor Averaging	$\mathcal{O}(Ndr)$	$\mathcal{O}(Ndr)$
FFA-LoRA	$\bm{B}_{i}$	Single-Factor Averaging	$\mathcal{O}(Ndr)$	$\mathcal{O}(Ndr)$
RoLoRA	$\bm{B}_{i}$ or $\bm{A}_{i}$	Single-Factor Averaging	$\mathcal{O}(Ndr)$	$\mathcal{O}(Ndr)$
FedSA-LoRA	$\bm{A}_{i}$	Single-Factor Averaging	$\mathcal{O}(Ndr)$	$\mathcal{O}(Ndr)$

Appendix

Table 10: Dominant per-layer server-side complexity under the homogeneous-rank setting with

Nr\ll d

. The randomized complexity assumes a sketch dimension

\ell=R_{g}+p=\mathcal{O}(r)

Server Operation	RoBERTa-Large $(N=10,\;R_{g}=4)$		Llama 3-8B $(N=6,\;R_{g}=8)$
Server Operation	Sketch Dim. $\bm{\ell}$	Time (s)	Sketch Dim. $\bm{\ell}$	Time (s)
Two-Factor Averaging	–	$0.0298$	–	$0.0199$
Single-Factor Averaging	–	$0.0206$	–	$0.0109$
Dense Product Aggregation and SVD	–	$4.7340$	–	$152.3957$
Reduced QR and Core SVD	–	$0.0840$	–	$0.1169$
Randomized Reconstruction ( $p=0$ )	$4$	$0.0473$	$8$	$0.0596$

Appendix

Table 11: Average per-round server-side aggregation wall-clock time under the homogeneous-rank setting. For randomized reconstruction, the sketch dimension is

\ell=R_{g}+p

. RoBERTa-Large uses

N=10

clients and

R_{g}=4

, yielding

r_{\mathrm{tot}}=NR_{g}=40

, whereas Llama 3-8B uses

N=6

clients and

R_{g}=8

, yielding

r_{\mathrm{tot}}=48

Figure 6: MNLI accuracy of randomized server-side reconstruction under different sketch dimensions

\ell=R_{g}+p\in\{4,5,6,8,12\}

. Experiments use the homogeneous-rank setting with

N=3

clients,

r=R_{g}=4

, and

\beta=0.5

. The corresponding oversampling parameters are

p\in\{0,1,2,4,8\}

, and

\ell=12=r_{\mathrm{tot}}

uses the full concatenated rank. Horizontal dashed lines denote baseline accuracies averaged over three random seeds.

Local Update Strategy	Representative Methods	RoBERTa-Large	Llama 3-8B
Single-Factor LoRA Update	FFA-LoRA, RoLoRA, FedSA-LoRA	$\mathbf{9.1388}$	$\mathbf{3.8015}$
Full LoRA Update	FedIT, FlexLoRA	$9.2053$	$3.8324$
LoRA Update + Alignment	FedRot-LoRA	$9.2121$	$3.9323$
Global and Local LoRA Update	FedDPA-LoRA	$10.3431$	$3.8371$
Product-Regularized LoRA Update	FedPA-LoRA	$9.4603$	$5.1343$

Appendix

Table 12: Average client-side wall-clock time in seconds, first averaged across participating clients within each communication round and then across rounds. The first round is excluded to avoid one-time initialization overhead.

Explore similar work

CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregation
Sep 27, 2026Yanan Ma, Qiyuan Chen, Zihan Fang +2Federated Fine-Tuning Federated Learning
City University of Hong Kong · The University of Hong Kong
Beyond Factor Aggregation: Gauge-Aware Low-Rank Server Representations for Federated LoRA
May 7, 2026Jinqian Chen, Chang Liu, Jihua ZhuFederated Learning Aggregation Low-Rank Adaptation
School of Software Engineering Xi’an Jiaotong University Xi’an, China 710000
CF-LoRA: Decoupled Factor Aggregation and Adaptation-Aware Client Clustering for Federated LoRA Fine-Tuning
Sep 29, 2026Mengjun Yi, Langxing Yang, Suhan Guo +2Federated Learning Aggregation Federated Fine-Tuning
State Key Laboratory for Novel Software Technology · School of Artificial Intelligence School of Electronic Science and Engineering Nanjing University

Sep 27, 2026, cs.LGJ/K move · Enter open · S save

CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregation

Yanan Ma, Qiyuan Chen, Zihan Fang, Xianhao Chen+1

City University of Hong Kong · The University of Hong Kong

Federated Fine-Tuning Federated Learning Low-Rank Adaptation Federated Learning Aggregation

Open paper AskPDF arXiv

Federated low-rank adaptation (LoRA) enables collaborative fine-tuning of large language models without centralizing private client data. Its factorized update, however, creates a structural mismatch in federated averaging: averaging the two LoRA factors separately does not equal averaging their products. Existing exact methods resolve this issue mainly by freezing an entire factor or alternating factors across rounds, but none can update factors simultaneously without aggregation errors or expanding communication ranks. To address this fundamental problem, we present \texttt{CFLoRA}, a federated LoRA scheme that partitions latent LoRA channels into two complementary sets in every communication round. By ensuring that columns and rows are complementary across factors, we eliminate bilinear terms in matrix multiplications, making federated aggregation exact. Crucially, our framework also supports clients with heterogeneous rank budgets. Convergence analysis validates \texttt{CFLoRA} achieves

\mathcal{O}(1/\sqrt{T})

convergence rate of the \textit{original} LoRA objective in homogeneous-rank cases. Extensive experiments with RoBERTa on the GLUE benchmark and with LLaMA-3.2-3B-Instruct on commonsense reasoning tasks demonstrate that \texttt{CFLoRA} achieves superior performance and training efficiency compared to state-of-the-art federated LoRA baselines.

Abstract

Explore similar work

CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregation

Beyond Factor Aggregation: Gauge-Aware Low-Rank Server Representations for Federated LoRA

CF-LoRA: Decoupled Factor Aggregation and Adaptation-Aware Client Clustering for Federated LoRA Fine-Tuning