FedRot-LoRA: Mitigating Rotational Misalignment in Federated LoRA

Haoran Zhang; Dongjun Kim; Seohyeon Cha; Haris Vikalo

cs.LGFeb 27, 2026

FedRot-LoRA: Mitigating Rotational Misalignment in Federated LoRA

Authors: Haoran Zhang, Dongjun Kim, Seohyeon Cha, Haris Vikalo

Organizations: Chandra Department of Electrical and Computer Engineering, The University of Texas at Austin, TX, USA.

Abstract

Federated LoRA provides a communication-efficient mechanism for fine-tuning large language models on decentralized data. In practice, however, a discrepancy between the factor-wise averaging used to preserve low rank and the mathematically correct aggregation of local updates can cause significant aggregation error and unstable training. We argue that a major source of this problem is rotational misalignment, arising from the rotational invariance of low-rank factorizations -- semantically equivalent updates can be represented in different latent subspaces across clients since $(B_i R_i)(R_i^\top A_i) = B_i A_i$ . When such misaligned factors are averaged directly, they interfere destructively and degrade the global update. To address this issue, we propose FedRot-LoRA, a federated LoRA framework that aligns client updates via orthogonal transformations prior to aggregation. This alignment preserves the semantic update while reducing cross-client subspace mismatch, without increasing communication cost or restricting model expressivity. We provide a convergence analysis that examines the aggregation error induced by factor-wise averaging and shows how rotational alignment yields a tighter upper bound on this error. Extensive experiments on natural language understanding and generative tasks demonstrate that FedRot-LoRA consistently outperforms existing federated LoRA baselines across a range of heterogeneity levels and LoRA ranks.

Figures & tables

Figure 1: Overview of rotational alignment in FedRot-LoRA. Top: Naive aggregation averages unaligned LoRA factors

(A_{i},B_{i})

, causing destructive interference due to cross-client subspace mismatch. Bottom: FedRot-LoRA applies client-specific rotations

R_{i}

to align local updates prior to aggregation. This preserves the semantic update while reducing subspace misalignment, leading to lower aggregation error and more stable training behavior.

Figure 2: Optimization trajectories with target

\Delta W^{*}=1.0

, initialized at

\bar{B}^{0}=0

and

\bar{A}^{0}=0.44

(black star). The red dashed curve denotes the optimal solution manifold. Each marker along a trajectory represents the global factors

(B,A)

after one communication round under a given method. Different aggregation schemes induce distinct optimization paths, illustrating the impact of rotational misalignment on training dynamics.

Clients Num	Methods	SST-2	QNLI	QQP	RTE	MNLI	Avg.
3	FedIT ( Zhang et al., 2024a )	$0.953\pm 0.001$	$0.926\pm 0.002$	$0.771\pm 0.098$	$0.840\pm 0.013$	$0.866\pm 0.001$	$0.8712$
	FlexLoRA ( Bai et al., 2024 )	$0.949\pm 0.003$	$0.910\pm 0.007$	$0.832\pm 0.008$	$0.813\pm 0.015$	$0.845\pm 0.008$	$0.8698$
	FFA-LoRA ( Sun et al., 2024 )	$0.768\pm 0.056$	$0.926\pm 0.002$	$0.838\pm 0.002$	$0.830\pm 0.005$	$0.862\pm 0.001$	$0.8448$
	RoLoRA ( Chen et al., 2025 )	$0.951\pm 0.004$	$0.917\pm 0.011$	$0.841\pm 0.012$	$0.854\pm 0.005$	$0.868\pm 0.005$	$0.8862$
	FedRot-LoRA (Ours)	$\mathbf{0.954\pm 0.001}$	$\mathbf{0.926\pm 0.002}$	$\mathbf{0.842\pm 0.002}$	$\mathbf{0.868\pm 0.014}$	$\mathbf{0.876\pm 0.002}$	$\mathbf{0.8932}$
10	FedIT ( Zhang et al., 2024a )	$0.949\pm 0.002$	$0.913\pm 0.004$	$0.849\pm 0.000$	$0.621\pm 0.054$	$0.849\pm 0.002$	$0.8362$

Table 1: GLUE benchmark results comparing different federated LoRA methods under three client scales (

N\in\{3,10,50\}

), using

r=4

and Dirichlet concentration parameter

h=0.5

. FedRot-LoRA achieves the highest average accuracy and shows improved stability across tasks.

Methods	SST-2	QNLI	QQP	RTE	MNLI
FedIT	$7.49\text{e-}3$	$1.98\text{e-}3$	$1.04\text{e-}2$	$1.91\text{e-}3$	$3.98\text{e-}3$
FedRot-LoRA	$4.72\text{e-}4$	$1.34\text{e-}4$	$8.70\text{e-}4$	$2.11\text{e-}4$	$1.48\text{e-}4$

Table 2: FedRot-LoRA achieves an order-of-magnitude reduction in aggregation error compared to FedIT.

Rank	Methods	ACC $\pm$ std@MNLI
2	FedIT ( Zhang et al., 2024a )	$0.844\pm 0.051$
	FlexLoRA ( Bai et al., 2024 )	$0.811\pm 0.108$
	FFA-LoRA ( Sun et al., 2024 )	$0.837\pm 0.008$
	RoLoRA ( Chen et al., 2025 )	$0.860\pm 0.002$
	FedRot-LoRA (Ours)	$\mathbf{0.877\pm 0.005}$
4	FedIT ( Zhang et al., 2024a )	$0.866\pm 0.001$

Table 3: MNLI accuracy under different LoRA ranks (

r\in\{{\color[rgb]{0,0,0}2},4,8,16,{\color[rgb]{0,0,0}24}\}

). FedRot-LoRA outperforms all baselines across ranks, demonstrating stable performance across ranks.

Methods	$h=100$	$h=1$	$h=0.5$
FedIT	$0.868\pm 0.003$	$0.864\pm 0.015$	$0.866\pm 0.001$
FlexLoRA	$0.871\pm 0.001$	$0.839\pm 0.017$	$0.845\pm 0.008$
FFA-LoRA	$0.869\pm 0.000$	$0.856\pm 0.003$	$0.862\pm 0.001$
RoLoRA	$0.767\pm 0.158$	$0.780\pm 0.140$	$0.868\pm 0.005$
FedRot-LoRA	$\mathbf{0.884\pm 0.001}$	$\mathbf{0.873\pm 0.001}$	$\mathbf{0.876\pm 0.002}$

Table 4: MNLI accuracy under varying levels of data heterogeneity (

h\in\{100,1,0.5\}

). FedRot-LoRA outperforms all baselines across settings, highlighting its robustness to both homogeneous and heterogeneous data distributions.

Methods	HumanEval (pass@1)	GSM8K (acc)
FedIT ( Zhang et al., 2024a )	$0.2877\pm 0.017$	$0.4293\pm 0.018$
FlexLoRA ( Bai et al., 2024 )	$0.2930\pm 0.048$	$0.3515\pm 0.126$
FFA-LoRA ( Sun et al., 2024 )	$0.3851\pm 0.009$	$0.4361\pm 0.001$
RoLoRA ( Chen et al., 2025 )	$0.2951\pm 0.081$	$0.3444\pm 0.172$
FedRot-LoRA (Ours)	$\mathbf{0.4088\pm 0.012}$	$\mathbf{0.4437\pm 0.009}$

Table 5: Performance on generative tasks. We report pass@1 scores for code generation (evaluated on HumanEval) and exact match accuracy for mathematical reasoning (evaluated on GSM8K). FedRot-LoRA outperforms all baselines on both tasks.

Panel of Figure 3 — Figure 3: Effect of soft rotation level $\lambda$ in FedRot-LoRA on MNLI and QQP. Red dashed line denotes the baseline’s average accuracy.

Methods	SST-2	QNLI	MNLI
Align A Only	$0.883\pm 0.015$	$0.913\pm 0.008$	$0.861\pm 0.001$
Align B Only	$0.879\pm 0.015$	$0.806\pm 0.149$	$0.862\pm 0.017$
FedRot-LoRA	$\mathbf{0.954\pm 0.001}$	$\mathbf{0.926\pm 0.002}$	$\mathbf{0.876\pm 0.002}$

Table 6: Ablation comparing alternating and single-factor alignment. FedRot-LoRA performs best, while single-factor alignment degrades accuracy, especially when aligning

B

alone.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Pure agg.	Agg. space	Communication	Extra computation
FlexLoRA	Yes	Full-parameter space	LoRA factors $B,A$ only	Full parameter SVD: $O(d^{3})$
FedSRD	No	Full-parameter space	Sparsified factors $O(\rho dr),\rho<1$	Sparsification + SVD: $O(d^{3})$
FedEx-LoRA	Yes	Mixed $\bar{B}\,\bar{A}+\Delta W_{res}$	Residual matrix $\Delta W_{res}\in\mathbb{R}^{d\times d}$	$O\!\big(Nd^{2}r\big)$ to form residual matrix
LoRA-FAIR	Yes	Full-parameter space	LoRA factors $B,A$ only	$O(Td^{2}r)$ for $T$ steps gradient descent
FLoRA	Yes	Full-parameter space	Stacked $B\in\mathbb{R}^{d\times Nr},A\in\mathbb{R}^{Nr\times d}$	server stacking $O\!\big(Nrd\big)$
FedSA-LoRA	No	Factor $A$ only (personalized FL)	Factor $A$ only	$0$

Appendix

Table 7: Comparison of representative federated LoRA methods in terms of aggregation type, aggregation space, communication payload, and additional computation. “Pure agg.” indicates whether the method mainly modifies the aggregation rule without introducing personalization, compression/reconstruction, or residual-transmission components. Computational costs are reported per communication round unless otherwise specified.

Experiments	Clients Num	Rank	Data distributions	Optimal learning rates (FedIT/FFA-LoRA/RoLRA/FedRot-LoRA)		Optimal $\lambda$
GLUE results (Table 1 )	$N=3,10$	$r=4$	Dirichlet $h=0.5$	SST-2	$N=3:(0.02/0.02/0.0005/0.02)$ $N=10:(0.005/0.02/0.0005/0.005)$	$\lambda=0.6$ $\lambda=0.6$
				QNLI	$N=3:(0.02/0.02/0.001/0.005)$ $N=10:(0.02/0.02/0.0005/0.005)$	$\lambda=0.2$ $\lambda=0.2$
				QQP	$N=3:(0.005/0.02/0.001/0.005)$ $N=10:(0.005/0.02/0.0005/0.005)$	$\lambda=0.4$ $\lambda=0.6$
				RTE	$N=3:(0.005/0.02/0.0005/0.005)$ $N=10:(0.005/0.02/0.0005/0.005)$	$\lambda=0.2$ $\lambda=0.4$
				MNLI	$N=3:(0.02/0.02/0.001/0.005)$ $N=10:(0.005/0.02/0.001/0.005)$	$\lambda=0.4$ $\lambda=0.4$
MNLI with different ranks (Table 3 )	$N=3$	$r=4,8,16$	Dirichlet $h=0.5$	$r=4:(0.02/0.02/0.001/0.005)$ $r=8:(0.02/0.02/0.001/0.005)$ $r=16:(0.02/0.02/0.0005/0.005)$		$\lambda=0.4$ $\lambda=0.4$ $\lambda=0.4$

Appendix

Table 8: The hyperparameters for each experimental setting.

Methods	SST-2	MNLI	GSM8K
FedIT (No Alignment)	$0.953\pm 0.001$	$0.866\pm 0.001$	$0.4293\pm 0.018$
Scalar Rescaling	$0.953\pm 0.002$	$0.865\pm 0.006$	$0.4286\pm 0.021$
FedRot-LoRA (Rotation)	$0.954\pm 0.001$	$0.876\pm 0.002$	$0.4437\pm 0.009$

Appendix

Table 9: Comparison of scalar rescaling and rotational alignment. Results show that rotational alignment substantially outperforms rescaling in higher-dimensional LoRA settings.

Reference Model	SST-2	MNLI
Random client local model	$0.954\pm 0.002$	$0.871\pm 0.006$
Old global model ( $W^{t-2}$ )	$0.951\pm 0.001$	$0.866\pm 0.002$
Previous global model ( $W^{t-1}$ )	$0.954\pm 0.001$	$0.876\pm 0.002$

Appendix

Table 10: Effect of reference model selection.

Methods	SST-2	MNLI
Random Rotation	$0.541\pm 0.182$	$0.318\pm 0.000$
FedRot-LoRA	$0.954\pm 0.001$	$0.876\pm 0.002$

Appendix

Table 11: Comparison with random rotation. FedRot-LoRA outperforms random rotation on GLUE tasks.

Panel of Figure 4 — Figure 4: Effect of soft rotation level $\lambda$ in FedRot-LoRA on MNLI and QQP.

Figure 5: Alignment gain

\alpha(\lambda)

across training rounds. The minimum observed alignment gain is

\alpha(\lambda)\geq 0.00043

. The shaded background regions denote the active alignment target during each respective round. The experimental configuration utilizes the SST-2 dataset distributed across 3 clients, consistent with the settings detailed in Table 1.

Figure 6: Evolution of the maximum relative alignment discrepancy, given by $\max_{i}\|R_{i}^{t,*}-I\|_{F}/\|A_{i}^{t}-A_{\mathrm{ref}}\|_{F}$ , among all clients over the course of training rounds $t$ . The y-axis is presented on a logarithmic scale to capture the magnitude shifts. The experimental configuration utilizes the SST-2 dataset distributed across 3 clients, consistent with the settings detailed in Table 1.

Figure 7: Evolution of the minimum client distance to the global reference matrices, quantified by the Frobenius norms $\min_{i}\|A_{i}^{t}-A_{\mathrm{ref}}\|_{F}$ and $\min_{i}\|B_{i}^{t}-B_{\mathrm{ref}}\|_{F}$ , among all clients at each training round $t$ . The experimental configuration utilizes the SST-2 dataset distributed across 3 clients, consistent with the settings detailed in Table 1.

Federated Fine-Tuning

Federated fine-tuning adapts pretrained models across decentralized data holders without pooling their private data.

See all 115 papers

FedPA-LoRA: Product-Aligned Framework for Mitigating Aggregation and Initialization Errors in Heterogeneous Federated LoRA
Aug 15, 2026Juseok Jeon, Ramy E. Ali, Doyun Kwon +3Federated Fine-Tuning Federated Learning
DGIST Daegu, Republic of Korea · Samsung San Diego, USA
CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregation
Sep 27, 2026Yanan Ma, Qiyuan Chen, Zihan Fang +2Federated Fine-Tuning Federated Learning
City University of Hong Kong · The University of Hong Kong
Beyond Factor Aggregation: Gauge-Aware Low-Rank Server Representations for Federated LoRA
May 7, 2026Jinqian Chen, Chang Liu, Jihua ZhuFederated Learning Aggregation Low-Rank Adaptation
School of Software Engineering Xi’an Jiaotong University Xi’an, China 710000

Aug 15, 2026J/K move · Enter open · S save

FedPA-LoRA: Product-Aligned Framework for Mitigating Aggregation and Initialization Errors in Heterogeneous Federated LoRA

Juseok Jeon, Ramy E. Ali, Doyun Kwon, Myungbeom Her+2

DGIST Daegu, Republic of Korea · Samsung San Diego, USA

Federated Fine-Tuning Federated Learning Federated Learning Aggregation Low-Rank Adaptation Heterogeneous Federated Learning

Open paper AskPDF arXiv

Low-Rank Adaptation (LoRA) enables efficient federated fine-tuning of large language models, but its factorized parameterization creates a tension between accurate aggregation of local updates and continuity of locally optimized factors. Factor-wise aggregation incurs aggregation mismatch but better preserves factor continuity, whereas product-space reconstruction reduces this mismatch at the cost of greater factor-level initialization mismatch from newly reconstructed factors. We propose FedPA-LoRA, a product-aligned federated LoRA framework that jointly addresses these limitations and provably converges under both homogeneous and heterogeneous client ranks. Each client preserves its local factors across communication rounds and aligns its product toward a rank-specific global reference, maintaining local optimization continuity while promoting global consistency under data heterogeneity. The server aggregates heterogeneous-rank updates in the common product space and efficiently reconstructs a rank-constrained global adapter without forming the dense aggregate. This design supports client-specific computation and communication budgets. Experiments on natural language understanding and generation tasks show that FedPA-LoRA consistently outperforms representative baselines across varying levels of data heterogeneity and homogeneous- and heterogeneous-rank settings, with up to a

6.82

percentage-point improvement in average GLUE accuracy under heterogeneous client ranks.

Abstract

Federated Fine-Tuning

Explore similar work

FedPA-LoRA: Product-Aligned Framework for Mitigating Aggregation and Initialization Errors in Heterogeneous Federated LoRA

CFLoRA: Federated Fine-tuning of LLMs with Complementary Factors for Error-free Aggregation

Beyond Factor Aggregation: Gauge-Aware Low-Rank Server Representations for Federated LoRA