FedTreeLoRA: Reconciling Statistical and Functional Heterogeneity in Federated LoRA Fine-Tuning

Jieming Bian; Lei Wang; Letian Zhang; Jie Xu

cs.LGFeb 27, 2026

FedTreeLoRA: Reconciling Statistical and Functional Heterogeneity in Federated LoRA Fine-Tuning

Authors: Jieming Bian, Lei Wang, Letian Zhang, Jie Xu

Organizations: University of Florida, Gainesville, FL 32611 · Middle Tennessee State University Murfreesboro, TN 37132

Abstract

Federated Learning (FL) with Low-Rank Adaptation (LoRA) has become a standard for privacy-preserving LLM fine-tuning. However, existing personalized methods predominantly operated under a restrictive Flat-Model Assumption: they addressed client-side statistical heterogeneity but treated the model as a monolithic block, ignoring the functional heterogeneity across LLM layers. We argue that these two statistical (horizontal) and functional (vertical) dimensions, are orthogonal in source yet coupled in interaction, implying that the optimal depth of parameter sharing is functionally dependent on client similarity. To address this, we propose FedTreeLoRA, a framework employing tree-structured aggregation for fine-grained, layer-wise alignment. By dynamically constructing an aggregation hierarchy, FedTreeLoRA allows clients to share broad consensus on shallow trunks' while progressively specializing on deep branches'. Experiments on NLU and NLG benchmarks demonstrate that FedTreeLoRA significantly outperforms state-of-the-art methods by effectively reconciling generalization and personalization.

Figures & tables

Figure 1: Vertical Heterogeneity. Aggregating only shallow layers significantly outperforms aggregating deep layers.

Figure 2: The Coupling Effect of Dual Heterogeneity. As client distributions diverge (from Homogeneous to Heterogeneous), the optimal sharing boundary shifts from deep to shallow layers.

Figure 3: Overview of FedTreeLoRA. (1) Global Topological Structure Modeling : A hierarchy tree is built via AHC on client LoRA $B$ matrices during warmup to capture cross-client relationships. (2) Adaptive Layer-wise Alignment : For each layer $l$ , the optimal cluster count $c_{l}^{*}$ is dynamically selected under a monotonicity constraint. (3) Cluster-External Expert Mechanism : Each client synthesizes parameters by mixing a Cluster Expert with an External Expert via a learnable coefficient $\lambda_{l,k}$ .

Methods	Rank	% Param	MNLI	QNLI	SST2	QQP	Average	$\mathbf{\Delta}$
FedIT ( Zhang et al., 2024 )	4	$0.1107\%$	$83.18\pm 0.74$	$87.03\pm 0.43$	$93.65\pm 0.63$	$84.93\pm 0.59$	$87.20$	-
FFA-LoRA ( Sun et al., 2024 )	4	$0.0553\%$	$83.02\pm 0.42$	$87.47\pm 0.13$	$93.73\pm 0.18$	$83.48\pm 0.36$	$86.93$	$-0.27$
FFA-LoRA ( Sun et al., 2024 )	8	$0.1107\%$	$83.13\pm 0.56$	$88.90\pm 0.30$	$93.95\pm 0.20$	$83.62\pm 0.48$	$87.40$	$+0.20$
FedSA ( Guo et al., 2025 )	4	$0.1107\%$	$83.63\pm 0.69$	$91.32\pm 0.10$	$95.87\pm 0.30$	$89.33\pm 0.43$	$90.04$	$+2.84$
FedDPA ( Yang et al., 2024 )	4	$0.1107\%$	$83.97\pm 0.75$	$91.31\pm 0.51$	$95.72\pm 0.12$	$89.74\pm 0.58$	$90.19$	$+2.99$
FedALT ( Bian et al., 2026 )	4	$0.1383\%$	$84.03\pm 0.59$	$90.77\pm 0.46$	$96.16\pm 0.29$	$89.27\pm 0.50$	$90.06$	$+2.86$

Table 1: Performance comparison on GLUE benchmarks (RoBERTa-Large-355M) under non-IID settings (

\alpha=0.5

Methods	Rank	% Param	Text Edit	Struct2Text	Sentiment	Reasoning	Average	$\mathbf{\Delta}$
FedIT ( Zhang et al., 2024 )	8	$0.0622\%$	$59.84\pm 1.17$	$51.71\pm 1.14$	$44.53\pm 1.30$	$74.42\pm 0.74$	$57.62$	-
FFA-LoRA ( Sun et al., 2024 )	8	$0.0311\%$	$58.64\pm 0.96$	$51.83\pm 0.48$	$44.26\pm 0.31$	$73.62\pm 0.43$	$57.09$	$-0.53$
FFA-LoRA ( Sun et al., 2024 )	16	$0.0622\%$	$59.15\pm 0.16$	$52.85\pm 0.54$	$44.96\pm 2.22$	$73.77\pm 0.28$	$57.68$	$+0.06$
FedSA ( Guo et al., 2025 )	8	$0.0622\%$	$63.80\pm 1.88$	$54.48\pm 0.53$	$46.98\pm 0.55$	$74.37\pm 0.94$	$59.91$	$+2.29$
FedDPA ( Yang et al., 2024 )	8	$0.0622\%$	$64.33\pm 0.92$	$54.18\pm 1.12$	$48.13\pm 1.02$	$75.55\pm 1.43$	$60.55$	$+2.93$
FedALT ( Bian et al., 2026 )	8	$0.0699\%$	$67.61\pm 1.80$	$54.06\pm 1.51$	$48.57\pm 1.26$	$76.84\pm 1.48$	$61.77$	$+4.15$

Table 2: Performance comparison on FLAN benchmarks (LLaMA-2-7B). We report ROUGE-1 scores.

Figure 4: Layer-wise cluster counts (

c_{l}^{*}

) across different datasets. FedTreeLoRA adaptively identifies the optimal aggregation granularity specific to each data distribution.

Interaction Variant	Params $\uparrow$	Comm.	Avg. Acc.
Isolationist (Cluster-Only)	$0\%$	Low	$91.40$
Decomposed Experts	$\approx 0.028\%$	High	$\mathbf{92.57}$
MoE Router	$25\%$	Low	$92.02$
Scalar-Mixed (Ours)	$\approx 0.020\%$	Low	$\underline{92.36}$

Table 3: Ablation on interaction mechanisms. “Params

\uparrow

” denotes the relative increase in trainable parameters vs. standard LoRA.

Aggregation Depth	Constraint	Avg. Acc.
Fixed ( $k=1$ )	Global Only	$87.20$
Fixed ( $k=4$ )	Coarse Clusters	$91.45$
Fixed ( $k=8$ )	Fine-grained	$90.74$
Layer-wise Adaptive	Dynamic ( $c_{l}^{*}$ )	$\mathbf{92.36}$

Table 4: Comparison between Fixed-Depth aggregation (Flat-Model Assumption) and Layer-wise Adaptive aggregation.

Structural Prior	Consistency	Avg. Acc.
Independent Clustering	Low	$89.47$
Global Tree Skeleton	High	$\mathbf{92.36}$

Table 5: Impact of the Global Tree Skeleton on performance.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Extended Motivational Studies. (a) Substantiates Observation 1 (Vertical Heterogeneity) on SST2 and QQP datasets. (b)–(d) Substantiate Observation 2 (Coupling Effect) across different tasks, confirming that the optimal sharing boundary consistently shifts towards shallower layers as data heterogeneity increases.

Figure 6: Sensitivity analysis of hyperparameters

\tau

and

K

. FedTreeLoRA demonstrates a clear robustness plateau within

\tau\in[0,0.06]

and

K\in\{3,4,5\}

, while still outperforming all baselines even in suboptimal settings.

Methods	MNLI	QNLI	SST2	QQP	Average
FedIT ( Zhang et al., 2024 )	$82.47\pm 0.36$	$86.36\pm 0.37$	$93.01\pm 0.99$	$83.83\pm 0.60$	$86.42$
FFA-LoRA ( $r=4$ ) ( Sun et al., 2024 )	$82.68\pm 0.49$	$90.32\pm 0.25$	$93.32\pm 0.10$	$83.16\pm 0.42$	$87.37$
FFA-LoRA ( $r=8$ ) ( Sun et al., 2024 )	$82.75\pm 0.59$	$90.73\pm 0.64$	$93.82\pm 0.18$	$83.66\pm 0.17$	$87.74$
FedSA ( Guo et al., 2025 )	$84.12\pm 0.64$	$92.15\pm 0.45$	$96.07\pm 0.41$	$89.15\pm 0.10$	$90.37$
FedDPA ( Yang et al., 2024 )	$84.69\pm 0.54$	$92.20\pm 0.53$	$95.91\pm 0.40$	$89.50\pm 0.25$	$90.58$
FedALT ( Bian et al., 2026 )	$86.28\pm 0.25$	$91.70\pm 0.38$	$96.51\pm 0.20$	$88.97\pm 0.53$	$90.87$

Appendix

Table 6: Performance with increased Local Epochs (

E=4

Methods	MNLI	QNLI	SST2	QQP	Average
FedIT ( Zhang et al., 2024 )	$81.08\pm 0.46$	$86.15\pm 0.58$	$92.42\pm 0.50$	$83.78\pm 0.46$	$85.86$
FFA-LoRA ( Sun et al., 2024 )	$81.74\pm 0.56$	$86.41\pm 0.19$	$93.13\pm 0.37$	$83.16\pm 0.73$	$86.11$
FedSA ( Guo et al., 2025 )	$82.55\pm 0.90$	$90.79\pm 0.27$	$95.24\pm 0.55$	$89.33\pm 0.49$	$89.48$
FedDPA ( Yang et al., 2024 )	$82.74\pm 0.60$	$91.07\pm 0.33$	$95.16\pm 0.19$	$89.69\pm 0.67$	$89.67$
FedALT ( Bian et al., 2026 )	$84.45\pm 0.30$	$90.90\pm 0.18$	$95.83\pm 0.17$	$89.49\pm 0.54$	$90.17$
FedLEASE ( Wang et al., 2025 )	$85.49\pm 0.29$	$92.77\pm 0.20$	$95.02\pm 0.27$	$90.35\pm 0.38$	$90.91$

Appendix

Table 7: Performance comparison under different LoRA Ranks (

r=2

Methods	MNLI	QNLI	SST2	QQP	Average
FedIT ( Zhang et al., 2024 )	$82.84\pm 0.24$	$87.78\pm 0.52$	$93.61\pm 0.52$	$84.17\pm 0.29$	$87.10$
FFA-LoRA ( Sun et al., 2024 )	$82.78\pm 0.93$	$88.43\pm 0.58$	$93.87\pm 0.69$	$83.49\pm 0.78$	$87.14$
FedSA ( Guo et al., 2025 )	$83.70\pm 0.24$	$91.30\pm 0.85$	$96.05\pm 0.14$	$89.83\pm 0.67$	$90.22$
FedDPA ( Yang et al., 2024 )	$84.03\pm 0.53$	$91.20\pm 0.82$	$96.02\pm 0.34$	$89.45\pm 0.53$	$90.18$
FedALT ( Bian et al., 2026 )	$81.45\pm 0.41$	$91.93\pm 0.13$	$96.12\pm 0.42$	$89.50\pm 0.18$	$89.75$
FedLEASE ( Wang et al., 2025 )	$86.03\pm 0.15$	$93.34\pm 0.36$	$95.25\pm 0.79$	$90.66\pm 0.20$	$91.32$

Appendix

Table 8: Performance comparison under different LoRA Ranks (

r=6

Methods	MNLI	QNLI	SST2	QQP	Average
FedIT ( Zhang et al., 2024 )	$82.33\pm 0.50$	$89.20\pm 0.35$	$93.14\pm 0.21$	$84.91\pm 0.57$	$87.40$
FFA-LoRA (r = 4) ( Sun et al., 2024 )	$82.40\pm 0.89$	$89.68\pm 0.85$	$93.34\pm 0.16$	$84.93\pm 0.74$	$87.59$
FFA-LoRA (r = 8) ( Sun et al., 2024 )	$83.33\pm 0.35$	$90.10\pm 0.21$	$93.60\pm 0.48$	$85.43\pm 0.73$	$88.12$
FedSA ( Guo et al., 2025 )	$83.52\pm 0.92$	$90.78\pm 0.35$	$94.75\pm 0.63$	$91.35\pm 0.17$	$90.10$
FedDPA ( Yang et al., 2024 )	$83.73\pm 0.30$	$90.25\pm 0.47$	$94.91\pm 0.23$	$91.15\pm 0.13$	$90.01$
FedALT ( Bian et al., 2026 )	$83.50\pm 0.71$	$90.68\pm 0.60$	$94.72\pm 0.49$	$91.14\pm 0.16$	$90.01$

Appendix

Table 9: Performance comparison under different Client Numbers (

N=10

Methods	MNLI	QNLI	SST2	QQP	Average
FedIT ( Zhang et al., 2024 )	$79.28\pm 0.56$	$85.39\pm 0.74$	$91.99\pm 0.71$	$80.05\pm 0.97$	$84.18$
FFA-LoRA (r = 4) ( Sun et al., 2024 )	$81.71\pm 0.46$	$86.62\pm 0.79$	$91.97\pm 0.74$	$82.41\pm 0.72$	$85.68$
FFA-LoRA (r = 8) ( Sun et al., 2024 )	$82.35\pm 0.48$	$86.98\pm 0.62$	$92.74\pm 0.85$	$82.71\pm 0.62$	$86.20$
FedSA ( Guo et al., 2025 )	$83.88\pm 0.41$	$89.13\pm 0.31$	$94.08\pm 0.54$	$85.96\pm 0.47$	$88.26$
FedDPA ( Yang et al., 2024 )	$83.45\pm 0.09$	$88.53\pm 0.77$	$93.52\pm 0.78$	$85.63\pm 0.69$	$87.78$
FedALT ( Bian et al., 2026 )	$82.53\pm 0.44$	$89.11\pm 0.33$	$91.78\pm 0.42$	$85.68\pm 0.05$	$87.28$

Appendix

Table 10: Performance comparison under different Client Numbers (

N=40

Methods	MNLI	QNLI	SST2	QQP	Average
FedIT ( Zhang et al., 2024 )	$83.13\pm 0.20$	$88.04\pm 0.56$	$93.67\pm 0.61$	$83.88\pm 0.36$	$87.18$
FFA-LoRA (r = 4) ( Sun et al., 2024 )	$83.41\pm 0.83$	$87.82\pm 0.99$	$94.12\pm 0.33$	$83.78\pm 0.53$	$87.28$
FFA-LoRA (r = 8) ( Sun et al., 2024 )	$83.69\pm 0.13$	$88.94\pm 0.23$	$94.27\pm 0.60$	$84.13\pm 0.23$	$87.76$
FedSA ( Guo et al., 2025 )	$85.32\pm 0.08$	$89.17\pm 0.97$	$94.50\pm 0.40$	$87.89\pm 0.39$	$89.22$
FedDPA ( Yang et al., 2024 )	$85.38\pm 0.21$	$89.93\pm 0.33$	$94.74\pm 0.63$	$88.07\pm 0.52$	$89.53$
FedALT ( Bian et al., 2026 )	$85.85\pm 0.88$	$90.10\pm 0.45$	$94.47\pm 0.29$	$87.95\pm 0.11$	$89.59$

Appendix

Table 11: Performance comparison under different Label Skew (

\alpha=1

Methods	MNLI	QNLI	SST2	QQP	Average
FedIT ( Zhang et al., 2024 )	$81.90\pm 0.15$	$87.28\pm 0.29$	$93.99\pm 0.10$	$82.87\pm 0.60$	$86.51$
FFA-LoRA (r = 4) ( Sun et al., 2024 )	$83.11\pm 0.31$	$89.18\pm 0.76$	$94.30\pm 0.93$	$83.23\pm 0.79$	$87.46$
FFA-LoRA (r = 8) ( Sun et al., 2024 )	$83.32\pm 0.93$	$89.60\pm 0.27$	$94.56\pm 0.36$	$83.51\pm 0.56$	$87.75$
FedSA ( Guo et al., 2025 )	$84.73\pm 0.39$	$89.84\pm 0.22$	$94.68\pm 0.39$	$88.98\pm 0.27$	$89.56$
FedDPA ( Yang et al., 2024 )	$84.19\pm 0.36$	$89.69\pm 0.39$	$94.79\pm 0.51$	$89.13\pm 0.39$	$89.45$
FedALT ( Bian et al., 2026 )	$83.23\pm 0.87$	$89.55\pm 0.11$	$90.86\pm 0.51$	$88.71\pm 0.19$	$88.09$

Appendix

Table 12: Performance comparison under different Label Skew (

\alpha=0.7

Figure 7: Layer-wise client clustering under task heterogeneity. Clients assigned to the same GLUE task form stable branches at deeper layers, while all tasks share common representations in shallow layers, illustrating the hierarchical specialization discovered by FedTreeLoRA.

Methods	MNLI	QNLI	SST2	QQP	Average
FedIT ( Zhang et al., 2024 )	$71.46\pm 1.74$	$83.71\pm 1.95$	$93.25\pm 0.31$	$78.54\pm 3.03$	$81.74$
FFA-LoRA ( Sun et al., 2024 )	$72.17\pm 2.74$	$82.75\pm 2.26$	$93.79\pm 0.68$	$81.25\pm 2.56$	$82.49$
FedSA ( Guo et al., 2025 )	$74.29\pm 1.09$	$86.33\pm 0.75$	$93.87\pm 0.18$	$82.12\pm 0.74$	$84.15$
FedDPA ( Yang et al., 2024 )	$75.00\pm 1.70$	$86.29\pm 0.16$	$93.58\pm 0.16$	$81.92\pm 2.01$	$84.20$
FedALT ( Bian et al., 2026 )	$73.25\pm 1.40$	$83.00\pm 1.35$	$93.69\pm 0.19$	$84.31\pm 0.44$	$83.56$
FedLEASE ( Wang et al., 2025 )	$79.91\pm 1.86$	$87.54\pm 1.35$	$94.17\pm 0.37$	$83.46\pm 0.79$	$86.27$

Appendix

Table 13: Performance under Task Heterogeneity (

N=16

clients, 4 per task).

Metric	MNLI	QNLI	SST2	QQP	Average
Cosine Distance	$88.29\pm 0.30$	$93.47\pm 0.25$	$96.41\pm 0.02$	$91.61\pm 0.09$	92.45
Frobenius Distance	$88.15\pm 0.25$	$93.37\pm 0.62$	$96.56\pm 0.07$	$91.35\pm 0.17$	92.36

Appendix

Table 14: Sensitivity analysis of distance metrics on GLUE benchmarks. Results demonstrate that FedTreeLoRA is metric-agnostic, maintaining high performance regardless of the specific similarity definition.

Warm-up Epochs	MNLI	QNLI	SST2	QQP	Average
5	82.94	89.31	94.19	84.75	87.80
10	82.66	89.83	94.62	84.13	87.81

Appendix

Table 15: Sensitivity analysis of warm-up duration on GLUE benchmarks. Results show that FedTreeLoRA remains highly stable across different warm-up stages, indicating that reliable client topology can be identified early during adaptation.

Join Round	Original Clients	Existing Clients	New Client	Local Only
10	87.90	87.43	84.50	82.00
20	87.90	87.50	86.50	79.00

Appendix

Table 16: Robustness to dynamic client participation on MNLI. Results show that integrating new clients causes negligible degradation to existing clients while significantly improving the performance of the newly joined client compared to local-only training.

Interaction Variant	MNLI	QNLI	SST2	QQP	Average
Isolationist (Cluster-Only)	$86.60\pm 0.21$	$92.57\pm 0.44$	$95.96\pm 0.39$	$90.48\pm 0.39$	$91.40$
Decomposed Experts	$\mathbf{88.46\pm 0.23}$	$\mathbf{93.91\pm 0.36}$	$96.54\pm 0.04$	$\mathbf{91.35\pm 0.13}$	$\mathbf{92.57}$
MoE Router	$87.92\pm 0.82$	$93.35\pm 0.58$	$96.15\pm 0.46$	$90.67\pm 0.45$	$92.02$
VA Scalar-Mixed	$88.15\pm 0.25$	$93.37\pm 0.62$	$\mathbf{96.56\pm 0.07}$	$\mathbf{91.35\pm 0.17}$	$\underline{92.36}$

Appendix

Table 17: Detailed performance of different Interaction Mechanisms across GLUE tasks. Accuracy is reported in (%).

Aggregation Depth	MNLI	QNLI	SST2	QQP	Average
Fixed ( $k=1$ , Global Only)	$83.18\pm 0.74$	$87.03\pm 0.43$	$93.65\pm 0.63$	$84.93\pm 0.59$	$87.20$
Fixed ( $k=4$ , Coarse)	$86.75\pm 0.63$	$92.95\pm 0.22$	$96.13\pm 0.16$	$89.98\pm 0.59$	$91.45$
Fixed ( $k=8$ , Fine-grained)	$85.81\pm 0.68$	$92.02\pm 0.71$	$95.17\pm 0.13$	$89.96\pm 0.89$	$90.74$
Layer-wise Adaptive ( $c_{l}^{*}$ )	$\mathbf{88.15\pm 0.25}$	$\mathbf{93.37\pm 0.62}$	$\mathbf{96.56\pm 0.07}$	$\mathbf{91.35\pm 0.17}$	$\mathbf{92.36}$

Appendix

Table 18: Detailed performance of Fixed-Depth vs. Layer-wise Adaptive aggregation.

Structural Prior	MNLI	QNLI	SST2	QQP	Average
Independent Clustering	$83.84\pm 0.32$	$89.74\pm 1.95$	$95.06\pm 0.21$	$89.23\pm 1.56$	$89.47$
Global Tree Skeleton	$\mathbf{88.15\pm 0.25}$	$\mathbf{93.37\pm 0.62}$	$\mathbf{96.56\pm 0.07}$	$\mathbf{91.35\pm 0.17}$	$\mathbf{92.36}$

Appendix

Table 19: Detailed impact of Global Structural Consistency on individual datasets.

Time (s)	FedIT	FFA-LoRA	FedSA	FedDPA	FedALT	FedLEASE	FedTreeLoRA
Local Training (Per Epoch)	1.0545	1.0369	1.0582	1.0668	1.2091	1.1883	1.0716
Global Aggregation	0.0628	0.0594	0.0693	0.0723	0.0782	0.0908	0.0770
Topology/Clustering Time	-	-	-	-	-	3.97	4.21
Total Training Time	65.154	63.996	65.571	66.177	74.892	77.992	70.816

Appendix

Table 20: Comparison of Computational Overhead. All times are measured in seconds (s).

Methods	Text Edit	Struct2Text	Sentiment	Reasoning	Average
FedIT ( Zhang et al., 2024 )	$77.21\pm 1.39$	$51.80\pm 0.89$	$49.73\pm 0.36$	$72.37\pm 0.58$	$62.78$
FFA-LoRA (r = 8) ( Sun et al., 2024 )	$77.05\pm 1.35$	$51.10\pm 0.66$	$48.72\pm 1.49$	$72.32\pm 0.46$	$62.30$
FFA-LoRA (r = 16) ( Sun et al., 2024 )	$78.77\pm 1.81$	$51.20\pm 0.60$	$49.05\pm 0.84$	$72.12\pm 0.86$	$62.79$
FedSA ( Guo et al., 2025 )	$84.93\pm 1.34$	$52.93\pm 0.23$	$49.38\pm 0.84$	$72.64\pm 1.18$	$64.94$
FedDPA ( Yang et al., 2024 )	$88.56\pm 0.86$	$53.78\pm 0.36$	$51.31\pm 0.51$	$72.16\pm 1.78$	$66.45$
FedALT ( Bian et al., 2026 )	$88.32\pm 0.19$	$53.58\pm 0.19$	$50.97\pm 1.13$	$73.22\pm 1.45$	$66.52$

Appendix

Table 21: Performance comparison on NLG benchmarks using BLOOM. We report ROUGE-1 scores averaged over three independent runs.

Figure 8: Layer-wise average pairwise Frobenius distance between two clients on QNLI under IID (identical label distributions) and Non-IID (

[0.8,0.2]

vs.

[0.2,0.8]

) settings. The upper-left and upper-right panels (Proj

A

at Query and Value) show nearly overlapping IID and Non-IID curves across all layers, indicating that

A

captures distribution-invariant shared representations. The lower-left and lower-right panels (Proj

B

at Query and Value) exhibit a progressively widening gap between IID and Non-IID as depth increases, revealing that client-specific semantics emerge primarily in deeper layers through the LoRA

B

matrices. This validates using

B

-matrix distances to quantify heterogeneity and to guide hierarchical, layer-wise aggregation in FedTreeLoRA.

Federated Fine-Tuning

Federated fine-tuning adapts pretrained models across decentralized data holders without pooling their private data.

See all 115 papers

Sep 29, 2026

FedLAFP: Low-Rank Aggregation Meets Full-Rank Personalization in Federated Fine-Tuning

Federated parameter-efficient fine-tuning enables clients to adapt pre-trained models without sharing raw data or communicating the full model, but statistical heterogeneity makes a single global adapter insufficient for personalized prediction. Existing personalized methods typically use the same low-rank structure for both shared and private adaptation, overlooking their distinct requirements for aggregation and personalization. We propose FedLAFP, a role-aware framework that couples a compact, globally aggregated LoRA branch with a client-private, full-rank-capable RandLoRA branch. The shared branch provides an efficient interface for transferring common knowledge, whereas the private branch combines fixed random low-rank bases with learned scaling coefficients to provide expressive client-specific adaptation without additional communication. Client- and layer-specific mixing coefficients jointly fuse the two branches, and only the shared LoRA parameters are exchanged. A controlled linear study supports this role assignment: LoRA yields more aligned client updates and lower aggregation error, while RandLoRA more accurately recovers client-specific residuals. Experiments across four visual recognition benchmarks show that FedLAFP consistently outperforms local-only and federated LoRA baselines, achieving an average personalized accuracy of

86.93\%

and exceeding the best baseline average by

1.30

percentage points.

Mengjun Yi, Huaian Gu, Yinghao Ai +2

State Key Laboratory for Novel Software Technology · School of Artificial Intelligence · School of Computer Science +1

Low-Rank Adaptation Federated Fine-Tuning Federated Learning Parameter-Efficient Fine-Tuning

Ask PDF

Sep 1, 2026

Breaking the Structural Identity: Personalized Federated LoRA Fine-tuning under Rank Heterogeneity

Large Language Models (LLMs) have achieved remarkable success across diverse domains, but their adaptation to privacy-sensitive, distributed datasets remains a challenge. While Federated Learning (FL) combined with Low-Rank Adaptation (LoRA) provides a resource-efficient paradigm for collaborative fine-tuning, practical deployments are hindered by the dual challenges of resource heterogeneity and data heterogeneity. Existing rank-heterogeneous methods primarily focus on bridging dimension mismatches for aggregation but typically provide a unified global model for all clients sharing the same rank, failing to capture client-specific features in non-IID scenarios. In this paper, we propose FedRoRA (Federated Rank-wise Personalized LoRA), a novel framework that enables fine-grained personalization within rank-heterogeneous federations. FedRoRA decouples adaptation into shared global directions and personalized rank-wise magnitudes governed by learnable diagonal scales. On the server side, it extracts a global subspace via singular value decomposition (SVD) and redistributes client-specific initializations through a personalized projection and top-

k

selection mechanism. Extensive experiments on NLU and NLG benchmarks demonstrate that FedRoRA consistently outperforms state-of-the-art methods.

Lei Wang, Jieming Bian, Letian Zhang +1

University of Florida · Gainesville, FL 32611 · Middle Tennessee State University +1

Non-IID Federated Learning Low-Rank Adaptation Federated Fine-Tuning Federated Learning Heterogeneous Federated Learning

Ask PDF

Aug 15, 2026

FedPA-LoRA: Product-Aligned Framework for Mitigating Aggregation and Initialization Errors in Heterogeneous Federated LoRA

Low-Rank Adaptation (LoRA) enables efficient federated fine-tuning of large language models, but its factorized parameterization creates a tension between accurate aggregation of local updates and continuity of locally optimized factors. Factor-wise aggregation incurs aggregation mismatch but better preserves factor continuity, whereas product-space reconstruction reduces this mismatch at the cost of greater factor-level initialization mismatch from newly reconstructed factors. We propose FedPA-LoRA, a product-aligned federated LoRA framework that jointly addresses these limitations and provably converges under both homogeneous and heterogeneous client ranks. Each client preserves its local factors across communication rounds and aligns its product toward a rank-specific global reference, maintaining local optimization continuity while promoting global consistency under data heterogeneity. The server aggregates heterogeneous-rank updates in the common product space and efficiently reconstructs a rank-constrained global adapter without forming the dense aggregate. This design supports client-specific computation and communication budgets. Experiments on natural language understanding and generation tasks show that FedPA-LoRA consistently outperforms representative baselines across varying levels of data heterogeneity and homogeneous- and heterogeneous-rank settings, with up to a

6.82

percentage-point improvement in average GLUE accuracy under heterogeneous client ranks.

Juseok Jeon, Ramy E. Ali, Doyun Kwon +3

DGIST Daegu, Republic of Korea · Samsung San Diego, USA

Federated Fine-Tuning Federated Learning Federated Learning Aggregation Low-Rank Adaptation Heterogeneous Federated Learning

Ask PDF

Abstract

Federated Fine-Tuning

Explore similar work

FedLAFP: Low-Rank Aggregation Meets Full-Rank Personalization in Federated Fine-Tuning

Breaking the Structural Identity: Personalized Federated LoRA Fine-tuning under Rank Heterogeneity

FedPA-LoRA: Product-Aligned Framework for Mitigating Aggregation and Initialization Errors in Heterogeneous Federated LoRA