FedTreeLoRA: Reconciling Statistical and Functional Heterogeneity in Federated LoRA Fine-Tuning

Jieming Bian; Lei Wang; Letian Zhang; Jie Xu

cs.LGFeb 27, 2026

FedTreeLoRA: Reconciling Statistical and Functional Heterogeneity in Federated LoRA Fine-Tuning

Authors: Jieming Bian, Lei Wang, Letian Zhang, Jie Xu

Organizations: University of Florida, Gainesville, FL 32611 · Middle Tennessee State University Murfreesboro, TN 37132

Abstract

Federated Learning (FL) with Low-Rank Adaptation (LoRA) has become a standard for privacy-preserving LLM fine-tuning. However, existing personalized methods predominantly operated under a restrictive Flat-Model Assumption: they addressed client-side statistical heterogeneity but treated the model as a monolithic block, ignoring the functional heterogeneity across LLM layers. We argue that these two statistical (horizontal) and functional (vertical) dimensions, are orthogonal in source yet coupled in interaction, implying that the optimal depth of parameter sharing is functionally dependent on client similarity. To address this, we propose FedTreeLoRA, a framework employing tree-structured aggregation for fine-grained, layer-wise alignment. By dynamically constructing an aggregation hierarchy, FedTreeLoRA allows clients to share broad consensus on shallow trunks' while progressively specializing on deep branches'. Experiments on NLU and NLG benchmarks demonstrate that FedTreeLoRA significantly outperforms state-of-the-art methods by effectively reconciling generalization and personalization.

Figures & tables

Figure 1: Vertical Heterogeneity. Aggregating only shallow layers significantly outperforms aggregating deep layers.

Figure 2: The Coupling Effect of Dual Heterogeneity. As client distributions diverge (from Homogeneous to Heterogeneous), the optimal sharing boundary shifts from deep to shallow layers.

Figure 3: Overview of FedTreeLoRA. (1) Global Topological Structure Modeling : A hierarchy tree is built via AHC on client LoRA $B$ matrices during warmup to capture cross-client relationships. (2) Adaptive Layer-wise Alignment : For each layer $l$ , the optimal cluster count $c_{l}^{*}$ is dynamically selected under a monotonicity constraint. (3) Cluster-External Expert Mechanism : Each client synthesizes parameters by mixing a Cluster Expert with an External Expert via a learnable coefficient $\lambda_{l,k}$ .

Methods	Rank	% Param	MNLI	QNLI	SST2	QQP	Average	$\mathbf{\Delta}$
FedIT ( Zhang et al., 2024 )	4	$0.1107\%$	$83.18\pm 0.74$	$87.03\pm 0.43$	$93.65\pm 0.63$	$84.93\pm 0.59$	$87.20$	-
FFA-LoRA ( Sun et al., 2024 )	4	$0.0553\%$	$83.02\pm 0.42$	$87.47\pm 0.13$	$93.73\pm 0.18$	$83.48\pm 0.36$	$86.93$	$-0.27$
FFA-LoRA ( Sun et al., 2024 )	8	$0.1107\%$	$83.13\pm 0.56$	$88.90\pm 0.30$	$93.95\pm 0.20$	$83.62\pm 0.48$	$87.40$	$+0.20$
FedSA ( Guo et al., 2025 )	4	$0.1107\%$	$83.63\pm 0.69$	$91.32\pm 0.10$	$95.87\pm 0.30$	$89.33\pm 0.43$	$90.04$	$+2.84$
FedDPA ( Yang et al., 2024 )	4	$0.1107\%$	$83.97\pm 0.75$	$91.31\pm 0.51$	$95.72\pm 0.12$	$89.74\pm 0.58$	$90.19$	$+2.99$
FedALT ( Bian et al., 2026 )	4	$0.1383\%$	$84.03\pm 0.59$	$90.77\pm 0.46$	$96.16\pm 0.29$	$89.27\pm 0.50$	$90.06$	$+2.86$

Table 1: Performance comparison on GLUE benchmarks (RoBERTa-Large-355M) under non-IID settings (

\alpha=0.5

Methods	Rank	% Param	Text Edit	Struct2Text	Sentiment	Reasoning	Average	$\mathbf{\Delta}$
FedIT ( Zhang et al., 2024 )	8	$0.0622\%$	$59.84\pm 1.17$	$51.71\pm 1.14$	$44.53\pm 1.30$	$74.42\pm 0.74$	$57.62$	-
FFA-LoRA ( Sun et al., 2024 )	8	$0.0311\%$	$58.64\pm 0.96$	$51.83\pm 0.48$	$44.26\pm 0.31$	$73.62\pm 0.43$	$57.09$	$-0.53$
FFA-LoRA ( Sun et al., 2024 )	16	$0.0622\%$	$59.15\pm 0.16$	$52.85\pm 0.54$	$44.96\pm 2.22$	$73.77\pm 0.28$	$57.68$	$+0.06$
FedSA ( Guo et al., 2025 )	8	$0.0622\%$	$63.80\pm 1.88$	$54.48\pm 0.53$	$46.98\pm 0.55$	$74.37\pm 0.94$	$59.91$	$+2.29$
FedDPA ( Yang et al., 2024 )	8	$0.0622\%$	$64.33\pm 0.92$	$54.18\pm 1.12$	$48.13\pm 1.02$	$75.55\pm 1.43$	$60.55$	$+2.93$
FedALT ( Bian et al., 2026 )	8	$0.0699\%$	$67.61\pm 1.80$	$54.06\pm 1.51$	$48.57\pm 1.26$	$76.84\pm 1.48$	$61.77$	$+4.15$

Table 2: Performance comparison on FLAN benchmarks (LLaMA-2-7B). We report ROUGE-1 scores.

Figure 4: Layer-wise cluster counts (

c_{l}^{*}

) across different datasets. FedTreeLoRA adaptively identifies the optimal aggregation granularity specific to each data distribution.

Interaction Variant	Params $\uparrow$	Comm.	Avg. Acc.
Isolationist (Cluster-Only)	$0\%$	Low	$91.40$
Decomposed Experts	$\approx 0.028\%$	High	$\mathbf{92.57}$
MoE Router	$25\%$	Low	$92.02$
Scalar-Mixed (Ours)	$\approx 0.020\%$	Low	$\underline{92.36}$

Table 3: Ablation on interaction mechanisms. “Params

\uparrow

” denotes the relative increase in trainable parameters vs. standard LoRA.

Aggregation Depth	Constraint	Avg. Acc.
Fixed ( $k=1$ )	Global Only	$87.20$
Fixed ( $k=4$ )	Coarse Clusters	$91.45$
Fixed ( $k=8$ )	Fine-grained	$90.74$
Layer-wise Adaptive	Dynamic ( $c_{l}^{*}$ )	$\mathbf{92.36}$

Table 4: Comparison between Fixed-Depth aggregation (Flat-Model Assumption) and Layer-wise Adaptive aggregation.

Structural Prior	Consistency	Avg. Acc.
Independent Clustering	Low	$89.47$
Global Tree Skeleton	High	$\mathbf{92.36}$

Table 5: Impact of the Global Tree Skeleton on performance.

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 5 — Figure 5: Extended Motivational Studies. (a) Substantiates Observation 1 (Vertical Heterogeneity) on SST2 and QQP datasets. (b)–(d) Substantiate Observation 2 (Coupling Effect) across different tasks, confirming that the optimal sharing boundary consistently shifts towards shallower layers as data heterogeneity increases.

Figure 6: Sensitivity analysis of hyperparameters

\tau

and

K

. FedTreeLoRA demonstrates a clear robustness plateau within

\tau\in[0,0.06]

and

K\in\{3,4,5\}

, while still outperforming all baselines even in suboptimal settings.

Methods	MNLI	QNLI	SST2	QQP	Average
FedIT ( Zhang et al., 2024 )	$82.47\pm 0.36$	$86.36\pm 0.37$	$93.01\pm 0.99$	$83.83\pm 0.60$	$86.42$
FFA-LoRA ( $r=4$ ) ( Sun et al., 2024 )	$82.68\pm 0.49$	$90.32\pm 0.25$	$93.32\pm 0.10$	$83.16\pm 0.42$	$87.37$
FFA-LoRA ( $r=8$ ) ( Sun et al., 2024 )	$82.75\pm 0.59$	$90.73\pm 0.64$	$93.82\pm 0.18$	$83.66\pm 0.17$	$87.74$
FedSA ( Guo et al., 2025 )	$84.12\pm 0.64$	$92.15\pm 0.45$	$96.07\pm 0.41$	$89.15\pm 0.10$	$90.37$
FedDPA ( Yang et al., 2024 )	$84.69\pm 0.54$	$92.20\pm 0.53$	$95.91\pm 0.40$	$89.50\pm 0.25$	$90.58$
FedALT ( Bian et al., 2026 )	$86.28\pm 0.25$	$91.70\pm 0.38$	$96.51\pm 0.20$	$88.97\pm 0.53$	$90.87$

Appendix

Table 6: Performance with increased Local Epochs (

E=4

Methods	MNLI	QNLI	SST2	QQP	Average
FedIT ( Zhang et al., 2024 )	$81.08\pm 0.46$	$86.15\pm 0.58$	$92.42\pm 0.50$	$83.78\pm 0.46$	$85.86$
FFA-LoRA ( Sun et al., 2024 )	$81.74\pm 0.56$	$86.41\pm 0.19$	$93.13\pm 0.37$	$83.16\pm 0.73$	$86.11$
FedSA ( Guo et al., 2025 )	$82.55\pm 0.90$	$90.79\pm 0.27$	$95.24\pm 0.55$	$89.33\pm 0.49$	$89.48$
FedDPA ( Yang et al., 2024 )	$82.74\pm 0.60$	$91.07\pm 0.33$	$95.16\pm 0.19$	$89.69\pm 0.67$	$89.67$
FedALT ( Bian et al., 2026 )	$84.45\pm 0.30$	$90.90\pm 0.18$	$95.83\pm 0.17$	$89.49\pm 0.54$	$90.17$
FedLEASE ( Wang et al., 2025 )	$85.49\pm 0.29$	$92.77\pm 0.20$	$95.02\pm 0.27$	$90.35\pm 0.38$	$90.91$

Appendix

Table 7: Performance comparison under different LoRA Ranks (

r=2

Methods	MNLI	QNLI	SST2	QQP	Average
FedIT ( Zhang et al., 2024 )	$82.84\pm 0.24$	$87.78\pm 0.52$	$93.61\pm 0.52$	$84.17\pm 0.29$	$87.10$
FFA-LoRA ( Sun et al., 2024 )	$82.78\pm 0.93$	$88.43\pm 0.58$	$93.87\pm 0.69$	$83.49\pm 0.78$	$87.14$
FedSA ( Guo et al., 2025 )	$83.70\pm 0.24$	$91.30\pm 0.85$	$96.05\pm 0.14$	$89.83\pm 0.67$	$90.22$
FedDPA ( Yang et al., 2024 )	$84.03\pm 0.53$	$91.20\pm 0.82$	$96.02\pm 0.34$	$89.45\pm 0.53$	$90.18$
FedALT ( Bian et al., 2026 )	$81.45\pm 0.41$	$91.93\pm 0.13$	$96.12\pm 0.42$	$89.50\pm 0.18$	$89.75$
FedLEASE ( Wang et al., 2025 )	$86.03\pm 0.15$	$93.34\pm 0.36$	$95.25\pm 0.79$	$90.66\pm 0.20$	$91.32$

Appendix

Table 8: Performance comparison under different LoRA Ranks (

r=6

Methods	MNLI	QNLI	SST2	QQP	Average
FedIT ( Zhang et al., 2024 )	$82.33\pm 0.50$	$89.20\pm 0.35$	$93.14\pm 0.21$	$84.91\pm 0.57$	$87.40$
FFA-LoRA (r = 4) ( Sun et al., 2024 )	$82.40\pm 0.89$	$89.68\pm 0.85$	$93.34\pm 0.16$	$84.93\pm 0.74$	$87.59$
FFA-LoRA (r = 8) ( Sun et al., 2024 )	$83.33\pm 0.35$	$90.10\pm 0.21$	$93.60\pm 0.48$	$85.43\pm 0.73$	$88.12$
FedSA ( Guo et al., 2025 )	$83.52\pm 0.92$	$90.78\pm 0.35$	$94.75\pm 0.63$	$91.35\pm 0.17$	$90.10$
FedDPA ( Yang et al., 2024 )	$83.73\pm 0.30$	$90.25\pm 0.47$	$94.91\pm 0.23$	$91.15\pm 0.13$	$90.01$
FedALT ( Bian et al., 2026 )	$83.50\pm 0.71$	$90.68\pm 0.60$	$94.72\pm 0.49$	$91.14\pm 0.16$	$90.01$

Appendix

Table 9: Performance comparison under different Client Numbers (

N=10

Methods	MNLI	QNLI	SST2	QQP	Average
FedIT ( Zhang et al., 2024 )	$79.28\pm 0.56$	$85.39\pm 0.74$	$91.99\pm 0.71$	$80.05\pm 0.97$	$84.18$
FFA-LoRA (r = 4) ( Sun et al., 2024 )	$81.71\pm 0.46$	$86.62\pm 0.79$	$91.97\pm 0.74$	$82.41\pm 0.72$	$85.68$
FFA-LoRA (r = 8) ( Sun et al., 2024 )	$82.35\pm 0.48$	$86.98\pm 0.62$	$92.74\pm 0.85$	$82.71\pm 0.62$	$86.20$
FedSA ( Guo et al., 2025 )	$83.88\pm 0.41$	$89.13\pm 0.31$	$94.08\pm 0.54$	$85.96\pm 0.47$	$88.26$
FedDPA ( Yang et al., 2024 )	$83.45\pm 0.09$	$88.53\pm 0.77$	$93.52\pm 0.78$	$85.63\pm 0.69$	$87.78$
FedALT ( Bian et al., 2026 )	$82.53\pm 0.44$	$89.11\pm 0.33$	$91.78\pm 0.42$	$85.68\pm 0.05$	$87.28$

Appendix

Table 10: Performance comparison under different Client Numbers (

N=40

Methods	MNLI	QNLI	SST2	QQP	Average
FedIT ( Zhang et al., 2024 )	$83.13\pm 0.20$	$88.04\pm 0.56$	$93.67\pm 0.61$	$83.88\pm 0.36$	$87.18$
FFA-LoRA (r = 4) ( Sun et al., 2024 )	$83.41\pm 0.83$	$87.82\pm 0.99$	$94.12\pm 0.33$	$83.78\pm 0.53$	$87.28$
FFA-LoRA (r = 8) ( Sun et al., 2024 )	$83.69\pm 0.13$	$88.94\pm 0.23$	$94.27\pm 0.60$	$84.13\pm 0.23$	$87.76$
FedSA ( Guo et al., 2025 )	$85.32\pm 0.08$	$89.17\pm 0.97$	$94.50\pm 0.40$	$87.89\pm 0.39$	$89.22$
FedDPA ( Yang et al., 2024 )	$85.38\pm 0.21$	$89.93\pm 0.33$	$94.74\pm 0.63$	$88.07\pm 0.52$	$89.53$
FedALT ( Bian et al., 2026 )	$85.85\pm 0.88$	$90.10\pm 0.45$	$94.47\pm 0.29$	$87.95\pm 0.11$	$89.59$

Appendix

Table 11: Performance comparison under different Label Skew (

\alpha=1

Methods	MNLI	QNLI	SST2	QQP	Average
FedIT ( Zhang et al., 2024 )	$81.90\pm 0.15$	$87.28\pm 0.29$	$93.99\pm 0.10$	$82.87\pm 0.60$	$86.51$
FFA-LoRA (r = 4) ( Sun et al., 2024 )	$83.11\pm 0.31$	$89.18\pm 0.76$	$94.30\pm 0.93$	$83.23\pm 0.79$	$87.46$
FFA-LoRA (r = 8) ( Sun et al., 2024 )	$83.32\pm 0.93$	$89.60\pm 0.27$	$94.56\pm 0.36$	$83.51\pm 0.56$	$87.75$
FedSA ( Guo et al., 2025 )	$84.73\pm 0.39$	$89.84\pm 0.22$	$94.68\pm 0.39$	$88.98\pm 0.27$	$89.56$
FedDPA ( Yang et al., 2024 )	$84.19\pm 0.36$	$89.69\pm 0.39$	$94.79\pm 0.51$	$89.13\pm 0.39$	$89.45$
FedALT ( Bian et al., 2026 )	$83.23\pm 0.87$	$89.55\pm 0.11$	$90.86\pm 0.51$	$88.71\pm 0.19$	$88.09$

Appendix

Table 12: Performance comparison under different Label Skew (

\alpha=0.7

Figure 7: Layer-wise client clustering under task heterogeneity. Clients assigned to the same GLUE task form stable branches at deeper layers, while all tasks share common representations in shallow layers, illustrating the hierarchical specialization discovered by FedTreeLoRA.

Methods	MNLI	QNLI	SST2	QQP	Average
FedIT ( Zhang et al., 2024 )	$71.46\pm 1.74$	$83.71\pm 1.95$	$93.25\pm 0.31$	$78.54\pm 3.03$	$81.74$
FFA-LoRA ( Sun et al., 2024 )	$72.17\pm 2.74$	$82.75\pm 2.26$	$93.79\pm 0.68$	$81.25\pm 2.56$	$82.49$
FedSA ( Guo et al., 2025 )	$74.29\pm 1.09$	$86.33\pm 0.75$	$93.87\pm 0.18$	$82.12\pm 0.74$	$84.15$
FedDPA ( Yang et al., 2024 )	$75.00\pm 1.70$	$86.29\pm 0.16$	$93.58\pm 0.16$	$81.92\pm 2.01$	$84.20$
FedALT ( Bian et al., 2026 )	$73.25\pm 1.40$	$83.00\pm 1.35$	$93.69\pm 0.19$	$84.31\pm 0.44$	$83.56$
FedLEASE ( Wang et al., 2025 )	$79.91\pm 1.86$	$87.54\pm 1.35$	$94.17\pm 0.37$	$83.46\pm 0.79$	$86.27$

Appendix

Table 13: Performance under Task Heterogeneity (

N=16

clients, 4 per task).

Metric	MNLI	QNLI	SST2	QQP	Average
Cosine Distance	$88.29\pm 0.30$	$93.47\pm 0.25$	$96.41\pm 0.02$	$91.61\pm 0.09$	92.45
Frobenius Distance	$88.15\pm 0.25$	$93.37\pm 0.62$	$96.56\pm 0.07$	$91.35\pm 0.17$	92.36

Appendix

Table 14: Sensitivity analysis of distance metrics on GLUE benchmarks. Results demonstrate that FedTreeLoRA is metric-agnostic, maintaining high performance regardless of the specific similarity definition.

Warm-up Epochs	MNLI	QNLI	SST2	QQP	Average
5	82.94	89.31	94.19	84.75	87.80
10	82.66	89.83	94.62	84.13	87.81

Appendix

Table 15: Sensitivity analysis of warm-up duration on GLUE benchmarks. Results show that FedTreeLoRA remains highly stable across different warm-up stages, indicating that reliable client topology can be identified early during adaptation.

Join Round	Original Clients	Existing Clients	New Client	Local Only
10	87.90	87.43	84.50	82.00
20	87.90	87.50	86.50	79.00

Appendix

Table 16: Robustness to dynamic client participation on MNLI. Results show that integrating new clients causes negligible degradation to existing clients while significantly improving the performance of the newly joined client compared to local-only training.

Interaction Variant	MNLI	QNLI	SST2	QQP	Average
Isolationist (Cluster-Only)	$86.60\pm 0.21$	$92.57\pm 0.44$	$95.96\pm 0.39$	$90.48\pm 0.39$	$91.40$
Decomposed Experts	$\mathbf{88.46\pm 0.23}$	$\mathbf{93.91\pm 0.36}$	$96.54\pm 0.04$	$\mathbf{91.35\pm 0.13}$	$\mathbf{92.57}$
MoE Router	$87.92\pm 0.82$	$93.35\pm 0.58$	$96.15\pm 0.46$	$90.67\pm 0.45$	$92.02$
VA Scalar-Mixed	$88.15\pm 0.25$	$93.37\pm 0.62$	$\mathbf{96.56\pm 0.07}$	$\mathbf{91.35\pm 0.17}$	$\underline{92.36}$

Appendix

Table 17: Detailed performance of different Interaction Mechanisms across GLUE tasks. Accuracy is reported in (%).

Aggregation Depth	MNLI	QNLI	SST2	QQP	Average
Fixed ( $k=1$ , Global Only)	$83.18\pm 0.74$	$87.03\pm 0.43$	$93.65\pm 0.63$	$84.93\pm 0.59$	$87.20$
Fixed ( $k=4$ , Coarse)	$86.75\pm 0.63$	$92.95\pm 0.22$	$96.13\pm 0.16$	$89.98\pm 0.59$	$91.45$
Fixed ( $k=8$ , Fine-grained)	$85.81\pm 0.68$	$92.02\pm 0.71$	$95.17\pm 0.13$	$89.96\pm 0.89$	$90.74$
Layer-wise Adaptive ( $c_{l}^{*}$ )	$\mathbf{88.15\pm 0.25}$	$\mathbf{93.37\pm 0.62}$	$\mathbf{96.56\pm 0.07}$	$\mathbf{91.35\pm 0.17}$	$\mathbf{92.36}$

Appendix

Table 18: Detailed performance of Fixed-Depth vs. Layer-wise Adaptive aggregation.

Structural Prior	MNLI	QNLI	SST2	QQP	Average
Independent Clustering	$83.84\pm 0.32$	$89.74\pm 1.95$	$95.06\pm 0.21$	$89.23\pm 1.56$	$89.47$
Global Tree Skeleton	$\mathbf{88.15\pm 0.25}$	$\mathbf{93.37\pm 0.62}$	$\mathbf{96.56\pm 0.07}$	$\mathbf{91.35\pm 0.17}$	$\mathbf{92.36}$

Appendix

Table 19: Detailed impact of Global Structural Consistency on individual datasets.

Time (s)	FedIT	FFA-LoRA	FedSA	FedDPA	FedALT	FedLEASE	FedTreeLoRA
Local Training (Per Epoch)	1.0545	1.0369	1.0582	1.0668	1.2091	1.1883	1.0716
Global Aggregation	0.0628	0.0594	0.0693	0.0723	0.0782	0.0908	0.0770
Topology/Clustering Time	-	-	-	-	-	3.97	4.21
Total Training Time	65.154	63.996	65.571	66.177	74.892	77.992	70.816

Appendix

Table 20: Comparison of Computational Overhead. All times are measured in seconds (s).

Methods	Text Edit	Struct2Text	Sentiment	Reasoning	Average
FedIT ( Zhang et al., 2024 )	$77.21\pm 1.39$	$51.80\pm 0.89$	$49.73\pm 0.36$	$72.37\pm 0.58$	$62.78$
FFA-LoRA (r = 8) ( Sun et al., 2024 )	$77.05\pm 1.35$	$51.10\pm 0.66$	$48.72\pm 1.49$	$72.32\pm 0.46$	$62.30$
FFA-LoRA (r = 16) ( Sun et al., 2024 )	$78.77\pm 1.81$	$51.20\pm 0.60$	$49.05\pm 0.84$	$72.12\pm 0.86$	$62.79$
FedSA ( Guo et al., 2025 )	$84.93\pm 1.34$	$52.93\pm 0.23$	$49.38\pm 0.84$	$72.64\pm 1.18$	$64.94$
FedDPA ( Yang et al., 2024 )	$88.56\pm 0.86$	$53.78\pm 0.36$	$51.31\pm 0.51$	$72.16\pm 1.78$	$66.45$
FedALT ( Bian et al., 2026 )	$88.32\pm 0.19$	$53.58\pm 0.19$	$50.97\pm 1.13$	$73.22\pm 1.45$	$66.52$

Appendix

Table 21: Performance comparison on NLG benchmarks using BLOOM. We report ROUGE-1 scores averaged over three independent runs.

Figure 8: Layer-wise average pairwise Frobenius distance between two clients on QNLI under IID (identical label distributions) and Non-IID (

[0.8,0.2]

vs.

[0.2,0.8]

) settings. The upper-left and upper-right panels (Proj

A

at Query and Value) show nearly overlapping IID and Non-IID curves across all layers, indicating that

A

captures distribution-invariant shared representations. The lower-left and lower-right panels (Proj

B

at Query and Value) exhibit a progressively widening gap between IID and Non-IID as depth increases, revealing that client-specific semantics emerge primarily in deeper layers through the LoRA

B

matrices. This validates using

B

-matrix distances to quantify heterogeneity and to guide hierarchical, layer-wise aggregation in FedTreeLoRA.

Federated Fine-Tuning

Federated fine-tuning adapts pretrained models across decentralized data holders without pooling their private data.

See all 115 papers

FedLAFP: Low-Rank Aggregation Meets Full-Rank Personalization in Federated Fine-Tuning
Sep 29, 2026Mengjun Yi, Huaian Gu, Yinghao Ai +2Low-Rank Adaptation Federated Fine-Tuning
State Key Laboratory for Novel Software Technology · School of Artificial Intelligence · School of Computer Science +1
Breaking the Structural Identity: Personalized Federated LoRA Fine-tuning under Rank Heterogeneity
Sep 1, 2026Lei Wang, Jieming Bian, Letian Zhang +1Non-IID Federated Learning Low-Rank Adaptation
University of Florida · Gainesville, FL 32611 · Middle Tennessee State University +1
FedPA-LoRA: Product-Aligned Framework for Mitigating Aggregation and Initialization Errors in Heterogeneous Federated LoRA
Aug 15, 2026Juseok Jeon, Ramy E. Ali, Doyun Kwon +3Federated Fine-Tuning Federated Learning
DGIST Daegu, Republic of Korea · Samsung San Diego, USA

Sep 29, 2026J/K move · Enter open · S save

FedLAFP: Low-Rank Aggregation Meets Full-Rank Personalization in Federated Fine-Tuning

Mengjun Yi, Huaian Gu, Yinghao Ai, Furao Shen+1

State Key Laboratory for Novel Software Technology · School of Artificial Intelligence · School of Computer Science+1

Low-Rank Adaptation Federated Fine-Tuning Federated Learning Parameter-Efficient Fine-Tuning

Open paper AskPDF arXiv

Federated parameter-efficient fine-tuning enables clients to adapt pre-trained models without sharing raw data or communicating the full model, but statistical heterogeneity makes a single global adapter insufficient for personalized prediction. Existing personalized methods typically use the same low-rank structure for both shared and private adaptation, overlooking their distinct requirements for aggregation and personalization. We propose FedLAFP, a role-aware framework that couples a compact, globally aggregated LoRA branch with a client-private, full-rank-capable RandLoRA branch. The shared branch provides an efficient interface for transferring common knowledge, whereas the private branch combines fixed random low-rank bases with learned scaling coefficients to provide expressive client-specific adaptation without additional communication. Client- and layer-specific mixing coefficients jointly fuse the two branches, and only the shared LoRA parameters are exchanged. A controlled linear study supports this role assignment: LoRA yields more aligned client updates and lower aggregation error, while RandLoRA more accurately recovers client-specific residuals. Experiments across four visual recognition benchmarks show that FedLAFP consistently outperforms local-only and federated LoRA baselines, achieving an average personalized accuracy of

86.93\%

and exceeding the best baseline average by

1.30

percentage points.

Abstract

Federated Fine-Tuning

Explore similar work

FedLAFP: Low-Rank Aggregation Meets Full-Rank Personalization in Federated Fine-Tuning

Breaking the Structural Identity: Personalized Federated LoRA Fine-tuning under Rank Heterogeneity

FedPA-LoRA: Product-Aligned Framework for Mitigating Aggregation and Initialization Errors in Heterogeneous Federated LoRA