What Should Federated LoRA Share? FedSAIL via Input-aware Subspace Alignment

Junye Du; Shuaida He; Long Feng

cs.LGSep 26, 2026

What Should Federated LoRA Share? FedSAIL via Input-aware Subspace Alignment

Authors: Junye Du, Shuaida He, Long Feng

Organizations: School of Computing & Data Science The University of Hong Kong

Abstract

Federated low-rank adaptation (LoRA) requires identifying an update structure that is shared across heterogeneous clients. Prior work reports strong similarity among trained LoRA projection matrices across clients; however, such agreement may be largely induced by common initialization and collapses toward random overlap under independent initialization. More crucially, relying solely on parameter similarity inherently ignores the influence of local input regime. To uncover a more robust shared structure, we introduce an input-aware action matrix that weights the adapter update by the second-moment statistics of local layer inputs. Empirically, while parameter similarity vanishes, the leading right singular directions of this action matrix remain strongly aligned across clients. This shared geometry preserves task-conditioned differences and naturally varies across network depths. Motivated by these findings, we propose Federated Subspace-Guided Action-Informed Learning (FedSAIL). Instead of averaging weights, FedSAIL estimates a shared action subspace to regularize local training while preserving client-specific coefficients. Across several benchmarks, our approach consistently improves predictive performance over competing federated LoRA methods while reducing communication cost significantly.

Figures & tables

Figure 1: Sharing in federated LoRA. (a) LoRA. (b) FedSA-LoRA shares

\mathbf{A}

and retains

\mathbf{B}_{k}

locally. (c) FedSAIL shares a spectral summary of the dominant right subspace of

\mathbf{G}_{k}=\mathbf{B}_{k}\mathbf{A}_{k}\boldsymbol{\Sigma}_{k}^{1/2}

Method	Shared object	Aggregation	Local training
FedIT ( Zhang et al., 2024 )	$\mathbf{A},\mathbf{B}$	$\bar{\mathbf{A}}=\sum_{k}\omega_{k}\mathbf{A}_{k}$ [0.5pt] $\bar{\mathbf{B}}=\sum_{k}\omega_{k}\mathbf{B}_{k}$	$\mathbf{A}_{k},\mathbf{B}_{k}$
FLoRA ( Wang et al., 2024 )	Adapter update $\boldsymbol{\Delta}$	$\bar{\boldsymbol{\Delta}}=\sum_{k}\omega_{k}\boldsymbol{\Delta}_{k}$	$\mathbf{A}_{k},\mathbf{B}_{k}$
FFA-LoRA ( Sun et al., 2024 )	$\mathbf{B}$ ; common frozen $\mathbf{A}^{(0)}$	$\bar{\mathbf{B}}=\sum_{k}\omega_{k}\mathbf{B}_{k}$	Only $\mathbf{B}_{k}$
FedSA-LoRA ( Guo et al., 2025 )	$\mathbf{A}$	$\bar{\mathbf{A}}=\sum_{k}\omega_{k}\mathbf{A}_{k}$	$\mathbf{A}_{k},\mathbf{B}_{k}$
FedDPA-LoRA ( Yang et al., 2024b )	Global $\mathbf{A}^{g},\mathbf{B}^{g}$	$\bar{\mathbf{A}}^{g}=\frac{1}{K}\sum_{k}\mathbf{A}_{k}^{g}$ [0.5pt] $\bar{\mathbf{B}}^{g}=\frac{1}{K}\sum_{k}\mathbf{B}_{k}^{g}$	Global $\mathbf{A}_{k}^{g},\mathbf{B}_{k}^{g}$ [0.5pt] Local $\mathbf{A}_{k}^{\ell},\mathbf{B}_{k}^{\ell}$
FedSAIL (ours)	Dominant right subspace [0.5pt]of $\mathbf{G}_{k}$ in ( 3 )	$\widehat{\mathbf{M}}=\sum_{k}\omega_{k}\widehat{\mathbf{M}}_{k}$ in ( 6 ) [0.5pt] $\widehat{\mathbf{V}}=\operatorname{TopEig}_{s}(\widehat{\mathbf{M}})$	$\mathbf{A}_{k},\mathbf{B}_{k}$

Table 1: Comparison of shared components, aggregation strategies, and local training rules across different federated LoRA frameworks.

Figure 2: Cross-client subspace agreement for RoBERTa-base on RTE. For each initialization group, label heterogeneity increases from left to right, with

\alpha\in\{0.5,1.0\}

controlling Dirichlet label skew.

Figure 3: Task and layer dependence of action-subspace agreement in GPT-2. Curves report rank-

1

Projector Dice averaged over query and value projections, while shaded bands indicate

95\%

confidence intervals.

$\alpha$	Method	RTE	SST-2	QNLI	MNLI-m	MNLI-mm	QQP	Avg.
$0.5$	LoRA	$78.70_{\pm 2.60}$	$95.83_{\pm 0.93}$	$90.13_{\pm 1.93}$	$88.68_{\pm 1.60}$	$88.96_{\pm 1.73}$	$89.41_{\pm 5.75}$	$88.62_{\pm 1.14}$
	FFA-LoRA	$70.04_{\pm 5.91}$	$90.75_{\pm 1.78}$	$79.46_{\pm 2.06}$	$65.46_{\pm 9.63}$	$66.65_{\pm 9.11}$	$81.28_{\pm 11.53}$	$75.61_{\pm 3.27}$
	FedDPA-LoRA	$78.22_{\pm 6.83}$	$95.99_{\pm 1.20}$	$91.62_{\pm 1.21}$	$\mathbf{89.06}_{\pm 1.58}$	$\mathbf{89.29}_{\pm 1.71}$	$89.62_{\pm 5.76}$	$88.97_{\pm 1.33}$
	FedSA-LoRA	$79.90_{\pm 8.61}$	$95.95_{\pm 1.13}$	$91.76_{\pm 0.61}$	$88.56_{\pm 1.72}$	$88.92_{\pm 1.76}$	$89.53_{\pm 5.63}$	$89.10_{\pm 1.60}$
	FedSAIL	$\mathbf{80.87}_{\pm 7.50}$	$\mathbf{96.37}_{\pm 1.17}$	$\mathbf{93.06}_{\pm 0.57}$	$88.98_{\pm 1.54}$	$89.07_{\pm 1.75}$	$\mathbf{89.80}_{\pm 5.39}$	$\mathbf{89.69}_{\pm 1.28}$
$0.75$	LoRA	$\mathbf{77.26}_{\pm 0.96}$	$95.45_{\pm 0.35}$	$90.76_{\pm 0.89}$	$88.39_{\pm 1.43}$	$88.62_{\pm 1.44}$	$88.68_{\pm 4.49}$	$88.19_{\pm 0.54}$

Table 2: Personalized GLUE validation accuracy (%, mean

\pm

sample std.) after

500

rounds, with the highest mean in each column for each

\alpha

shown in bold. Smaller

\alpha

indicates stronger label heterogeneity.

$r_{\rm L}$	$s$	Params. (M)	Comm. (GiB)	$\alpha=0.5$	$\alpha=0.75$	$\alpha=1.0$
$8$	$1$	$0.786$	$0.4949$	$89.69_{\pm 1.28}$	$88.77_{\pm 0.56}$	$89.52_{\pm 1.14}$
$8$	$2$	$0.786$	$0.9895$	$89.83_{\pm 1.08}$	$88.87_{\pm 0.49}$	$\mathbf{89.65}_{\pm 0.98}$
$8$	$4$	$0.786$	$1.9787$	$\mathbf{89.91}_{\pm 1.48}$	$89.15_{\pm 0.18}$	$89.47_{\pm 1.03}$
$2$	$1$	$0.197$	$0.4949$	$88.74_{\pm 1.43}$	$87.92_{\pm 0.55}$	$88.65_{\pm 1.41}$
$4$	$1$	$0.393$	$0.4949$	$89.21_{\pm 1.51}$	$88.24_{\pm 0.81}$	$89.10_{\pm 1.09}$
$16$	$1$	$1.573$	$0.4949$	$89.84_{\pm 1.43}$	$\mathbf{89.22}_{\pm 0.29}$	$89.61_{\pm 0.95}$

Table 3: Sensitivity to shared and local ranks on personalized GLUE . We report averaged validation accuracy (%, mean

\pm

sample std.), adapter parameter counts, and total communication costs.

Shared message	Input-aware	$\alpha=0.5$	$\alpha=0.75$	$\alpha=1.0$	Overall
$\mathbf{A}_{k}$	No	$68.75_{\pm 2.92}$	$66.17_{\pm 1.41}$	$68.95_{\pm 3.47}$	$67.96_{\pm 1.20}$
$\boldsymbol{\Delta}_{k}=\mathbf{B}_{k}\mathbf{A}_{k}$	No	$68.79_{\pm 3.05}$	$66.17_{\pm 1.42}$	$69.04_{\pm 3.41}$	$68.00_{\pm 1.22}$
$\widehat{\mathbf{G}}_{k}=\boldsymbol{\Delta}_{k}\widehat{\boldsymbol{\Sigma}}_{k}^{1/2}$	Yes	$\mathbf{89.69}_{\pm 1.28}$	$\mathbf{88.77}_{\pm 0.56}$	$\mathbf{89.52}_{\pm 1.14}$	$\mathbf{89.33}_{\pm 0.36}$

Table 4: Ablation of the shared message on personalized GLUE. We report validation accuracy (%, mean

\pm

sample std.) with rank-

1

payloads.

FedSAIL variant	Accuracy (%)	Comm. (MiB)	Saving (%)
Full	$90.033\pm 0.225$	$96.094$	–
Selection, $\tau=0.6$	$90.133\pm 0.038$	$77.589$	$19.26$
Selection, $\tau=0.7$	$89.908\pm 0.800$	$58.904$	$38.70$

Table 5: Layer selection performance on DDDT. We report averaged held-out accuracy (%, mean

\pm

sample std.) at round

64

and total communication cost, including selector overhead.

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Illustration of Remark 1 , with

a=10

and

\mathbf{B}_{1}=\mathbf{B}_{2}=1

. (a) The raw

\mathbf{A}_{k}

row spaces have Dice

0.039

. (b) After client-specific input weighting,

\mathbf{G}_{1}=\mathbf{G}_{2}=\bigl(1,1\bigr)

, so the right action subspaces coincide with Dice one. Blue and orange denote clients 1 and 2, respectively.

Setting	RoBERTa-base / RTE	GPT-2 124M
Clients	3 per partition setting	$2$ domains $\times$ $2$ tasks $\times$ $2$ replicas
Train / calibration / evaluation	$161$ – $1543$ / $9$ – $86$ / $10$ – $87$	$500/250/250$
Local LoRA rank / scale	$8/16$	$4/8$
LoRA dropout	$0.05$	$0$
Steps / effective batch size	$200/128$	$200/16$
Maximum sequence length	$128$	$64$

Appendix

Table 6: Local-training protocols for the structural diagnostics.

Partition	Initialization	$\mathbf{A}$	$\mathbf{B}$	$\mathbf{G}$ : top 1	$\mathbf{G}$ : top 2	$\mathbf{G}$ : full
IID	Common	$1.000$	$0.199$	$0.978$	$0.825$	$0.876$
$\alpha=1$	Common	$1.000$	$0.185$	$0.974$	$0.834$	$0.871$
$\alpha=0.5$	Common	$1.000$	$0.177$	$0.965$	$0.818$	$0.835$
IID	Independent	$0.011$	$0.193$	$0.814$	$0.452$	$0.159$
$\alpha=1$	Independent	$0.011$	$0.180$	$0.810$	$0.454$	$0.160$
$\alpha=0.5$	Independent	$0.011$	$0.177$	$0.810$	$0.455$	$0.159$

Appendix

Table 7: Cross-client projector Dice in RoBERTa-base/RTE averaged across client pairs, layers, and projections. All entries use the fixed step-200 checkpoint.

Figure 5: Query-projection subspace agreement in RoBERTa-base/RTE. For each initialization group, label heterogeneity increases from left to right, with

\alpha\in\{0.5,1.0\}

controlling Dirichlet label skew.

Initialization	$S_{11}$	$S_{10}$	$S_{01}$	$S_{00}$
Common	$0.9412$	$0.8288$	$0.9082$	$0.8221$
Independent	$0.0806$	$0.0383$	$0.0687$	$0.0355$

Appendix

Table 8: Rank-

4

\mathbf{A}

-rowspace Dice for GPT-2, averaged over layers and query/value projections.

S_{11}

: same domain and task,

S_{10}

: same domain and different tasks,

S_{01}

: different domains and same task,

S_{00}

: different domains and tasks.

Figure 6: Ordered-direction diagnostics on RoBERTa-base/RTE with independent initialization and LoRA rank eight. Panels show (a) evaluation action energy fraction, (b) cross-client rank-one projector Dice, and (c) LOO action-energy capture. Metrics are averaged across partitions, layers, projections, and client pairs.

Initialization	Tokens	$C_{\rm task}$	$C_{\rm domain}$	$S_{01}-S_{10}$
Common	Instruction	$.224[.207,.240]$	$.008[.001,.016]$	$.215[.195,.235]$
Common	Content	$.212[.202,.222]$	$.047[.042,.052]$	$.165[.153,.176]$
Common	Decision	$.242[.221,.263]$	$.023[.017,.030]$	$.218[.196,.241]$
Independent	Instruction	$.201[.190,.213]$	$.011[.003,.018]$	$.191[.178,.203]$
Independent	Content	$.201[.192,.211]$	$.039[.033,.045]$	$.162[.153,.171]$
Independent	Decision	$.237[.222,.252]$	$.023[.017,.029]$	$.214[.198,.230]$

Appendix

Table 9: Task and domain organization of the fixed rank-

1

dominant right direction of

\mathbf{G}_{k}

. Mean values and

95\%

t

intervals (in brackets) are averaged over query and value projections.

Initialization	Tokens	Projection	$s=1$	$s=2$	$s=3$	$s=4$
Common	Instruction	Q	$.211$	$.246$	$.288$	$.318$
		V	$.220$	$.287$	$.328$	$.356$
	Content	Q	$.126$	$.097$	$.016$	$-.061$
		V	$.204$	$.150$	$.016$	$-.067$
	Decision	Q	$.210$	$.188$	$.129$	$.083$
		V	$.227$	$.207$	$.144$	$.082$

Appendix

Table 10: Task-minus-domain contrasts in GPT-2 across projections and retained ranks, averaged over replicates and 12 layers. Entries report

S_{01}-S_{10}=C_{\rm task}-C_{\rm domain}

, with negative values indicating a larger domain contrast.

Q

and

V

denote query and value.

Setting	Configuration
Backbone and adapters	RoBERTa-large; query and value projections in all $24$ layers; default $r_{\rm L}=8$ , $\alpha_{\rm LoRA}=16$ , LoRA dropout $0.05$ .
Initialization	Independent client $\mathbf{A}_{k}$ , zero $\mathbf{B}_{k}$ ; identical initial classification heads, trained locally.
Data and batching	Full task training/validation sets; maximum length $128$ ; batch and micro-batch size $128$ .
Optimization	SGD with CUDA AMP; $500$ rounds; $10$ local steps per client and phase per round.
Main-comparison seeds	$123$ , $124$ , and $125$ .
Evaluation	Personalized validation every $25$ rounds; no global-model evaluation.

Appendix

Table 11: Common training settings for personalized GLUE. FedSAIL-specific settings also apply to its message and rank ablations unless stated otherwise.

Method	Federated exchange	Learning rate	Steps
LoRA	Local $\mathbf{A}_{k}$ and $\mathbf{B}_{k}$	$0.005$	$10$
FFA-LoRA	Average $\mathbf{B}_{k}$ ; freeze local $\mathbf{A}_{k}$	$0.005$	$10$
FedSA-LoRA	Average $\mathbf{A}_{k}$ ; keep $\mathbf{B}_{k}$ local	$0.005$	$10$
FedDPA-LoRA	Average global factors; retain local branch	$0.005$	$10+10$
FedSAIL	Share action spectra and consensus bases	$0.005$	$10$

Appendix

Table 12: Method-specific GLUE settings. Steps are local steps per client and round; FedDPA-LoRA uses separate global and local phases.

$\alpha$	Shared message	RTE	SST-2	QNLI	MNLI-m	MNLI-mm	QQP	Avg.
$0.5$	$\mathbf{A}_{k}$	$74.73$	$81.42$	$70.49$	$48.04$	$49.14$	$69.11$	$65.49$
	$\boldsymbol{\Delta}_{k}$	$74.37$	$81.65$	$70.55$	$48.13$	$49.15$	$68.56$	$65.40$
	$\mathbf{A}_{k}\widehat{\boldsymbol{\Sigma}}_{k}^{1/2}$	$\mathbf{85.20}$	$95.30$	$93.25$	$87.45$	$87.39$	$\mathbf{86.01}$	$89.10$
	$\widehat{\mathbf{G}}_{k}$	$\mathbf{85.20}$	$\mathbf{95.53}$	$\mathbf{93.37}$	$\mathbf{87.56}$	$\mathbf{87.55}$	$85.92$	$\mathbf{89.19}$
$0.75$	$\mathbf{A}_{k}$	$63.54$	$76.72$	$69.25$	$62.65$	$63.10$	$68.35$	$67.27$
	$\boldsymbol{\Delta}_{k}$	$61.73$	$78.67$	$69.34$	$62.69$	$63.13$	$68.87$	$67.40$

Appendix

Table 13: Ablation of shared-message on personalized GLUE. We report validation accuracy (%) over

500

rounds.

$\alpha$	$s$	RTE	SST-2	QNLI	MNLI-m	MNLI-mm	QQP	Avg.	Comm. (GiB)
$0.5$	$1$	$80.87_{\pm 7.50}$	$96.37_{\pm 1.17}$	$93.06_{\pm 0.57}$	$88.98_{\pm 1.54}$	$89.07_{\pm 1.75}$	$89.80_{\pm 5.39}$	$89.69_{\pm 1.28}$	$0.4949$
	$2$	$80.39_{\pm 5.53}$	$96.56_{\pm 1.30}$	$93.56_{\pm 0.57}$	$89.12_{\pm 1.60}$	$89.23_{\pm 1.80}$	$90.11_{\pm 5.34}$	$89.83_{\pm 1.08}$	$0.9895$
	$4$	$80.51_{\pm 7.82}$	$96.64_{\pm 1.32}$	$93.40_{\pm 0.31}$	$89.37_{\pm 1.50}$	$89.45_{\pm 1.81}$	$90.07_{\pm 5.39}$	$89.91_{\pm 1.48}$	$1.9787$
	$8$	$80.75_{\pm 8.08}$	$96.60_{\pm 1.30}$	$93.37_{\pm 0.59}$	$89.34_{\pm 1.53}$	$89.41_{\pm 1.80}$	$90.19_{\pm 5.44}$	$89.94_{\pm 1.50}$	$3.9573$
$0.75$	$1$	$77.26_{\pm 5.05}$	$95.64_{\pm 0.20}$	$93.23_{\pm 0.61}$	$88.61_{\pm 1.53}$	$88.84_{\pm 1.60}$	$89.05_{\pm 4.69}$	$88.77_{\pm 0.56}$	$0.4949$
	$2$	$76.90_{\pm 5.32}$	$95.80_{\pm 0.52}$	$93.84_{\pm 1.04}$	$88.72_{\pm 1.49}$	$88.85_{\pm 1.60}$	$89.10_{\pm 4.79}$	$88.87_{\pm 0.49}$	$0.9895$

Appendix

Table 14: Shared-rank sensitivity with fixed local rank

r_{\rm L}=8

. We report personalized validation accuracy and total communication cost, with accuracy averaged across the six splits.

$r_{\rm L}$	Params. (M)	$\alpha=0.5$	$\alpha=0.75$	$\alpha=1.0$
$2$	$0.197$	$88.74_{\pm 1.43}$	$87.92_{\pm 0.55}$	$88.65_{\pm 1.41}$
$4$	$0.393$	$89.21_{\pm 1.51}$	$88.24_{\pm 0.81}$	$89.10_{\pm 1.09}$
$8$	$0.786$	$89.69_{\pm 1.28}$	$88.77_{\pm 0.56}$	$89.52_{\pm 1.14}$
$16$	$1.573$	$89.84_{\pm 1.43}$	$89.22_{\pm 0.29}$	$89.61_{\pm 0.95}$

Appendix

Table 15: Local-rank sensitivity with fixed shared rank

s=1

. We report six-split validation accuracy (%, mean

\pm

sample std.) and per-client LoRA parameter counts. Total communication remains fixed at

0.4949

GiB per run.

Component	Setting
Clients and splits	Eight clients; per client: $1000$ training, $256$ calibration, $244$ validation, and $500$ final-test examples. All splits are label-balanced and disjoint across splits and replicas.
Data sources	Training/calibration: official training pools. Validation/final test: IMDb/Amazon test pools and labeled QQP/QNLI validation pools; no GLUE test-server evaluation.
Seeds	Data partition: $2027$ . Training seeds: $83,97,109$ .
Backbone and adapters	Frozen RoBERTa-large and MLM head; query/value LoRA in all $24$ layers, $r_{\rm L}=4$ , $\alpha_{\rm LoRA}=16$ , dropout $0.05$ . Independent $\mathbf{A}_{k}$ initialization and $\mathbf{B}_{k}=0$ .
Optimization	AdamW, learning rate $2\times 10^{-4}$ , weight decay $0$ , gradient clipping $1$ , batch size $16$ , CUDA AMP; $10$ local iterations per client per round. Optimizer/scaler states remain local across rounds.
Training schedule	$64$ rounds: $32$ local warmup rounds and $32$ guided rounds, with shared rank $s=1$ and guidance coefficient $\beta=1$ .

Appendix

Table 16: DDDT data, training, and evaluation protocol.

FedSAIL variant	$\tau$	IMDb	Amazon	QQP	QNLI	Avg.	Comm.
Full	–	$93.33$	$95.93$	$\mathbf{83.27}$	$87.60$	$90.033_{\pm 0.225}$	$96.094$
Selection	$0.4$	$93.20$	$95.70$	$83.07$	$88.07$	$90.008_{\pm 0.644}$	$104.282$
Selection	$0.5$	$93.23$	$96.17$	$82.37$	$\mathbf{88.37}$	$90.033_{\pm 0.284}$	$89.601$
Selection	$0.6$	$93.57$	$96.37$	$83.13$	$87.47$	$\mathbf{90.133}_{\pm 0.038}$	$77.589$
Selection	$0.7$	$\mathbf{93.77}$	$\mathbf{96.40}$	$81.97$	$87.50$	$89.908_{\pm 0.800}$	$\mathbf{58.904}$

Appendix

Table 17: FedSAIL threshold sensitivity on DDDT. We report round-

64

accuracy (%, mean

\pm

sample std.) and total communication cost, including selector snapshots and mask delivery.

Component	Specification
GPUs	$4\times$ NVIDIA GeForce RTX 3090, $24$ GiB memory each
CPUs	$2\times$ Intel Xeon Gold 5218R, $40$ physical cores in total
Host memory	Approximately $504$ GiB (OS-reported)
Operating system	Ubuntu 22.04.5 LTS
Training runtime	Python 3.9.21; PyTorch 2.5.1 with CUDA 12.4
Libraries	Transformers 4.42.3; Datasets 2.20.0

Appendix

Table 18: Primary compute server and benchmark training environment.

Explore similar work

Aug 10, 2026cs.LG

Rethinking Factor Sharing in Federated LoRA: A Rank-Aware Adaptive Approach

Low-rank adaptation (LoRA) represents large language model (LLM) updates with two compact matrix factors, i.e.,

A

and

B

, providing an efficient way to fine-tune large models in federated learning paradigm. Inspired by the asymmetric roles of the LoRA factors, we study whether

A

should be shared across clients while

B

remains client-specific (Share-A/Local-B), or whether

B

should instead be shared while

A

remains client-specific (Share-B/Local-A). With a least-squares surrogate, we reveal that Share-A/Local-B requires the client-specific LoRA update matrices to use a common rank-

r

input-side space, whereas Share-B/Local-A requires a common rank-

r

output-side space. The two strategies therefore incur different projection residuals, indicating that the preferred strategy is the one with the smaller aggregate residual across clients. With this insight, we propose Federated Adaptive Factor Sharing Low-Rank Adaptation (FedAS-LoRA), which selects the sharing side before training to enhance fine-tuning performance. To enable adaptive factor selection before training, we design a Rank-Aware Shared-Subspace Sufficiency (RSS) metric, which effectively assesses whether a shared rank-

r

input subspace is sufficient for the local data distributions using representations extracted from a frozen LLM backbone. Experiments across different tasks, data distributions, LoRA ranks, and participation settings confirm the effectiveness of RSS and the superior performance of FedAS-LoRA.

Xinyi Xu, Bingnan Xiao, Shuang Qin +2

University of Electronic Science and Technology of China · Fudan University · Singapore University of Technology and Design

LLM Fine-Tuning Low-Rank Adaptation Federated Fine-Tuning Federated Learning

Ask PDF

Jun 21, 2026cs.LG

Subspace-Constrained Federated Learning with Low-Rank Adaptation

Federated low-rank adaptation methods are attractive for fine-tuning large models under communication and privacy constraints, but heterogeneous client data can induce geometric misalignment between local low-rank updates. We study whether this subspace misalignment leads to destructive aggregation and slower convergence in LoRA-based federated learning. We propose a subspace-regularized federated LoRA objective that encourages local client updates to remain close to a shared global reference subspace. We present a complete empirical evaluation on two pretrained models, RoBERTa-large and SmolLM-360M, over HellaSwag in a non-IID 10-client federated setting, across 3 random seeds (42, 43, 44), yielding 24 total experimental runs (4 methods x 3 seeds x 2 models). On RoBERTa-large, Subspace-Reg achieves the strongest mean best accuracy (0.454 +/- 0.023), mean final accuracy (0.429 +/- 0.011), and lowest final loss (1.363) across all three seeds, outperforming FedAvg, SVD redistribution, and FedSVD baselines by a large margin. On SmolLM-360M, FedAvg leads on accuracy, revealing that accuracy gains are model-dependent. Crucially, Subspace-Reg achieves near-perfect basis overlap, approximately 0.9999, on both models and across all seeds, versus 0.958 to 0.991 for all baselines, providing robust support for the geometric alignment hypothesis. The code is publicly available at https://github.com/sadia-sigma-lab/Subspace-Constrained-Federated-learning-with-Lora.

Neranjan Senarath, Rohit Muralitharan, Sadia Asif

Department of ECSE Rensselaer Polytechnic Institute Troy, NY 12180, USA · Department of Computer Science Rensselaer Polytechnic Institute Troy, NY 12180, USA

Federated Learning Aggregation Non-IID Federated Learning Low-Rank Adaptation Federated Fine-Tuning Representation Alignment Latent Space Alignment+1

Ask PDF

Oct 1, 2026cs.AI

FedLore: Communication and Memory Efficient Federated Learning via Shared Gradient Low-Rank Projection

Federated training of foundation models is constrained by client memory and communication costs. LoRA-based methods reduce these costs through low-rank adapters, but their fixed rank budget can limit adaptation. Gradient low-rank optimization offers greater flexibility, yet independently chosen client subspaces create a problem we term \emph{subspace fragmentation}: local projections interact with data heterogeneity to bias aggregated directions, while aggregation can increase update rank and communication cost. Thus, accurate local gradient compression need not preserve global descent. We propose \texttt{FedLore}, which shares a low-rank optimization basis within each round and refreshes it across rounds. The shared basis enables exact aggregation in low-rank coordinates and eliminates the identified projection bias. Subspace refresh allows the accumulated model update to exceed the per-round rank budget. We characterize the aggregation bias and establish an

O(T^{-1/2})

stationarity bound for the projected-SGD variant under a global-gradient coverage condition and standard smoothness and variance assumptions, with bounded gradient heterogeneity. Experiments on vision and language tasks, including federated pre-training, show that \texttt{FedLore} outperforms the evaluated low-rank adapter baselines and matches or exceeds full-parameter training, while reducing communication and optimizer-state memory.

Junkang Liu

Tianjin University

Federated Learning Aggregation Memory-Efficient Fine-Tuning Communication-Efficient Distributed Training Federated Learning

Ask PDF