FedLore: Communication and Memory Efficient Federated Learning via Shared Gradient Low-Rank Projection

Junkang Liu

cs.AIOct 1, 2026

FedLore: Communication and Memory Efficient Federated Learning via Shared Gradient Low-Rank Projection

Authors: Junkang Liu

Organizations: Tianjin University

Abstract

Federated training of foundation models is constrained by client memory and communication costs. LoRA-based methods reduce these costs through low-rank adapters, but their fixed rank budget can limit adaptation. Gradient low-rank optimization offers greater flexibility, yet independently chosen client subspaces create a problem we term \emph{subspace fragmentation}: local projections interact with data heterogeneity to bias aggregated directions, while aggregation can increase update rank and communication cost. Thus, accurate local gradient compression need not preserve global descent. We propose \texttt{FedLore}, which shares a low-rank optimization basis within each round and refreshes it across rounds. The shared basis enables exact aggregation in low-rank coordinates and eliminates the identified projection bias. Subspace refresh allows the accumulated model update to exceed the per-round rank budget. We characterize the aggregation bias and establish an $O(T^{-1/2})$ stationarity bound for the projected-SGD variant under a global-gradient coverage condition and standard smoothness and variance assumptions, with bounded gradient heterogeneity. Experiments on vision and language tasks, including federated pre-training, show that \texttt{FedLore} outperforms the evaluated low-rank adapter baselines and matches or exceeds full-parameter training, while reducing communication and optimizer-state memory.

Figures & tables

Panel of Figure 1 — Figure 1: Comparison between FedLore and Local GaLore and FedIT on ViT-Tiny over CIFAR-100 with non-iid data. (a) Test accuracy: FedLore vs FedIT vs Local GaLore. (b) Effective Rank: Dynamic expansion of optimization space of FedLore . (c) Subspace fragmentation of Local GaLore. We measure subspace fragmentation by the difference between projection matrices: $\Delta_{\mathcal{D}}=\frac{1}{Nr}\sum_{i\in\mathcal{N}}\|\Pi_{i}-\bar{\Pi}\|_{F}^{2},\Pi_{i}=P_{i}P_{i}^{\top},\bar{\Pi}=\frac{1}{N}\sum_{i\in\mathcal{N}}\Pi_{i}$ . Client-specific projections introduce severe subspace fragmentation under heterogeneous data. (d) Local GaLore degrades under heterogeneity due to misaligned client-wise optimization subspaces.

Figure 2: Fixed-rank LoRA vs ours low-rank updates. FedLore shares a low-rank update basis within each round and refreshes it across rounds, allowing the accumulated model change to exceed the per-round rank.

Method	Venue	CIFAR-100		Tiny-ImageNet		Food-101		Avg.
Method	Venue	Swin-Base	ViT-Base	Swin-Base	ViT-Base	Swin-Base	ViT-Base	Avg.
FedIT	ICASSP’24	$90.17\pm 0.28$	$89.57\pm 0.28$	$86.50\pm 0.22$	$84.88\pm 0.12$	$91.55\pm 0.18$	$91.02\pm 0.20$	$88.95$
FFA-LoRA	ICLR’24	$90.07\pm 0.18$	$86.26\pm 0.14$	$85.34\pm 0.18$	$77.89\pm 0.11$	$91.25\pm 0.12$	$90.68\pm 0.22$	$86.92$
FlexLoRA	ICLR’25	$90.91\pm 0.19$	$89.59\pm 0.19$	$85.94\pm 0.19$	$85.55\pm 0.19$	$91.82\pm 0.16$	$91.34\pm 0.10$	$89.19$
LoRA-FAIR	ICCV’25	$90.26\pm 0.19$	$89.86\pm 0.19$	$86.59\pm 0.15$	$85.11\pm 0.19$	$91.41\pm 0.12$	$91.23\pm 0.14$	$89.08$
RoLoRA	NeurIPS’25	$90.55\pm 0.26$	$88.55\pm 0.26$	$86.03\pm 0.28$	$82.97\pm 0.28$	$90.54\pm 0.15$	$90.32\pm 0.08$	$88.16$

Table 1: Test accuracy (%) under Dir-0.1 over

T=100

. Experiments are conducted with 50 clients, batch size 16, local steps

K=50

, and LoRA rank

r=8

, projection rank

r=8

. Best results are shown in bold and second-best results are underlined.

Method	Venue	SNLI	AG News	QQP	DBPedia 14	QNLI	MNLI	Avg.
FedIT	ICASSP’24	$86.50\pm 0.13$	$92.68\pm 0.09$	$85.06\pm 0.14$	$98.94\pm 0.07$	$89.47\pm 0.11$	$81.77\pm 0.15$	$89.07$
FFA-LoRA	ICLR’24	$80.70\pm 0.17$	$90.14\pm 0.12$	$81.48\pm 0.16$	$98.22\pm 0.10$	$85.61\pm 0.14$	$72.44\pm 0.18$	$84.77$
FlexLoRA	ICLR’25	$86.64\pm 0.15$	$93.30\pm 0.11$	$63.18\pm 0.19$	$98.06\pm 0.09$	$49.46\pm 0.20$	$35.44\pm 0.18$	$71.01$
LoRA-FAIR	ICCV’25	$86.76\pm 0.10$	$92.76\pm 0.08$	$85.33\pm 0.12$	$98.88\pm 0.07$	$89.47\pm 0.09$	$82.04\pm 0.11$	$89.21$
RoLoRA	NeurIPS’25	$85.95\pm 0.12$	$92.06\pm 0.10$	$84.51\pm 0.13$	$98.78\pm 0.08$	$88.30\pm 0.12$	$80.58\pm 0.14$	$88.36$
FLoRA	NeurIPS’24	$69.36\pm 0.18$	$89.77\pm 0.13$	$79.78\pm 0.15$	$98.22\pm 0.11$	$81.93\pm 0.16$	$78.23\pm 0.17$	$82.88$

Table 2: Test accuracy (%) of RoBERTa-Base with LoRA on six text classification tasks under FL. Experiments are conducted for

T=100

rounds with 20 clients, Dir-0.5 data partition, batch size 16, local steps

K=50

, and LoRA rank

r=8

Method	60M		130M		350M		1B		Comm. $\downarrow$	Time/Round $\downarrow$
Method	Loss	PPL	Loss	PPL	Loss	PPL	Loss	PPL	Comm. $\downarrow$	on LLaMA 350M (s)
FedIT	8.811	6707.62	8.544	5135.85	8.072	3203.50	7.561	1921.77	$O(rd)$	118.6
Local GaLore	4.326	75.64	4.215	67.69	3.962	52.56	3.781	43.86	$O(d^{2})$	136.6
FedFull	4.134	62.43	4.056	57.74	3.798	44.61	3.561	35.20	$O(d^{2})$	132.1
FedLore-Random	4.112	61.07	4.036	56.60	3.781	43.86	3.523	33.89	$O(rd)$	119.8
FedLore	3.833	46.20	3.611	37.00	3.554	34.95	3.345	28.36	$O(rd)$	122.4

Table 3: Comparison with low-rank federated algorithms for pre-training LLaMA models on the C4 dataset. We report validation loss (Val Loss), validation perplexity (Val PPL), communication cost (Comm.), and local training time. All methods are trained for 100 communication rounds (20 clients, batch size 16,

K=50

). The projection rank of FedLore is identical to LoRA rank.

Panel of Figure 3 — Figure 3: (a,b,c): Validation loss convergence curves on C4 with LLaMA models of different scales. (d): Effect of the projection rank $r$ on validation loss for LLaMA 350M.

Method	Comm.	Opt. Memory	Ablation Setting	Val Loss / PPL	Comm. $\downarrow$
FedIT	$0.36$	$0.72$	Local GaLore	3.962 / 52.56	$0.68$
FedFull	$0.68$	$1.37$	FedLore	3.554 / 34.95	$0.36$
Local GaLore	$0.68$	$0.54$	w/o Shared Projector	3.890 / 48.91	$0.68$
FedLore	$0.36$	$0.54$	w/o Dynamic Subspace	5.952 / 384.7	$0.18$

Table 4: Communication and optimizer memory on LLaMA 350M (left), and component ablations on C4 pre-training (right). All costs are in GB.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Hidden	Intermediate	Heads	Layers
LLaMA-60M	512	1376	8	8
LLaMA-130M	768	2048	12	12
LLaMA-350M	1024	2736	16	24
LLaMA-1B	2048	5461	32	24

Appendix

Table 5: LLaMA architecture configurations used in federated pre-training. Model names follow the GaLore implementation.

Configuration	Value
Dataset	C4
Number of clients	20
Client participation	100%
Communication rounds	100
Local steps per round	50
Batch size	16

Appendix

Table 6: Federated pre-training configurations.

Method	Communication Cost	Optimizer Memory
FedIT	$O(2r(m+n))$	$O(r(m+n))$
FedFull	$O(2mn)$	$O(mn)$
Local GaLore	$O(2mn)$	$O(rn)$
FedLore	$O(2r(m+n))$	$O(rn)$

Appendix

Table 7: Communication and optimizer memory comparison.

r\ll\min(m,n)

Method	LLaMA-60M	LLaMA-130M	LLaMA-350M	LLaMA-1B
FedIT	$1\times 10^{-3}$	$8\times 10^{-4}$	$5\times 10^{-4}$	$5\times 10^{-4}$
Local GaLore	$3\times 10^{-3}$	$3\times 10^{-3}$	$1\times 10^{-3}$	$1\times 10^{-3}$
FedFull	$1\times 10^{-3}$	$8\times 10^{-4}$	$5\times 10^{-4}$	$5\times 10^{-4}$
FedLore-Random	$3\times 10^{-3}$	$3\times 10^{-3}$	$3\times 10^{-3}$	$1\times 10^{-3}$
FedLore	$3\times 10^{-3}$	$3\times 10^{-3}$	$3\times 10^{-3}$	$1\times 10^{-3}$

Appendix

Table 8: Optimal learning rates for different methods and LLaMA model scales. The learning rate is selected according to validation perplexity on C4.

Configuration	Value
Datasets	CIFAR-100, Tiny-ImageNet, Food-101
Backbone	Swin-Base, ViT-Base
Number of clients	50
Data partition	Dirichlet ( $\alpha=0.1$ )
Communication rounds	100
Local steps $K$	50

Appendix

Table 9: Experimental configurations for vision federated fine-tuning.

Configuration	Value
Datasets	SNLI, AG News, QQP, DBPedia-14, QNLI, MNLI
Backbone	RoBERTa-Base
Number of clients	20
Data partition	Dirichlet ( $\alpha=0.5$ )
Communication rounds	100
Local steps $K$	50

Appendix

Table 10: Experimental configurations for language federated fine-tuning.

Method	CIFAR-100		Tiny-ImageNet		Food-101
	Swin-Base	ViT-Base	Swin-Base	ViT-Base	Swin-Base	ViT-Base
FedIT	$1e^{-3}$	$8e^{-4}$	$1e^{-3}$	$8e^{-4}$	$1e^{-3}$	$8e^{-4}$
FFA-LoRA	$3e^{-3}$	$3e^{-3}$	$3e^{-3}$	$3e^{-3}$	$3e^{-3}$	$3e^{-3}$
FlexLoRA	$1e^{-3}$	$1e^{-3}$	$1e^{-3}$	$1e^{-3}$	$1e^{-3}$	$1e^{-3}$
LoRA-FAIR	$3e^{-3}$	$3e^{-3}$	$3e^{-3}$	$3e^{-3}$	$3e^{-3}$	$3e^{-3}$
RoLoRA	$1e^{-3}$	$1e^{-3}$	$1e^{-3}$	$1e^{-3}$	$1e^{-3}$	$1e^{-3}$

Appendix

Table 11: Selected learning rates for vision federated fine-tuning experiments.

Method	SNLI	AG News	QQP	DBPedia-14	QNLI	MNLI
FedIT	$5e^{-4}$	$1e^{-3}$	$5e^{-4}$	$1e^{-3}$	$5e^{-4}$	$5e^{-4}$
FFA-LoRA	$1e^{-3}$	$3e^{-3}$	$1e^{-3}$	$3e^{-3}$	$1e^{-3}$	$1e^{-3}$
FlexLoRA	$5e^{-4}$	$1e^{-3}$	$5e^{-4}$	$1e^{-3}$	$5e^{-4}$	$5e^{-4}$
LoRA-FAIR	$1e^{-3}$	$3e^{-3}$	$1e^{-3}$	$3e^{-3}$	$1e^{-3}$	$1e^{-3}$
RoLoRA	$1e^{-3}$	$3e^{-3}$	$1e^{-3}$	$3e^{-3}$	$1e^{-3}$	$1e^{-3}$
FLoRA	$1e^{-3}$	$3e^{-3}$	$1e^{-3}$	$3e^{-3}$	$1e^{-3}$	$1e^{-3}$

Appendix

Table 12: Selected learning rates for RoBERTa-Base federated fine-tuning on different NLP datasets.

Method	Communication Cost	Optimizer Memory
FedIT	$\mathcal{O}(2r(m+n))$	$\mathcal{O}(r(m+n))$
FedFull	$\mathcal{O}(2mn)$	$\mathcal{O}(mn)$
Local GaLore	$\mathcal{O}(2mn)$	$\mathcal{O}(rn)$
FedLore	$\mathcal{O}(2r(m+n))$	$\mathcal{O}(rn)$

Appendix

Table 13: Communication and optimizer memory complexity comparison.

Method	60M	130M	350M	1B
Full-Rank	0.23G	0.51G	1.37G	5.20G
GaLore	0.13G	0.28G	0.54G	1.78G
Low-Rank	0.17G	0.37G	0.72G	2.38G
LoRA	0.17G	0.37G	0.72G	2.38G
FedLore	0.13G	0.28G	0.54G	1.78G

Appendix

Table 14: Memory estimation of optimizer states under BF16 precision.

Explore similar work

Subspace-Constrained Federated Learning with Low-Rank Adaptation
Jun 21, 2026Neranjan Senarath, Rohit Muralitharan, Sadia AsifFederated Learning Subspace
Department of ECSE Rensselaer Polytechnic Institute Troy, NY 12180, USA · Department of Computer Science Rensselaer Polytechnic Institute Troy, NY 12180, USA
FedSmoothLoRA: Toward Smoother and Faster Convergence in Federated Low-Rank Adaptation
May 28, 2026Zehao Wang, Guanglei Yang, Yihan Zeng +4Multi-Lora Smoothing
1Harbin Institute of Technology · 2Huawei Noah’s Ark Lab · University College Dublin
Rethinking Factor Sharing in Federated LoRA: A Rank-Aware Adaptive Approach
Aug 10, 2026Xinyi Xu, Bingnan Xiao, Shuang Qin +2Federated Learning Subspace
University of Electronic Science and Technology of China · Fudan University · Singapore University of Technology and Design

Jun 21, 2026, cs.LGJ/K move · Enter open · S save

Subspace-Constrained Federated Learning with Low-Rank Adaptation

Neranjan Senarath, Rohit Muralitharan, Sadia Asif

Department of ECSE Rensselaer Polytechnic Institute Troy, NY 12180, USA · Department of Computer Science Rensselaer Polytechnic Institute Troy, NY 12180, USA

Federated Learning Subspace

Open paper AskPDF arXiv

Federated low-rank adaptation methods are attractive for fine-tuning large models under communication and privacy constraints, but heterogeneous client data can induce geometric misalignment between local low-rank updates. We study whether this subspace misalignment leads to destructive aggregation and slower convergence in LoRA-based federated learning. We propose a subspace-regularized federated LoRA objective that encourages local client updates to remain close to a shared global reference subspace. We present a complete empirical evaluation on two pretrained models, RoBERTa-large and SmolLM-360M, over HellaSwag in a non-IID 10-client federated setting, across 3 random seeds (42, 43, 44), yielding 24 total experimental runs (4 methods x 3 seeds x 2 models). On RoBERTa-large, Subspace-Reg achieves the strongest mean best accuracy (0.454 +/- 0.023), mean final accuracy (0.429 +/- 0.011), and lowest final loss (1.363) across all three seeds, outperforming FedAvg, SVD redistribution, and FedSVD baselines by a large margin. On SmolLM-360M, FedAvg leads on accuracy, revealing that accuracy gains are model-dependent. Crucially, Subspace-Reg achieves near-perfect basis overlap, approximately 0.9999, on both models and across all seeds, versus 0.958 to 0.991 for all baselines, providing robust support for the geometric alignment hypothesis. The code is publicly available at https://github.com/sadia-sigma-lab/Subspace-Constrained-Federated-learning-with-Lora.

Abstract

Explore similar work

Subspace-Constrained Federated Learning with Low-Rank Adaptation

FedSmoothLoRA: Toward Smoother and Faster Convergence in Federated Low-Rank Adaptation

Rethinking Factor Sharing in Federated LoRA: A Rank-Aware Adaptive Approach