FedLore: Communication and Memory Efficient Federated Learning via Shared Gradient Low-Rank Projection

Junkang Liu

cs.AIOct 1, 2026

FedLore: Communication and Memory Efficient Federated Learning via Shared Gradient Low-Rank Projection

Authors: Junkang Liu

Organizations: Tianjin University

Abstract

Federated training of foundation models is constrained by client memory and communication costs. LoRA-based methods reduce these costs through low-rank adapters, but their fixed rank budget can limit adaptation. Gradient low-rank optimization offers greater flexibility, yet independently chosen client subspaces create a problem we term \emph{subspace fragmentation}: local projections interact with data heterogeneity to bias aggregated directions, while aggregation can increase update rank and communication cost. Thus, accurate local gradient compression need not preserve global descent. We propose \texttt{FedLore}, which shares a low-rank optimization basis within each round and refreshes it across rounds. The shared basis enables exact aggregation in low-rank coordinates and eliminates the identified projection bias. Subspace refresh allows the accumulated model update to exceed the per-round rank budget. We characterize the aggregation bias and establish an $O(T^{-1/2})$ stationarity bound for the projected-SGD variant under a global-gradient coverage condition and standard smoothness and variance assumptions, with bounded gradient heterogeneity. Experiments on vision and language tasks, including federated pre-training, show that \texttt{FedLore} outperforms the evaluated low-rank adapter baselines and matches or exceeds full-parameter training, while reducing communication and optimizer-state memory.

Figures & tables

Panel of Figure 1 — Figure 1: Comparison between FedLore and Local GaLore and FedIT on ViT-Tiny over CIFAR-100 with non-iid data. (a) Test accuracy: FedLore vs FedIT vs Local GaLore. (b) Effective Rank: Dynamic expansion of optimization space of FedLore . (c) Subspace fragmentation of Local GaLore. We measure subspace fragmentation by the difference between projection matrices: $\Delta_{\mathcal{D}}=\frac{1}{Nr}\sum_{i\in\mathcal{N}}\|\Pi_{i}-\bar{\Pi}\|_{F}^{2},\Pi_{i}=P_{i}P_{i}^{\top},\bar{\Pi}=\frac{1}{N}\sum_{i\in\mathcal{N}}\Pi_{i}$ . Client-specific projections introduce severe subspace fragmentation under heterogeneous data. (d) Local GaLore degrades under heterogeneity due to misaligned client-wise optimization subspaces.

Figure 2: Fixed-rank LoRA vs ours low-rank updates. FedLore shares a low-rank update basis within each round and refreshes it across rounds, allowing the accumulated model change to exceed the per-round rank.

Method	Venue	CIFAR-100		Tiny-ImageNet		Food-101		Avg.
Method	Venue	Swin-Base	ViT-Base	Swin-Base	ViT-Base	Swin-Base	ViT-Base	Avg.
FedIT	ICASSP’24	$90.17\pm 0.28$	$89.57\pm 0.28$	$86.50\pm 0.22$	$84.88\pm 0.12$	$91.55\pm 0.18$	$91.02\pm 0.20$	$88.95$
FFA-LoRA	ICLR’24	$90.07\pm 0.18$	$86.26\pm 0.14$	$85.34\pm 0.18$	$77.89\pm 0.11$	$91.25\pm 0.12$	$90.68\pm 0.22$	$86.92$
FlexLoRA	ICLR’25	$90.91\pm 0.19$	$89.59\pm 0.19$	$85.94\pm 0.19$	$85.55\pm 0.19$	$91.82\pm 0.16$	$91.34\pm 0.10$	$89.19$
LoRA-FAIR	ICCV’25	$90.26\pm 0.19$	$89.86\pm 0.19$	$86.59\pm 0.15$	$85.11\pm 0.19$	$91.41\pm 0.12$	$91.23\pm 0.14$	$89.08$
RoLoRA	NeurIPS’25	$90.55\pm 0.26$	$88.55\pm 0.26$	$86.03\pm 0.28$	$82.97\pm 0.28$	$90.54\pm 0.15$	$90.32\pm 0.08$	$88.16$

Table 1: Test accuracy (%) under Dir-0.1 over

T=100

. Experiments are conducted with 50 clients, batch size 16, local steps

K=50

, and LoRA rank

r=8

, projection rank

r=8

. Best results are shown in bold and second-best results are underlined.

Method	Venue	SNLI	AG News	QQP	DBPedia 14	QNLI	MNLI	Avg.
FedIT	ICASSP’24	$86.50\pm 0.13$	$92.68\pm 0.09$	$85.06\pm 0.14$	$98.94\pm 0.07$	$89.47\pm 0.11$	$81.77\pm 0.15$	$89.07$
FFA-LoRA	ICLR’24	$80.70\pm 0.17$	$90.14\pm 0.12$	$81.48\pm 0.16$	$98.22\pm 0.10$	$85.61\pm 0.14$	$72.44\pm 0.18$	$84.77$
FlexLoRA	ICLR’25	$86.64\pm 0.15$	$93.30\pm 0.11$	$63.18\pm 0.19$	$98.06\pm 0.09$	$49.46\pm 0.20$	$35.44\pm 0.18$	$71.01$
LoRA-FAIR	ICCV’25	$86.76\pm 0.10$	$92.76\pm 0.08$	$85.33\pm 0.12$	$98.88\pm 0.07$	$89.47\pm 0.09$	$82.04\pm 0.11$	$89.21$
RoLoRA	NeurIPS’25	$85.95\pm 0.12$	$92.06\pm 0.10$	$84.51\pm 0.13$	$98.78\pm 0.08$	$88.30\pm 0.12$	$80.58\pm 0.14$	$88.36$
FLoRA	NeurIPS’24	$69.36\pm 0.18$	$89.77\pm 0.13$	$79.78\pm 0.15$	$98.22\pm 0.11$	$81.93\pm 0.16$	$78.23\pm 0.17$	$82.88$

Table 2: Test accuracy (%) of RoBERTa-Base with LoRA on six text classification tasks under FL. Experiments are conducted for

T=100

rounds with 20 clients, Dir-0.5 data partition, batch size 16, local steps

K=50

, and LoRA rank

r=8

Method	60M		130M		350M		1B		Comm. $\downarrow$	Time/Round $\downarrow$
Method	Loss	PPL	Loss	PPL	Loss	PPL	Loss	PPL	Comm. $\downarrow$	on LLaMA 350M (s)
FedIT	8.811	6707.62	8.544	5135.85	8.072	3203.50	7.561	1921.77	$O(rd)$	118.6
Local GaLore	4.326	75.64	4.215	67.69	3.962	52.56	3.781	43.86	$O(d^{2})$	136.6
FedFull	4.134	62.43	4.056	57.74	3.798	44.61	3.561	35.20	$O(d^{2})$	132.1
FedLore-Random	4.112	61.07	4.036	56.60	3.781	43.86	3.523	33.89	$O(rd)$	119.8
FedLore	3.833	46.20	3.611	37.00	3.554	34.95	3.345	28.36	$O(rd)$	122.4

Table 3: Comparison with low-rank federated algorithms for pre-training LLaMA models on the C4 dataset. We report validation loss (Val Loss), validation perplexity (Val PPL), communication cost (Comm.), and local training time. All methods are trained for 100 communication rounds (20 clients, batch size 16,

K=50

). The projection rank of FedLore is identical to LoRA rank.

Panel of Figure 3 — Figure 3: (a,b,c): Validation loss convergence curves on C4 with LLaMA models of different scales. (d): Effect of the projection rank $r$ on validation loss for LLaMA 350M.

Method	Comm.	Opt. Memory	Ablation Setting	Val Loss / PPL	Comm. $\downarrow$
FedIT	$0.36$	$0.72$	Local GaLore	3.962 / 52.56	$0.68$
FedFull	$0.68$	$1.37$	FedLore	3.554 / 34.95	$0.36$
Local GaLore	$0.68$	$0.54$	w/o Shared Projector	3.890 / 48.91	$0.68$
FedLore	$0.36$	$0.54$	w/o Dynamic Subspace	5.952 / 384.7	$0.18$

Table 4: Communication and optimizer memory on LLaMA 350M (left), and component ablations on C4 pre-training (right). All costs are in GB.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Hidden	Intermediate	Heads	Layers
LLaMA-60M	512	1376	8	8
LLaMA-130M	768	2048	12	12
LLaMA-350M	1024	2736	16	24
LLaMA-1B	2048	5461	32	24

Appendix

Table 5: LLaMA architecture configurations used in federated pre-training. Model names follow the GaLore implementation.

Configuration	Value
Dataset	C4
Number of clients	20
Client participation	100%
Communication rounds	100
Local steps per round	50
Batch size	16

Appendix

Table 6: Federated pre-training configurations.

Method	Communication Cost	Optimizer Memory
FedIT	$O(2r(m+n))$	$O(r(m+n))$
FedFull	$O(2mn)$	$O(mn)$
Local GaLore	$O(2mn)$	$O(rn)$
FedLore	$O(2r(m+n))$	$O(rn)$

Appendix

Table 7: Communication and optimizer memory comparison.

r\ll\min(m,n)

Method	LLaMA-60M	LLaMA-130M	LLaMA-350M	LLaMA-1B
FedIT	$1\times 10^{-3}$	$8\times 10^{-4}$	$5\times 10^{-4}$	$5\times 10^{-4}$
Local GaLore	$3\times 10^{-3}$	$3\times 10^{-3}$	$1\times 10^{-3}$	$1\times 10^{-3}$
FedFull	$1\times 10^{-3}$	$8\times 10^{-4}$	$5\times 10^{-4}$	$5\times 10^{-4}$
FedLore-Random	$3\times 10^{-3}$	$3\times 10^{-3}$	$3\times 10^{-3}$	$1\times 10^{-3}$
FedLore	$3\times 10^{-3}$	$3\times 10^{-3}$	$3\times 10^{-3}$	$1\times 10^{-3}$

Appendix

Table 8: Optimal learning rates for different methods and LLaMA model scales. The learning rate is selected according to validation perplexity on C4.

Configuration	Value
Datasets	CIFAR-100, Tiny-ImageNet, Food-101
Backbone	Swin-Base, ViT-Base
Number of clients	50
Data partition	Dirichlet ( $\alpha=0.1$ )
Communication rounds	100
Local steps $K$	50

Appendix

Table 9: Experimental configurations for vision federated fine-tuning.

Configuration	Value
Datasets	SNLI, AG News, QQP, DBPedia-14, QNLI, MNLI
Backbone	RoBERTa-Base
Number of clients	20
Data partition	Dirichlet ( $\alpha=0.5$ )
Communication rounds	100
Local steps $K$	50

Appendix

Table 10: Experimental configurations for language federated fine-tuning.

Method	CIFAR-100		Tiny-ImageNet		Food-101
	Swin-Base	ViT-Base	Swin-Base	ViT-Base	Swin-Base	ViT-Base
FedIT	$1e^{-3}$	$8e^{-4}$	$1e^{-3}$	$8e^{-4}$	$1e^{-3}$	$8e^{-4}$
FFA-LoRA	$3e^{-3}$	$3e^{-3}$	$3e^{-3}$	$3e^{-3}$	$3e^{-3}$	$3e^{-3}$
FlexLoRA	$1e^{-3}$	$1e^{-3}$	$1e^{-3}$	$1e^{-3}$	$1e^{-3}$	$1e^{-3}$
LoRA-FAIR	$3e^{-3}$	$3e^{-3}$	$3e^{-3}$	$3e^{-3}$	$3e^{-3}$	$3e^{-3}$
RoLoRA	$1e^{-3}$	$1e^{-3}$	$1e^{-3}$	$1e^{-3}$	$1e^{-3}$	$1e^{-3}$

Appendix

Table 11: Selected learning rates for vision federated fine-tuning experiments.

Method	SNLI	AG News	QQP	DBPedia-14	QNLI	MNLI
FedIT	$5e^{-4}$	$1e^{-3}$	$5e^{-4}$	$1e^{-3}$	$5e^{-4}$	$5e^{-4}$
FFA-LoRA	$1e^{-3}$	$3e^{-3}$	$1e^{-3}$	$3e^{-3}$	$1e^{-3}$	$1e^{-3}$
FlexLoRA	$5e^{-4}$	$1e^{-3}$	$5e^{-4}$	$1e^{-3}$	$5e^{-4}$	$5e^{-4}$
LoRA-FAIR	$1e^{-3}$	$3e^{-3}$	$1e^{-3}$	$3e^{-3}$	$1e^{-3}$	$1e^{-3}$
RoLoRA	$1e^{-3}$	$3e^{-3}$	$1e^{-3}$	$3e^{-3}$	$1e^{-3}$	$1e^{-3}$
FLoRA	$1e^{-3}$	$3e^{-3}$	$1e^{-3}$	$3e^{-3}$	$1e^{-3}$	$1e^{-3}$

Appendix

Table 12: Selected learning rates for RoBERTa-Base federated fine-tuning on different NLP datasets.

Method	Communication Cost	Optimizer Memory
FedIT	$\mathcal{O}(2r(m+n))$	$\mathcal{O}(r(m+n))$
FedFull	$\mathcal{O}(2mn)$	$\mathcal{O}(mn)$
Local GaLore	$\mathcal{O}(2mn)$	$\mathcal{O}(rn)$
FedLore	$\mathcal{O}(2r(m+n))$	$\mathcal{O}(rn)$

Appendix

Table 13: Communication and optimizer memory complexity comparison.

Method	60M	130M	350M	1B
Full-Rank	0.23G	0.51G	1.37G	5.20G
GaLore	0.13G	0.28G	0.54G	1.78G
Low-Rank	0.17G	0.37G	0.72G	2.38G
LoRA	0.17G	0.37G	0.72G	2.38G
FedLore	0.13G	0.28G	0.54G	1.78G

Appendix

Table 14: Memory estimation of optimizer states under BF16 precision.

Explore similar work

Jun 21, 2026cs.LG

Subspace-Constrained Federated Learning with Low-Rank Adaptation

Federated low-rank adaptation methods are attractive for fine-tuning large models under communication and privacy constraints, but heterogeneous client data can induce geometric misalignment between local low-rank updates. We study whether this subspace misalignment leads to destructive aggregation and slower convergence in LoRA-based federated learning. We propose a subspace-regularized federated LoRA objective that encourages local client updates to remain close to a shared global reference subspace. We present a complete empirical evaluation on two pretrained models, RoBERTa-large and SmolLM-360M, over HellaSwag in a non-IID 10-client federated setting, across 3 random seeds (42, 43, 44), yielding 24 total experimental runs (4 methods x 3 seeds x 2 models). On RoBERTa-large, Subspace-Reg achieves the strongest mean best accuracy (0.454 +/- 0.023), mean final accuracy (0.429 +/- 0.011), and lowest final loss (1.363) across all three seeds, outperforming FedAvg, SVD redistribution, and FedSVD baselines by a large margin. On SmolLM-360M, FedAvg leads on accuracy, revealing that accuracy gains are model-dependent. Crucially, Subspace-Reg achieves near-perfect basis overlap, approximately 0.9999, on both models and across all seeds, versus 0.958 to 0.991 for all baselines, providing robust support for the geometric alignment hypothesis. The code is publicly available at https://github.com/sadia-sigma-lab/Subspace-Constrained-Federated-learning-with-Lora.

Neranjan Senarath, Rohit Muralitharan, Sadia Asif

Department of ECSE Rensselaer Polytechnic Institute Troy, NY 12180, USA · Department of Computer Science Rensselaer Polytechnic Institute Troy, NY 12180, USA

Federated Learning Subspace

Ask PDF

May 28, 2026cs.CV

FedSmoothLoRA: Toward Smoother and Faster Convergence in Federated Low-Rank Adaptation

Federated fine-tuning of foundation models with Low-Rank Adaptation (LoRA) provides an efficient solution for reducing communication and computation costs while preserving data locality. However, the direct combination of FedAvg and LoRA suffers from three key issues: limited update space, which restricts the model's effective learning capacity; inter-round state mismatch, which disrupts cross-round local optimization continuity; and a client-agnostic starting state, which slows local convergence on clients. Although recent methods mitigate the limited update space issue by merging LoRA updates into the backbone across communication rounds, inter-round state mismatch and the client-agnostic starting state remain insufficiently addressed. To address these issues, we propose FedSmoothLoRA, a federated LoRA tuning framework that preserves the enlarged update space, improves cross-round local optimization continuity, and provides a client-aware starting state for local training. At each communication round, FedSmoothLoRA constructs the local LoRA initialization using two matrices: a Round-Matching matrix that preserves cross-round local state continuity, and a Gradient-Aligned matrix that provides client-specific optimization guidance from gradient signals estimated on local data. Together, these designs enable smoother and faster convergence. Extensive experiments on image classification and natural language generation tasks demonstrate that FedSmoothLoRA consistently outperforms existing federated LoRA tuning methods. Code: https://github.com/wangzehao0704/FedSmoothLoRA

Zehao Wang, Guanglei Yang, Yihan Zeng +4

1Harbin Institute of Technology · 2Huawei Noah’s Ark Lab · University College Dublin

Multi-Lora Smoothing Foundation Model Convergence Constrained Optimization

Ask PDF

Aug 10, 2026cs.LG

Rethinking Factor Sharing in Federated LoRA: A Rank-Aware Adaptive Approach

Low-rank adaptation (LoRA) represents large language model (LLM) updates with two compact matrix factors, i.e.,

A

and

B

, providing an efficient way to fine-tune large models in federated learning paradigm. Inspired by the asymmetric roles of the LoRA factors, we study whether

A

should be shared across clients while

B

remains client-specific (Share-A/Local-B), or whether

B

should instead be shared while

A

remains client-specific (Share-B/Local-A). With a least-squares surrogate, we reveal that Share-A/Local-B requires the client-specific LoRA update matrices to use a common rank-

r

input-side space, whereas Share-B/Local-A requires a common rank-

r

output-side space. The two strategies therefore incur different projection residuals, indicating that the preferred strategy is the one with the smaller aggregate residual across clients. With this insight, we propose Federated Adaptive Factor Sharing Low-Rank Adaptation (FedAS-LoRA), which selects the sharing side before training to enhance fine-tuning performance. To enable adaptive factor selection before training, we design a Rank-Aware Shared-Subspace Sufficiency (RSS) metric, which effectively assesses whether a shared rank-

r

input subspace is sufficient for the local data distributions using representations extracted from a frozen LLM backbone. Experiments across different tasks, data distributions, LoRA ranks, and participation settings confirm the effectiveness of RSS and the superior performance of FedAS-LoRA.

Xinyi Xu, Bingnan Xiao, Shuang Qin +2

University of Electronic Science and Technology of China · Fudan University · Singapore University of Technology and Design

Federated Learning Subspace Exploratory Factor Analysis

Ask PDF