FedSubMuon: Communication-Efficient Federated LLM Fine-Tuning via Structured Subspace Muon

Shaolong Chen; Youming Tao; Shuzhen Chen; Falko Dressler; Qingqing Ye; Di Wang

cs.LGSep 5, 2026

FedSubMuon: Communication-Efficient Federated LLM Fine-Tuning via Structured Subspace Muon

Authors: Shaolong Chen, Youming Tao, Shuzhen Chen, Falko Dressler, Qingqing Ye, Di Wang

Organizations: King Abdullah University of Science and Technology (KAUST) · Provable Responsible AI and Data Analytics (PRADA) Lab · Imperial College London · Technische Universität Berlin (TU Berlin) ·

Abstract

Federated fine-tuning adapts large language models (LLMs) to decentralized client data, but its scalability in cross-device training is often limited by the high communication cost. Muon is an optimizer that improves optimization performance by orthogonalizing momentum for matrix-valued parameters. Existing federated Muon methods demonstrate the benefit of matrix-aware optimization in federated learning, but still require transmitting full layer-size updates and optimizer state. A natural way to reduce communication is to directly apply Muon to LoRA factors, but this changes the optimized object and weakens Muon's matrix-aware update geometry. We propose FedSubMuon, a communication-efficient federated Muon fine-tuning method that optimizes compact coefficient matrices within shared structured subspaces. This design keeps Muon on a single matrix-valued trainable object, while reducing the client upload to compact coefficient matrices. We further introduce FedSubMuon-GT, an accuracy-oriented extension that uses projected gradients to adapt tracked subspace bases toward task-relevant gradient directions. Experiments on instruction tuning and mathematical reasoning show that FedSubMuon-GT achieves the best overall accuracy on four of five dataset-model pairs, while FedSubMuon performs best under all matched communication budgets. On Dolly-15K, the closest communication baseline requires 5.5 times and 1.4 times more total communication on Llama-1B and Qwen-4B, respectively.

Figures & tables

Method	AdamW	Muon	SGD
FedIT	$30.96$	$30.81$	$27.94$
FLoRA	$31.14$	$30.99$	$27.85$

Table 1: Optimizer control for LoRA-factor federated baselines on Natural Instructions with Llama 1B. Each cell reports mean ROUGE-L% over two runs.

Method	Natural Instructions	Dolly-15K		GSM8K	MATH
Method	Llama 1B	Llama 1B	Qwen 4B	Qwen 8B	Qwen 8B
FedIT	30.14 $\pm$ 0.82	28.32 $\pm$ 0.41	36.33 $\pm$ 0.17	79.59 $\pm$ 0.33	45.24 $\pm$ 1.22
FeDeRA	31.07 $\pm$ 0.43	29.63 $\pm$ 0.53	36.78 $\pm$ 0.37	80.29 $\pm$ 0.26	47.52 $\pm$ 0.65
FLoRA	31.12 $\pm$ 0.28	29.30 $\pm$ 0.45	36.39 $\pm$ 0.36	76.58 $\pm$ 0.82	43.48 $\pm$ 0.42
FedEx-LoRA	30.29 $\pm$ 0.51	28.83 $\pm$ 0.10	36.57 $\pm$ 0.76	79.51 $\pm$ 0.25	45.82 $\pm$ 1.26
FLoRG	29.83 $\pm$ 0.42	28.53 $\pm$ 0.54	36.79 $\pm$ 0.70	80.12 $\pm$ 0.51	43.48 $\pm$ 0.42

Table 2: Performance comparison across datasets (%). Each cell reports mean

\pm

standard deviation over four runs. Best and second-best values are highlighted.

Method	Model / Dataset
	Llama/NI	Llama/Dolly	Qwen/Dolly
FedSubMuon	30.59	30.09	36.87
FedSubMuon-GT	29.19	29.32	36.83
FedIT	30.12	24.70	34.62
FeDeRA	30.10	26.68	36.49
FLoRA	10.28	16.95	28.77

Table 3: Performance under matched communication budgets. Each cell reports the mean ROUGE-L% over two runs under the matched communication budget.

Panel of Figure 2 — Figure 2: Validation-loss dynamics on Qwen 8B.

Model	Metric	FedSubMuon	FedSubMuon-GT	FedIT	FeDeRA	FLoRA	FedEx-LoRA	FLoRG	FedKRSO
Llama 1B	Uplink (MB)	105 (1 $\times$ )	923 (8.8 $\times$ )	920 (8.8 $\times$ )	1159 (11 $\times$ )	2045 (19 $\times$ )	920 (8.8 $\times$ )	721 (6.9 $\times$ )	1311 (12 $\times$ )
	Total Comm. (MB)	264 (1 $\times$ )	11848 (45 $\times$ )	1840 (7.0 $\times$ )	2317 (8.8 $\times$ )	22492 (85 $\times$ )	11042 (42 $\times$ )	1442 (5.5 $\times$ )	85679 (325 $\times$ )
	GPU Mem. (MB)	5530	5555	5761	5762	5764	5762	6567	24136
	Wall time (s)	77.19	75.33	83.47	81.25	82.65	84.61	114.18	77.61
Qwen 4B	Uplink (MB)	1410 (1 $\times$ )	7078 (5.0 $\times$ )	4720 (3.3 $\times$ )	5782 (4.1 $\times$ )	7078 (5.0 $\times$ )	6960 (4.9 $\times$ )	2490 (1.8 $\times$ )	13920 (9.9 $\times$ )
	Total Comm. (MB)	3457 (1 $\times$ )	83802 (24 $\times$ )	9440 (2.7 $\times$ )	11564 (3.3 $\times$ )	77880 (23 $\times$ )	83519 (24 $\times$ )	4980 (1.4 $\times$ )	1127508 (326 $\times$ )

Table 4: Communication and computational efficiency on Dolly-15K.

Panel of Figure 3 — Figure 3: FedSubMuon component ablations on Dolly-15K with Llama 1B. Each plotted value is the mean over two runs.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Method group	Rank	Trainable params (ratio)
Llama 1B	FedSubMuon/ FedSubMuon-GT	$64$	$131{,}072$ ( $1\times$ )
	FedIT/FeDeRA/FLoRA/ FedEx-LoRA	$8$	$851{,}968$ ( $6.5\times$ )
	FLoRG	$8$	$327{,}680$ ( $2.5\times$ )
	FedKRSO	$8$	$1{,}310{,}720$ ( $10\times$ )
Qwen 4B	FedSubMuon/ FedSubMuon-GT	$128$	$1{,}179{,}648$ ( $1\times$ )
	FedIT/FeDeRA/FLoRA/ FedEx-LoRA	$8$	$2{,}949{,}120$ ( $2.5\times$ )

Appendix

Table 5: Per-round per-client trainable parameter counts under the

q/v

-only adaptation setup.

Panel of Figure 4 — Figure 4: Federation ablations over client participation and data heterogeneity on Dolly-15K with Llama 1B; each plotted value is averaged over two runs.

Subspace Optimization

See all 12 papers

FedMomentum: Preserving LoRA Training Momentum in Federated Fine-Tuning
Mar 9, 2026Peishen Yan, Yang Hua, Hao Wang +4Federated Fine-Tuning Federated Learning Aggregation
Shanghai Jiao Tong University · Queen’s University Belfast · Stevens Institute of Technology +2
FedFit: Federated Fine-Tuning of LLMs via Vector-Bank Parameterization and Quantization
Oct 1, 2026Hang Zou, Chao Zhang, Yuzhi Yang +3Federated Learning Aggregation Low-Rank Adaptation
6G Research Center, Khalifa University, 127788 Abu Dhabi, UAE · Department of Computer Science, Central South University, 410083, Changsha, China · Université de Lorraine, CNRS, CRAN, F-54000 Nancy, France
FED-FSTQ: Fisher-Guided Token Quantization for Communication-Efficient Federated Fine-Tuning of LLMs on Edge Devices
Apr 28, 2026Changyu Li, Shuanghong Huang, Jiashen Liu +5Federated Fine-Tuning Federated Learning
School of Computing and Information Technology, Great Bay University, Dongguan, China · Beijing Institute of Technology, Beijing, China · University of Warwick, Coventry, U.K. +3

Mar 9, 2026J/K move · Enter open · S save

FedMomentum: Preserving LoRA Training Momentum in Federated Fine-Tuning

Peishen Yan, Yang Hua, Hao Wang, Jiaru Zhang+3

Shanghai Jiao Tong University · Queen’s University Belfast · Stevens Institute of Technology+2

Federated Fine-Tuning Federated Learning Aggregation Federated Learning Low-Rank Adaptation

Open paper AskPDF arXiv

Federated fine-tuning of large language models (LLMs) with low-rank adaptation (LoRA) offers a communication-efficient and privacy-preserving solution for task-specific adaptation. Naive aggregation of LoRA modules introduces noise due to mathematical incorrectness when averaging the downsampling and upsampling matrices independently. However, existing noise-free aggregation strategies inevitably compromise the structural expressiveness of LoRA, limiting its ability to retain client-specific adaptations by either improperly reconstructing the low-rank structure or excluding partially trainable components. We identify this problem as loss of training momentum, where LoRA updates fail to accumulate effectively across rounds, resulting in slower convergence and suboptimal performance. To address this, we propose FedMomentum, a novel framework that enables structured and momentum-preserving LoRA aggregation via singular value decomposition (SVD). Specifically, after aggregating low-rank updates in a mathematically correct manner, FedMomentum applies SVD to extract the dominant components that capture the main update directions. These components are used to reconstruct the LoRA modules with the same rank, while residual components can be retained and later merged into the backbone to preserve semantic information and ensure robustness. Extensive experiments across multiple tasks demonstrate that FedMomentum consistently outperforms prior state-of-the-art methods in convergence speed and final accuracy.

Abstract

Subspace Optimization

Explore similar work

FedMomentum: Preserving LoRA Training Momentum in Federated Fine-Tuning

FedFit: Federated Fine-Tuning of LLMs via Vector-Bank Parameterization and Quantization

FED-FSTQ: Fisher-Guided Token Quantization for Communication-Efficient Federated Fine-Tuning of LLMs on Edge Devices