cs.LGApr 12, 2026

Communication-Efficient Gluon in Federated Learning

Authors: Xun Qian, Alexander Gaponov, Grigory Malinovsky, Peter Richtárik

Organizations: Center of Excellence for Generative AI, King Abdullah University of Science and Technology, Thuwal, Saudi Arabia.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Recent developments have shown that Muon-type optimizers based on linear minimization oracles (LMOs) over non-Euclidean norm balls have the potential to get superior practical performance than Adam-type methods in the training of large language models. Since large-scale neural networks are trained across massive machines, communication cost becomes the bottleneck. To address this bottleneck, we investigate Gluon, which is an extension of Muon under the more general layer-wise (L0,L1)(L^0, L^1)-smooth setting, with both unbiased and contraction compressors. In order to reduce the compression error, we employ the variance reduced technique in SARAH in our compressed methods. The convergence rates and improved communication cost are achieved under certain conditions. As a byproduct, a new variance reduced algorithm with faster convergence rate than Gluon is obtained. We also incorporate momentum variance reduction (MVR) to these compressed algorithms and comparable communication cost is derived under weaker conditions when Li1≠0L_i^1 \neq 0. Finally, several numerical experiments are conducted to verify the superior performance of our compressed algorithms in terms of communication cost.

Figures & tables

MethodCommunication CostOracle Complexity
Compressed GluonO(nd+1c2ϵ4n+ndϵ2){\cal O}\left(nd+\frac{1}{c^{2}\epsilon^{4}\sqrt{n}}+\frac{\sqrt{nd}}{\epsilon^{2}}\right)O(1ϵ2+(1+1c2)1ϵ4n){\cal O}\left(\frac{1}{\epsilon^{2}}+(1+\frac{1}{c^{2}})\frac{1}{\epsilon^{4}\sqrt{n}}\right) , for any 0<c2≤1ϵ2n3/20<c^{2}\leq\frac{1}{\epsilon^{2}n^{3/2}}
Compressed Gluon with MVRO(nd+1c2ϵ4n+ndϵ2){\cal O}\left(nd+\frac{1}{c^{2}\epsilon^{4}\sqrt{n}}+\frac{\sqrt{nd}}{\epsilon^{2}}\right)O(1ϵ2+(1+1c2)1ϵ4n){\cal O}\left(\frac{1}{\epsilon^{2}}+(1+\frac{1}{c^{2}})\frac{1}{\epsilon^{4}\sqrt{n}}\right) , for any 0<c2≤1ϵ2n3/20<c^{2}\leq\frac{1}{\epsilon^{2}n^{3/2}}
Compressed Gluon with Error FeedbackO(1c2ϵ4+ndϵ2){\cal O}\left(\frac{1}{c^{2}\epsilon^{4}}+\frac{nd}{\epsilon^{2}}\right)O(1c2ϵ4+c2ϵ2){\cal O}\left(\frac{1}{c^{2}\epsilon^{4}}+\frac{c^{2}}{\epsilon^{2}}\right) , for any 0<c2≤1ϵ2n0<c^{2}\leq\frac{1}{\epsilon^{2}n}
Compressed Gluon with Error Feedback and MVRO(1c2ϵ4+ndϵ2){\cal O}\left(\frac{1}{c^{2}\epsilon^{4}}+\frac{nd}{\epsilon^{2}}\right)O(1c2ϵ4+c2ϵ2){\cal O}\left(\frac{1}{c^{2}\epsilon^{4}}+\frac{c^{2}}{\epsilon^{2}}\right) , for any 0<c2≤1ϵ2n0<c^{2}\leq\frac{1}{\epsilon^{2}n}
Table 1: Summary of the results to reach the ϵ\epsilon -precision for the problem ( 1 ), i.e., min⁡k=0,...,K−1∑i=1ptiE[∥∇if(Xk)∥(i)⋆]≤ϵ\min_{k=0,...,K-1}\sum_{i=1}^{p}t_{i}\mathbb{E}\left[\|\nabla_{i}f(X^{k})\|_{(i)\star}\right]\leq\epsilon . Dependences on the numerical constants,“quality” of the starting point, smoothness constants, and Hessian variance constants, etc., are omitted in the complexity bounds. We assume the compression ratios of unbiased compressor and contraction compressor are Θ(ω+1)\Theta(\omega+1) and Θ(1δ)\Theta(\frac{1}{\delta}) , respectively. Abbreviations: “Communication cost” = the total communication cost needed to reach the ϵ\epsilon -precision in expectation; “Oracle complexity” = the total number of (stochastic) first-order oracle calls needed to reach the ϵ\epsilon -precision in expectation. The value of parameter cc in the Communication cost is the same as in the corresponding Oracle complexity.
 Xik+1=LMOBik(Mik):=arg⁡min⁡Xi∈Bik⟨Mik,Xi⟩(i)X_{i}^{k+1}=\text{LMO}_{\mathcal{B}_{i}^{k}}(M_{i}^{k}):=\arg\min_{X_{i}\in\mathcal{B}_{i}^{k}}\langle M_{i}^{k},X_{i}\rangle_{(i)} (7)
Algorithm 1 Compressed Gluon
 Xik+1=LMOBik(Mik):=arg⁡min⁡Xi∈Bik⟨Mik,Xi⟩(i)X_{i}^{k+1}=\text{LMO}_{\mathcal{B}_{i}^{k}}(M_{i}^{k}):=\arg\min_{X_{i}\in\mathcal{B}_{i}^{k}}\langle M_{i}^{k},X_{i}\rangle_{(i)} (15)
Algorithm 2 Compressed Gluon with Error Feedback
Figure 1: Logreg, a5a. f(x)−f∗f(x)-f^{*} vs. communication cost. Algorithms 1 , 2 and VR-MARINA utilize compression parameter K=1%K=1\%
Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

 Xik+1=LMOBik(Mik):=arg⁡min⁡Xi∈Bik⟨Mik,Xi⟩(i)X_{i}^{k+1}=\text{LMO}_{\mathcal{B}_{i}^{k}}(M_{i}^{k}):=\arg\min_{X_{i}\in\mathcal{B}_{i}^{k}}\langle M_{i}^{k},X_{i}\rangle_{(i)} (30)
Appendix
Algorithm 3 Compressed Gluon with MVR
 Xik+1=LMOBik(Mik):=arg⁡min⁡Xi∈Bik⟨Mik,Xi⟩(i)X_{i}^{k+1}=\text{LMO}_{\mathcal{B}_{i}^{k}}(M_{i}^{k}):=\arg\min_{X_{i}\in\mathcal{B}_{i}^{k}}\langle M_{i}^{k},X_{i}\rangle_{(i)} (40)
Appendix
Algorithm 4 Compressed Gluon with Error Feedback and MVR
Figure 3: f(x)−f∗f(x)-f^{*} vs. Step, Algorithm 1 , Logistic Regression, a5a dataset, B=1B=1 across various β\beta , lr, compression factor K%K\% and qq .
Figure 4: f(x)−f∗f(x)-f^{*} vs. Step, Algorithm 1 , Logistic Regression, a5a dataset, B=1B=1 across various β\beta , lr, compression factor K%K\% and qq .
Figure 5: f(x)−f∗f(x)-f^{*} vs. Step, Algorithm 1 , Logistic Regression, a5a dataset, B=16B=16 across various β\beta , lr, compression factor K%K\% and qq .
Figure 6: f(x)−f∗f(x)-f^{*} vs. training step, Algorithm 2 , logistic regression ablation on the a5a dataset, B=16B=16 , Top 1%1\% compression. Only stable trajectories are presented.
Figure 7: f(x)−f∗f(x)-f^{*} vs. training step, VR-MARINA, logistic regression on the a5a dataset, B=16B=16 (to ensure a fair comparison with our methods), Rand 1%1\% compression. Only stable trajectories are presented.
Figure 8: Loss vs. training step, CIFAR10, compression parameter K=1%K=1\% , learning rate and momentum tuning for Non-Euclidean SGD baseline, Algorithm 1 , and VR-MARINA.
More in this topic

Communication-Efficient Distributed Training

Communication-efficient distributed training reduces the data exchanged among devices so models can be trained faster and over limited-bandwidth networks.

See all 225 papers

Explore similar work

CardsList

  1. Sign compression for Muon: SignMuon, MuonSign, and the Limits of Error Feedback

    Jul 31, 2026Maria Smirnova, Alexey KravatskiySign-Based OptimizationCommunication-Efficient Distributed Training

    MIRIAI Moscow, Russia

  2. Communication-Efficient Agnostic Federated Learning via Faster Convergence and Compression

    Sep 29, 2026Haomin Bai, Junyan Sun, Sifan Yang +2Error FeedbackOnline Federated Learning

    State Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China · School of Artificial Intelligence, Nanjing University, Nanjing, China · Department of Computer Science, City University of Hong Kong, Hong Kong, China

  3. FedSubMuon: Communication-Efficient Federated LLM Fine-Tuning via Structured Subspace Muon

    Sep 5, 2026Shaolong Chen, Youming Tao, Shuzhen Chen +3Subspace OptimizationFederated Fine-Tuning

    King Abdullah University of Science and Technology (KAUST) · Provable Responsible AI and Data Analytics (PRADA) Lab · Imperial College London +3