DisMUSE: An Efficient Distributed Singular-Spectrum-Based Optimization Approach with Theoretical Convergence Guarantee
Organizations: Department of Electrical and Computer Engineering, The Ohio State University, Columbus, Ohio, USA
Abstract
In recent years, singular-spectrum-based optimizers have attracted increasing attention due to their ability to preserve the matrix structure of neural network gradients, thereby enabling geometry-aware optimization. However, most singular-spectrum-based (SSB) approaches remain centralized, limiting their use in distributed learning infrastructures that exploit massive parallelism. Distributed SSB optimization is still at an early stage, with open questions about convergence guarantees and communication efficiency. To fill this gap, we propose the Distributed Momentum-based Spectral Estimator (DisMUSE), an efficient distributed SSB optimization framework for both data-center and federated settings. Using Muon-style exact orthogonalized momentum as the spectral update, DisMUSE integrates a recursive variance-reduced momentum estimator and achieves an convergence rate, where and denote the number of nodes and local iterations per node, respectively. This rate matches the best-known rates for variance-reduced nonconvex distributed optimization. Building on this variance-reduced framework, we study two extensions. First, our DisMUSE method with finite Newton-Schulz (NS) iterations preserves the same convergence order under a uniform normalized support-spectrum condition, up to a multiplicative factor. Second, we develop a communication-efficient variant based on low-rank (LR) momentum transmission and quantify the error it introduces to the stationarity bound, characterizing the trade-off between accuracy and communication cost.
Figures & tables
| Parameter | Definition |
|---|---|
| Number of distributed nodes. | |
| Number of local optimization iterations per node. | |
| Constant model-update stepsize. | |
| STORM momentum parameter, where . | |
| Communication period; synchronization occurs every local steps. | |
| Initial mini-batch size used to initialize the momentum estimator. |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Test accuracy (%) |
|---|---|
| FedAvg | |
| FedAvg-SGDM | |
| FedMuon-FA | |
| FedMuon-SF | |
| DisMUSE-SVD | |
| DisMUSE-NS |
| Method | Perplexity |
|---|---|
| FedAvg | |
| FedAdam | |
| DisMUSE-SVD | |
| DisMUSE-NS | |
| DisMUSE-LR-SVD ( ) | |
| DisMUSE-LR-NS ( ) |
Distributed Optimization
Distributed optimization coordinates computation across multiple agents or machines while balancing convergence, communication, privacy, and reliability.
See all 99 papers