cs.LGNov 6, 2026

DisMUSE: An Efficient Distributed Singular-Spectrum-Based Optimization Approach with Theoretical Convergence Guarantee

Authors: Zhen Qin, Zhiyao Zhang, Yingbin Liang, Jia Liu

Organizations: Department of Electrical and Computer Engineering, The Ohio State University, Columbus, Ohio, USA

Abstract

In recent years, singular-spectrum-based optimizers have attracted increasing attention due to their ability to preserve the matrix structure of neural network gradients, thereby enabling geometry-aware optimization. However, most singular-spectrum-based (SSB) approaches remain centralized, limiting their use in distributed learning infrastructures that exploit massive parallelism. Distributed SSB optimization is still at an early stage, with open questions about convergence guarantees and communication efficiency. To fill this gap, we propose the Distributed Momentum-based Spectral Estimator (DisMUSE), an efficient distributed SSB optimization framework for both data-center and federated settings. Using Muon-style exact orthogonalized momentum as the spectral update, DisMUSE integrates a recursive variance-reduced momentum estimator and achieves an O((KT)−1/3)O((KT)^{-1/3}) convergence rate, where KK and TT denote the number of nodes and local iterations per node, respectively. This rate matches the best-known rates for variance-reduced nonconvex distributed optimization. Building on this variance-reduced framework, we study two extensions. First, our DisMUSE method with finite Newton-Schulz (NS) iterations preserves the same convergence order under a uniform normalized support-spectrum condition, up to a multiplicative factor. Second, we develop a communication-efficient variant based on low-rank (LR) momentum transmission and quantify the error it introduces to the stationarity bound, characterizing the trade-off between accuracy and communication cost.

Figures & tables

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. SignMuon: Communication-Efficient Distributed Muon Optimization

    May 4, 2026Neel Mishra, Jichao Jiang, Kushagara Trivedi +3Sign-Based OptimizationCommunication-Efficient Distributed Training

  2. Orth-Dion: Eliminating Geometric Mismatch in Distributed Low-Rank Spectral Optimization

    May 7, 2026Tatsuhiro Nakamori, Laura Gomezjurado Gonzalez, Ganesh Talluri +5Communication-Efficient Distributed TrainingDistributed Optimization

  3. Scalar Communication via Random Direction Refreshing for Distributed Optimization

    Oct 5, 2026Mohammadreza Rostami, Solmaz S. KiaDistributed OptimizationDecentralized Optimization