Subspace Optimization

Latest papers

12 papers2 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Memory-Efficient Subspace Optimization with Residual Correction: From Momentum SGD to Muon
Oct 14, 2026Sifan Yang, Hongyi Tao, Dingzhi Yu +5Subspace Optimization Memory-Efficient Training
State Key Laboratory of Novel Software Technology, Nanjing University · School of Artificial Intelligence, Nanjing University · Alibaba Token Hub, Alibaba Group +1
AIM-ZO: Activation-Informed Subspace Maintenance for Zeroth-Order LLM Fine-Tuning
Sep 28, 2026Yue Xie, Zhi Zheng, Yunpeng Ba +5Subspace Optimization Memory-Efficient Fine-Tuning
Southern University of Science and Technology · National University of Singapore · Huawei Technologies Ltd. +1
Gradient Descent with Stochastic Subspaces via Persistence of Memory
Sep 16, 2026Subhroshekhar Ghosh, Clement Z. Q. Ng, Pierre-Louis Poirion +1Gradient Descent Stochastic Optimization Convergence
Department of Mathematics, National University of Singapore, Singapore · Center for Advanced Intelligence Project, RIKEN, Tokyo, Japan · Department of Mathematical Informatics, The University of Tokyo, Tokyo, Japan
FedSubMuon: Communication-Efficient Federated LLM Fine-Tuning via Structured Subspace Muon
Sep 5, 2026Shaolong Chen, Youming Tao, Shuzhen Chen +3Subspace Optimization Federated Fine-Tuning
King Abdullah University of Science and Technology (KAUST) · Provable Responsible AI and Data Analytics (PRADA) Lab · Imperial College London +3
Predicting When Random Low-Dimensional Reparameterizations Train Neural Networks
Aug 12, 2026Andrew Cheng, Ali Eslamian, Jie Cheng +2Subspace Optimization
Department of Computer Science, Tsinghua University, Beijing, China · Department of Computer Science, The University of Manchester, Manchester, UK · Department of Computer Science, University of Kentucky, Lexington, KY, USA +3
Dysco: Dynamic Subspace Boosting to Mitigate LoRA Interference in Federated Learning
Jul 15, 2026Haobo Zhang, Jiankun Wang, Suraj Rajendran +5Subspace Optimization Federated Learning Aggregation
University of Michigan · Cornell University · University of Pennsylvania
Stochastic global optimization of continuous functions via random walks on Grassmannians
May 13, 2026Kartik Gupta, Stephen D. Miller, Pradeep Ravikumar +1Stochastic Optimization Subspace Optimization
BROS: Bias-Corrected Randomized Subspaces for Memory-Efficient Single-Loop Bilevel Optimization
May 11, 2026Hengrui Zhang, Boao Kong, Engao Zhang +1Stochastic Bilevel Optimization Bilevel Optimization
1Sichuan University · 2Peking University
Randomized Subspace Nesterov Accelerated Gradient
May 1, 2026Gaku Omiya, Pierre-Louis Poirion, Akiko TakedaGradient Descent Randomized Sketching
Department of Mathematical Informatics, The University of Tokyo, Tokyo, Japan · Center for Advanced Intelligence Project, RIKEN, Tokyo, Japan
Subspace Optimization for Efficient Federated Learning under Heterogeneous Data
Apr 28, 2026Shuchen Zhu, Zhengyang Huang, Yuqi Xu +1Subspace Optimization Non-IID Federated Learning
Peking University · Beihang University
Subspace Optimization for Backpropagation-Free Continual Test-Time Adaptation
Mar 30, 2026Damian Sójka, Sebastian Cygert, Marc MasanaSubspace Optimization Continual Test-Time Adaptation
Poznan University of Technology · Akces NCBR · NASK - National Research Institute +3
Linearized subspace refinement framework to expose hidden accuracy in trained neural networks
Jan 20, 2026Wenbo Cao, Weiwei ZhangSubspace Optimization Neural Network Optimization
Institute of AI for Industries, Chinese Academy of Sciences, Nanjing 211135, China · Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China · School of Aeronautics, Northwestern Polytechnical University, Xi’an 710072, China +2

Oct 14, 2026J/K move · Enter open · S save

Memory-Efficient Subspace Optimization with Residual Correction: From Momentum SGD to Muon

Sifan Yang, Hongyi Tao, Dingzhi Yu, Haomin Bai+4

State Key Laboratory of Novel Software Technology, Nanjing University · School of Artificial Intelligence, Nanjing University · Alibaba Token Hub, Alibaba Group+1

Subspace Optimization Memory-Efficient Training Stochastic Gradient Descent

Open paper AskPDF arXiv

Memory-efficient subspace optimization has emerged as a practical approach to reducing the memory cost of LLM training while preserving optimization performance. Previous algorithms typically rely on the leading singular subspace of stochastic gradients, yet the gradient noise may dominate the subspace in later training stages and can prevent convergence. To overcome this limitation, we adopt a residual correction framework, which maintains the optimizer states in the subspace and adds the orthogonal residual gradient to the parameter update. We first integrate this scheme into momentum SGD to develop SMRC, which attains an

O(T^{-1/4})

convergence rate for a fixed orthonormal basis, matching that of full-space momentum SGD. Our analysis introduces a subspace virtual iterate to resolve the coupling between the momentum and the residual correction. Furthermore, we extend SMRC to a multi-stage variant which periodically recomputes bases to track the evolving dominant subspace and retains the same rate. Since each basis refresh breaks the consistency of the virtual iterates, we devise a boundary correction approach that aligns them across stages. Beyond momentum SGD, we incorporate our framework into Muon and propose PMRC, which applies the matrix sign to the projected momentum. Through a weighted momentum tracking analysis that accounts for the residual gradient, we establish an

O(T^{-1/4})

convergence rate for PMRC, which matches the rate of full-space Muon. We also design a multi-stage PMRC with periodically refreshed bases, which maintains a convergence rate of the same order. Extensive experimental results confirm the effectiveness and efficiency of our algorithms.