Matrix Optimization

Follow this topic Ask about this topic

Momentum

3 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 48

All topics

Normalize-Then-Precondition: A Hierarchical Approach to Marginal Scale and Interaction Geometry for LLM Training
Sep 29, 2026Zixuan Gong, Zeyu Gan, Jiaye Teng +1Language Model Pretraining Matrix Optimization
Gaoling School of Artificial Intelligence Renmin University of China · School of Statistics and Management Shanghai University of Finance and Economics
Riemannian Structure and Optimization for a Class of Low-Parametric Orthogonal Matrices
Sep 23, 2026Ali Aliev, Maxim RakhubaRiemannian Optimization Matrix Optimization
HSE University
Optimization over covariance matrices with a parameterized metric
Sep 15, 2026Yibang Li, Bamdev Mishra, Pratik Jawanpuria +1Riemannian Optimization Second-Order Optimization
Nanyang Technological University, Singapore · Microsoft India · Indian Institute of Technology Bombay, India
MOON: Multi-Objective OrthoNormalized Updates for Multitask Learning
Aug 12, 2026Shiji Zhou, Kunlin Lyu, Lei Zhang +2Neural Network Optimization Multi-Task Learning
Institute of Artificial Intelligence, Beihang University · Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, Beihang University · Beijing Academy of Artificial Intelligence (BAAI) +1
Exact Rank-Space KL Projection for Shared-Marginal Low-Rank Factors: Application to Doubly Stochastic Clustering
Aug 9, 2026Enliang HuGraph Structure Learning Low-Rank Matrix Decomposition
Muon on the Stiefel Manifold Admits an Exact Closed-Form Update
Aug 6, 2026Mikhail Solonko, Molozhavenko Alexander, Maxim RakhubaStiefel Manifold Optimization Riemannian Optimization
HSE University, Moscow, Russia.
The Loss Does Not See the Basis, but Adam Does
Aug 5, 2026Devender SinghLow-Rank Matrix Decomposition Matrix Optimization
Department of Computer Science Memorial University of Newfoundland
MESH: Memory-Efficient Sinkhorn Optimization for Mixture-of-Experts Training
Aug 5, 2026Masato FujitakeNeural Network Optimization Memory-Efficient Optimization
FA Research, Fast Accounting Co., Ltd.
Joint Affine Spectral Shaping: Coupling Weight and Bias Updates Beyond Weight-Only Muon
Aug 4, 2026Gongyue Zhang, Honghai LiuNeural Network Optimization Muon Optimizer
State Key Laboratory of Robotics and Systems, Harbin Institute of Technology, Shenzhen, Shenzhen, 518055, China.
Low-Rank Dependence Decomposition via Accelerated Symmetric Non-negative Matrix Factorization
Jul 27, 2026Lavinia Ghita, Dhruv Desai, Jake Goldberg +1Non-Negative Matrix Factorization Low-Rank Matrix Decomposition
NVIDIA
Graph-Based Correlation Matrix Generation: A Convex Optimization Approach
Jul 24, 2026Ali Fakhar, K{é}vin Polisano, Ir{è}ne Gannaz +1Matrix Optimization Matrix Completion
UGA · Univ. Grenoble Alpes, CNRS, Grenoble INP, Inria, LJK, F-38000 Grenoble, France · G-SCOP\_GROG, G-SCOP, Grenoble INP, UGA +2
Muse: Representation Geometry of Muon Beyond Normalized Momentum
Jul 16, 2026Da Chang, Qiankun Shi, Lvgang Zhang +4Representation Geometry Momentum Methods
1Pengcheng Laboratory · 2Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences +3
Graph-Regularized Low-Rank Matrix Completion by Variable Projection
Jul 10, 2026Benoît Loucheur, P. -A. Absil, Michel JournéeLow-Rank Approximation Matrix Optimization
ICTEAM Institute, UCLouvain, 1348 Louvain-la-Neuve, Belgium. · Department of Climatology, Royal Meteorological Institute of Belgium, 1180 Uccle, Belgium.
Beyond Adam: SOAP and Muon for Faster, Label-Efficient Training of Machine Learning Interatomic Potentials
Jul 2, 2026Gil Harari, Yoel Zimmermann, Ola Tangen Kulseng +4Machine Learning Interatomic Potentials Neural Network Optimization
John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA, USA · Robert Bosch LLC Research and Technology Center, Watertown, MA, USA
Direction-Magnitude Decomposition for Low-Rank Matrix Optimization: Faster Convergence and Saddle-to-saddle Dynamics
Jun 30, 2026Yudong Wei, Liang Zhang, Bingcong Li +1Low-Rank Matrix Decomposition Nonconvex Optimization
Department of Mathematics, ETH Zurich, Zurich, Switzerland · Department of Computer Science, ETH Zurich, Zurich, Switzerland
ITSPACE: Monotone Gaussian Optimal Transport Updates
Jun 29, 2026Woojoo Na, Jennifer DyMatrix Optimization Optimal Transport
Department of Computer Engineering, Northeastern University, Boston, USA.
Muon learns balanced solutions in matrix factorization without slow saddle-to-saddle dynamics
Jun 29, 2026Mark Rhee, Jamie Simon, Dhruva KarkadaMuon Optimizer Matrix Optimization
UC Berkeley · Imbue
Aurora: A Leverage-Aware Spectral Optimizer
Jun 26, 2026Alec Dewulf, Dhruv Pai, Li Yang +2Neural Network Optimization Muon Optimizer
Tilde Research
Spectral DPPs via NEPv: A Scalable Continuous Relaxation of Determinantal MAP for Diversity-Aware Data Selection
Jun 17, 2026Richard Yi Da XuDeterminantal Point Process Training Data Selection
Hong Kong Baptist University · TadReamk Limited
Schattor: Schatten-family methods for deep learning optimization
Jun 14, 2026Bohao Ma, Junyu Zhang, Chuan HeDeep Learning Optimization Stochastic Optimization Convergence
School of Data Science, The Chinese University of Hong Kong (Shenzhen), Shenzhen, Guangdong, China · Department of Industrial Systems Engineering and Management, National University of Singapore, Singapore · Department of Mathematics, Linköping University, Sweden
Free Heavy-Tailed Lunch for Muon: A Theoretical Justification of Empirical Success
Jun 12, 2026Florian Hübler, Thomas Pethick, Suvrit SraStochastic Optimization Nonconvex Stochastic Optimization
Department of Computer Science, ETH Zurich, Switzerland · Department of Mathematics, Technical University of Munich, Germany · Munich Center for Machine Learning (MCML)
Zeta: Dual Whitening for Matrix Optimization via Coordinate-Adaptive Preconditioning
Jun 12, 2026Kaiwen Chen, Shuhai Zhang, Zimo Liu +7Neural Network Optimization Matrix Optimization
South China University of Technology · AIGCode · Hong Kong Baptist University
Scalable Deep Unfolding of Conic Optimizers
Jun 11, 2026Alex Oshin, Rahul Vodeb Ghosh, Evangelos A. TheodorouSemidefinite Programming Algorithm Unrolling
Georgia Institute of Technology
The Spectral Dynamics and Noise Geometry of Muon
Jun 7, 2026Pierfrancesco Beneventano, Mahmoud Abdelmoneum, Tomaso PoggioMuon Optimizer Matrix Optimization
Massachusetts Institute of Technology
On solving symmetric multi-type orthogonal non-negative matrix tri-factorization problem
Jun 6, 2026Rok Hribar, Gregor Papa, Janez Povh +1Non-Negative Matrix Factorization Graph Clustering
Computer Systems Department, Jožef Stefan Institute, Jamova cesta 39, Ljubljana, 1000, Slovenia. · Laboratory for Engineering Design, Faculty of Mechanical Engineering, University of Ljubljana, Aškerčeva cesta 6, Ljubljana, 1000, Slovenia. · Rudolfovo – Science and Technology Centre Novo mesto, Podbreznik 15, Novo mesto, 8000, Slovenia. +1
A Nonmonotone Gradient-Based Algorithm for Symmetric Nonnegative Matrix Factorization and Graph Clustering
Jun 1, 2026Ryan Swart, Johannes BrustNon-Negative Matrix Factorization Graph Clustering
School of Mathematical and Statistical Sciences, Arizona State University, Tempe, AZ
Riemannian Gradient Descent for Low-Rank Architectures
Jun 1, 2026Nicholas KnightRiemannian Optimization Low-Rank Approximation
NVIDIA
Riemannian Optimization for Hadamard Products of Low-Rank Matrices
May 31, 2026Pratik Jawanpuria, Ankish Chandresh, Bamdev MishraRiemannian Optimization Low-Rank Approximation
Centre for Machine Intelligence and Data Science, Indian Institute of Technology Bombay, India · Microsoft India
Manifold-based Algorithms for the Hadamard Decomposition
May 27, 2026Nicolas Gillis, Subhayan Saha, Stefano Sicilia +1Low-Rank Matrix Decomposition Low-Rank Approximation
Department of Mathematics and Operational Research, University of Mons, Mons, Belgium. · SS is a member of the Gruppo Nazionale Calcolo Scientifico-Istituto Nazionale di Alta Matematica (GNCS-INdAM).
Implicit Regularization in Perturbed Deep Matrix Factorization: Spectral Conditions and Stability
May 27, 2026Jingzhe Wang, Hung-Hsu ChouAlgorithmic Stability Low-Rank Matrix Decomposition
Department of Informatics and Networked Systems, University of Pittsburgh · Department of Mathematics, University of Pittsburgh

Sep 29, 2026, cs.LGJ/K move · Enter open · S save

Normalize-Then-Precondition: A Hierarchical Approach to Marginal Scale and Interaction Geometry for LLM Training

Zixuan Gong, Zeyu Gan, Jiaye Teng, Yong Liu

Gaoling School of Artificial Intelligence Renmin University of China · School of Statistics and Management Shanghai University of Finance and Economics

Language Model Pretraining Matrix Optimization LLM Training

Open paper AskPDF arXiv

Matrix optimizers have emerged as a promising direction, with Muon standing out as a prominent design. Revisiting Muon through its full-Gram representation, we observe that it jointly processes marginal-scale and interaction information. This opens an alternative way to organize geometric information hierarchically, motivating the Normalize-Then-Precondition framework. Specifically, it first uses diagonal-Gram information to construct a marginally normalized update, then applies spectral preconditioning to its directional interaction geometry. Building on this framework, we develop NormPre with NormPre-G and NormPre-L adopting global and localized spectral preconditioning, grounded in spectral-norm steepest descent and a regularized formulation followed by leading mode selection, respectively. To enable large-scale training, NormPre-G uses Newton-Schulz iterations and NormPre-L employs randomized sketching to approximate the leading interaction eigenspace. Theoretically, we establish

\mathcal{O}(T^{-1/2})

convergence guarantees for simplified versions of NormPre. Across extensive pretraining experiments on GPT-2 Small, LLaMA and Qwen3, both variants consistently outperform AdamW, Muon and MANO under matched training budgets. Further efficiency and spectral analyses reveal the complementary strengths of two variants and characterize their performance-efficiency trade-off. We open-source our code through a GitHub repository at https://github.com/zx-gong/NormPre.

Momentum

Related topics

Latest papers 48