Matrix Optimization

Follow this topic Ask about this topic

Momentum

3 papers in the last four weeks, against 1 the four weeks before. 0.0% of all new papers.

Jul 13Week of Sep 28

Latest papers 47

All topics

Normalize-Then-Precondition: A Hierarchical Approach to Marginal Scale and Interaction Geometry for LLM Training
Sep 29, 2026Zixuan Gong, Zeyu Gan, Jiaye Teng +1Language Model Pretraining Matrix Optimization
Gaoling School of Artificial Intelligence Renmin University of China · School of Statistics and Management Shanghai University of Finance and Economics
Riemannian Structure and Optimization for a Class of Low-Parametric Orthogonal Matrices
Sep 23, 2026Ali Aliev, Maxim RakhubaRiemannian Optimization Matrix Optimization
HSE University
Optimization over covariance matrices with a parameterized metric
Sep 15, 2026Yibang Li, Bamdev Mishra, Pratik Jawanpuria +1Riemannian Optimization Second-Order Optimization
Nanyang Technological University, Singapore · Microsoft India · Indian Institute of Technology Bombay, India
MOON: Multi-Objective OrthoNormalized Updates for Multitask Learning
Aug 12, 2026Shiji Zhou, Kunlin Lyu, Lei Zhang +2Neural Network Optimization Multi-Task Learning
Institute of Artificial Intelligence, Beihang University · Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, Beihang University · Beijing Academy of Artificial Intelligence (BAAI) +1
Exact Rank-Space KL Projection for Shared-Marginal Low-Rank Factors: Application to Doubly Stochastic Clustering
Aug 9, 2026Enliang HuGraph Structure Learning Low-Rank Matrix Decomposition
Muon on the Stiefel Manifold Admits an Exact Closed-Form Update
Aug 6, 2026Mikhail Solonko, Molozhavenko Alexander, Maxim RakhubaStiefel Manifold Optimization Riemannian Optimization
HSE University, Moscow, Russia.
The Loss Does Not See the Basis, but Adam Does
Aug 5, 2026Devender SinghLow-Rank Matrix Decomposition Matrix Optimization
Department of Computer Science Memorial University of Newfoundland
MESH: Memory-Efficient Sinkhorn Optimization for Mixture-of-Experts Training
Aug 5, 2026Masato FujitakeNeural Network Optimization Memory-Efficient Optimization
FA Research, Fast Accounting Co., Ltd.
Joint Affine Spectral Shaping: Coupling Weight and Bias Updates Beyond Weight-Only Muon
Aug 4, 2026Gongyue Zhang, Honghai LiuNeural Network Optimization Muon Optimizer
State Key Laboratory of Robotics and Systems, Harbin Institute of Technology, Shenzhen, Shenzhen, 518055, China.
Low-Rank Dependence Decomposition via Accelerated Symmetric Non-negative Matrix Factorization
Jul 27, 2026Lavinia Ghita, Dhruv Desai, Jake Goldberg +1Non-Negative Matrix Factorization Low-Rank Matrix Decomposition
NVIDIA
Graph-Based Correlation Matrix Generation: A Convex Optimization Approach
Jul 24, 2026Ali Fakhar, K{é}vin Polisano, Ir{è}ne Gannaz +1Matrix Optimization Matrix Completion
UGA · Univ. Grenoble Alpes, CNRS, Grenoble INP, Inria, LJK, F-38000 Grenoble, France · G-SCOP\_GROG, G-SCOP, Grenoble INP, UGA +2
Muse: Representation Geometry of Muon Beyond Normalized Momentum
Jul 16, 2026Da Chang, Qiankun Shi, Lvgang Zhang +4Representation Geometry Momentum Methods
1Pengcheng Laboratory · 2Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences +3
Graph-Regularized Low-Rank Matrix Completion by Variable Projection
Jul 10, 2026Benoît Loucheur, P. -A. Absil, Michel JournéeLow-Rank Approximation Matrix Optimization
ICTEAM Institute, UCLouvain, 1348 Louvain-la-Neuve, Belgium. · Department of Climatology, Royal Meteorological Institute of Belgium, 1180 Uccle, Belgium.
Beyond Adam: SOAP and Muon for Faster, Label-Efficient Training of Machine Learning Interatomic Potentials
Jul 2, 2026Gil Harari, Yoel Zimmermann, Ola Tangen Kulseng +4Machine Learning Interatomic Potentials Neural Network Optimization
John A. Paulson School of Engineering and Applied Sciences, Harvard University, Cambridge, MA, USA · Robert Bosch LLC Research and Technology Center, Watertown, MA, USA
Direction-Magnitude Decomposition for Low-Rank Matrix Optimization: Faster Convergence and Saddle-to-saddle Dynamics
Jun 30, 2026Yudong Wei, Liang Zhang, Bingcong Li +1Low-Rank Matrix Decomposition Nonconvex Optimization
Department of Mathematics, ETH Zurich, Zurich, Switzerland · Department of Computer Science, ETH Zurich, Zurich, Switzerland
ITSPACE: Monotone Gaussian Optimal Transport Updates
Jun 29, 2026Woojoo Na, Jennifer DyMatrix Optimization Optimal Transport
Department of Computer Engineering, Northeastern University, Boston, USA.
Muon learns balanced solutions in matrix factorization without slow saddle-to-saddle dynamics
Jun 29, 2026Mark Rhee, Jamie Simon, Dhruva KarkadaMuon Optimizer Matrix Optimization
UC Berkeley · Imbue
Aurora: A Leverage-Aware Spectral Optimizer
Jun 26, 2026Alec Dewulf, Dhruv Pai, Li Yang +2Neural Network Optimization Muon Optimizer
Tilde Research
Spectral DPPs via NEPv: A Scalable Continuous Relaxation of Determinantal MAP for Diversity-Aware Data Selection
Jun 17, 2026Richard Yi Da XuDeterminantal Point Process Training Data Selection
Hong Kong Baptist University · TadReamk Limited
Schattor: Schatten-family methods for deep learning optimization
Jun 14, 2026Bohao Ma, Junyu Zhang, Chuan HeDeep Learning Optimization Stochastic Optimization Convergence
School of Data Science, The Chinese University of Hong Kong (Shenzhen), Shenzhen, Guangdong, China · Department of Industrial Systems Engineering and Management, National University of Singapore, Singapore · Department of Mathematics, Linköping University, Sweden
Free Heavy-Tailed Lunch for Muon: A Theoretical Justification of Empirical Success
Jun 12, 2026Florian Hübler, Thomas Pethick, Suvrit SraStochastic Optimization Nonconvex Stochastic Optimization
Department of Computer Science, ETH Zurich, Switzerland · Department of Mathematics, Technical University of Munich, Germany · Munich Center for Machine Learning (MCML)
Zeta: Dual Whitening for Matrix Optimization via Coordinate-Adaptive Preconditioning
Jun 12, 2026Kaiwen Chen, Shuhai Zhang, Zimo Liu +7Neural Network Optimization Matrix Optimization
South China University of Technology · AIGCode · Hong Kong Baptist University
Scalable Deep Unfolding of Conic Optimizers
Jun 11, 2026Alex Oshin, Rahul Vodeb Ghosh, Evangelos A. TheodorouSemidefinite Programming Algorithm Unrolling
Georgia Institute of Technology
The Spectral Dynamics and Noise Geometry of Muon
Jun 7, 2026Pierfrancesco Beneventano, Mahmoud Abdelmoneum, Tomaso PoggioMuon Optimizer Matrix Optimization
Massachusetts Institute of Technology
On solving symmetric multi-type orthogonal non-negative matrix tri-factorization problem
Jun 6, 2026Rok Hribar, Gregor Papa, Janez Povh +1Non-Negative Matrix Factorization Graph Clustering
Computer Systems Department, Jožef Stefan Institute, Jamova cesta 39, Ljubljana, 1000, Slovenia. · Laboratory for Engineering Design, Faculty of Mechanical Engineering, University of Ljubljana, Aškerčeva cesta 6, Ljubljana, 1000, Slovenia. · Rudolfovo – Science and Technology Centre Novo mesto, Podbreznik 15, Novo mesto, 8000, Slovenia. +1
A Nonmonotone Gradient-Based Algorithm for Symmetric Nonnegative Matrix Factorization and Graph Clustering
Jun 1, 2026Ryan Swart, Johannes BrustNon-Negative Matrix Factorization Graph Clustering
School of Mathematical and Statistical Sciences, Arizona State University, Tempe, AZ
Riemannian Gradient Descent for Low-Rank Architectures
Jun 1, 2026Nicholas KnightRiemannian Optimization Low-Rank Approximation
NVIDIA
Riemannian Optimization for Hadamard Products of Low-Rank Matrices
May 31, 2026Pratik Jawanpuria, Ankish Chandresh, Bamdev MishraRiemannian Optimization Low-Rank Approximation
Centre for Machine Intelligence and Data Science, Indian Institute of Technology Bombay, India · Microsoft India
Manifold-based Algorithms for the Hadamard Decomposition
May 27, 2026Nicolas Gillis, Subhayan Saha, Stefano Sicilia +1Low-Rank Matrix Decomposition Low-Rank Approximation
Department of Mathematics and Operational Research, University of Mons, Mons, Belgium. · SS is a member of the Gruppo Nazionale Calcolo Scientifico-Istituto Nazionale di Alta Matematica (GNCS-INdAM).
Implicit Regularization in Perturbed Deep Matrix Factorization: Spectral Conditions and Stability
May 27, 2026Jingzhe Wang, Hung-Hsu ChouAlgorithmic Stability Low-Rank Matrix Decomposition
Department of Informatics and Networked Systems, University of Pittsburgh · Department of Mathematics, University of Pittsburgh
An Exterior Method for Nonnegative Matrix Factorization
May 19, 2026Qiujing Lu, Tonmoy Monsoor, Ehsan Ebrahimzadeh +2Non-Negative Matrix Factorization Low-Rank Matrix Decomposition
1ECE, UCLA · 2eBay Search Science Team
Learning Fill-in Reduction Ordering via Graph Policy Optimization for Sparse Matrices
May 17, 2026Ziwei Li, Shuzi Niu, Huiyuan Li +2Policy Optimization Matrix Optimization
Institute of Software, Chinese Academy of Sciences, Beijing, China
Sample-efficient inductive matrix completion with noise and inexact side-information
May 16, 2026Yuepeng Yang, Cong MaMatrix Optimization Matrix Completion
Department of Statistics and Data Sciences, Yale University · Department of Statistics, University of Chicago
Intrinsic Muon: Spectral Optimization on Riemannian Matrix Manifolds
May 10, 2026Yibang Li, Bihari Lal Pandey, Ravi Sah +4Stiefel Manifold Optimization Riemannian Optimization
Nanyang Technological University, Singapore. · Indian Institute of Technology Bombay, India. · University of Sydney, Australia. +1
Navigating LLM Valley: From AdamW to Memory-Efficient and Matrix-Based Optimizers
May 9, 2026Aditya RanganathDeep Learning Optimization Memory-Efficient Optimization
Center for Applied Scientific Computing Lawrence Livermore National Laboratory Livermore, CA, USA
OrScale: Orthogonalised Optimization with Layer-Wise Trust-Ratio Scaling
May 8, 2026Yuxuan Lou, Yang YouNeural Network Optimization Muon Optimizer
National University of Singapore
PolarAdamW: Disentangling Spectral Control and Schur Gauge-Equivariance in Matrix Optimisation
May 8, 2026Haozhou ZhangMuon Optimizer Matrix Optimization
Department of Mathematics and Statistics University of Idaho
Muon with Nesterov Momentum: Heavy-Tailed Noise and (Randomized) Inexact Polar Decomposition
May 7, 2026Sayantan Choudhury, Xiaoran Cheng, Martin Takáč +2Stochastic Optimization Momentum Methods
MBZUAI · Penn State University · Georgia Institute of Technology +1
Pro-KLShampoo: Projected KL-Shampoo with Whitening Recovered by Orthogonalization
May 7, 2026Ruotong Sun, Ermin WeiLanguage Model Pretraining Deep Learning Optimization
†Department of Industrial Engineering & Management Sciences, Northwestern University · ∗Department of Electrical & Computer Engineering, Northwestern University
DiBA: Diagonal and Binary Matrix Approximation for Neural Network Weight Compression
May 7, 2026Nobutaka OnoEfficient Neural Network Inference Matrix Optimization
Tokyo Metropolitan University, 6–6 Asahigaoka, Hino-shi, Tokyo 191–0065, Japan.
Nora: Normalized Orthogonal Row Alignment for Scalable Matrix Optimizer
May 5, 2026Jinghui Yuan, Jiaxuan Zou, Shuo Wang +2Neural Network Optimization Matrix Optimization
1. School of Artificial Intelligence, Optics and Electronics (iOPEN), Northwestern Polytechnical University. · 2. School of Mathematics and Statistics, Xi’an Jiaotong University. · 4. Gaoling School of Artificial Intelligence, Renmin University of China. +1
Hankel and Toeplitz Rank-1 Decomposition of Arbitrary Matrices with Applications to Signal Direction-of-Arrival Estimation
Apr 29, 2026Georgios I. OrfanidisLow-Rank Matrix Decomposition Direction-of-Arrival Estimation
Center for Connected Autonomy and AI and the Dept. of Electrical Engineering and Computer Science, Florida Atlantic University, Boca Raton, FL 33431 USA
State-Dependent Lyapunov Analysis of Rank-1 Matrix Factorization
Apr 28, 2026Jaehong MoonEdge of Stability Lyapunov Stability
Industrial & Enterprise Systems Engineering University of Illinois at Urbana-Champaign Urbana, IL 61801
Provably Finding a Hidden Dense Submatrix among Many Planted Dense Submatrices via Convex Programming
Jan 7, 2026Valentine Olanubi, Phineas Agar, Brendan AmesStochastic Block Model Matrix Optimization
University of Alabama, Department of Mathematics · Department of Mathematics, University of Alabama, Tuscaloosa, AL, USA · University of Southampton, School of Mathematical Sciences +1
Ky Fan Norms and Beyond: Dual Norms and Combinations for Matrix Optimization
Dec 10, 2025Alexey Kravatskiy, Ivan Kozyrev, Nikolai Kozlov +3Deep Learning Optimization Muon Optimizer
MIRIAI · Russia · MIPT, INM RAS +3
The Optimization Landscape of Carathéodory Decomposition of Toeplitz Covariances
Nov 3, 2025Daniel Busbib, Ami WieselNonconvex Optimization Matrix Optimization
School of Computer Science and Engineering, The Hebrew University of Jerusalem, Jerusalem 91904, Israel
Low-rank Orthogonalization for Large-scale Matrix Optimization with Applications to Foundation Model Training
Sep 15, 2025Chuan He, Zhanwang Deng, Zhaosong LuNeural Network Optimization Low-Rank Approximation
Department of Mathematics Linköping University, Sweden · Academy for Advanced Interdisciplinary Studies Peking University, Beijing, People’s Republic of China · Department of Industrial and Systems Engineering University of Minnesota, USA
Doubly Stochastic Adaptive Neighbors Clustering via the Marcus Mapping
Aug 6, 2024Jinghui Yuan, Chusheng Zeng, Fangyuan Xie +5Clustering Graph Clustering
School of Artificial Intelligence, OPtics and ElectroNics (iOPEN), Northwestern Polytechnical University, Xi’an 710072, Shaanxi, China · Institute of Artificial Intelligence (TeleAI), China Telecom, P. R. China

Sep 29, 2026, cs.LGJ/K move · Enter open · S save

Normalize-Then-Precondition: A Hierarchical Approach to Marginal Scale and Interaction Geometry for LLM Training

Zixuan Gong, Zeyu Gan, Jiaye Teng, Yong Liu

Gaoling School of Artificial Intelligence Renmin University of China · School of Statistics and Management Shanghai University of Finance and Economics

Language Model Pretraining Matrix Optimization LLM Training

Open paper AskPDF arXiv

Matrix optimizers have emerged as a promising direction, with Muon standing out as a prominent design. Revisiting Muon through its full-Gram representation, we observe that it jointly processes marginal-scale and interaction information. This opens an alternative way to organize geometric information hierarchically, motivating the Normalize-Then-Precondition framework. Specifically, it first uses diagonal-Gram information to construct a marginally normalized update, then applies spectral preconditioning to its directional interaction geometry. Building on this framework, we develop NormPre with NormPre-G and NormPre-L adopting global and localized spectral preconditioning, grounded in spectral-norm steepest descent and a regularized formulation followed by leading mode selection, respectively. To enable large-scale training, NormPre-G uses Newton-Schulz iterations and NormPre-L employs randomized sketching to approximate the leading interaction eigenspace. Theoretically, we establish

\mathcal{O}(T^{-1/2})

convergence guarantees for simplified versions of NormPre. Across extensive pretraining experiments on GPT-2 Small, LLaMA and Qwen3, both variants consistently outperform AdamW, Muon and MANO under matched training budgets. Further efficiency and spectral analyses reveal the complementary strengths of two variants and characterize their performance-efficiency trade-off. We open-source our code through a GitHub repository at https://github.com/zx-gong/NormPre.

Momentum

Related topics

Latest papers 47