cs.LGSep 18, 2026

Matrix AdaGrad: Row-wise and Column-wise Adaptive Subgradient Methods

Authors: Wenpeng Zhang, Runsheng Yu, Peilin Zhao

Organizations: Independent Reasearcher · School of Artificial Intelligence Shanghai Jiao Tong University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Adaptive optimization methods such as AdaGrad and Adam are widely used in modern deep neural network training, but their adaptive scaling is primarily designed for vector-valued parameters and does not explicitly exploit matrix structure. Recent matrix-aware optimizers demonstrate the benefits of structured optimization, yet a general theoretical framework for deriving matrix-aware adaptivity comparable to that of AdaGrad remains lacking. In this work, we develop an Online Mirror Descent framework with adaptive proximal functions for matrix-valued parameters, providing a principled methodology for deriving matrix-aware adaptive optimization through online regret minimization. By introducing row-wise and column-wise matrix proximal functions, our framework explicitly reveals the trade-off governing adaptive scaling: increasing the scaling factors reduces the gradient-dependent dual norm term while increasing the cost of evolving the proximal geometry. In the row-wise setting, this trade-off becomes separable under diagonal parameterization, allowing the adaptive scaling for each row to be derived independently by minimizing its corresponding row-wise regret bound. The column-wise counterpart follows directly by applying the row-wise construction to the transposed matrix. This framework yields Row-wise Matrix AdaGrad and Column-wise Matrix AdaGrad as concrete instantiations, with regret guarantees that are strictly tighter than those of entry-wise AdaGrad under row-sparse or column-sparse gradient structures. Experiments on matrix factorization and stacked deep MLP training further demonstrate the benefits of matrix-aware adaptive scaling, yielding improved optimization performance in both settings and enhanced optimization stability and trainability at larger learning rates and greater network depths in the latter.

Figures & tables

 st,ir=∑k=1t∥Gk,i,:∥22+δ,    ⁣i=1,…,ms_{t,i}^{\mathrm{r}}=\sqrt{\textstyle\sum_{k=1}^{t}\|G_{k,i,:}\|_{2}^{2}}+\delta,\,\,\,\!i=1,\dots,m\qquad\qquad\qquad 
Algorithm 1 Row-wise Matrix AdaGrad (Row-AdaGrad)
 st,jc=∑k=1t∥Gk,:,j∥22+δ,    ⁣j=1,…,ns_{t,j}^{\mathrm{c}}=\sqrt{\textstyle\sum_{k=1}^{t}\|G_{k,:,j}\|_{2}^{2}}+\delta,\,\,\,\!j=1,\dots,n\qquad\qquad\qquad 
Algorithm 2 Column-wise Matrix AdaGrad (Column-AdaGrad)
Figure 1: MovieLens 100K results with row-wise organization of user and item embeddings. Row-AdaGrad significantly outperforms entry-wise AdaGrad, while Column-AdaGrad performs the worst when its column-wise structure is misaligned with the parameter organization.
Figure 2: Training loss of Adam and Row-wise Momentum on stacked MLPs of different depths with lr=0.01lr=0.01 . Entry-wise scaling becomes highly unstable at 1515 layers and fails to train at 2525 layers, while row-wise scaling remains trainable.
Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Training loss of Adam and Row-wise Momentum on stacked MLPs of different depths with lr=0.001lr=0.001 . Compared with lr=0.01lr=0.01 , both methods exhibit reduced loss fluctuations at 1515 layers, while entry-wise scaling still fails to train at 2525 layers.
More in this topic

Adaptive Gradient Methods

Adaptive Gradient Methods adjust updates using gradient history, curvature, or noise estimates to improve optimization across diverse problems.

See all 157 papers

Explore similar work

CardsList
Jul 14, 2026

Reassessing Muon for Matrix Factorization

Muon has recently emerged as a strong optimizer for large-scale deep learning, where it reshapes gradient updates through approximate orthogonalization and has been reported to outperform Adam and AdamW in large language model training. Its empirical success has motivated a growing body of theoretical work that interprets Muon as steepest descent under the spectral norm. Yet it remains unclear which of Muon's advantages stem from its update rule itself and which are artifacts of the scale, architecture, and data of modern deep networks. In this work, we isolate the optimizer from these confounding factors by studying Muon on a simple, well-understood, and spectrally structured problem: low-rank matrix factorization. Through a controlled comparison against carefully tuned adaptive baselines, we find that Muon does not consistently outperform AdamW in this setting and that several previously reported advantages are sensitive to hyperparameter choices. Our results provide a more nuanced picture of when spectrum-aware orthogonalization is beneficial and argue for evaluating modern optimizers on controlled problems in addition to end-to-end benchmarks.

Ali Parviz, Gal Mishne, Alex Cloninger

†. Halicioğlu Data Science Institute, UC San Diego. · ∗. Department of Mathematics, UC San Diego.

Deep Learning OptimizationMuon OptimizerAdaptive Gradient Methods
AskPDF
May 5, 2026

Nora: Normalized Orthogonal Row Alignment for Scalable Matrix Optimizer

Matrix-based optimizers have demonstrated immense potential in training Large Language Models (LLMs), however, designing an ideal optimizer remains a formidable challenge. A superior optimizer must satisfy three core desiderata: efficiency, achieving Muon-like preconditioning to accelerate optimization; stability, strictly adhering to the scale-invariance inherent in neural networks; and speed, minimizing computational overhead. While existing methods address these aspects to varying degrees, they often fail to unify them, either incurring prohibitive computational costs like Muon, or allowing radial jitters that compromise stability like RMNP. To bridge this gap, we propose Nora, an optimizer that rigorously satisfies all three requirements. Nora achieves training stability by explicitly stabilizing weight norms and angular velocities through row-wise momentum projection onto the orthogonal complement of the weights. Simultaneously, by leveraging the block-diagonal dominance of the Transformer Hessian, Nora effectively approximates structured preconditioning while maintaining an optimal computational complexity of O(mn)\mathcal{O}(mn). Furthermore, we prove that Nora is a scalable optimizer and establish its corresponding scaling theorems. With a streamlined implementation requiring only two lines of code, our preliminary experiments validate Nora as an efficient and highly promising optimizer for large-scale training.

Jinghui Yuan, Jiaxuan Zou, Shuo Wang +2

1. School of Artificial Intelligence, Optics and Electronics (iOPEN), Northwestern Polytechnical University. · 2. School of Mathematics and Statistics, Xi’an Jiaotong University. · 4. Gaoling School of Artificial Intelligence, Renmin University of China. +1

Neural Network OptimizationMatrix OptimizationLLM Training
AskPDF
Mar 10, 2026

On the Width Scaling of Neural Optimizers Under Matrix Operator Norms I: Row/Column Normalization and Hyperparameter Transfer

A central question in modern deep learning is how to design optimizers whose behavior remains stable as the network width ww increases. We address this question by interpreting several widely used neural-network optimizers, including AdamW and Muon, as instances of steepest descent under matrix operator norms. This perspective links optimizer geometry with the Lipschitz structure of the network forward map, and enables width-independent control of both Lipschitz and smoothness constants. However, steepest-descent rules induced by standard p→qp \to q operator norms lack layerwise composability and therefore cannot provide width-independent bounds in deep architectures. We overcome this limitation by introducing a family of mean-normalized operator norms, denoted \pmean→\qmean\pmean \to \qmean, that admit layerwise composability, yield width-independent smoothness bounds, and give rise to practical optimizers such as rescaled AdamW, row normalization, and column normalization. The resulting learning rate width-aware scaling rules recover μμP scaling~\cite{yang2021tensor} as a special case and provide a principled mechanism for cross-width learning-rate transfer across a broad class of optimizers. We further show that Muon can suffer an O(w)\mathcal{O}(\sqrt{w}) worst-case growth in the smoothness constant, whereas a new family of row-normalized optimizers we propose achieves width-independent smoothness guarantees. Based on the observations, we propose MOGA (Matrix Operator Geometry Aware), a width-aware optimizer based only on row/column-wise normalization that enables stable learning-rate transfer across model widths. Large-scale pre-training on GPT-2 and LLaMA shows that MOGA, especially with row normalization, is competitive with Muon while being notably faster in large-token and low-loss regimes.

Ruihan Xu, Jiajin Li, Yiping Lu

University of Chicago · Sauder School of Business, University of British Columbia · Industrial Engineering & Management Science, Northwestern University

Neural Network OptimizationHyperparameter TransferMatrix Optimization
AskPDF