Orthogonal Optimization

Follow this topic Ask about this topic

Latest papers

14 papers1 paper in the last four weeks, with none the four weeks before. 0.0% of all new papers.

ORCA: The Annealed Spectral Conditioning Optimizer for Faster, Better LLM Training
Oct 5, 2026Yuanshi Liu, Boyuan Jiang, Liang Hou +4Spectral Regularization Deep Learning Optimization
School of Intelligence Science and Technology, Peking University · Kling Team
Dion3: Full-Stack Orthogonal Updates
Aug 12, 2026Noah Amsel, Jack Zhang, Kwangjun Ahn +5Newton-Schulz Iteration GPU Kernel Optimization
New York University · Microsoft Research · Princeton University +3
Muon on the Stiefel Manifold Admits an Exact Closed-Form Update
Aug 6, 2026Mikhail Solonko, Molozhavenko Alexander, Maxim RakhubaStiefel Manifold Optimization Riemannian Optimization
HSE University, Moscow, Russia.
A Note on Stability for Orthogonalized Matrix Momentum with Client Sampling
Jun 1, 2026Da Chang, Qiankun Shi, Lvgang Zhang +2Algorithmic Stability Distributed Optimization
University of Chinese Academy of Sciences · Sun Yat-sen University · Southern University of Science and Technology +1
How Much Orthogonalization Does Muon Need?
May 29, 2026Hua HuangMuon Optimizer Matrix Factorization
NVIDIA
When Muon Optimizer Meets Adversarial Training: A Theoretical and Empirical Study
May 26, 2026Jun Yan, Weiquan Huang, Jiankai Zuo +4Adversarial Training Orthogonal Optimization
IT College, Shanghai Ocean University, Shanghai, 201306 · School of Computer Science and Technology, Tongji University, Shanghai, 201804 · SEIE, Suzhou University of Science and Technology, Suzhou, 215009 +2
OrScale: Orthogonalised Optimization with Layer-Wise Trust-Ratio Scaling
May 8, 2026Yuxuan Lou, Yang YouOrthogonal Optimization Neural Network Optimization
National University of Singapore
AdamO: A Collapse-Suppressed Optimizer for Offline RL
May 3, 2026Nan Qiao, Sheng Yue, Shuning Wang +1Reinforcement Learning Orthogonal Optimization
1Tsinghua University, China · 2Central South University, China · 3Sun Yat-sen University, China
Deflation-Free Optimal Scoring
Apr 28, 2026Sharmin Afroz, Brendan AmesOrthogonal Optimization Constrained Optimization
Department of Computational & Informational Sciences, Stillman College, Tuscaloosa, AL, USA · School of Mathematical Sciences, University of Southampton, Southampton, UK
Delve into the Applicability of Advanced Optimizers for Multi-Task Learning
Apr 10, 2026Zhipeng Zhou, Linxiao Cao, Pengcheng Wu +2Orthogonal Optimization Multi-Task Learning
Nanyang Technological University · Hong Kong University of Science and Technology (Guangzhou) · School of Artificial Intelligence, Shanghai Jiao Tong University
Primal-Dual Methods for Nonsmooth Nonconvex Optimization with Orthogonality Constraints
Apr 5, 2026Linglingzhi Zhu, Wentao Ding, Shangyuan Liu +1Orthogonal Optimization Constrained Optimization
H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, GA, USA · Department of Systems Engineering and Engineering Management, The Chinese University of Hong Kong, Shatin, N.T., Hong Kong
MuonEq: Balancing Before Orthogonalization with Lightweight Equilibration
Mar 30, 2026Da Chang, Qiankun Shi, Lvgang Zhang +5Neural Network Optimization Muon Optimizer
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences · Pengcheng Laboratory · University of Chinese Academy of Sciences +4
An Embarrassingly Simple Way to Optimize Orthogonal Matrices at Scale
Feb 16, 2026Adrián Javaloy, Antonio VergariOrthogonal Optimization Matrix Optimization
Institute for Machine Learning, University of Edinburgh, GB.
Turbo-Muon: Almost-Orthogonal Pre-Conditioning for Fast Muon Updates
Dec 4, 2025Thibaut Boissin, Thomas Massena, Franck Mamalet +1Newton-Schulz Iteration Orthogonal Optimization
IRIT-MISFIT · Institut de Recherche Technologique Saint-Exupery, France · DTIPG - SNCF, IRIT-MISFIT +2

Oct 5, 2026J/K move · Enter open · S save

ORCA: The Annealed Spectral Conditioning Optimizer for Faster, Better LLM Training

Yuanshi Liu, Boyuan Jiang, Liang Hou, Xin Tao+3

School of Intelligence Science and Technology, Peking University · Kling Team

Spectral Regularization Deep Learning Optimization LLM Training Spectral Optimization Orthogonal Optimization

Open paper AskPDF arXiv

Modern LLM optimizers such as Muon often produce weight matrices with higher effective rank than Adam, yet further spectral control has delivered only modest gains. We identify a tension behind this result: concentrated spectra can suppress gradient directions in coupled weight matrices and slow optimization, while constraints maintained throughout training can limit task-specific adaptation and raise the attainable loss floor. We introduce ORCA (Orthogonal Regularization, Cooled After), a minimal optimizer intervention that applies strong but temporary soft orthogonality regularization early in training, then removes it. This allows the weights to benefit from a broader spectrum early on and adapt freely afterward. Across LLaMA, Qwen3, and fine-grained mixture-of-experts models ranging from 130M to 8B parameters, ORCA achieves lower final validation loss than Muon. Its loss reduction relative to Muon matches or exceeds Muon's reduction relative to Adam. Ablations support the early-shaping, later-release design. Further, ORCA requires no architectural changes and adds minimal overhead.