Spectral Regularization

Follow this topic Ask about this topic

Momentum

6 papers in the last four weeks, against 1 the four weeks before. 0.1% of all new papers.

Jul 13Week of Sep 28

Latest papers 31

All topics

ORCA: The Annealed Spectral Conditioning Optimizer for Faster, Better LLM Training
Oct 5, 2026Yuanshi Liu, Boyuan Jiang, Liang Hou +4Spectral Regularization Deep Learning Optimization
School of Intelligence Science and Technology, Peking University · Kling Team
Principal Component Regression Dominates all Monotone Spectral Filters for Linear Regression
Sep 30, 2026Juno Kim, Hengyu Fu, Peter Bartlett +2Spectral Regularization Linear Regression
University of California, Berkeley · Google DeepMind
$λ$ -JEPA Spectral Anti-Collapse Regularization for Self-Supervised Learning
Sep 28, 2026Berker Demirel, Clémentine Dominé, Valentino Maiorca +3Spectral Regularization Joint-Embedding Predictive Architecture
Institute of Science and Technology Austria (ISTA) Am Campus 1, 3400 Klosterneuburg, Austria
Prox-Friendly Log-Magnitude Prior on Complex-Valued Signal
Sep 28, 2026Kazuki Matsumoto, Keidai Arai, Kohei YatabeSpectral Regularization Speech Enhancement
Tokyo University of Agriculture and Technology (TUAT), Tokyo, Japan
Graph Learning with Spectral Connectivity Priors for Scarce Data
Sep 23, 2026Mingxiao Liu, Bahar Oveisgharan, Bingyan Zou +3Graph Signal Processing Graph Structure Learning
Tsinghua University, China · York University, Canada
Why Learning Rediscovers the Closed-Form Diagonal Regularizer
Sep 9, 2026Jeahn Han, Pyojin KimSpectral Regularization PDE Inverse Problems
GIST
Think Wider: Mitigating Latent Rank Collapse in Implicit Chain-of-Thought Reasoning
Sep 7, 2026Yuwen Hao, Menglin YangSpectral Regularization Continuous Latent Reasoning
The Hong Kong University of Science and Technology (Guangzhou)
Spectral Initialization and Scheduled Graph Smoothness for Uncertain Knowledge Graph Completion
Sep 2, 2026Md Abrar Jahin, Taufikur Rahman Fuad, Jay Pujara +1KG Completion Link Prediction
University of Southern California · Islamic University of Technology
Spectral Truncation in Synthetic Control
Jul 27, 2026Mojtaba EslamiSpectral Regularization Causal Effect Estimation
University of Calgary
Spectral-Aware Analytic Class-Incremental Learning for Long-Tailed Distributions
Jul 24, 2026Quyen Tran, Hai Nguyen, Quan Dao +4Class-Incremental Learning Spectral Regularization
1Rutgers University · 2Tuft University · 3HUST +1
Beyond Negative-Ridge Endpoints: Mixed-Sign Spectral Regularization via Negative-Shifted Gradient Descent
Jul 24, 2026Peng ZhaoSpectral Regularization Early Stopping
Department of Applied Economics and Statistics · University of Delaware
Regularized Optimization on Grassmann Manifold: Theory, Algorithm and Applications
Jul 23, 2026Zhuan Liang, Zheng ZhaiSpectral Regularization Riemannian Optimization
Department of Statistics, Faculty of Arts and Sciences at Beijing Normal University, Zhuhai
An Isotropy-Preserving Spectral Cap for Muon: Theory and Three Case Studies
Jul 22, 2026Jiachun LiSpectral Regularization Muon Optimizer
How to Tame Grokking: Representation Geometry as a Control Signal
Jul 13, 2026Maksim A KazanskiiNeural Network Generalization Spectral Regularization
Independent Researcher
Spectrally Deconfounded Gradient Boosting
Jul 10, 2026Andrea Nava, Peter Bühlmann, Fabio SigristSpectral Regularization Gradient-Boosted Decision Trees
Seminar for Statistics, ETH Zürich, Zürich, Switzerland · IFZ, Lucerne University of Applied Sciences and Arts, Rotkreuz, Switzerland
PIEFS: Physics-Informed Eigenfunction Features with Learnable Scaling
Jul 4, 2026Varvara Nazarenko, Timur Lidzhiev, Alexander TarakanovSpectral Regularization Representation Learning
Faculty of Computer Science, HSE University, Moscow, Russia · AI VK, Moscow, Russia
Regularized Variational and Spectral Log-Density-Ratio Estimation in the Gaussian Location Model
Jul 2, 2026Francis BachSpectral Regularization Density Ratio Estimation
SIERRA · Inria - Ecole Normale Sup´erieure PSL Research University
Residual-Guided Dictionary Learning for Spectrally Accurate Koopman Approximation
Jun 27, 2026George Coote, Matthew J. ColbrookSpectral Regularization Koopman Operator Learning
Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Wilberforce Road, CB3 0WA, United Kingdom
Differential Spectral Damping Gap Adaptive Regularization for Ill-Conditioned Kernel Methods
Jun 22, 2026Praveg VashishthaSpectral Regularization Support Vector Machines
Department of Computer Science and Engineering, Indian Institute of Technology Patna, India · Sopra Banking Software India
The Pitfall of Scaling Up: Uncovering and Mitigating Popularity Bias Amplification in Scaling Transformer-based Recommenders
Jun 20, 2026Weiqin Yang, Yue Pan, Chongming Gao +4Spectral Regularization Sequential Recommendation
Zhejiang University · State Key Laboratory of Blockchain and Data Security, Zhejiang University · College of Computer Science and Technology, Zhejiang University +2
Spectrally Safe Neural Operator Warm-Starts for Large-Scale Newton Solvers
Jun 20, 2026Jaemin Oh, Youngkyu Lee, Jerome Darbon +1Spectral Regularization PDE Solving
Division of Applied Mathematics, Brown University, Providence, RI 02912, USA
Lung-SRAD: Spectral-Aware Regularized Audio DASS with Dual-Axis Patch-Mix Contrastive Learning for Respiratory Sound Classification
Jun 10, 2026Hemansh Shridhar, Miika Toikkanen, June-Woo KimContrastive Learning Spectral Regularization
RSC LAB, MODULABS, Republic of Korea · Department of Electronic Engineering, Wonkwang University, Republic of Korea · AI Convergence Research Institute, Wonkwang University, Republic of Korea
Spectrally Regularized Latent Flow Matching for Turbulence Generation
Jun 10, 2026Khalid Rafiq, Aditya G. NairSpectral Regularization Flow Matching
Department of Mechanical Engineering University of Nevada, Reno, NV, USA
Closed-Form Spectral Regularization for Multi-Task Model Merging
Jun 5, 2026Yongxian Wei, Runxi Cheng, Xingxuan Zhang +4Spectral Regularization Multi-Task Learning
Shenzhen International Graduate School, Tsinghua University, Shenzhen 518071, China · Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China · Sun Yat-sen University, Shenzhen 510275, China +1
Low-Rank Decay for Grokking in Scale-Invariant Transformers: A Spectral-Geometric View
Jun 3, 2026Mingyu LiSpectral Regularization Weight Decay
Beijing Normal University
Stochastic Rounding Increases Small Singular Values
May 29, 2026Linkai Ma, Tingzhou Yu, Petros DrineasSpectral Regularization Implicit Regularization
Department of Mathematics University of Alberta
Ridge Regression from Poisson Resetting: A Renewal Perspective on Spectral Regularization
May 28, 2026Petar JolakoskiSpectral Regularization Linear Regression
Research Center for Computer Science and Information Technologies, Macedonian Academy of Sciences and Arts, Bul. Krste Misirkov 2, 1000 Skopje, Macedonia
FragileFlow: Spectral Control of Correct-but-Fragile Predictions for Foundation Model Robustness
May 9, 2026Zhuoyun Li, Boxuan Wang, Jinwei Hu +2VLM Robustness Spectral Regularization
School of Computer Science and Informatics, University of Liverpool, UK
Detecting Adversarial Data via Provable Adversarial Noise Amplification
May 4, 2026Furkan Mumcu, Yasin YilmazSpectral Regularization Adversarial Robustness
University of South Florida
Delving into Latent Spectral Biasing of Video VAEs for Superior Diffusability
Dec 5, 2025Shizhan Liu, Xinran Deng, Zhuoyi Yang +3Variational Autoencoders Spectral Regularization
Zhipu AI, Beijing, China · University of Chinese Academy of Sciences, Beijing, China · Tsinghua University, Beijing, China

Oct 5, 2026, cs.LGJ/K move · Enter open · S save

ORCA: The Annealed Spectral Conditioning Optimizer for Faster, Better LLM Training

Yuanshi Liu, Boyuan Jiang, Liang Hou, Xin Tao+3

School of Intelligence Science and Technology, Peking University · Kling Team

Spectral Regularization Deep Learning Optimization LLM Training

Open paper AskPDF arXiv

Modern LLM optimizers such as Muon often produce weight matrices with higher effective rank than Adam, yet further spectral control has delivered only modest gains. We identify a tension behind this result: concentrated spectra can suppress gradient directions in coupled weight matrices and slow optimization, while constraints maintained throughout training can limit task-specific adaptation and raise the attainable loss floor. We introduce ORCA (Orthogonal Regularization, Cooled After), a minimal optimizer intervention that applies strong but temporary soft orthogonality regularization early in training, then removes it. This allows the weights to benefit from a broader spectrum early on and adapt freely afterward. Across LLaMA, Qwen3, and fine-grained mixture-of-experts models ranging from 130M to 8B parameters, ORCA achieves lower final validation loss than Muon. Its loss reduction relative to Muon matches or exceeds Muon's reduction relative to Adam. Ablations support the early-shaping, later-release design. Further, ORCA requires no architectural changes and adds minimal overhead.

Momentum

Related topics

Latest papers 31