Spectral Optimization

Follow this topic Ask about this topic

Latest papers

22 papers6 papers in the last four weeks, up 50% on the four weeks before. 0.0% of all new papers.

Corrected Block Sparse Spectral Optimization: Exact Models, Finite Identification, and Local Dynamics
Oct 19, 2026Omid Shams SolariSpectral Optimization Structured Sparsity
sAIm Labs
Bridging Compressed Communication and Local Steps via Spectral Method in Federated Learning
Oct 16, 2026Dmitriy Bystrov, George Chirkov, Daniil Medyakov +2Federated Learning Communication-Efficient Distributed Training
Basic Research of Artificial Intelligence Laboratory (BRAIn Lab) · Federated Learning Problems Laboratory · Innopolis University
Which Geometry on Which Layer? Measured Sign-Spectral Assignment in Mixed-Optimizer Training
Oct 12, 2026Hiroki Naganuma, Laura Gomezjurado, Mahdi Ghaznavi +5Spectral Optimization Deep Learning Optimization
Mila · Université de Montréal · Stanford University +4
Spectrally Targeted Muon
Oct 7, 2026Vishrut Goyal, Rohan RamkumarSpectral Optimization Deep Learning Optimization
Department of Computer Science, Stanford University · Department of Electrical Engineering, Stanford University.
ORCA: The Annealed Spectral Conditioning Optimizer for Faster, Better LLM Training
Oct 5, 2026Yuanshi Liu, Boyuan Jiang, Liang Hou +4Spectral Regularization Deep Learning Optimization
School of Intelligence Science and Technology, Peking University · Kling Team
Neural networks for spectral optimization
Sep 28, 2026Alexis de Villeroché, Beniamin Bogosel, Stéphane Breuils +2Neural Surrogate Modeling Spectral Optimization
Orthogonal Witness Control for Muon Optimization via Sigmoid Spectral Reshaping
Sep 27, 2026Dat Phi Van, Ngo Vu Minh, Tuc Nguyen +3Neural Network Optimization Spectral Optimization
Center for AI Research, VinUniversity, Hanoi, Vietnam · Indiana University, Bloomington, IN, USA · Monash University, Clayton, VIC, Australia
Online Supervised Dimension Reduction with Random Features: Diagnostics and Computational Trade-offs
Sep 17, 2026Zhenlin Yao, Wei XiongPrincipal Component Analysis Random Feature Methods
School of Statistics University of International Business and Economics Beijing, China
The Drift Contract: Spectral Updates for Depth-Robust Local Learning
Sep 16, 2026Fabien PollySpectral Optimization Local Learning
Independent researcher
Beyond the Matrix Sign: Quadratic Spectral Descent
Sep 7, 2026Qiaozhe Zhang, Jun Sun, Yingzhuang LiuSign-Based Optimization Muon Optimizer
School of Electronic Information and Communications Huazhong University of Science and Technology
Spectral Allocation: Why Muon Outperforms Adam, and How to Improve Muon
Aug 26, 2026Xiaodong Wu, Wenyi Yu, Chao Zhang +1Deep Learning Optimization Muon Optimizer
Department of Engineering University of Cambridge · Department of Electronic Engineering Tsinghua University
Joint Affine Spectral Shaping: Coupling Weight and Bias Updates Beyond Weight-Only Muon
Aug 4, 2026Gongyue Zhang, Honghai LiuNeural Network Optimization Muon Optimizer
State Key Laboratory of Robotics and Systems, Harbin Institute of Technology, Shenzhen, Shenzhen, 518055, China.
Aurora: A Leverage-Aware Spectral Optimizer
Jun 26, 2026Alec Dewulf, Dhruv Pai, Li Yang +2Neural Network Optimization Muon Optimizer
Tilde Research
Muon $^p$ : Muon with Fractional Spectral Powers
Jun 11, 2026Yihe Dong, Will SawinNeural Network Optimization Muon Optimizer
Princeton University
The Spectral Dynamics and Noise Geometry of Muon
Jun 7, 2026Pierfrancesco Beneventano, Mahmoud Abdelmoneum, Tomaso PoggioMuon Optimizer Matrix Optimization
Massachusetts Institute of Technology
Anytime Training with Schedule-Free Spectral Optimization
May 21, 2026Anuj Apte, Pranav Deshpande, Niraj Kumar +2Deep Learning Optimization Learning Rate Scheduling
Global Technology Applied Research, JPMorganChase New York, NY 10001, USA
Constrained Stochastic Spectral Preconditioning Converges for Nonconvex Objectives
May 12, 2026Konstantinos Oikonomidis, Jan Quan, Kimon Antonakopoulos +3Nonconvex Stochastic Optimization Muon Optimizer
KU Leuven, Department of Electrical Engineering, STADIUS Center for Dynamical Systems, Signal Processing, and Data Analytics · ´Ecole Polytechnique F´ed´erale de Lausanne (EPFL), Laboratory for Information and Inference Systems (LIONS) · CVN, CentraleSup´elec, Universit´e Paris-Saclay, Inria
Phases of Muon: When Muon Eclipses SignSGD
May 10, 2026Elliot Paquette, Noah Marshall, Lucas Benigni +3Sign-Based Optimization Muon Optimizer
Mathematics and Statistics Department, McGill University · Mathematics and Statistics Department, Université de Montréal · Google DeepMind
Accelerating Zeroth-Order Spectral Optimization with Partial Orthogonalization from Power Iteration
May 9, 2026Jiahe Chen, Ziye MaZeroth-Order Optimization LLM Fine-Tuning
Department of Computer Science City University of Hong Kong
Orth-Dion: Eliminating Geometric Mismatch in Distributed Low-Rank Spectral Optimization
May 7, 2026Tatsuhiro Nakamori, Laura Gomezjurado Gonzalez, Ganesh Talluri +5Communication-Efficient Distributed Training Distributed Optimization
1Keio University · 2Stanford University · 3Midwestern University +3
Subspace Control: Turning Constrained Model Steering into Controllable Spectral Optimization
Apr 5, 2026Yancheng Huang, Changsheng Wang, Chongyu Fan +7Spectral Optimization Spectral Methods
OPTML, Michigan State University · MIT-IBM Watson AI Lab, IBM Research · University of Minnesota +2
Sharp Capacity Scaling of Spectral Optimizers in Learning Associative Memory
Mar 27, 2026Juno Kim, Eshaan Nichani, Denny Wu +2Spectral Optimization Neural Network Optimization
UC Berkeley · Princeton University · New York University +1

Oct 19, 2026J/K move · Enter open · S save

Corrected Block Sparse Spectral Optimization: Exact Models, Finite Identification, and Local Dynamics

Omid Shams Solari

sAIm Labs

Spectral Optimization Structured Sparsity Principal Component Analysis Riemannian Optimization

Open paper AskPDF arXiv

The block generalized-power method pairs threshold-based support selection with inexpensive Stiefel updates, but the standard block formulation conflates two nonequivalent objectives and leaves inactive components, threshold ties, and post-selection refitting ill-specified. We rebuild the model from the primal. Eliminating each loading column from the normalized loading-ball problem gives a sum of soft-thresholded Euclidean norms (A-SUM); the squared-threshold objective in common use arises from a quadratic lift (A-SQ). We prove both identities, exhibit a two-component instance whose maximizers disagree, and specify explicit conventions for zero-response columns, ties, rank deficiency, and mask-before-refit. For A-SQ we separate the historical full-polar iteration from a safeguarded Riemannian ascent and prove, for the latter, whole-sequence convergence via a sufficient-ascent lemma, a relative-error lemma, and the Kurdyka-Lojasiewicz property, which applies because the soft-threshold objective and the Stiefel constraint are semi-algebraic. A strict threshold margin then forces finite identification of the support and sign pattern. The post-identification map is ordinary power iteration on the active covariance for a rank-one hard threshold, and the Frechet derivative of a coupled polar map for component-specific supports; a radius-based safe-screening rule follows from the same local analysis. Planted-spike experiments match the predicted identification time and local rates. On modern-AI panels (GPT-2 residual stream, ViT CLS stream, Pitprops) the thresholded block-sparse components match dense PCA on reconstruction and improve on it in monosemanticity and class specialisation. On three native-label scRNA datasets (Zheng17 FACS-sorted PBMCs, 68k donor-A with FACS-correlated labels, Tabula Muris) A-SQ beats dense PCA on cluster-recovery AMI by +0.023 to +0.044 at matched sparsity.