Spectral Methods

Follow this topic Ask about this topic

Momentum

17 papers in the last four weeks, up 42% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 169

All topics

Accelerating Zeroth-Order Spectral Optimization with Partial Orthogonalization from Power Iteration
May 9, 2026Jiahe Chen, Ziye MaZeroth-Order Optimization LLM Fine-Tuning
Department of Computer Science City University of Hong Kong
Anchoring the Eigengap: Cross-Modal Spectral Stabilization for Sample-Efficient Representation Learning
May 9, 2026Nikhil J. Dhinagar, Vidhi Chhatbar, Chirag Jagad +6Representation Learning Spectral Methods
Imaging Genetics Center, Mark & Mary Stevens Neuroimaging & Informatics Institute, Keck School of Medicine, University of Southern California, Los Angeles, CA, USA · Neuroscience Graduate Program, Mark & Mary Stevens Neuroimaging & Informatics Institute, Chan Division of Occupational Science & Occupational Therapy, Biomedical Engineering, University of Southern California, Los Angeles, CA, USA
Diagnosing Spectral Ceilings in Equivariant Neural Force Fields
May 8, 2026Hyunmog KimMachine Learning Interatomic Potentials Spectral Bias
Independent Researcher.
Orth-Dion: Eliminating Geometric Mismatch in Distributed Low-Rank Spectral Optimization
May 7, 2026Tatsuhiro Nakamori, Laura Gomezjurado Gonzalez, Ganesh Talluri +5Communication-Efficient Distributed Training Distributed Optimization
1Keio University · 2Stanford University · 3Midwestern University +3
Full-Spectrum Graph Neural Networks: Expressive and Scalable
May 7, 2026Xiaohan Wang, Deyu Bo, Longlong Li +1Graph Neural Networks GNN Expressivity
Division of Mathematical Sciences, School of Physical and Mathematical Sciences, Nanyang Technological University, Singapore 637371, Singapore.
Self-Attention as Transport: Limits of Symmetric Spectral Diagnostics
May 6, 2026Dominik Dahlem, Diego Maniloff, Mac MisiuraTransformer Interpretability Attention Head Analysis
Red Hat AI
Free Decompression with Algebraic Spectral Curves
May 5, 2026Siavash Ameli, Chris van der Heide, Liam Hodgkinson +1Random Matrix Theory Spectral Methods
1ICSI and Department of Statistics, University of California, Berkeley · 2Dept. of Electrical and Electronic Engineering, University of Melbourne · School of Mathematics and Statistics, University of Melbourne +1
Geometric and Spectral Alignment for Deep Neural Network I
May 4, 2026Ziran Liu, Wei Wang, Jinhao Wang +5Residual Learning Spectral Methods
Shanghai Institute for Mathematics and Interdisciplinary Sciences (SIMIS), Shanghai 200433, China · Futurewei Technologies, Inc., San Jose, CA 95131 · Dept. of Computer Science and Engineering, Santa Clara University, Santa Clara, CA 95050 +2
RamanBench: A Large-Scale Benchmark for Machine Learning on Raman Spectroscopy
May 3, 2026Mario Koddenbrock, Christoph Lange, Robin Legner +6Benchmark Design Scientific ML
HTW Berlin · TU Berlin · KWS SAAT, Einbeck +5
Complex Diffusion Maps with $ω$ -Parameterized Kernels Revealing Inherent Harmonic Representations
May 3, 2026Tongzhen Dang, Weiyang Ding, Michael K. NgManifold Learning Kernel Methods
Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai, China. · Institute of Science and Technology for Brain-Inspired Intelligence, Fudan University, Shanghai, China; Key Laboratory of Computational Neuroscience and Brain-Inspired Intelligence (Fudan University), Ministry of Education, China. · Department of Mathematics, Hong Kong Baptist University, Hong Kong, China.
Local Hessian Spectral Filtering for Robust Intrinsic Dimension Estimation
May 2, 2026Genki OsadaIntrinsic Dimensionality Spectral Methods
LY Corporation.
Spectral Graph Sparsification Preserves Representation Geometry in Graph Neural Networks
May 1, 2026Sanjukta KrishnagopalGraph Neural Networks Neural Representation Geometry
Department of Computer Science, University of California Santa Barbara
Disease Is a Spectral Perturbation
May 1, 2026John D. Mayfield, Matthew S. RosenClinical Outcome Prediction Spectral Methods
Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital · Department of Radiology, Harvard Medical School, Boston, MA 02129
A Short Note on Batch-efficient Divide-and-Conquer Algorithm for EigenDecomposition
Apr 30, 2026Yue SongSpectral Methods
College of AI, Tsinghua University, China
Large-scale semi-supervised learning with online spectral graph sparsification
Apr 29, 2026Daniele Calandriello, Alessandro Lazaric, Michal ValkoGraph Sparsification Graph-Based Semi-Supervised Learning
Team SequeL Inria Lille – Nord Europe, France
Cheeger--Hodge Contrastive Learning for Structurally Robust Graph Representation Learning
Apr 29, 2026Mengyang Zhao, Longlong Li, Cunquan QuGraph Representation Learning Graph Contrastive Learning
School of Physical and Mathematical Sciences, Nanyang Technological University, Singapore 637371, Singapore · Data Science Institute, Shandong University, Jinan 250100, China
TimeMM: Time-as-Operator Spectral Filtering for Dynamic Multimodal Recommendation
Apr 29, 2026Wei Yang, Rui Zhong, Zihan Lin +4Dynamic Graph Learning Recommender Systems
Xiaohongshu Inc. · Beijing, China
Sparse Graph Learning from Sparse Data via Fiedler Number Maximization
Apr 28, 2026Bahar Oveisgharan, Gene Cheung, Andrew EckfordGraph Structure Learning Spectral Methods
York University, Canada
Diffusion-Guided Feature Selection via Nishimori Temperature: Noise-Based Spectral Embedding
Apr 27, 2026Vasiliy S. Usatyuk, Denis A. Sapozhnikov, Sergey I. EgorovFeature Selection Dimensionality Reduction
R&D Department T8 LLC Moscow, Russia · Department of Computer Science South-West State University Kursk, Russia
Learning Curves and Benign Overfitting of Spectral Algorithms in Large Dimensions
Apr 25, 2026Weihao Lu, Qian Lin, Yingcun Xia +1Kernel Ridge Regression Benign Overfitting
Department of Statistics and Data Science, National University of Singapore. · Department of Statistics and Data Science, Tsinghua University.
A Brain-Inspired Deep Separation Network for Single Channel Raman Spectra Unmixing
Apr 24, 2026Gaoruishu Long, Jinchao Liu, Bo Liu +2Raman Spectroscopy Spectral Methods
College of Artificial Intelligence, Nankai University, Tianjin, China · Guangdong Laboratory of Chemistry and Fine Chemical Industry, Guangdong, China · Smekal Tech (Shantou) Ltd. +1
Spectral Embeddings Leak Graph Topology: Theory, Benchmark, and Adaptive Reconstruction
Apr 22, 2026Thinh Nguyen-Cong, Truong-Son Hy, Thang N. DinhSpectral Methods
Department of Computer Science Virginia Commonwealth University Richmond, VA, USA · Department of Computer Science The University of Alabama at Birmingham Birmingham, AL, USA
Improved large-scale graph learning through ridge spectral sparsification
Apr 22, 2026Daniele Calandriello, Ioannis Koutis, Alessandro Lazaric +1Graph Sparsification Spectral Methods
SequeL team, INRIA Lille - Nord Europe, France · LCSL, IIT, Italy · New Jersey Institute of Technology, USA +1
Analysis of Nystrom method with sequential ridge leverage scores
Apr 22, 2026Daniele Calandriello, Alessandro Lazaric, Michal ValkoKernel Ridge Regression Kernel Regression
SequeL team INRIA Lille - Nord Europe
Achieving the Kesten-Stigum bound in the non-uniform hypergraph stochastic block model
Apr 21, 2026Manuel Fernandez, Ludovic Stephan, Yizhe ZhuSpectral Clustering Stochastic Block Model
Department of Mathematics, University of Southern California · Univ Rennes, Ensai, CNRS, CREST-UMR 9194, F-35000 Rennes, France
Predicting LLM Compression Degradation from Spectral Statistics
Apr 20, 2026Mingxue XuLLM Compression Low-Rank Matrix Decomposition
Department of Electrical and Electronic Engineering, Imperial College London, London, United Kingdom
Spectral Forensics of Diffusion Attention Graphs for Copy-Move Forgery Detection
Apr 19, 2026H. M. Shadman Tabib, Tasriad Ahmed Tias, Nafis TahmidImage Forgery Detection Digital Image Forensics
Department of Computer Science and Engineering Bangladesh University of Engineering and Technology (BUET) Dhaka, Bangladesh
Spectral Analysis of Fake News Propagation
Apr 18, 2026Weibin Cai, Reza ZafaraniSocial Media Analysis Fake News Detection
Data Lab, Department of EECS, Syracuse University, New York, USA.
Convergence theory for Hermite approximations under adaptive coordinate transformations
Apr 18, 2026Yahya SalehSpectral Methods
Spectral Kernel Dynamics for Planetary Surface Graphs: Distinction Dynamics and Topological Conservation
Apr 17, 2026Jnaneshwar DasDynamical Systems Spectral Methods
School of Earth and Space Exploration, Arizona State University

May 9, 2026, cs.LGJ/K move · Enter open · S save

Accelerating Zeroth-Order Spectral Optimization with Partial Orthogonalization from Power Iteration

Jiahe Chen, Ziye Ma

Department of Computer Science City University of Hong Kong

Zeroth-Order Optimization LLM Fine-Tuning Muon Optimizer Spectral Methods

Open paper AskPDF arXiv

Zeroth-order (ZO) optimization has become increasingly popular and important in fine-tuning large language models (LLMs), especially on edge devices due to its ability to adjust the model to local data without the need for memory-intensive back-propagation. Recent works try to reduce ZO variance through low-dimensional subspace search, but subspace restriction alone leaves key optimization geometry under-exploited, motivating additional acceleration. In this work, we focus on the hidden layer training problem in which spectral optimizers like Muon outperform AdamW due to its ability to exploit weak spectral directions by orthogonalization. However, we have discovered that unlike in the first-order setting, full orthogonalization works poorly in the ZO setting since the gradient estimates are highly noisy and unreliable. To address this issue, we propose applying partial spectral orthogonalization to accelerate ZO optimization. To do so, we replace the iconic Newton-Schulz procedure in Muon with the faster, more concentrated power-iteration method so that it only amplifies dominant spectral directions. Furthermore, to improve the efficiency and generalization of the algorithm, we adopted a streaming variant of power-iteration that requires low variance in gradients, which was achieved through constraining our search inside a subspace obtained through the projection of momentum, echoing recent advances. Experiments on LLM fine-tuning show that our method can achieve from 1.5x to 4x the convergence speed of ZO-Muon, the current SOTA algorithm, across SuperGlue datasets in the OPT-13B model. Across different models, we also reach competitive final accuracies with less time in most cases compared with strong ZO baselines such as MeZO, LOZO and ZO-Muon. Code is available at https://github.com/MOFA-LAB/ZO-MOPI.git.

Momentum

Related topics

Latest papers 169