Spectral Methods

Follow this topic Ask about this topic

Momentum

17 papers in the last four weeks, up 42% on the four weeks before. 0.2% of all new papers.

Jul 13Week of Sep 28

Latest papers 169

Jun 1, 2026cs.LG

Pseudospectral Bounds for Transient Amplification in Coupled Gradient Descent

Coupled gradient descent - where the update of one parameter depends on another - arises naturally in bilevel optimization, two-time-scale stochastic approximation, and generative adversarial networks. When the coupled Jacobian is block-triangular, asymptotic stability is determined by the spectral radii of the diagonal blocks, yet transient amplification before convergence can be arbitrarily large due to non-normality. We develop a sharp pseudospectral theory for block-triangular Jacobians J = [[A, 0], [C, D]], proving Kreiss-constant bounds of the form K(J) <= 2/(1-γ) + ||C||/(4(1-γ)) when ρ(A), ρ(D) <= γ< 1 and A, D are symmetric, and establishing matching minimax lower bounds. We characterize the critical coupling threshold for spectral instability and extend the theory to nearly self-referential systems via a Neumann-series perturbation framework. As a consequence, we obtain a finite-horizon O(K(J)^2 log(1/δ)) iteration complexity bound. Framed as scaling laws for stochastic two-time-scale optimization, our results expose a non-asymptotic, instance-dependent regime of high-dimensional learning dynamics that is invisible to spectral-radius analysis. Experiments on linear-quadratic problems, IQC-based comparisons, and neural-network training confirm the theory.

Ahanaf Hasan Ariq

Ideal School and College

Stochastic Approximation Two-Timescale Stochastic Approximation Spectral Methods Gradient Descent Dynamics

Jun 1, 2026eess.AS

Localizing broadband noise sources using the Loève spectrum and a 2.5D approach

The localization of moving sound sources using a microphone array is typically based on modifying the signal to compensate for the Doppler effect. In the time domain this compensation is done on a sample-by-sample basis. In the frequency domain short time segments need to be used in which the Doppler effect is assumed to be approximately constant and a discrete Fourier transform is done on each segment. In contrast, the authors developed an inverse 2.5D localization method for uniformly moving single-frequency sources that works in the spectral domain and allows for the use of longer windows. This was achieved by modifying the 2.5D forward model to directly compute the effect of the motion in the static observer position. The method does neither require to modify the measured signal nor does it require quasi-stationary of the measurements within the window used. Unfortunately, this approach is not directly suitable for broad-band stochastic sources, and in the present work we will investigate how the statistical properties of a uniformly moving stochastic source change when observed at a static observer. Using a 2.5D setting, the relation between the power spectral density of the moving source and the Loève spectrum, which is a generalization of the cross-spectral density at the static receivers, was derived. Based on simulated data with speeds up to 100 m,s

^{-1}

, the work presented here provides a proof of concept for a method based on multi-taper estimates for the Loève spectrum to localize moving broad-band stochastic sources . Currently, the method requires a stationary source signal and that the spectral density is flat within a certain range around the frequency of interest. Also, correlations between sources are currently not considered.

Christian H. Kasess, Wolfgang Kreuzer, Holger Waubke

Acoustics Research Institute, Austrian Academy of Sciences, Dominikanerbastei 16, Vienna, A-1010, Austria

Direction-of-Arrival Estimation Spectral Methods

May 30, 2026cs.DS

Easy, robust approximate message passing for planted spike models

We present a simple and efficient algorithm for robust approximate message passing (AMP) in the spiked matrix setting. In particular, let

\varepsilon

be a sufficiently small constant, and suppose that

X \in \mathbb R^{n \times n}

is a Gaussian matrix with a planted rank-

1

spike, and

E \in \mathbb R^{n \times n}

is an adversarially chosen matrix supported on an

\varepsilon n \times \varepsilon n

principal minor. Let

v_{\mathrm{AMP}}(X)

be the output of an AMP iteration on the uncorrupted matrix

X

. We give a procedure that, given access only to the corrupted matrix

Y = X + E

, computes a vector

v_{\mathrm{ALG}}(Y)

which is

\tilde{O}(\sqrt{\varepsilon})

-close to

v_{\mathrm{AMP}}(X)

, for any of a class of AMP iterations which includes sparse Principal Component Analysis (PCA), non-negative PCA, and

\mathbb Z_2

synchronization. Our algorithm consists of a spectral pre-processing step combined with a robust spectral initialization procedure; given these inputs, we prove that (perhaps surprisingly) AMP is robust out-of-the-box.

Misha Ivkov, Tselil Schramm

*Stanford University. · †Stanford University.

Random Matrix Theory Principal Component Analysis Adversarial Robustness Spectral Methods

May 29, 2026physics.comp-ph

Data-Driven Spectral Prediction for Accelerating Large-Scale Electronic Structure Calculations

Simulating large molecular systems comprising thousands of atoms requires highly scalable methodologies. While modern Density Functional Theory (DFT) codes exhibit linear scaling, solving the associated large, sparse generalized eigenproblems remains a critical computational bottleneck on exascale architectures. In the context of the LimitX project, we propose a data-driven framework to accelerate these calculations. By shifting the machine learning target from discrete eigenvalues to the coefficients of an interpolating Chebyshev polynomial, and by comparing both all-atom and fragment-based structural representations, we successfully overcome the dimensionality constraints of large-scale spectral prediction. We investigate three machine learning models (Kernel Ridge Regression, Graph Neural Networks, and Random Forests) trained on a novel 2 TB dataset of protein dimers. The predicted spectra provide initial guesses that effectively bypass early Self-Consistent Field (SCF) iterations in BigDFT. Ultimately, these spectral predictors will be deployed to dynamically optimize upcoming rational filter-based eigensolvers, such as FrASE, which is currently in initial development.

Abhiram Badrinarayanan, Davor Davidovic, Edoardo Di Napoli +4

Jülich Supercomputing Centre, Forschungszentrum Jülich, Germany · CEA / IRIG / MEM / L_Sim, France

Quantum Chemistry Scientific ML Spectral Methods

May 29, 2026cs.LG

Modeling Spectral Energy Shifts in Spatio-Temporal Graph Anomaly Detection

Graph anomaly detection methods aim to distinguish anomalous nodes. While prior methods characterize anomalies through increased variation in the spectral energy distributions, they overlook those that result in decreased variation, i.e., camouflaged anomalies that appear normal. We show that this type of anomaly persists across multiple datasets and remains undetectable by existing spectral approaches. To address this limitation, we propose a node-level spectral energy formulation that is fully compatible with message passing and enables the detection of camouflaged anomalies. Building on this formulation, we introduce an energy-aware graph learning framework that models spectral shifts through energy-driven message passing in both static and time-series graphs. Besides, our unified architecture extends to temporal settings without introducing specialized sequence modules, enabling efficient learning under long sliding windows. Extensive experiments on large-scale benchmarks demonstrate the effectiveness and scalability of our approach.

Yilin Liu, Hongchao Zhang, Taylor T. Johnson +2

College of Connected Computing, Vanderbilt University, Nashville, USA · Department of Civil and Environmental Engineering, Electrical and Computer Engineering, Vanderbilt University, Nashville, USA

Spectral Methods Spatiotemporal GNNs Anomaly Detection

May 29, 2026cs.LG

Spectral Anatomy of Quantum Gaussian Process Kernels

Two recent results have reshaped quantum Gaussian processes (QGPs). On the one hand, \citet{lowe2025assessing} rule out the exponential speedups claimed by HHL-based QGP regression in the typical, well-conditioned regime; on the other, an independent line of work shows that highly expressive quantum kernels suffer posterior pathologies that break Bayesian optimization. We show that these seemingly unrelated phenomena are governed by the same quantity: the normalized spectral entropy

S(K)/\log n

of the kernel Gram matrix. We prove a Cauchy--Schwarz tail bound on Nyström approximation error, a finite-sample variance-contraction identity in terms of Bach's degrees of freedom

d_σ(K)

, and a characterization of the \emph{target-dependent} optimal entropy via the intrinsic dimension of the target in the kernel eigenbasis. Empirically, the diagnostic is kernel-agnostic: hardware-efficient, matchgate, IQP \emph{and} RBF/Matérn/RFF/deep-kernel families all collapse onto identical

S/\log n

curves on dequantization, ECE, and variance-contraction panels. The NLL sweet spot lives at high entropy for smooth targets and at low entropy for band-limited quantum-data targets. The diagnostic transfers from simulator to IBM Heron hardware with median absolute error

3.2\%

and mean

5.2\%

in

S/\log n

across

24

configurations at

n_q = 4

, with matchgate and IQP within

5\%

mean and a single HE configuration returning a

30\%

outlier that drops to

0.5\%

on rerun (attributed to calibration drift); the same diagnostic transfers to a second Heron backend (mean error

2.7\%

) and to a

n_q = 6

scale-up on the original backend (mean error

1.7\%

). No error mitigation is applied throughout.

Jian Xu, Chao Li, Guang Lin +4

1RIKEN iTHEMS · 2RIKEN AIP · 3South China University of Technology +1

Quantum Machine Learning Gaussian Processes Quantum Kernel Methods Spectral Methods

May 26, 2026cs.CV

Rotation-Invariant Spherical Watermarking via Third-Order SO(3) Representation Coupling

Reliable watermarking of panoramic imagery is fundamentally challenged by arbitrary 3D rotations. As panoramas are defined on the sphere, they naturally transform under the action of

SO(3)

, rendering conventional planar representations and augmentation-based robustness strategies inadequate and devoid of theoretical guarantees. To address this, we formulate panoramas as spherical signals and leverage

SO(3)

representation theory to derive provably rotation-invariant descriptors. While spherical harmonic coefficients transform equivariantly under rotations, the natural invariant constructions are typically limited to zeroth-order statistics which eliminate directional information and severely constrain embedding capacity. In this work, we introduce a principled third-order invariant construction by coupling higher-order

SO(3)

irreducible representations via tensor products and projecting onto the trivial representation. This yields a spherical invariant bispectrum that preserves phase information while remaining strictly rotation-invariant. Leveraging this property, we embed watermarks into higher-order spherical harmonic coefficients and recover them from invariant bispectral scalars, enabling reliable extraction under arbitrary 3D rotations. We provide a theoretical proof of

SO(3)

invariance for it and demonstrate experimentally its near-perfect robustness to continuous rotations while maintaining high visual fidelity.

Pengzhen Chen, Yanwei Liu, Xiaoyan Gu +3

Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · State Key Laboratory of Cyberspace Security Defense +2

Robust Watermarking Spherical Harmonics Spectral Methods

May 26, 2026cs.LG

MuCon: Clipped Muon Updates for LLM Training

Muon-style optimizers take a matrix-valued momentum or preconditioned update

B = U \operatorname{diag}(σ_1,\ldots,σ_r) V^\top

and replace it with its canonical partial polar factor

\operatorname{Pol}(B) = U V^\top

. This maps every nonzero singular value to one. MuCon is the clipped-Muon variant studied here: it applies singular-value clipping to the same Muon matrix,

D^{\mathrm{MuCon}}\_τ(B) = \operatorname{MClip}\_τ(B) = U \operatorname{diag}\bigl(\min\{σ\_i,τ\}\bigr) V^\top, \qquad τ> 0

. Thus,

\operatorname{MClip}\_τ

denotes the mathematical clipping operator, while MuCon denotes the optimizer primitive that substitutes this clipped direction for Muon's polar direction. The Muon/MuCon scaling parameterization used in this work is called

\text{SpectralP}

: it is the hidden-matrix scaling recipe under which polar Muon or clipped MuCon directions are applied. The map

\operatorname{MClip}\_τ

is the Frobenius projection onto the spectral-norm ball

\{X : \|X\|_2 \le τ\}

: it leaves singular values at or below

τ

unchanged and modifies only the violating singular directions. This paper asks when the MuCon clipping step can be approximated without a full dense SVD. We record two exact identities, a polar/absolute-value formula and a scalar-root formulation leading to a rational Newton filter for the clipped positive-semidefinite factor, and identify the numerical obstruction common to both: singular values near the threshold make sign decisions and rational solves ill-conditioned. Matrix-function methods are therefore useful only when paired with stable polar/square-root primitives or explicit regularization near the clipping boundary.

Albert Yi

Muon Optimizer Spectral Methods LLM Training

May 25, 2026stat.ML

Mean-Shift PCA by Knockoff Mean

Removing noise is difficult, but adding noise is easy. In this work, we show how to eliminate mean-shift noisy components from PCA by deliberately introducing knockoff mean-shift perturbation. Standard PCA is highly sensitive to shifts in the sample mean: a small fraction of samples from a shifted distribution can cause large deviations in the leading principal components. In high-dimensional regimes, existing Robust PCA approaches cannot handle the mean-shift contamination structure inherent in the mixture model. Using tools from Random Matrix Theory, we prove that the mean-shift spikes are spectrally separable from the stable eigenvalues of the original covariance. Furthermore, the original eigenspace remains asymptotically invariant to the contamination, independent of the mixture weight. Exploiting this spectral stability, we propose a simple, two-stage PCA algorithm by adding knockoff mean that identifies and removes the mean-shift component using only standard PCA operations.

Mengda Li, Zeng Li, Jianfeng Yao

School of Data Science, The Chinese University of Hong Kong, Shenzhen, China · Department of Statistics and Data Science, Southern University of Science and Technology, Shenzhen, China

Distribution Shift Robustness Principal Component Analysis Spectral Methods Noise Robustness

May 23, 2026cs.LG

Learning Laplacian Eigenspace with Mass-Aware Neural Operators on Point Clouds

The eigendecomposition of the Laplace--Beltrami Operator (LBO) is fundamental to geometric analysis, yet computing its low-frequency eigenmodes remains a significant bottleneck due to the high cost of iterative solvers on large-scale data. To amortize this cost, we introduce the Neural Eigenspace Operator (NEO), a feed-forward framework designed to predict the spectrum directly from point clouds. Crucially, NEO circumvents the ill-posed nature of standard eigenvector regression, which suffers from intrinsic sign flips and rotation ambiguities, by learning the stable, invariant low-frequency subspace instead. Specifically, the network predicts a redundant set of basis functions whose span robustly covers the target eigenspace, allowing for the recovery of accurate eigenpairs via a lightweight Rayleigh--Ritz refinement. To handle irregular sampling, we propose a mass-aware neural operator that incorporates per-point area weights into attention-based aggregation, improving robustness to non-uniform densities and enabling zero-shot generalization across resolutions. Our approach achieves near-linear runtime scaling and substantial wall-clock speedups over iterative solvers at comparable accuracy, and exhibits strong zero-shot transfer to high-resolution point clouds. The resulting eigenpairs support standard spectral geometry tasks, while the raw basis functions provide effective point-wise features for downstream learning. Code: https://github.com/Adversarr/NEO.

Zherui Yang, Tao Du, Ligang Liu

University of Science and Technology of China, China · Tsinghua University, China · Shanghai Qi Zhi Institute, China +1

Point Cloud Learning Geometric Representation Learning Manifold Learning Spectral Methods Neural Operators

May 22, 2026cs.LG

Entrywise Error Bounds for Spectral Ranking with Semi-Random Adversaries

Bradley-Terry-Luce (BTL) model estimation is a well-established strategy to rank a collection of items given a dataset of pairwise comparisons. Although the theoretical performance of BTL estimation methods, such as spectral and maximum likelihood estimation, is well studied in the regime of uniformly sampled graphs, generalizing such results to a wider class of random graphs has proved challenging. In this work, we investigate the entry-wise error of spectral algorithms against a semi-random adversary that can arbitrarily boost the sampling probabilities of certain edges. We find that the performance of the unweighted spectral method is heavily dependent on the spectral properties of the generated graph. Furthermore, we show that asymptotic performance approaching that of uniformly sampled graphs can be recovered by appropriately reweighting the observed edges to counteract the adversary and restore the spectral gap. Finally, we provide numerical simulations that support our theoretical findings.

Dongmin Lee, Anuran Makur, Japneet Singh

Department of Computer Science Purdue University West Lafayette, IN, USA · Department of Computer Science Elmore Family School of Electrical and Computer Engineering Purdue University West Lafayette, IN, USA

Learning to Rank Pairwise Ranking Bradley–Terry Model Spectral Methods

May 21, 2026stat.ML

The ASE-LSE Disagreement Landscape: An End-to-End Characterisation of Extremes and Structural Drivers

Two of the most widely used methods for analysing graph data, Adjacency Spectral Embedding and Laplacian Spectral Embedding, often produce different results when applied to the same graph. Yet the structural reasons behind this disagreement remain incompletely understood. This paper provides an end-to-end account of ASE-LSE latent subspace disagreement. We first prove that the two methods produce identical latent subspaces for every embedding dimension whenever the Laplacian is a scalar multiple of the adjacency matrix, and show that this scalar relationship holds if and only if the graph is either regular or bipartite biregular. This anchor result identifies a sufficient condition for perfect agreement that pins down the floor of the disagreement spectrum and supplies the baseline for the perturbation analysis. We then prove that no maximal-disagreement graph or family of graphs exists: the disagreement is always strictly below its theoretical ceiling, and we exhibit a witness family demonstrating that no finite maximum is attainable, so the disagreement landscape has no maximiser. With both endpoints established, we derive a Regularity Departure Bound whose two terms isolate degree heterogeneity and eigengap as the primary structural factors influencing disagreement in the middle regime. Empirical validation across thousands of simulated graphs confirms the mechanisms predicted by the bound: heterogeneity pushes disagreement up, eigengap suppresses it, and their joint ratio emerges as a unified predictor of ASE-LSE disagreement, suggesting when the two embeddings can be treated as interchangeable and when they cannot.

Minh Triet Pham, Ian Gallagher

School of Mathematics and Statistics The University of Melbourne, Australia

Graph Embedding Spectral Methods

May 19, 2026cs.LG

Spectral Unforgetting: Post-Hoc Recovery of Damaged Capabilities Without Retraining

Fine-tuning a language model for a target task routinely degrades capabilities the training data never explicitly threatened. We study this phenomenon, known as catastrophic forgetting, and propose a post-hoc repair solution that uses only the pretrained checkpoint

W_{\mathrm{base}}

and its fine-tuned descendant

W_{\mathrm{ft}}

. The goal is not merely to revert the model toward the base checkpoint, but to recover capabilities damaged by fine-tuning while preserving both the target-task gains and any beneficial held-out improvements. We introduce DG-Hard, a checkpoint-only spectral repair method for the fine-tuning update

Δ= W_{\mathrm{ft}} - W_{\mathrm{base}}

. DG-Hard treats

Δ

as a low-rank task-aligned signal embedded in an IID-like noise residual that gradient descent has no incentive to remove, and applies the Donoho-Gavish hard singular-value threshold to each weight-delta matrix, keeping the structured high-energy part of the update and removing the spectral bulk. This reduces repair to a closed-form SVD filtering step requiring no data-dependent tuning. A central difficulty is evaluation: average accuracy hides per-benchmark failures, while naive recovery scores reward models that simply revert toward the base. We therefore introduce a partition-conditional metric that separately tracks healing, preservation, non-damage, and target-task retention. Across

14

(model, task) settings and nine cross-domain held-out benchmarks, DG-Hard achieves the strongest balanced repair among post-hoc baselines. DG-Hard also restores safety alignment degraded by benign fine-tuning on three independent safety axes, despite using no alignment data. These results suggest that part of fine-tuning-induced capability loss is not an unavoidable consequence of specialization, but a removable spectral residue in the weight update itself.

Aarash Abro, Muhammad Tahir

Zeta Labs · Lahore University of Management Sciences

LLM Fine-Tuning Spectral Methods Catastrophic Forgetting

May 18, 2026cs.SI

Neural Acceleration for Graph Partitioning

Graph Partitioning is a critical problem in numerous scientific and engineering domains including social network analysis, VLSI design, and many more. Spectral methods are known to produce quality partitions while minimizing edge cuts for a wide range of problems. However, the computational cost associated with the calculation of the Fiedler vector, an eigenvector associated with the second smallest eigenvalue of the graph Laplacian, remains a significant bottleneck due to memory issues and computational costs. In this paper, we present an accelerated approach to spectral bisection partitioning by replacing the traditional eigenvalue calculation with a simple artificial neural network model to approximate the Fiedler vector. We demonstrate that our approach achieves partitioning quality comparable to spectral bisection while significantly reducing the computational overhead, making it more scalable and efficient for large-scale problems

Joshua Dennis Booth, Vishvam Patel

Department of Computer Science University of Alabama in Huntsville Huntsville AL, USA

Graph Clustering Spectral Methods

May 18, 2026cs.AI

A Practical Noise2Noise Denoising Pipeline for High-Throughput Raman Spectroscopy

A lightweight and reproducible denoising pipeline for high-throughput Raman spectroscopy is presented. The approach relies on a one-dimensional convolutional autoencoder trained using a Noise2Noise strategy, requiring neither external spectral libraries nor high signal-to-noise reference spectra for training. From a reduced training subset composed of repeated short-exposure acquisitions, the model learns to reconstruct Raman spectra while efficiently suppressing stochastic noise. The method is evaluated on a heterogeneous mineral sample, using both quantitative spectral fidelity metrics (RMSE, SNR, SSIM) and task-oriented criteria based on unsupervised K-means classification. Results demonstrate that integration times as short as 5 ms per spectrum, which are typically insufficient for reliable interpretation, yield denoised spectra with high fidelity to the reference data while preserving chemically coherent maps. This work provides a practical trade-off between spectral quality and acquisition speed, enabling fast, adaptable Raman workflows compatible with routine laboratory use. It also offers a transferable framework for other one-dimensional spectroscopic modalities.

David Martin-Calle, Cesar Alvarez Llamas, Vincent Motto- Ros +3

ILM,UCBL,CNRS · Universit´e Claude Bernard Lyon 1, CNRS, Institut Lumi`ere Mati`ere, F-69622 Villeurbanne, France · ILM,UCBL,CNRS,IUF +1

Denoising Autoencoders Convolutional Autoencoder Raman Spectroscopy Spectral Methods Noise Robustness

May 18, 2026cs.SI

Prism: Structural Symmetry Scanning via Duality-Constrained Laplacian Projection

We introduce \textbf{Prism}, a framework for structural symmetry diagnosis in complex networks. Given a graph Laplacian

L

and a duality operator

P

(a symmetric involution), Prism computes the \emph{duality defect}

δ(L,P) = \|LP - PL\|_F / \|L\|_F

-- a scalar measuring how far the network deviates from structural self-consistency. When

P

encodes the network's true symmetry,

δ

starts near zero and rises monotonically as structure degrades; an arbitrary

P

gives noise. We prove that the optimal

L'

satisfying

[L', P] = 0

is given by a closed-form block-diagonal projection, and provide an unsupervised alternating optimization that learns

P

from the graph's own Fiedler vector. Experiments on synthetic networks show the true-

P

defect is

3.38\times

more sensitive to structural degradation than an index-reversal baseline and more sensitive than modularity. On Zachary's Karate Club with edge noise, Prism achieves

94.5\%

community detection accuracy at

5\%

noise versus

76.6\%

for the raw Laplacian baseline. Applied to live S&P~500 data (2026-05-17), Prism detects rising structural stress (defect

0.43 \to 0.73

over 90 days) while surface correlations remain low -- a signal invisible to correlation-based methods. In a historical backtest spanning five major stress events (2011--2020), the duality defect exhibits a consistent pattern: it reaches elevated levels \emph{before} the correlation spike that accompanies each crisis, and sustains high readings during periods of structural fragility that conventional metrics classify as calm. The duality defect is a first-principles structural admissibility condition, requiring no training data and computable in milliseconds.

Jiatong Xie

Independent researcher.

Graph Clustering Spectral Methods Anomaly Detection

May 18, 2026math.ST

Self-Distillation is Optimal Among Spectral Shrinkage Estimators in Spiked Covariance Models

Self-distillation has emerged as a promising technique for improving model performance in modern machine learning systems. We develop the statistical foundations of self-distillation in spiked covariance models, by introducing and analyzing a broad class of estimators, namely spectral shrinkage estimators. We establish that for spiked covariance matrices with

s

spikes,

s

-step self-distillation achieves optimal performance among spectral shrinkage estimators, outperforming well-known estimators in statistics and machine learning. Moreover, we show that

s

steps are necessary for optimality: any

(s-k)

-step distilled estimator is strictly suboptimal for

1 \leq k \leq s

. For the special subclass of isotropic covariances, we show that optimally tuned Ridge regression performs best among spectral shrinkage estimators. We also study a federated approach where multiple data centers share spectral shrinkage estimators and a common server seeks to aggregate them to achieve optimal performance. In this case, we find that the best local rule again takes the form of self-distillation, though it differs from the optimal rule when data are hosted centrally on a single server. Together, our results elucidate why self-distillation improves predictive performance and provide a broader statistical framework connecting it with classical shrinkage-based methods.

Radu Lecoiu, Debarghya Mukherjee, Pragya Sur

Department of Statistics, Harvard University · Department of Mathematics & Statistics, Boston University

Self-Distillation Spectral Methods Covariance Estimation Statistical Learning Theory

May 14, 2026cs.CV

CHASM: Cross-frequency Harmonized Axis-Separable Mixing for Spectral Token Operators

Spectral token mixers based on Fourier transforms provide an efficient way to model global interactions in visual feature maps. Existing designs often either apply filter-wise spectral responses along fixed channel axes, or learn adaptive frequency-indexed channel mixing without explicitly aligning the channel directions used across frequencies. We propose CHASM, a Cross-frequency Harmonized Axis-Separable Mixer, as a structured middle ground. CHASM separates what should be shared from what should remain frequency-specific: all frequencies share a learned channel eigenbasis, while each frequency retains its own positive spectral gains. The shared basis makes channel directions comparable across the spectrum, whereas the positive gains preserve local spectral adaptivity. CHASM applies this structured operator separably along the height and width axes and is used as a drop-in replacement mixer inside existing backbones. We provide a structural characterization of the shared-basis operator family and evaluate CHASM through controlled same-backbone comparisons. Across accelerated MRI reconstruction, undersampled MRI segmentation, and natural-image reconstruction, CHASM consistently improves over same-backbone spectral-mixer baselines. Ablations show that removing the shared-basis constraint weakens performance, and randomizing coherent sampling geometry substantially reduces the gain, supporting cross-frequency harmonization as a useful inductive bias for spectral token operators.

Pengcheng Fang, Hongli Chen, Yuxia Chen +3

University of Southampton · University of Queensland · 3Chengdu University of Technology +1

Frequency-Domain Feature Learning Accelerated MRI Reconstruction Spectral Methods

May 14, 2026stat.ML

Scaling Laws from Sequential Feature Recovery: A Solvable Hierarchical Model

We propose a simple mechanism by which scaling laws emerge from feature learning in multi-layer networks. We study a high-dimensional hierarchical target that is a globally high-degree function, but that can be represented by a combination of latent compositional features whose weights decrease as a power law. We show that a layer-wise spectral algorithm adapted to this compositional structure achieves improved scaling relative to shallow, non-adaptive methods, and recovers the latent directions sequentially: strong features become detectable at small sample sizes, while weaker features require more data. We prove sharp feature-wise recovery thresholds and show that aggregating these transitions yields an explicit power-law decay of the prediction error. Technically, the analysis relies on random matrix methods and a resolvent-based perturbation argument, which gives matching upper and lower bounds for individual eigenvector recovery beyond what standard gap-based perturbation bounds provide. Numerical experiments confirm the predicted sequential recovery, finite-size smoothing of the thresholds, and separation from non-hierarchical kernel baselines. Together, these results show how smooth scaling laws can emerge from a cascade of sharp feature-learning transitions.

Arie Wortsman-Zurich, Hugo Tabanelli, Yatin Dandi +2

Département d’Informatique, Ecole Normale Supérieure, PSL & CNRS · Information Learning and Physics Laboratory, École Polytechnique Fédérale de Lausanne (EPFL) · Statistical Physics of Computation Laboratory, École Polytechnique Fédérale de Lausanne (EPFL)

Hierarchical Representation Learning Scaling Laws Spectral Methods Statistical Learning Theory

May 14, 2026cs.LG

Dynamics of the Transformer Residual Stream: Coupling Spectral Geometry to Network Topology

Large language models are remarkably capable, yet how computation propagates through their layers remains poorly understood. A growing line of work treats depth as discrete time and the residual stream as a dynamical system, where each layer's nonlinear update has a local linear description. However, previous analyses have relied on scalar summaries or approximate linearizations, leaving the full spectral geometry of trained LLMs unknown. We perform full Jacobian eigendecomposition across three production--scale LLMs and show that training installs a monotonic spectral gradient through depth -- from non-normal, rotation-dominated early layers to near--symmetric late layers -- together with a cumulative low-rank bottleneck that funnels perturbations into a small fraction of the residual stream's effective dimensions. Our experiments reveal that this gradient and the dimensional collapse are learned rather than architectural, and is largely dissolved when structured non-normality is removed. We further show that the topological positioning of graph communities predicts whether the Jacobian amplifies or suppresses them, with the sign of the coupling determined by the local operator type, a relationship absent at initialization. These results map a learned spectral geometry in LLMs that links perturbation propagation and compression to the network's functional topology.

Jesseba Fernando, Grigori Guitchounts

Network Science Institute, Northeastern University · Flagship Pioneering

Dynamical Systems Transformer LLM Interpretability Spectral Methods Neural Network Training Dynamics

May 13, 2026stat.ML

Reframing preprocessing selection as model-internal calibration in near-infrared spectroscopy: A large-scale benchmark of operator-adaptive PLS and Ridge models

Preprocessing screening is often the most expensive part of a near-infrared spectroscopy calibration workflow. It works because smoothing, derivatives, detrending and related filters change the spectral directions seen by partial least squares (PLS) or Ridge regression, but a full external search repeatedly refits nearly the same linear model. This paper studies the case where that search can be collapsed into one calibration step. For a strict linear preprocessing operator A acting on row spectra as XA^T, the transformed PLS cross-covariance satisfies (XA^T)^T Y = A X^T Y, and Ridge regression depends on the operator-induced kernel X A^T A X^T. These identities let a finite operator bank be screened inside the model while retaining original-wavelength coefficients, and the same identity extends to cheaply evaluated linear operator chains. Sample-adaptive or fitted corrections such as SNV, MSC, EMSC and ASLS are not strict linear; we prove the boundary and keep them as fold-local branches. The cohort has 61 regression and 17 classification rows, with a strict paired regression denominator of N=32 for the eight paper variants. There, AOM-PLS reaches median RMSEP ratios of 0.991/0.990 (simple) and 0.985/1.002 (best) against PLS-default/PLS-HPO, and AOM-Ridge reaches 0.974/0.984 (simple) and 0.918/0.966 (best) against Ridge-default/Ridge-HPO. The operator-adaptive classifier AOM-PLS-DA improves balanced accuracy by a median 0.159 on N=13 datasets (12/13 wins). The practical result is the runtime gap: PLS-HPO takes a median 710.81 s per run, whereas AOM-PLS takes 1.18-1.63 s -- 436 to 602 times less PLS fitting time. Linear operator-adaptive calibration thus gives prediction quality comparable to exhaustive preprocessing screening, with orders-of-magnitude less fitting time for PLS.

Gregory Beurier, Robin Reiter, Camille Noûs +2

CIRAD, UMR AGAP Institut, Montpellier, France · UMR AGAP Institut, Univ Montpellier, CIRAD, INRAE, Institut Agro, Montpellier, France · CIRAD, UMR AGAP Institut, F-34398 Montpellier, France +1

Linear Regression Spectral Methods Ridge Regression

May 13, 2026cs.LG

Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergence

Muon orthogonalizes the momentum buffer before each update, replacing its singular values with ones via Newton-Schulz iterations. This simple change lets Muon tolerate far larger learning rates and converge faster than other optimizers, but why? We show that the mechanism is spectral flattening, and develop two results around it. First, we prove that Muon's maximal stable step size scales with the average singular value of the gradient rather than the largest, which bottlenecks standard gradient descent. Second, we recast Muon as a preconditioned gradient method and show, under a Kronecker-factored curvature model, that it improves the effective convergence factor, with the improvement controlled by the spectrum of the gradient covariance. Extensive experiments validate both results: Muon remains stable at learning rates that cause SGD to diverge within the first few iterations, and reaches accuracy milestones several epochs earlier even at identical step sizes. Taken together, our results offer a principled, geometric explanation for Muon's empirical success.

Tien-Phat Nguyen, Truong Nguyen, Minh-Phuc Truong +3

Hanoi University of Science and Technology Hanoi, Vietnam · Indiana University 107 S. Indiana Ave, Bloomington, IN 47405, USA · Monash University Clayton, VIC 3800, Australia

Momentum Methods Gradient Descent Muon Optimizer Spectral Methods Stochastic Gradient Descent

May 12, 2026cs.LG

Modeling Heterophily in Multiplex Graphs: An Adaptive Approach for Node Classification

Existing multiplex graph models often assume homophily, where connected nodes tend to belong to the same class or share similar attributes. Consequently, these models may struggle with graphs exhibiting heterophily, where connected nodes typically belong to different classes and have dissimilar attributes. While recent methods have been developed to learn reliable node representations from unidimensional graphs with heterophily, they do not fully address the complexities of multiplex graphs. In a multiplex graph, nodes are linked through multiple types of edges (referred to as dimensions), which can simultaneously exhibit homophilic and heterophilic interactions. To address this gap, we propose \methodname, a novel method for node classification in multiplex graphs that adapts to both homophilic and heterophilic dimensions. \methodname introduces dimension-specific compatibility matrices to model varying degrees of homophily and heterophily across dimensions. A key innovation is its use of a product of trainable low-pass and high-pass filters, approximated via Chebyshev polynomials, to capture both smooth and abrupt changes in the graph signal. By composing these filters and optimizing label predictions using a proximal-gradient method, \methodname dynamically adjusts to the heterophilic characteristics of each dimension. Extensive experiments on synthetic and real-world datasets provide evidence that \methodname captures the complex interplay of homophilic and heterophilic interactions in multiplex graphs, and tends to yield improved node classification performance compared to state-of-the-art methods.

Kamel Abdous, Nairouz Mrabah, Mohamed Bouguessa

Department of Computer Science, University of Quebec at Montreal, Montreal, QC, Canada

Graph Representation Learning Spectral Methods Node Classification Graph Heterophily

May 12, 2026stat.ML

Minimax Rates and Spectral Distillation for Tree Ensembles

Tree ensembles such as random forests (RFs) and gradient boosting machines (GBMs) are among the most widely used supervised learners, yet their theoretical properties remain incompletely understood. We adopt a spectral perspective on these algorithms, with two main contributions. First, we derive minimax-optimal convergence for RF regression, showing that, under mild regularity conditions on tree growth, the eigenvalue decay of the induced kernel operator governs the statistical rate. Second, we exploit this spectral viewpoint to develop compression schemes for tree ensembles. For RFs, leading eigenfunctions of the kernel operator capture the dominant predictive directions; for GBMs, leading singular vectors of the smoother matrix play an analogous role. Learning nonlinear maps for these spectral representations yields distilled models that are orders of magnitude smaller than the originals while maintaining competitive predictive performance. Our methods compare favorably to state of the art algorithms for forest pruning and rule extraction, with applications to resource constrained computing.

Binh Duc Vu, David S. Watson

King’s College London

Model Compression Gradient-Boosted Decision Trees Decision Tree Learning Spectral Methods Statistical Learning Theory Knowledge Distillation+1

May 12, 2026physics.optics

Bin Latent Transformer (BiLT): A shift-invariant autoencoder for calibration-free spectral unmixing of turbid media

The accurate recovery of constituent-level optical properties from integrating sphere measurements is a central analytical challenge in pharmaceutical analysis, food science, and biomedical diagnostics. Neural network autoencoders can extract spectrally resolved absorption and scattering coefficients for each constituent without prior knowledge, but their fully connected encoders bind learned features to absolute wavelength indices, causing accuracy loss under spectrometer calibration drift or hardware exchange. This work introduces the Bin Latent Transformer (BiLT)-Autoencoder, in which the dense encoder is replaced by a cross-attention scanner: 16 learnable probe vectors query a convolutional feature map, aggregating morphological spectral information independently of absolute wavelength position. A physics-constrained linear decoder with enforced absorption/scattering separation and a three-phase curriculum augmentation strategy complete the architecture. On a liquid phantom benchmark (intralipid and two ink absorbers; 496 samples), the model achieves

R^2 = 0.979

and

0.975

for

μ_a(λ)

and

μ_s'(λ)

, respectively, on held-out test spectra, maintaining

R^2 > 0.90

for

μ_a

and

R^2 \approx 0.99

for

μ_s'

across the full tested shift range of

\pm 10

spectral bands. The model generalises to a simulated spectrometer with a broader instrument line shape (

{\approx}24

nm FWHM) without retraining, retaining

R^2 \approx 0.96

and

0.974

for the two channels. Attention map analysis reveals a physically interpretable two-component probe strategy: sparse anchor probes at absorption-edge wavelengths combined with a diffuse, SNR-driven ensemble at the high-transmittance long-wavelength region, which recruits additional probes dynamically under noise to provide implicit spectral averaging.

Martin Hohmann

Institute of Photonic Technologies (LPT), Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU), Konrad-Zuse-Straße 3/5, 91052 Erlangen, Germany

Autoencoders Transformer Attention Physics-Informed ML Spectral Methods

May 11, 2026cs.LG

A Spectral Framework for Closed-Form Relative Density Estimation

We propose a closed-form spectral framework for relative log-density estimation in linearly parameterized probabilistic models, including unnormalized and conditional models. This is achieved by representing the Kullback-Leibler (KL) divergence as an integral of weighted chi-squared divergences, converting KL estimation into a family of least-squares problems. We derive an explicit spectral formula based only on first- and second-order feature moments, yielding closed-form estimators of both divergences and log-density potentials for fixed features. The framework extends to a broad class of f-divergences and can be combined with kernelization or feature learning with neural networks. We prove convergence guarantees for the resulting estimators and empirically compare them on synthetic data with optimization-based variational formulations, including logistic and softmax regression for normalized conditional models.

Francis Bach

SIERRA · Inria - Ecole Normale Supérieure · PSL Research University

Density Ratio Estimation Spectral Methods Statistical Learning Theory Density Estimation

May 11, 2026cs.LG

Don't Fix the Basis -- Learn It: Spectral Representation with Adaptive Basis Learning for PDEs

Spectral neural operators achieve strong performance for PDE learning, but rely on fixed global bases that limit their ability to represent spatially heterogeneous and multiscale dynamics. We propose Adaptive Basis Learning (ABLE), a framework that learns data-dependent spectral representations instead of relying on predefined bases. ABLE constructs a spatially adaptive Parseval frame via a learned ancillary density, enabling the operator to act in a lifted spectral space while preserving invertibility and maintaining

O(N\log N)

complexity through FFT-based implementation. This shifts the source of expressivity from spectral coefficients to the representation itself, allowing the model to capture localized structures and non-translation-invariant interactions more efficiently. ABLE integrates seamlessly into existing neural operator architectures as a drop-in replacement for spectral layers. Across a range of benchmarks ABLE improves accuracy over strong baselines, with the largest gains in regimes characterized by sharp gradients and multiscale behavior. Moreover, augmenting existing models (e.g., U-FNO, HPM) with ABLE further enhances their performance, demonstrating its role as a general and complementary spectral refinement. Our results highlight that the data-driven choice of representation, rather than operator complexity alone, is a key bottleneck in neural operator design. By learning the basis itself, ABLE provides a principled and efficient framework for improving spectral methods in PDE learning.

Xuxiang Zhao, Angelica I. Aviles-Rivero

Qiuzhen College, Tsinghua University · YMSC, Tsinghua University

Fourier Neural Operator Representation Learning PDE Operator Learning Spectral Methods Neural Operators

May 11, 2026cs.LG

Stable Long-Horizon PDE Forecasting via Latent Structured Spectral Propagators

Long-horizon forecasting of time-dependent partial differential equations (PDEs) is critical for characterizing the sustained evolution of physical systems. While neural operators have emerged as efficient surrogates, they typically learn implicit finite-time transitions from discrete observations. When deployed autoregressively, such propagators often suffer from rapid error accumulation and dynamic drift. To address this, we propose a neural forecasting framework that reformulates PDE rollout as learning a Structured Spectral Propagator (SSP) in a propagation-oriented latent space. Following an analysis-propagation-synthesis design, our framework: (i) maps physical states into a shared, time-consistent spatial representation; (ii) projects this space into a compact propagation state to isolate recurrent dynamics from fine-grained spatial details, thereby decoupling reconstruction fidelity from rollout regularity; and (iii) evolves retained spectral modes using a frequency-conditioned linear backbone complemented by a nonlinear spectral closure to account for truncated interactions. This explicit structuring endows the propagator with a strong inductive bias for coherent modal evolution. Extensive experiments demonstrate that SSP significantly outperforms state-of-the-art baselines, reducing relative

L_2

errors by up to 48.9% and exhibiting improved stability in temporal extrapolation beyond the supervised horizon.

Xiaoxiao Lu, Ye Yuan, Jiahao Shi

School of AIA, Huazhong University of Science and Technology

Latent Dynamics Modeling PDE Surrogate Modeling PDE Operator Learning Spatiotemporal Forecasting Spectral Methods

May 10, 2026math.NA

Accelerating Power Method with Fast Sketching for Stronger Low-Rank Approximation

The power method is one of the most fundamental tools for extracting top principal components from data through low-rank matrix approximation. Yet, when the target rank is large, the cost of matrix multiplication associated with this procedure becomes a major bottleneck. We develop an algorithmic and theoretical framework for accelerating the power method using fast sketching, which is a popular paradigm in randomized linear algebra. Our framework leads to simple and provably efficient methods for singular value decomposition, low-rank factorization, and Nyström approximation, which attain strong numerical performance on benchmark problems. The key novelty in our analysis is the use of regularized spectral approximation, a property of fast sketching methods which proves more flexible in generalizing power method guarantees than traditional arguments.

Shabarish Chenakkod, Michał Dereziński

University of Michigan, Ann Arbor, MI 48105.

Low-Rank Matrix Decomposition Randomized Sketching Low-Rank Approximation Spectral Methods Singular Value Decomposition

May 9, 2026cs.LG

Bridging Spectral Operator Learning and U-Net Hierarchies: SpectraNet for Stable Autoregressive PDE Surrogates

Neural operators for time-dependent PDEs face a structural tension: spectral architectures (FNO and descendants) inherit exponential rollout-error growth from their one-step Lipschitz constant, while hierarchical U-Net operators trade resolution invariance for multi-scale detail. We introduce SpectraNet, an autoregressive neural operator that composes truncated spectral convolutions inside a U-Net hierarchy with a Residual-Target Spectral Block trained under a Semigroup-Consistency Loss. The residual-target parametrization replaces L^T stability blow-up with linear T*delta drift, and the spectral path's parameter count is Theta(L w^2 M^2), independent of grid N. Under a single unified protocol against 16 published neural-operator baselines on Navier-Stokes nu=1e-5 at 64x64, SpectraNet reaches test relative L2 = 0.0822 at 2.04M parameters -- 2.33x fewer than canonical FNO at ~20% lower error -- and wins five of six rows in a cross-PDE comparison against FNO (NS at nu in {1e-4, 1e-3}, PDEBench Shallow-Water 2D and Diffusion-Reaction, with the Active-Matter row going to FNO inside its seed spread). Trained from scratch at native 128^2 under the same protocol, SpectraNet improves to 0.0724 while FNO regresses to 0.3080. Free rollout stays bounded for T=100 where FNO diverges across all 200 test trajectories. On consumer CPU at B=1, SpectraNet runs sub-200ms while the full-attention Transformer that wins raw L2 pays ~60x latency; we do not claim to beat that Transformer on raw L2, only to dominate the lightweight (<=5M parameter, sub-200ms CPU) Pareto frontier. Source code: https://github.com/Enrikkk/spectranet

Enrique Hernández Noguera, Md Meftahul Ferdaus, Elias Ioup +2

University of New Orleans, New Orleans, LA, USA · Naval Research Laboratory, Stennis Space Center, MS, USA

U-Net PDE Surrogate Modeling Convolutional Neural Networks Spectral Methods Neural Operators