Singular Learning Theory

Follow this topic Ask about this topic

Singular Learning Theory examines models whose parameters are redundant or whose loss has singular structure, where standard statistical approximations can fail. It uses quantities such as the real log canonical threshold and local learning coefficients to characterize Bayesian evidence, predictive error, and model complexity in settings including neural networks, matrix factorization, and mixture models. Researchers derive these quantities through algebraic geometry and exact computation, and estimate them using thermodynamic methods or directional Fisher information; the resulting tools also inform PAC-Bayes bounds and analyses of training dynamics. A central challenge is connecting tractable local or finite-scale measurements to global asymptotic quantities, especially when noise, parameter symmetries, and the choice of localization affect interpretation.

Latest papers

16 papers2 papers in the last four weeks, against 2 the four weeks before. 0.0% of all new papers.

Singular Geometry of Low-Rank Adaptation
Oct 23, 2026Kamillo Ferry, Anthea Monod, Keisuke YanoLow-Rank Adaptation Singular Learning Theory
Department of Mathematics, Imperial College London, UK · Institute of Statistical Mathematics, Japan
Singularities of Non-negative Matrix Factorization and their application to Bayesian inference
Sep 28, 2026Naoki Hayashi, Yota Maeda, Yasushi EsakiNon-Negative Matrix Factorization Low-Rank Matrix Decomposition
Toyota Central R&D Labs., Inc.
Symmetries and Singularities
Sep 13, 2026Vishnu Varadarajan, Mihir More, Aritra Das +1Singular Learning Theory Graph Attention Networks
Truth Audit Labs
Patterning in Practice: Debiasing Reward Models with Susceptibilities
Sep 1, 2026George Wang, Elizabeth Donoway, Daniel MurfetReward Modeling Debiased ML
Resolution
Exact Algebraic Computation of Learning Coefficients for Two-Dimensional Singular Models
Aug 20, 2026Grégoire Sergeant-Perthuis, Elias Tsigaridas, Jules TsukaharaSingular Learning Theory
CQSB, Sorbonne Université · Ouragan Team, INRIA · Ouragan team, Inria Paris
The Off-Support Barrier: Why Semantic Safety Constraints Are Not Learning-Problem Invariants, and What Follows for Prior Design, Containment, and Verification
Aug 1, 2026Yoshinori WatanabeSingular Learning Theory AI Safety
Measuring Dead Directions: Decomposing and Classifying Singular Structure off Canonical Alignment
Jul 1, 2026Tejas Pradeep ShirodkarSingular Learning Theory Statistical Learning Theory
IIIT, Hyderabad
Singular Learning and Occam's Razor in Deep Monomial Networks
Jun 26, 2026Kathlén Kohn, Giovanni Luca Marchetti, Farhan Shabir +2Neural Network Optimization Singular Learning Theory
KTH Stockholm · No Affiliation · Umeå University +1
Noise-Debiased Thermodynamic Variance for Local Learning Coefficient Probes
Jun 21, 2026Yingjia CaiSingular Learning Theory Neural Network Training Dynamics
Dead-Direction Signatures: A Cheap Spectral Reading of Singular Complexity
Jun 19, 2026Tejas Pradeep Shirodkar, P. J. NarayananSingular Learning Theory Spectral Methods
IIIT, Hyderabad
Dead Directions: Geometric Singular Learning
Jun 4, 2026Tejas Pradeep ShirodkarInformation Geometry Singular Learning Theory
IIIT, Hyderabad
PAC-Bayes Bounds for Gibbs Posteriors via Singular Learning Theory
Apr 19, 2026Chenyang Wang, Yun YangNeural Network Generalization PAC-Bayesian Generalization Bounds
Department of Statistics, University of Illinois Urbana-Champaign · Department of Mathematics, University of Maryland, College Park
On Higher-Order Geometric Refinements of Classical Covariance Asymptotics: An Approach via Intrinsic and Extrinsic Information Geometry
Apr 14, 2026Malik Amir, Sourangshu GhoshInformation Geometry Covariance Estimation
Upper Bounds for Local Learning Coefficients of Three-Layer Neural Networks
Mar 13, 2026Yuki KurumadaniTwo-Layer Neural Networks Singular Learning Theory
The University of Osaka, 1 Chome-3 Machikaneyamacho, Toyonaka, Osaka 560-8531, Japan
Thermodynamic Response Functions in Singular Bayesian Models
Mar 5, 2026Sean PlummerSingular Learning Theory Statistical Physics of Learning
A Basin-Selection Perspective on Grokking via Singular Learning Theory
Mar 1, 2026Ben Cullen, Sergio Estan-Ruiz, Riya Danait +1Grokking Neural Network Training Dynamics
Department of Computer Science, University of Pisa, Italy · Department of Mathematics, Imperial College London, United Kingdom · Mathematical Institute, University of Oxford, United Kingdom +1

Oct 23, 2026J/K move · Enter open · S save

Singular Geometry of Low-Rank Adaptation

Kamillo Ferry, Anthea Monod, Keisuke Yano

Department of Mathematics, Imperial College London, UK · Institute of Statistical Mathematics, Japan

Low-Rank Adaptation Singular Learning Theory Low-Rank Matrix Decomposition

Open paper AskPDF arXiv

Low-rank adaptation (LoRA) is now a standard approach to parameter-efficient fine-tuning of large neural networks. Its low-rank factorization is inherently non-identifiable, making LoRA a singular statistical model whose learning complexity cannot be described by parameter dimension alone. We study this complexity using the real log canonical threshold (RLCT) from singular learning theory and how it depends on both the low-rank parameterization and the pretrained network. Under a local linearization, LoRA induces a low-rank trace regression model whose Kullback--Leibler geometry is controlled by a forward-backward covariance, constructed from activations and backpropagated sensitivities at the adapted layer. This covariance determines which low-rank perturbations are statistically observable. When it is nondegenerate along the relevant low-rank directions, we recover the classical learning complexity of reduced-rank regression; when its kernel meets these directions, additional statistically indistinguishable parameters change the singular geometry. We characterize this geometry locally through observable tangent directions and a residual singularity, and globally through affine linear sections of determinantal varieties. The global learning complexity need not be determined at a ground-truth parameter, as singularities on another rank stratum can dominate. We also introduce a curvature-corrected multi-temperature estimator of the RLCT. Experiments on multilayer perceptron and transformer architectures reveal strongly layer-dependent forward-backward geometry and show that ambient covariance degeneracy does not imply degeneracy along the low-rank directions relevant to adaptation.