Singularity-Induced Local Complexity for High-Dimensional Learning Models

Atsuyoshi Muta; Masaaki Imaizumi

cs.LGNov 2, 2026

Singularity-Induced Local Complexity for High-Dimensional Learning Models

Authors: Atsuyoshi Muta, Masaaki Imaizumi

Organizations: The University of Tokyo, Tokyo, Japan. · RIKEN Center for Advanced Intelligence Project, Tokyo, Japan. · Kyoto University, Kyoto, Japan.

Abstract

Many overparameterized models that have achieved remarkable success are singular models. That is, the set of optimal parameters consists of multiple points, or the Hessian of the generalization error at such parameters is degenerate, reflecting redundancy in the model parametrization. Therefore, understanding the effective dimension and intrinsic model complexity, beyond the apparent dimension of the parameter space, is essential for studying the generalization ability of such overparameterized models. In this work, we focus on the real log canonical threshold (RLCT) and its localized counterpart, the local RLCT, which are fundamental quantities for characterizing the generalization of singular models. We clarify the theoretical relationship between the local RLCT and existing two key metrics related to generalization under some localization assumption: (I) the rank of the Hessian matrix of the generalization error and (II) the covering number of a family of loss functions. This result provides novel insights into effective dimension and intrinsic complexity of overparameterized models.

Figures & tables

Panel of Figure 1 — Figure 1: (a) The definition of RLCT. The RLCT is defined as the absolute value of the largest pole of the $\zeta$ function. The $\zeta$ function is initially defined for $s\in\mathbb{C}$ with $\operatorname{Re}(s)>0$ . and it can then be extended to the entire complex plane by analytic continuation Atiyah (1970) . Then their poles (Remark 2 ) arise on the negative real axis Kashiwara (1976) ; Watanabe (2009) . It was found that this mathematical quantity is deeply related to statistical models ( Watanabe, 2009 ) . (b) The image of $\zeta(s)$ . At each pole, $|\zeta(s)|$ diverges to infinity.

Panel of Figure 2 — Figure 2: Hessian Eigenvalues vs. Local RLCT : $\operatorname{rank}\nabla^{2}R(w_{i})\leq 2\lambda(w_{i})\;(i=1,2,3)$ holds at all displayed parameter values. The red dots indicate the values of the eigenvalues, and the orange vertical line marks $2\lambda(w_{i})$ . The rank of the Hessian equals the number of its positive eigenvalues at global minimizers. The parameters and their corresponding local RLCTs are given in ( 8 ) for reduced-rank regression, and in ( 9 ) for the linear neural network.

Panel of Figure 3 — Figure 3: Hessian Eigenvalues vs. Local RLCT (GELU Network) : This plot shows the eigenvalues of the Hessian and the estimated local RLCTs $\hat{\lambda}_{\mathrm{SGLD}}(w_{i})$ (orange lines) at five parameters $w_{i}\;(i=1,\ldots,5)$ , which are chosen from a GELU network so that the output function is $0$ . The number of dominant eigenvalues is roughly bounded above by $2\lambda(w_{i})\;(i=1,\ldots,5)$ .

Panel of Figure 4 — Figure 4: Hessian Eigenvalues vs. Local RLCT (ReLU Network) : This plot shows the eigenvalues of the Hessian and the estimated local RLCTs $\hat{\lambda}_{\mathrm{SGLD}}(w_{i})$ (orange lines) at five parameters $w_{i}\;(i=1,\ldots,5)$ , which are chosen from a ReLU network so that the output function is zero. Since ReLU is not analytic, the experimental results do not clearly support the claim of the Main Theorem.

Singular Learning Theory

Singular Learning Theory studies how non-identifiability and geometric singularities shape learning, Bayesian inference, and generalization.

See all 18 papers

Singular Geometry of Low-Rank Adaptation
Oct 23, 2026Kamillo Ferry, Anthea Monod, Keisuke YanoLow-Rank Adaptation Singular Learning Theory
Department of Mathematics, Imperial College London, UK · Institute of Statistical Mathematics, Japan
Symmetries and Singularities
Sep 13, 2026Vishnu Varadarajan, Mihir More, Aritra Das +1Singular Learning Theory Graph Attention Networks
Truth Audit Labs
Implicit Regularization and Generalization in Overparameterized Neural Networks
Apr 8, 2026Zeran JohannsenOverparameterized Neural Network Neural Network Generalization

Oct 23, 2026J/K move · Enter open · S save

Singular Geometry of Low-Rank Adaptation

Kamillo Ferry, Anthea Monod, Keisuke Yano

Department of Mathematics, Imperial College London, UK · Institute of Statistical Mathematics, Japan

Low-Rank Adaptation Singular Learning Theory Low-Rank Matrix Decomposition

Open paper AskPDF arXiv

Low-rank adaptation (LoRA) is now a standard approach to parameter-efficient fine-tuning of large neural networks. Its low-rank factorization is inherently non-identifiable, making LoRA a singular statistical model whose learning complexity cannot be described by parameter dimension alone. We study this complexity using the real log canonical threshold (RLCT) from singular learning theory and how it depends on both the low-rank parameterization and the pretrained network. Under a local linearization, LoRA induces a low-rank trace regression model whose Kullback--Leibler geometry is controlled by a forward-backward covariance, constructed from activations and backpropagated sensitivities at the adapted layer. This covariance determines which low-rank perturbations are statistically observable. When it is nondegenerate along the relevant low-rank directions, we recover the classical learning complexity of reduced-rank regression; when its kernel meets these directions, additional statistically indistinguishable parameters change the singular geometry. We characterize this geometry locally through observable tangent directions and a residual singularity, and globally through affine linear sections of determinantal varieties. The global learning complexity need not be determined at a ground-truth parameter, as singularities on another rank stratum can dominate. We also introduce a curvature-corrected multi-temperature estimator of the RLCT. Experiments on multilayer perceptron and transformer architectures reveal strongly layer-dependent forward-backward geometry and show that ambient covariance degeneracy does not imply degeneracy along the low-rank directions relevant to adaptation.

Abstract

Singular Learning Theory

Explore similar work

Singular Geometry of Low-Rank Adaptation

Symmetries and Singularities

Implicit Regularization and Generalization in Overparameterized Neural Networks