Singularity-Induced Local Complexity for High-Dimensional Learning Models
Authors: Atsuyoshi Muta, Masaaki Imaizumi
Organizations: The University of Tokyo, Tokyo, Japan. · RIKEN Center for Advanced Intelligence Project, Tokyo, Japan. · Kyoto University, Kyoto, Japan.
Many overparameterized models that have achieved remarkable success are singular models. That is, the set of optimal parameters consists of multiple points, or the Hessian of the generalization error at such parameters is degenerate, reflecting redundancy in the model parametrization. Therefore, understanding the effective dimension and intrinsic model complexity, beyond the apparent dimension of the parameter space, is essential for studying the generalization ability of such overparameterized models. In this work, we focus on the real log canonical threshold (RLCT) and its localized counterpart, the local RLCT, which are fundamental quantities for characterizing the generalization of singular models. We clarify the theoretical relationship between the local RLCT and existing two key metrics related to generalization under some localization assumption: (I) the rank of the Hessian matrix of the generalization error and (II) the covering number of a family of loss functions. This result provides novel insights into effective dimension and intrinsic complexity of overparameterized models.
Figures & tables
Figure 1: (a) The definition of RLCT. The RLCT is defined as the absolute value of the largest pole of the ζ function. The ζ function is initially defined for s∈C with Re(s)>0 . and it can then be extended to the entire complex plane by analytic continuation Atiyah (1970) . Then their poles (Remark 2 ) arise on the negative real axis Kashiwara (1976) ; Watanabe (2009) . It was found that this mathematical quantity is deeply related to statistical models ( Watanabe, 2009 ) . (b) The image of ζ(s) . At each pole, ∣ζ(s)∣ diverges to infinity.
Figure 2: Hessian Eigenvalues vs. Local RLCT : rank∇2R(wi)≤2λ(wi)(i=1,2,3) holds at all displayed parameter values. The red dots indicate the values of the eigenvalues, and the orange vertical line marks 2λ(wi) . The rank of the Hessian equals the number of its positive eigenvalues at global minimizers. The parameters and their corresponding local RLCTs are given in ( 8 ) for reduced-rank regression, and in ( 9 ) for the linear neural network.
Figure 3: Hessian Eigenvalues vs. Local RLCT (GELU Network) : This plot shows the eigenvalues of the Hessian and the estimated local RLCTs λ^SGLD(wi) (orange lines) at five parameters wi(i=1,…,5) , which are chosen from a GELU network so that the output function is 0 . The number of dominant eigenvalues is roughly bounded above by 2λ(wi)(i=1,…,5) .
Figure 4: Hessian Eigenvalues vs. Local RLCT (ReLU Network) : This plot shows the eigenvalues of the Hessian and the estimated local RLCTs λ^SGLD(wi) (orange lines) at five parameters wi(i=1,…,5) , which are chosen from a ReLU network so that the output function is zero. Since ReLU is not analytic, the experimental results do not clearly support the claim of the Main Theorem.
Low-rank adaptation (LoRA) is now a standard approach to parameter-efficient fine-tuning of large neural networks. Its low-rank factorization is inherently non-identifiable, making LoRA a singular statistical model whose learning complexity cannot be described by parameter dimension alone. We study this complexity using the real log canonical threshold (RLCT) from singular learning theory and how it depends on both the low-rank parameterization and the pretrained network. Under a local linearization, LoRA induces a low-rank trace regression model whose Kullback--Leibler geometry is controlled by a forward-backward covariance, constructed from activations and backpropagated sensitivities at the adapted layer. This covariance determines which low-rank perturbations are statistically observable. When it is nondegenerate along the relevant low-rank directions, we recover the classical learning complexity of reduced-rank regression; when its kernel meets these directions, additional statistically indistinguishable parameters change the singular geometry. We characterize this geometry locally through observable tangent directions and a residual singularity, and globally through affine linear sections of determinantal varieties. The global learning complexity need not be determined at a ground-truth parameter, as singularities on another rank stratum can dominate. We also introduce a curvature-corrected multi-temperature estimator of the RLCT. Experiments on multilayer perceptron and transformer architectures reveal strongly layer-dependent forward-backward geometry and show that ambient covariance degeneracy does not imply degeneracy along the low-rank directions relevant to adaptation.
Kamillo Ferry, Anthea Monod, Keisuke Yano
Department of Mathematics, Imperial College London, UK · Institute of Statistical Mathematics, Japan
Deep neural networks are highly over-parameterized, and different parameter values represent the same predictive function. This makes their effective complexity difficult to measure using only the number of parameters or the rank of the Hessian. Singular Learning Theory addresses this issue through the local learning coefficient (LLC), which characterizes the effective complexity of a model near a given solution. Existing methods for estimating the LLC often rely on posterior sampling, which can be computationally expensive for large neural networks. This makes accurate LLC estimation difficult at scale. In this work, we use known structures in the model to simplify the analysis and make LLC estimation more tractable. Specifically, we study the LLC of a graph attention model by exploiting symmetries in both the graph structure and the attention parameters. An analytic framework through a teacher--student setting, and explicit LLC estimates after considering the symmetry--induced degeneracies are developed.
Classical statistical learning theory predicts that overparameterized models should exhibit severe overfitting, yet modern deep neural networks with far more parameters than training samples consistently generalize well. This contradiction has become a central theoretical question in machine learning. This study investigates the role of optimization dynamics and implicit regularization in enabling generalization in overparameterized neural networks through controlled experiments. We examine stochastic gradient descent (SGD) across batch sizes, the geometry of flat versus sharp minima via Hessian eigenvalue estimation and weight perturbation analysis, the Neural Tangent Kernel (NTK) regime through wide-network experiments, double descent across model scales, and the Lottery Ticket Hypothesis through iterative magnitude pruning. All experiments use PyTorch on CIFAR-10 and MNIST with multiple random seeds. Our findings demonstrate that generalization is strongly influenced by the interaction between network architecture, optimization algorithms, and loss landscape geometry. Smaller batch sizes consistently produced lower test error and flatter minima, with an 11.8x difference in top Hessian eigenvalue between small-batch and large-batch solutions corresponding to 1.61 percentage points higher test accuracy. Sparse subnetworks retaining only 10% of parameters achieved within 1.15 percentage points of full model performance when retrained from their original initialization. These results highlight the need for revised learning-theoretic frameworks capable of explaining generalization in high-dimensional model regimes.