Low-rank adaptation (LoRA) is now a standard approach to parameter-efficient fine-tuning of large neural networks. Its low-rank factorization is inherently non-identifiable, making LoRA a singular statistical model whose learning complexity cannot be described by parameter dimension alone. We study this complexity using the real log canonical threshold (RLCT) from singular learning theory and how it depends on both the low-rank parameterization and the pretrained network. Under a local linearization, LoRA induces a low-rank trace regression model whose Kullback--Leibler geometry is controlled by a forward-backward covariance, constructed from activations and backpropagated sensitivities at the adapted layer. This covariance determines which low-rank perturbations are statistically observable. When it is nondegenerate along the relevant low-rank directions, we recover the classical learning complexity of reduced-rank regression; when its kernel meets these directions, additional statistically indistinguishable parameters change the singular geometry. We characterize this geometry locally through observable tangent directions and a residual singularity, and globally through affine linear sections of determinantal varieties. The global learning complexity need not be determined at a ground-truth parameter, as singularities on another rank stratum can dominate. We also introduce a curvature-corrected multi-temperature estimator of the RLCT. Experiments on multilayer perceptron and transformer architectures reveal strongly layer-dependent forward-backward geometry and show that ambient covariance degeneracy does not imply degeneracy along the low-rank directions relevant to adaptation.
Figures & tables
J
κ
∥α∥2≍
(λ,m)
Theorem 4.2.1
∅
0
u2+a22+b22
(23,1)
s=3 , χ≡0
{(1,2)}
1
u2+b22(a12+a22)
(1,1)
s=2 , χ≡0
{(1,2),(2,1)}
2
u2+(a2b2)2
(1,2)
s=1 , χ=a2b2
Table 1: The local data of Example 4.2.4 at (A0,B0) , for three choices of J .
Figure 1: Bipartite graphs of the three cases of Example 4.2.4 , one for each row of Table 1 . Top: on each input regime Xi , the FB feature joins the nonzero coordinates of g(x) to those of h(x) ; dotted coordinates vanish on Xi . Bottom: the union over the regimes, with the entries in J dashed.
Figure 2: The FB covariance spectra in the synthetic trace regression.
Figure 3: The RLCT estimates based on the localized prior and the MALA in the synthetic trace regression. The left panel shows the results for the equal rank case in which the true rank r0 is equal to the LoRA rank r . The right panel shows the results for the excess rank case in which r0<r . For RLCT estimates, we use the methods in Section 6.1 . The solid bar is ±2 Monte Carlo SE. For multi-temperature estimates, the shaded bar describes ±2 (Monte Carlo SE + model-form error). The green dashed lines denote the quadratic prediction λquad(β⋆) of the Laplace approximation, and the black dashed lines denote the Aoyagi and Watanabe (2005) theoretical values.
Figure 4: The FB covariance spectra in the rotated-digits task. The left panel shows the FB covariance spectra for the MLP. The right panel shows the FB covariance spectra for the transformer.
Figure 5: The RLCT estimates based on the localized prior and the MALA in the rotated-digits task. The green dashed lines denote the quadratic prediction λquad(β⋆) of the Laplace approximation, and the black dashed lines denote the Aoyagi and Watanabe (2005) theoretical values for the equal rank cases.
Figure 6: Linearization error in the rotated-digits task. At each LoRA step, the deviation of the change of the model output from the pretrained network from its first-order approximation (the Frobenius inner product between the FB feature at the pretrained weight and the LoRA update), relative to the norm of that change over the the target-domain training data.
Figure 7: The FB covariance spectra in the prompt-based surgical fine tuning. The left panel shows the FB covariance spectra for the query projection units. The right panel shows the FB covariance spectra for the value projection units. The spectra are computed using 4096 target-domain data.
Figure 8: The RLCT estimates based on the localized prior and the MALA in the prompt-based surgical fine tuning. The green solid lines denote the quadratic prediction λquad(β⋆) of the Laplace approximation, and the green dashed lines denote the ranks of the Fisher information matrices for the LoRA parameters at the fitted LoRA updates.