How Much Rank Does LoRA Need? Rank-Error Bounds for Transformer Attention

Gerard Conangla Planes

cs.LGAug 26, 2026

How Much Rank Does LoRA Need? Rank-Error Bounds for Transformer Attention

Authors: Gerard Conangla Planes

Organizations: Aily Labs

Abstract

Choosing the rank of a low-rank adaptation (LoRA) update is usually an empirical task. In this paper, we provide a task-dependent theory of the approximation error achievable at each LoRA rank for Transformer attention. We fix a pretrained attention head, a target attention function, and a distribution over inputs from the downstream task, and bound the smallest expected Kullback--Leibler (KL) error achievable by a rank- $r$ query LoRA update. When target attention probabilities are bounded away from zero, we prove a lower bound of the error proportional to $ψ(\|d\|_2)$ , where $d$ is the difference between candidate and target attention scores and $ψ(t)=\min\{t^2,t\}$ . We also prove an unconditional upper bound $\min\{\|d\|_2^2/4,\sqrt2\|d\|_2\}$ . Under explicit realizability, geometry, and moment conditions, we then bound the best rank- $r$ error between an explicit multiple of $ψ(\sqrt{T_r})$ and $\min\{T_r/4,\sqrt{2T_r}\}$ , where $T_r$ is the downstream-weighted tail energy of the target update. We also provide target-Fisher bounds when candidate scores remain within a fixed range of the target scores, and an unrestricted lower bound when a subset of tokens carries most of the probability mass. These spectral bounds describe finite-score approximation. We then construct explicit families in which softmax saturation makes the rank required to match the attention function strictly smaller than the rank required to match the finite logits. Finally, we extend the analysis to fused multi-head LoRA and joint query/key updates, exposing the effects of rank sharing and query/key factorization constraints.

Figures & tables

Panel of Figure 1 — Figure 1: Transformer attention with LoRA and the scope of our analysis. Read from top to bottom, the diagram traces an input through the query, key, and value projections, attention scores, row-wise softmax, and multi-head aggregation. Our central question is how closely a rank- $r$ LoRA update can reproduce a target attention function on inputs from the downstream task. The main theorem treats query-only adaptation in one head; later results cover a fused update shared across heads and simultaneous query/key updates. For an input $u$ , $X(u)$ contains $n(u)$ token representations of width $d$ . For each head index $h\in\{1,\ldots,H\}$ , query/key width is $p$ and value width is $p_{v}$ , with $Q_{h}(u),K_{h}(u)\in\mathbb{R}^{n(u)\times p}$ , $V_{h}(u)\in\mathbb{R}^{n(u)\times p_{v}}$ , and $S_{h}(u),P_{h}(u)\in\mathbb{R}^{n(u)\times n(u)}$ . Here $\sigma_{\rm row}$ applies softmax row-wise and $\beta$ is typically $1/\sqrt{p}$ . The pretrained projections $Q_{0},K_{0},V_{0}$ are fixed, while $A_{h}$ and $B_{h}$ are rank- $r$ LoRA updates. The multi-head result bounds the sum of headwise attention KL errors before $W_{O}$ ; the remaining Transformer computation and the final model output are outside this guarantee.

Object	Meaning	What changes?
$u$	One input from the downstream task	Varies across examples
$z_{0}(u)$	Attention scores produced by the pretrained head	Depends on $u$
$p_{*}(\cdot\mid u)$	Target attention probabilities to be reproduced	Depends on $u$ ; the target model is fixed
$M$	Candidate query update with $\operatorname{rank}(M)\leq r$	One fixed matrix for all inputs
$p_{M}(\cdot\mid u)$	Attention probabilities produced with update $M$	Depends on $u$ and on the chosen $M$
$d_{M}(u)$	Candidate–target score difference after removing a common shift	Depends on $u$ and on the chosen $M$

Table 1: Objects in the target–candidate comparison. For each downstream input

u

, the pretrained head and fixed target determine the behavior to be matched;

M

ranges over candidate updates of rank at most

r

Panel of Figure 2 — Figure 2: Which rank–KL theorem to use . Start at the left and stop at the first result whose assumptions you can justify. An upper bound shows that some rank- $r$ update achieves at most the stated error; a lower bound shows that every rank- $r$ update incurs at least the stated error. The Fisher bounds apply only to candidates whose scores remain within a fixed range of the target scores. Consequently, their lower bound does not apply to the unrestricted optimum $\mathcal{E}_{r}$ .

Route	Upper bound	Lower bound	Candidate class	Main price
Global	$\checkmark$	$\checkmark$	unrestricted rank- $r$	minimum target probability and moment assumptions on keys/queries
Target Fisher	$\checkmark$	$\checkmark$	rank- $r$ with a justified range of score differences	span-dependent constants and the target Fisher inner product
High mass	$\times$	$\checkmark$	unrestricted rank- $r$	tokens carrying most target mass, a conditional probability floor, and the key/query Gram on that set

Table 2: The three rank–KL routes and their logical scope.

Explore similar work

Sep 25, 2026

What Does the Rank Buy? A Spectral and Distributional Analysis of Low-Rank Adaptation

The rank

r

in LoRA is widely treated as a capacity control: a smaller rank is assumed to yield a simpler model that generalizes better. We show that, under hard per-factor norm budgets---the idealization of the weight decay and norm control used in practice---this intuition breaks down. The reason is structural: under such budgets, the updates LoRA can reach are exactly the matrices of rank at most

r

inside a nuclear-norm ball, and every complexity and displacement functional we analyze is maximized over this set by a rank-one update---so the rank cap never binds. The consequences follow directly. The linear-readout model class we study is identical for every

r \ge 1

, its Rademacher complexity carries no dependence on

r

, and the distance the adaptation can move the source distribution obeys a rank-independent upper bound that we show is sharp. If rank does not control capacity, where does it act? We identify two places. Statistically, replacing the per-factor budgets with a joint budget on the product restores a data-dependent, rank-sensitive complexity bound---though the gain appears only for well-spread feature distributions, and the worst case remains rank-free. Spectrally, rank sets the price of adaptation: canceling the leading singular directions of the pretrained weight requires both sufficient rank and sufficient budget. We bound the smallest rank achieving a desired source--target alignment, with upper and lower bounds that match under two-sided spectral decay. Together, these results recast rank as governing which updates are reachable and what cancellation costs---not how much capacity the model has.

Babak Barazandeh

Cribl AI Research Lab

Low-Rank Adaptation Statistical Learning Theory

Ask PDF

Jul 25, 2026

The Entropic Bound for Transformers: Why Static Rank Fails and Attention-Native Rank Recovers

Neural scaling laws describe how loss decreases as models, data, and compute grow, but they do not answer a prior question: for a fixed task, what is the minimum model capacity required to solve it? We study this through the Entropic Bound, a spectral notion of task-intrinsic capacity for Transformers. We first prove that, in a linear attention surrogate, the intrinsic rank

r^*

of the token-mixing operator is a tight lower bound: any rank-deficient model incurs unavoidable excess risk, and the bound is achievable at

r^*

. We further show that gradient descent recovers this rank under standard low-rank implicit-bias assumptions, confirm all three properties empirically, and show

r^*

is recoverable from data before training. We then ask whether this transfers to real attention. A naive transfer fails, and a controlled interpolation ladder localizes the cause precisely: it is not softmax and not a rank constraint, but the input-conditioned nature of attention's mixing operator, which a static weight kernel cannot summarize. Motivated by this, we introduce an attention-native intrinsic rank -- the minimum query-key kernel rank realizing the task within the attention class -- and show that under this definition the full Entropic Bound structure (deficiency, achievability, recovery) is restored for both linear and softmax attention, with the energy effective rank as the estimator robust to softmax distortion. Finally, we map the boundary of data-only predictability:

r^*

is exactly recoverable for linear QK attention, even without the value map at scale, while softmax attention admits only partial pre-training recovery due to nonlinear inversion and kernel-value identifiability effects. Our results reframe the Entropic Bound from a post-hoc descriptor into an attention-native capacity measure with a precisely characterized predictability frontier.

Byeong Hoon Yoon

Independent Researcher

Softmax Attention Transformer Expressivity Low-Rank Approximation Attention Mechanisms

Ask PDF

May 16, 2026

FIM-LoRA: Task-Informative Rank Allocation for LoRA via Calibration-Time Gradient-Variance Estimation

Low-rank adaptation (LoRA) assigns a uniform rank to every adapted weight matrix - a practical convenience that ignores a fundamental reality: different layers contribute unequally to task adaptation. We address this with a lightweight engineering solution: before fine-tuning begins, run eight calibration backward passes, compute the gradient variance of each LoRA-B matrix as a proxy for layer informativeness, and redistribute the rank budget proportionally. The resulting adapter is a standard LoRA with a per-layer rank pattern - no new parameters, no training overhead, no changes to serving infrastructure. We implement this via an efficient approximation of the empirical Fisher Information Matrix (eFIM) diagonal, restricted to LoRA adapter matrices only, which reduces memory cost by approximately 256x compared to full-model Fisher estimation. On GLUE with DeBERTa-v3-base, FIM-LoRA matches LoRA (88.6 vs. 88.7) at the same parameter budget, and on commonsense reasoning with LLaMA-3-8B reaches 68.5 vs. 68.7 for LoRA. The per-layer rank maps are interpretable: value projections and early-to-middle layers consistently receive higher rank, consistent with established findings on transformer layer roles.

Ramakrishnan Sathyavageeswaran

Intuit

Adapter Tuning Low-Rank Adaptation Adaptive Rank Allocation Parameter-Efficient Fine-Tuning

Ask PDF