cs.AISep 25, 2026

What Does the Rank Buy? A Spectral and Distributional Analysis of Low-Rank Adaptation

Authors: Babak Barazandeh

Organizations: Cribl AI Research Lab

Abstract

The rank rr in LoRA is widely treated as a capacity control: a smaller rank is assumed to yield a simpler model that generalizes better. We show that, under hard per-factor norm budgets---the idealization of the weight decay and norm control used in practice---this intuition breaks down. The reason is structural: under such budgets, the updates LoRA can reach are exactly the matrices of rank at most rr inside a nuclear-norm ball, and every complexity and displacement functional we analyze is maximized over this set by a rank-one update---so the rank cap never binds. The consequences follow directly. The linear-readout model class we study is identical for every r≥1r \ge 1, its Rademacher complexity carries no dependence on rr, and the distance the adaptation can move the source distribution obeys a rank-independent upper bound that we show is sharp. If rank does not control capacity, where does it act? We identify two places. Statistically, replacing the per-factor budgets with a joint budget on the product restores a data-dependent, rank-sensitive complexity bound---though the gain appears only for well-spread feature distributions, and the worst case remains rank-free. Spectrally, rank sets the price of adaptation: canceling the leading singular directions of the pretrained weight requires both sufficient rank and sufficient budget. We bound the smallest rank achieving a desired source--target alignment, with upper and lower bounds that match under two-sided spectral decay. Together, these results recast rank as governing which updates are reachable and what cancellation costs---not how much capacity the model has.

Explore similar work

CardsList
  1. Rank-Efficient LoRA via Joint Tangent-Space Optimization under Isotropic Curvature

    Sep 10, 2026Zihan Zhu, Zhehang Du, Xuyang Chen +5Low-Rank AdaptationParameter-Efficient Fine-Tuning

  2. A Fine-Grained Analysis of the LoRA Fine-Tuning Landscape with Implications for Data Selection

    Oct 5, 2026Bowen Zhang, Changrui Fang, Xinsong Ma +2Low-Rank AdaptationTraining Data Selection