cs.LG · 2607.21975 Copy arXiv ID · Jul 24, 2026 Save On the Convergence of Stochastic Low-Rank Adaptation Authors: Ru Wang , Chengchang Liu , John C. S. Lui
Organizations: The Chinese University of Hong Kong · Westlake University
Abstract Low-rank adaptation (LoRA) optimizes J ( B , A ) = L ( W b a s e + s B A ) J(B,A)=\mathcal L(W_\mathrm{base}+sBA) J ( B , A ) = L ( W base + s B A ) B ∈ R m × r B \in \mathbb{R}^{m \times r} B ∈ R m × r A ∈ R r × n A \in \mathbb{R}^{r \times n} A ∈ R r × n W b a s e ∈ R m × n W_\mathrm{base} \in \mathbb{R}^{m \times n} W base ∈ R m × n exp { O ( ε − 2 ) } \exp\{\mathcal{O}(ε^{-2})\} exp { O ( ε − 2 )} ε ε ε ∥ ∇ J ( B , A ) ∥ ≤ ε \|\nabla J(B,A)\|\leq ε ∥∇ J ( B , A ) ∥ ≤ ε O ( ε − 4 ) \mathcal{O}(ε^{-4}) O ( ε − 4 ) ε ε ε O ( ε − 8 ) \mathcal{O}(ε^{-8}) O ( ε − 8 ) O ( ε − 6 ) \mathcal{O}(ε^{-6}) O ( ε − 6 )
Explore similar work Aug 31, 2026 · Jiale Kang, Ziyin Yue, Zheng Zhan +2 Low-Rank Adaptation Parameter-Efficient Adaptation
Jun 15, 2026 · Junghun Oh, Sungyong Baik, Kyoung Mu Lee Low-Rank Adaptation Low-Rank Factorization
Jul 30, 2026 · cs.LG J/K move · Enter open · S save
Arunan J
Independent Researcher, Chennai, India.
Low-Rank Adaptation (LoRA) has become the standard mechanism for fine-tuning large pretrained models, yet its statistical properties remain only partially understood. Existing generalization results provide upper bounds of the form O~(sqrt(rd/n)) or O~(rd/n), but a matching lower bound is missing, and the question of how to choose the LoRA rank r has no formal answer. Both gaps are closed here. A local Rademacher argument establishes an upper bound of O~(rd/n) on the excess risk of the empirical risk minimizer over rank-r LoRA, whenever the target adaptation has rank at most r. A matching minimax lower bound of Omega(rd/n) is then proved via a Fano-type packing of the rank-r subspace of R^{d x d}; the bound applies to any estimator whose output lies in the rank-r LoRA class. Combining the two yields a rank-selection dichotomy. For the constrained empirical risk minimizer, the optimal rank equals the intrinsic rank r*, and over-ranking strictly hurts. For adaptive estimators of the nuclear-norm-then-truncate type, over-ranking is harmless and the rate saturates at Theta~(r* d / n) regardless of r. Taken together, the three results characterize the statistical complexity of LoRA fine-tuning within the well-specified locally quadratic regime, and identify the empirically observed over-parameterization penalty as a property of unregularized empirical risk minimization rather than of the LoRA class itself. Predictions of the theory are verified on a synthetic trace-regression benchmark and on real LoRA fine-tuning across three (model, task) configurations covering DistilBERT and RoBERTa on SST-2 and MRPC. All configurations exhibit the predicted U-shape in validation loss, with two showing statistically significant loss inflation at large ranks (paired permutation p = 0.016).